Mike Slinn's Blog

Installing JDK 17 on Ubuntu

2023-09-28T00:00:00-04:00

The Java Development Kit (JDK) version 17 is compatible with v11 and v8. Installing the JDK also installs the JRE, as you can see from the highlighted dependency shown below:

Shell

$ apt show -a openjdk-17-jdk
Package: openjdk-17-jdk
  Version: 17.0.8.1+1~us1-0ubuntu1~23.04
  Priority: optional
  Section: java
  Source: openjdk-17
  Origin: Ubuntu
  Maintainer: Ubuntu Developers 
  Original-Maintainer: OpenJDK Team 
  Bugs: https://bugs.launchpad.net/ubuntu/+filebug
  Installed-Size: 1544 kB
  Provides: java-compiler, java-sdk (= 17), java10-sdk, java11-sdk, java12-sdk, java13-sdk, java14-sdk, java15-sdk, java16-sdk, java17-sdk, java2-sdk, java5-sdk, java6-sdk, java7-sdk, java8-sdk, java9-sdk
  Depends: openjdk-17-jre (= 17.0.8.1+1~us1-0ubuntu1~23.04), openjdk-17-jdk-headless (= 17.0.8.1+1~us1-0ubuntu1~23.04), libc6 (>= 2.34), zlib1g (>= 1:1.1.4)
  Recommends: libxt-dev
  Suggests: openjdk-17-demo, openjdk-17-source, visualvm
  Homepage: https://openjdk.java.net/
  Download-Size: 1486 kB
  APT-Sources: http://archive.ubuntu.com/ubuntu lunar-updates/main amd64 Packages
  Description: OpenJDK Development Kit (JDK)
    OpenJDK is a development environment for building applications,
    applets, and components using the Java programming language.

  Package: openjdk-17-jdk
  Version: 17.0.6+10-1ubuntu2
  Priority: optional
  Section: java
  Source: openjdk-17
  Origin: Ubuntu
  Maintainer: Ubuntu Developers 
  Original-Maintainer: OpenJDK Team 
  Bugs: https://bugs.launchpad.net/ubuntu/+filebug
  Installed-Size: 4732 kB
  Provides: java-compiler, java-sdk, java10-sdk, java11-sdk, java12-sdk, java13-sdk, java14-sdk, java15-sdk, java16-sdk, java17-sdk, java2-sdk, java5-sdk, java6-sdk, java7-sdk, java8-sdk, java9-sdk
  Depends: openjdk-17-jre (= 17.0.6+10-1ubuntu2), openjdk-17-jdk-headless (= 17.0.6+10-1ubuntu2), libc6 (>= 2.34)
  Recommends: libxt-dev
  Suggests: openjdk-17-demo, openjdk-17-source, visualvm
  Homepage: https://openjdk.java.net/
  Download-Size: 4585 kB
  APT-Sources: http://archive.ubuntu.com/ubuntu lunar/main amd64 Packages
  Description: OpenJDK Development Kit (JDK)
    OpenJDK is a development environment for building applications,
    applets, and components using the Java programming language.

Install OpenJDK 17 and its dependencies:

Shell

$ sudo apt install openjdk-17-jdk

Setting the Default Java Version

The man page for update-java-alternatives is:

Shell

$ man update-java-alternatives
UPDATE-JAVA-ALTERNATIVESystem Manager's MUPDATE-JAVA-ALTERNATIVES(8)

NAME
        update-java-alternatives  -  update  alternatives for jre/sdk
        installations

SYNOPSIS
        update-java-alternatives [--jre] [--plugin] [-v|--verbose]
              -l|--list [<jname>]
              -s|--set <jname>
              -a|--auto
              -h|-?|--help

DESCRIPTION
        update-java-alternatives updates all  alternatives  belonging
        to  one  runtime or development kit for the Java language.  A
        package does provide these information of  it's  alternatives
        in /usr/lib/jvm/.<jname>.jinfo.

OPTIONS
        -l|--list [<jname>]
              List  all installed packages (or just <jname>) provid‐
              ing information to set a bunch of  java  alternatives.
              Verbose  output shows each alternative provided by the
              packages.

        -a|--auto
              Switch all alternatives of registered jre/sdk  instal‐
              lations to automatic mode.

        -s|--set <jname>
              Set all alternatives of the registered jre/sdk instal‐
              lation to the program path provided by the <jname> in‐
              stallation.

        --jre  Limit  the actions to alternatives belong to a runtime
              environment, not a development kit.

        --jre-headless
              Limit the actions to alternatives belong to the  head‐
              less part of a runtime environment.

        --plugin
              Limit  the  actions  to alternatives providing browser
              plugins.

        -h|--help
              Display a help message.

        -v|--verbose
              Verbose output.

FILES
        /usr/lib/jvm/.*.jinfo
              A text file describing a  jre/sdk  installation.  Con‐
              sists  of some variables of the form <var>=<value> and
              a list of alternatives  of  the  form  jre|jdk  <name>
              <path>.

AUTHOR
        update-java-alternatives  and this manual page was written by
        Matthias Klose <doko@ubuntu.com>.

                              May 2006   UPDATE-JAVA-ALTERNATIVES(8)

The above man page neglects to say that a package version’s priority is set from its version number. Thus a newer version would normally have a higher priority. To get a list of the installed Java versions, type:

Shell

$ sudo update-java-alternatives --list
java-1.11.0-openjdk-amd64      1111       /usr/lib/jvm/java-1.11.0-openjdk-amd64
java-1.17.0-openjdk-amd64      1711       /usr/lib/jvm/java-1.17.0-openjdk-amd64

As you can see, my machine had Java 11 and 17 installed. To set the newest version to be the default, type:

Shell

$ sudo update-java-alternatives -a

$ java -version
openjdk version "17.0.8.1" 2023-08-24
OpenJDK Runtime Environment (build 17.0.8.1+1-Ubuntu-0ubuntu123.04)
OpenJDK 64-Bit Server VM (build 17.0.8.1+1-Ubuntu-0ubuntu123.04, mixed mode, sharing)

C++ Boost library

2023-09-14T00:00:00-04:00

About Boost

Boost is a general-purpose open source library of utility functions for C++. The list of categories of fuctionality is formidable.

Boost is used in many commercial products, in-house projects, and open-source projects.

Ostensibly a C++ library for wide usage, it has become an techical underpinning for Python; that is why Boost documentation also contains information about Python.

Getting Started

I started by reading the online Getting Started guide.

For most Boost functionality, only headers are required, and libraries need not be built. The exceptions are described here.

The Boost C++ Libraries is a good free online book, and you can order non-free printed copies.

Ubuntu Installation

Discover the Latest Boost Version Number

The latest version of Boost is shown on https://www.boost.org/users/history/?ref=learnubuntu.com:

Let’s scrape the lastest version number from the above web page:

Shell

$ sudo apt install libxml2-utils

$ VER=$(
  wget -q -O - https://www.boost.org/users/history/?ref=learnubuntu.com | \
  xmllint --html --xpath '//*[@id="intro"]/div/h2[1]/a[2]' - 2>/dev/null | \
  grep -oEi 'Version ([0-9].)*' | \
  cut -d' ' -f 2
)

$ echo $VER
1.83

Download Source & Dependencies

Download and unpack the library source to a new directory within your home directory as follows:

Shell

$ cd

$ V_R=`echo $VER | tr . _`
$ echo $V_R
1_83 

$ NAME=boost_${V_R}_0
$ echo $NAME
boost_1_83_0 

$ wget -O $NAME.tar.gz \
  https://sourceforge.net/projects/boost/files/boost/$VER.0/$NAME.tar.gz/download

$ tar xzvf $NAME.tar.gz

$ cd $NAME/

You should now have the following files and directories:

Shell

$ ls -w72
INSTALL          boost/           boostcpp.jam   index.htm   rst.css
Jamroot          boost-build.jam  bootstrap.bat  index.html  status/
LICENSE_1_0.txt  boost.css        bootstrap.sh*  libs/       tools/
README.md        boost.png        doc/           more/

Install the required dependencies for building Boost.

Shell

$ sudo apt-get update

$ sudo apt install autotools-dev build-essential g++ \
  libbz2-dev libicu-dev python3-dev

Build Installer Using Bootstrap

Two scripts for building the installation program Boost are provided: bootstrap.sh (for Bash) and bootstrap.bat (for Windows/DOS). By default the scripts build all possible static/shared debug/release Boost libraries. Another default is that the script uses all the CPU cores your computer has for the build process. Even using all the cores on a fast laptop, the build might take 10 minutes.

Here is the bootstrap.sh help message:

Shell

$ ./bootstrap.sh -h
`./bootstrap.sh\' builds the Boost build system B2 and prepares Boost for
building. This includes setting defaults in the project-config.jam which you
can adjust prior to invoking B2.

Usage: ./bootstrap.sh [OPTION]...

Defaults for the options are specified in brackets.

Configuration:
  -h, --help                display this help and exit
  --with-bjam=BJAM          use existing Boost.Jam executable (bjam)
                            [automatically built]
  --with-toolset=TOOLSET    use specific TOOLSET to build B2 and as default
                            for building Boost
                            [automatically detected]
  --show-libraries          show the set of libraries that require build
                            and installation steps (i.e., those libraries
                            that can be used with --with-libraries or
                            --without-libraries), then exit
  --with-libraries=list     build only a particular set of libraries,
                            describing using either a comma-separated list of
                            library names or "all"
                            [all]
  --without-libraries=list  build all libraries except the ones listed []
  --with-icu                enable Unicode/ICU support in Regex
                            [automatically detected]
  --without-icu             disable Unicode/ICU support in Regex
  --with-icu=DIR            specify the root of the ICU library installation
                            and enable Unicode/ICU support in Regex
                            [automatically detected]
  --with-python=PYTHON      specify the Python executable [python]
  --with-python-root=DIR    specify the root of the Python installation
                            [automatically detected]
  --with-python-version=X.Y specify the Python version as X.Y
                            [automatically detected]

Installation directories:
  --prefix=PREFIX           install Boost into the given PREFIX
                            [/usr/local]
  --exec-prefix=EPREFIX     install Boost binaries into the given EPREFIX
                            [PREFIX]

More precise control over installation directories:
  --libdir=DIR              install libraries here [EPREFIX/lib]
  --includedir=DIR          install headers here [PREFIX/include]

Bootstrap.sh compiles and links a build program called b2, which by default installs all of Boost into /usr/local/. To be more specific, the default location for Boost include files to be installed into is /usr/local/include/boost/, and Boost libraries are installed by default into /usr/local/lib/.

Create (and recreate) the b2 installation program for Boost for each Python virtual environment. By default, bootstrap.sh uses the currently active Python virtual environment, like this:

Shell

$ ./bootstrap.sh

Following is how to build b2 for the virtual environment at ~/venv/blah/.

Shell

$ ./bootstrap.sh --with-python-root=~/venv/blah
Building B2 engine..

###
###
### Using 'gcc' toolset.
###
###

g++ (Ubuntu 12.3.0-1ubuntu1~23.04) 12.3.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


###
###

> g++ -x c++ -std=c++11 -O2 -s -DNDEBUG builtins.cpp class.cpp command.cpp compile.cpp constants.cpp cwd.cpp debug.cpp debugger.cpp execcmd.cpp execnt.cpp execunix.cpp filesys.cpp filent.cpp fileunix.cpp frames.cpp function.cpp glob.cpp hash.cpp hcache.cpp hdrmacro.cpp headers.cpp jam_strings.cpp jam.cpp jamgram.cpp lists.cpp make.cpp make1.cpp md5.cpp mem.cpp modules.cpp native.cpp object.cpp option.cpp output.cpp parse.cpp pathnt.cpp pathsys.cpp pathunix.cpp regexp.cpp rules.cpp scan.cpp search.cpp startup.cpp subst.cpp sysinfo.cpp timestamp.cpp variable.cpp w32_getreg.cpp modules/order.cpp modules/path.cpp modules/property-set.cpp modules/regex.cpp modules/sequence.cpp modules/set.cpp -o b2
tools/build/src/engine/b2
Detecting Python version... 3.11
Unicode/ICU support for Boost.Regex?... /usr
Backing up existing B2 configuration in project-config.jam.2
Generating B2 configuration in project-config.jam for gcc...

Bootstrapping is done. To build, run:

    ./b2

To generate header files, run:

    ./b2 headers

The configuration generated uses gcc to build by default. If that is
unintended either use the --with-toolset option or adjust configuration, by
editing 'project-config.jam'.

Further information:

   - Command line help:
     ./b2 --help

   - Getting started guide:
     http://www.boost.org/more/getting_started/unix-variants.html

   - B2 documentation:
     http://www.boost.org/build/

Running B2, the Boost Installer

Following is the help message for the newly created b2 Boost installation program. As you can see, the defaults specified when creating the b2 script can be overidden. Note that not all of the available options are described in this help message; for example, the -j option is only described in the online help for b2.

Shell

$ ./b2 --help
B2 4.10-git

Project-specific help:

  Project has jamfile at Jamroot

Usage:

  b2 [options] [properties] [install|stage]

  Builds and installs Boost.

Targets and Related Options:

  install                 Install headers and compiled library files to the
  =======                 configured locations (below).

  --prefix=<PREFIX>       Install architecture independent files here.
                          Default: C:\Boost on Windows
                          Default: /usr/local on Unix, Linux, etc.

  --exec-prefix=<EPREFIX> Install architecture dependent files here.
                          Default: <PREFIX>

  --libdir=<LIBDIR>       Install library files here.
                          Default: <EPREFIX>/lib

  --includedir=<HDRDIR>   Install header files here.
                          Default: <PREFIX>/include

  --cmakedir=<CMAKEDIR>   Install CMake configuration files here.
                          Default: <LIBDIR>/cmake

  --no-cmake-config       Do not install CMake configuration files.

  stage                   Build and install only compiled library files to the
  =====                   stage directory.

  --stagedir=<STAGEDIR>   Install library files here
                          Default: ./stage

Other Options:

  --build-type=<type>     Build the specified pre-defined set of variations of
                          the libraries. Note, that which variants get built
                          depends on what each library supports.

                              -- minimal -- (default) Builds a minimal set of
                              variants. On Windows, these are static
                              multithreaded libraries in debug and release
                              modes, using shared runtime. On Linux, these are
                              static and shared multithreaded libraries in
                              release mode.

                              -- complete -- Build all possible variations.

  --build-dir=DIR         Build in this location instead of building within
                          the distribution tree. Recommended!

  --show-libraries        Display the list of Boost libraries that require
                          build and installation steps, and then exit.

  --layout=<layout>       Determine whether to choose library names and header
                          locations such that multiple versions of Boost or
                          multiple compilers can be used on the same system.

                              -- versioned -- Names of boost binaries include
                              the Boost version number, name and version of
                              the compiler and encoded build properties. Boost
                              headers are installed in a subdirectory of
                              <HDRDIR> whose name contains the Boost version
                              number.

                              -- tagged -- Names of boost binaries include the
                              encoded build properties such as variant and
                              threading, but do not including compiler name
                              and version, or Boost version. This option is
                              useful if you build several variants of Boost,
                              using the same compiler.

                              -- system -- Binaries names do not include the
                              Boost version number or the name and version
                              number of the compiler. Boost headers are
                              installed directly into <HDRDIR>. This option is
                              intended for system integrators building
                              distribution packages.

                          The default value is 'versioned' on Windows, and
                          'system' on Unix.

  --buildid=ID            Add the specified ID to the name of built libraries.
                          The default is to not add anything.

  --python-buildid=ID     Add the specified ID to the name of built libraries
                          that depend on Python. The default is to not add
                          anything. This ID is added in addition to --buildid.

  --help                  This message.

  --with-<library>        Build and install the specified <library>. If this
                          option is used, only libraries specified using this
                          option will be built.

  --without-<library>     Do not build, stage, or install the specified
                          <library>. By default, all libraries are built.

Properties:

  toolset=toolset         Indicate the toolset to build with.

  variant=debug|release   Select the build variant

  link=static|shared      Whether to build static or shared libraries

  threading=single|multi  Whether to build single or multithreaded binaries

  runtime-link=static|shared
                          Whether to link to static or shared C and C++
                          runtime.


General command line usage:

    b2 [options] [properties] [targets]

  Options, properties and targets can be specified in any order.

Important Options:

  * --clean Remove targets instead of building
  * -a Rebuild everything
  * -n Don't execute the commands, only print them
  * -d+2 Show commands as they are executed
  * -d0 Suppress all informational messages
  * -q Stop at first error
  * --reconfigure Rerun all configuration checks
  * --durations[=N] Report top N targets by execution time
  * --debug-configuration Diagnose configuration
  * --debug-building Report which targets are built with what properties
  * --debug-generator Diagnose generator search/execution

Further Help:

  The following options can be used to obtain additional documentation.

  * --help-options Print more obscure command line options.
  * --help-internal B2 implementation details.
  * --help-doc-options Implementation details doc formatting.

...found 1 target...

The following runs the newly created b2 program, which builds and installs Boost using all CPU cores.

The Python version is stored into an environment variable called PVER.
The CPLUS_INCLUDE_PATH environment variable is pointed at the location of the Python 3 include files. PVER was used to set the proper value for CPLUS_INCLUDE_PATH.
B2 generates screen upon screen of compiler warnings. The -d 1 option suppresses them.

Shell

$ PVER="$(python --version | cut -d' ' -f 2 | grep -oEi '(3.[0-9]*)')"

$ echo $PVER
3.11 

$ sudo CPLUS_INCLUDE_PATH=/usr/include/python$PVER ./b2 -d 1 install
Performing configuration checks

  - default address-model    : 64-bit (cached) [1]
  - default architecture     : x86 (cached) [1]
  - compiler supports SSE2   : yes (cached) [2]
  - compiler supports SSE4.1 : yes (cached) [2]
  - has std::atomic_ref      : no  (cached) [2]
  - has -Wl,--no-undefined   : yes (cached) [2]
  - has statx                : yes (cached) [2]
  - has init_priority attribute : yes (cached) [2]
  - has stat::st_blksize     : yes (cached) [2]
  - has stat::st_mtim        : yes (cached) [2]
  - has stat::st_mtimensec   : no  (cached) [2]
  - has stat::st_mtimespec   : no  (cached) [2]
  - has stat::st_birthtim    : no  (cached) [2]
  - has stat::st_birthtimensec : no  (cached) [2]
  - has stat::st_birthtimespec : no  (cached) [2]
  - has fdopendir(O_NOFOLLOW) : yes (cached) [2]
  - has dirent::d_type       : yes (cached) [2]
  - has POSIX *at APIs       : yes (cached) [2]
  - cxx11_auto_declarations  : yes (cached) [2]
  - cxx11_constexpr          : yes (cached) [2]
  - cxx11_defaulted_functions : yes (cached) [2]
  - cxx11_final              : yes (cached) [2]
  - cxx11_hdr_mutex          : yes (cached) [2]
  - cxx11_hdr_tuple          : yes (cached) [2]
  - cxx11_lambdas            : yes (cached) [2]
  - cxx11_noexcept           : yes (cached) [2]
  - cxx11_nullptr            : yes (cached) [2]
  - cxx11_rvalue_references  : yes (cached) [2]
  - cxx11_template_aliases   : yes (cached) [2]
  - cxx11_thread_local       : yes (cached) [2]
  - cxx11_variadic_templates : yes (cached) [2]
  - has_icu builds           : yes (cached) [2]
warning: Graph library does not contain MPI-based parallel components.
note: to enable them, add "using mpi ;" to your user-config.jam.
note: to suppress this message, pass "--without-graph_parallel" to bjam.
  - zlib                     : yes (cached)
  - bzip2                    : yes (cached)
  - lzma                     : yes (cached)
  - zstd                     : yes (cached)
  - has_lzma_cputhreads builds : yes (cached) [2]
  - cxx11_decltype           : yes (cached) [2]
  - cxx11_basic_alignas      : yes (cached) [2]
  - iconv (libc)             : yes (cached) [2]
  - icu                      : yes (cached) [2]
  - cxx11_defaulted_moves    : yes (cached) [2]
  - cxx11_hdr_functional     : yes (cached) [2]
  - cxx11_hdr_type_traits    : yes (cached) [2]
  - cxx11_override           : yes (cached) [2]
  - cxx11_range_based_for    : yes (cached) [2]
  - cxx11_scoped_enums       : yes (cached) [2]
  - cxx11_smart_ptr          : yes (cached) [2]
  - cxx11_static_assert      : yes (cached) [2]
  - lockfree boost::atomic_flag : yes (cached) [2]
  - native atomic int32 supported : yes (cached) [2]
  - native syslog supported  : yes (cached) [2]
  - pthread supports robust mutexes : yes (cached) [2]
  - compiler supports SSSE3  : yes (cached) [2]
  - compiler supports AVX2   : yes (cached) [2]
  - gcc visibility           : yes (cached) [2]
  - sfinae_expr              : yes (cached) [2]
  - cxx11_unified_initialization_syntax : yes (cached) [2]
  - cxx11_hdr_initializer_list : yes (cached) [2]
  - cxx11_hdr_chrono         : yes (cached) [2]
  - cxx11_numeric_limits     : yes (cached) [2]
  - cxx11_hdr_array          : yes (cached) [2]
  - cxx11_hdr_atomic         : yes (cached) [2]
  - cxx11_allocator          : yes (cached) [2]
  - cxx11_explicit_conversion_operators : yes (cached) [2]
  - long double support      : yes (cached) [2]
warning: skipping optional Message Passing Interface (MPI) library.
note: to enable MPI support, add "using mpi ;" to user-config.jam.
note: to suppress this message, pass "--without-mpi" to bjam.
note: otherwise, you can safely ignore this message.
  - cxx11_char16_t           : yes (cached) [2]
  - cxx11_char32_t           : yes (cached) [2]
  - Has Large File Support   : yes (cached) [2]
  - Has attribute init_priority : yes (cached) [2]
  - libbacktrace builds      : yes (cached) [2]
  - addr2line builds         : yes (cached) [2]
  - WinDbg builds            : no  (cached) [2]
  - WinDbg builds            : no  (cached) [3]
  - WinDbgCached builds      : no  (cached) [2]
  - WinDbgCached builds      : no  (cached) [3]
  - BOOST_COMP_GNUC >= 4.3.0 : yes (cached) [2]
  - BOOST_COMP_GNUC >= 4.3.0 : yes (cached) [4]
  - cxx11_hdr_thread         : yes (cached) [2]
  - cxx11_hdr_regex          : yes (cached) [2]
  - compiler supports SSE2   : yes (cached) [4]
  - compiler supports SSE4.1 : yes (cached) [4]
  - has std::atomic_ref      : no  (cached) [4]
  - has statx                : yes (cached) [4]
  - has init_priority attribute : yes (cached) [4]
  - has stat::st_blksize     : yes (cached) [4]
  - has stat::st_mtim        : yes (cached) [4]
  - has stat::st_mtimensec   : no  (cached) [4]
  - has stat::st_mtimespec   : no  (cached) [4]
  - has stat::st_birthtim    : no  (cached) [4]
  - has stat::st_birthtimensec : no  (cached) [4]
  - has stat::st_birthtimespec : no  (cached) [4]
  - has fdopendir(O_NOFOLLOW) : yes (cached) [4]
  - has dirent::d_type       : yes (cached) [4]
  - has POSIX *at APIs       : yes (cached) [4]
  - cxx11_auto_declarations  : yes (cached) [4]
  - cxx11_constexpr          : yes (cached) [4]
  - cxx11_defaulted_functions : yes (cached) [4]
  - cxx11_final              : yes (cached) [4]
  - cxx11_hdr_mutex          : yes (cached) [4]
  - cxx11_hdr_tuple          : yes (cached) [4]
  - cxx11_lambdas            : yes (cached) [4]
  - cxx11_noexcept           : yes (cached) [4]
  - cxx11_nullptr            : yes (cached) [4]
  - cxx11_rvalue_references  : yes (cached) [4]
  - cxx11_template_aliases   : yes (cached) [4]
  - cxx11_thread_local       : yes (cached) [4]
  - cxx11_variadic_templates : yes (cached) [4]
  - has_icu builds           : yes (cached) [4]
  - zlib                     : yes (cached) [5]
  - bzip2                    : yes (cached) [5]
  - lzma                     : yes (cached) [5]
  - zstd                     : yes (cached) [5]
  - has_lzma_cputhreads builds : yes (cached) [4]
  - cxx11_decltype           : yes (cached) [4]
  - cxx11_basic_alignas      : yes (cached) [4]
  - iconv (libc)             : yes (cached) [4]
  - icu                      : yes (cached) [4]
  - cxx11_defaulted_moves    : yes (cached) [4]
  - cxx11_hdr_functional     : yes (cached) [4]
  - cxx11_hdr_type_traits    : yes (cached) [4]
  - cxx11_override           : yes (cached) [4]
  - cxx11_range_based_for    : yes (cached) [4]
  - cxx11_scoped_enums       : yes (cached) [4]
  - cxx11_smart_ptr          : yes (cached) [4]
  - cxx11_static_assert      : yes (cached) [4]
  - lockfree boost::atomic_flag : yes (cached) [4]
  - native atomic int32 supported : yes (cached) [4]
  - native syslog supported  : yes (cached) [4]
  - pthread supports robust mutexes : yes (cached) [4]
  - compiler supports SSSE3  : yes (cached) [4]
  - compiler supports AVX2   : yes (cached) [4]
  - gcc visibility           : yes (cached) [4]
  - sfinae_expr              : yes (cached) [4]
  - cxx11_unified_initialization_syntax : yes (cached) [4]
  - cxx11_hdr_initializer_list : yes (cached) [4]
  - cxx11_hdr_chrono         : yes (cached) [4]
  - cxx11_numeric_limits     : yes (cached) [4]
  - cxx11_hdr_array          : yes (cached) [4]
  - cxx11_hdr_atomic         : yes (cached) [4]
  - cxx11_allocator          : yes (cached) [4]
  - cxx11_explicit_conversion_operators : yes (cached) [4]
  - long double support      : yes (cached) [4]
  - cxx11_char16_t           : yes (cached) [4]
  - cxx11_char32_t           : yes (cached) [4]
  - Has Large File Support   : yes (cached) [4]
  - Has attribute init_priority : yes (cached) [4]
  - libbacktrace builds      : yes (cached) [4]
  - addr2line builds         : yes (cached) [4]
  - WinDbg builds            : no  (cached) [4]
  - WinDbg builds            : no  (cached) [6]
  - WinDbgCached builds      : no  (cached) [4]
  - WinDbgCached builds      : no  (cached) [6]
  - cxx11_hdr_thread         : yes (cached) [4]
  - cxx11_hdr_regex          : yes (cached) [4]

[1] gcc-12
[2] gcc-12/release/python-3.11/threading-multi/visibility-hidden
[3] gcc-12/release/build-no/python-3.11/threading-multi/visibility-hidden
[4] gcc-12/release/link-static/python-3.11/threading-multi/visibility-hidden
[5] link-static
[6] gcc-12/release/build-no/link-static/python-3.11/threading-multi/visibility-hidden

Component configuration:

  - atomic                   : building
  - chrono                   : building
  - container                : building
  - context                  : building
  - contract                 : building
  - coroutine                : building
  - date_time                : building
  - exception                : building
  - fiber                    : building
  - filesystem               : building
  - graph                    : building
  - graph_parallel           : building
  - headers                  : building
  - iostreams                : building
  - json                     : building
  - locale                   : building
  - log                      : building
  - math                     : building
  - mpi                      : building
  - nowide                   : building
  - program_options          : building
  - python                   : building
  - random                   : building
  - regex                    : building
  - serialization            : building
  - stacktrace               : building
  - system                   : building
  - test                     : building
  - thread                   : building
  - timer                    : building
  - type_erasure             : building
  - url                      : building
  - wave                     : building

...patience...
...patience...
...patience...
...patience...
...patience...
...patience...
...patience...
...found 52031 targets...

The b2 installation program stored the following include files in /usr/local/include/boost/

Shell

$ ls -w72 /usr/local/include/boost/
accumulators/                 make_unique.hpp
algorithm/                    math/
align/                        math_fwd.hpp
align.hpp                     mem_fn.hpp
aligned_storage.hpp           memory_order.hpp
any/                          metaparse/
any.hpp                       metaparse.hpp
archive/                      move/
array.hpp                     mp11/
asio/                         mp11.hpp
asio.hpp                      mpi/
assert/                       mpi.hpp
assert.hpp                    mpl/
assign/                       msm/
assign.hpp                    multi_array/
atomic/                       multi_array.hpp
atomic.hpp                    multi_index/
beast/                        multi_index_container.hpp
beast.hpp                     multi_index_container_fwd.hpp
bimap/                        multiprecision/
bimap.hpp                     mysql/
bind/                         mysql.hpp
bind.hpp                      next_prior.hpp
blank.hpp                     non_type.hpp
blank_fwd.hpp                 noncopyable.hpp
call_traits.hpp               nondet_random.hpp
callable_traits/              none.hpp
callable_traits.hpp           none_t.hpp
cast.hpp                      nowide/
cerrno.hpp                    numeric/
checked_delete.hpp            operators.hpp
chrono/                       operators_v1.hpp
chrono.hpp                    optional/
circular_buffer/              optional.hpp
circular_buffer.hpp           outcome/
circular_buffer_fwd.hpp       outcome.hpp
compat/                       parameter/
compatibility/                parameter.hpp
compressed_pair.hpp           pending/
compute/                      pfr/
compute.hpp                   pfr.hpp
concept/                      phoenix/
concept_archetype.hpp         phoenix.hpp
concept_check/                pointee.hpp
concept_check.hpp             pointer_cast.hpp
config/                       pointer_to_other.hpp
config.hpp                    poly_collection/
container/                    polygon/
container_hash/               polymorphic_cast.hpp
context/                      polymorphic_pointer_cast.hpp
contract/                     pool/
contract.hpp                  predef/
contract_macro.hpp            predef.h
convert/                      preprocessor/
convert.hpp                   preprocessor.hpp
core/                         process/
coroutine/                    process.hpp
coroutine2/                   program_options/
crc.hpp                       program_options.hpp
cregex.hpp                    progress.hpp
cstdfloat.hpp                 property_map/
cstdint.hpp                   property_tree/
cstdlib.hpp                   proto/
current_function.hpp          ptr_container/
cxx11_char_types.hpp          python/
date_time/                    python.hpp
date_time.hpp                 qvm/
describe/                     qvm.hpp
describe.hpp                  qvm_lite.hpp
detail/                       random/
dll/                          random.hpp
dll.hpp                       range/
dynamic_bitset/               range.hpp
dynamic_bitset.hpp            ratio/
dynamic_bitset_fwd.hpp        ratio.hpp
enable_shared_from_this.hpp   rational.hpp
endian/                       ref.hpp
endian.hpp                    regex/
exception/                    regex.h
exception_ptr.hpp             regex.hpp
fiber/                        regex_fwd.hpp
filesystem/                   safe_numerics/
filesystem.hpp                scope_exit.hpp
flyweight/                    scoped_array.hpp
flyweight.hpp                 scoped_ptr.hpp
foreach.hpp                   serialization/
foreach_fwd.hpp               shared_array.hpp
format/                       shared_container_iterator.hpp
format.hpp                    shared_ptr.hpp
function/                     signals2/
function.hpp                  signals2.hpp
function_equal.hpp            smart_ptr/
function_output_iterator.hpp  smart_ptr.hpp
function_types/               sort/
functional/                   spirit/
functional.hpp                spirit.hpp
fusion/                       stacktrace/
generator_iterator.hpp        stacktrace.hpp
geometry/                     statechart/
geometry.hpp                  static_assert.hpp
get_pointer.hpp               static_string/
gil/                          static_string.hpp
gil.hpp                       stl_interfaces/
graph/                        swap.hpp
hana/                         system/
hana.hpp                      system.hpp
heap/                         test/
histogram/                    thread/
histogram.hpp                 thread.hpp
hof/                          throw_exception.hpp
hof.hpp                       timer/
icl/                          timer.hpp
implicit_cast.hpp             token_functions.hpp
indirect_reference.hpp        token_iterator.hpp
integer/                      tokenizer.hpp
integer.hpp                   tti/
integer_fwd.hpp               tuple/
integer_traits.hpp            type.hpp
interprocess/                 type_erasure/
intrusive/                    type_index/
intrusive_ptr.hpp             type_index.hpp
io/                           type_traits/
io_fwd.hpp                    type_traits.hpp
iostreams/                    typeof/
is_placeholder.hpp            units/
iterator/                     unordered/
iterator.hpp                  unordered_map.hpp
iterator_adaptors.hpp         unordered_set.hpp
json/                         url/
json.hpp                      url.hpp
lambda/                       utility/
lambda2/                      utility.hpp
lambda2.hpp                   uuid/
leaf/                         variant/
leaf.hpp                      variant.hpp
lexical_cast/                 variant2/
lexical_cast.hpp              variant2.hpp
limits.hpp                    version.hpp
local_function/               visit_each.hpp
local_function.hpp            vmd/
locale/                       wave/
locale.hpp                    wave.hpp
lockfree/                     weak_ptr.hpp
log/                          winapi/
logic/                        xpressive/
make_default.hpp              yap/
make_shared.hpp

The b2 installation program stored the following libraries in /usr/local/lib/:

Shell

$ ls -w72 /usr/local/lib
cmake/
libboost_atomic.a
libboost_atomic.so@
libboost_atomic.so.1.83.0*
libboost_chrono.a
libboost_chrono.so@
libboost_chrono.so.1.83.0*
libboost_container.a
libboost_container.so@
libboost_container.so.1.83.0*
libboost_context.a
libboost_context.so@
libboost_context.so.1.83.0*
libboost_contract.a
libboost_contract.so@
libboost_contract.so.1.83.0*
libboost_coroutine.a
libboost_coroutine.so@
libboost_coroutine.so.1.83.0*
libboost_date_time.a
libboost_date_time.so@
libboost_date_time.so.1.83.0*
libboost_exception.a
libboost_fiber.a
libboost_fiber.so@
libboost_fiber.so.1.83.0*
libboost_filesystem.a
libboost_filesystem.so@
libboost_filesystem.so.1.83.0*
libboost_graph.a
libboost_graph.so@
libboost_graph.so.1.83.0*
libboost_iostreams.a
libboost_iostreams.so@
libboost_iostreams.so.1.83.0*
libboost_json.a
libboost_json.so@
libboost_json.so.1.83.0*
libboost_locale.a
libboost_locale.so@
libboost_locale.so.1.83.0*
libboost_log.a
libboost_log.so@
libboost_log.so.1.83.0*
libboost_log_setup.a
libboost_log_setup.so@
libboost_log_setup.so.1.83.0*
libboost_math_c99.a
libboost_math_c99.so@
libboost_math_c99.so.1.83.0*
libboost_math_c99f.a
libboost_math_c99f.so@
libboost_math_c99f.so.1.83.0*
libboost_math_c99l.a
libboost_math_c99l.so@
libboost_math_c99l.so.1.83.0*
libboost_math_tr1.a
libboost_math_tr1.so@
libboost_math_tr1.so.1.83.0*
libboost_math_tr1f.a
libboost_math_tr1f.so@
libboost_math_tr1f.so.1.83.0*
libboost_math_tr1l.a
libboost_math_tr1l.so@
libboost_math_tr1l.so.1.83.0*
libboost_nowide.a
libboost_nowide.so@
libboost_nowide.so.1.83.0*
libboost_prg_exec_monitor.a
libboost_prg_exec_monitor.so@
libboost_prg_exec_monitor.so.1.83.0*
libboost_program_options.a
libboost_program_options.so@
libboost_program_options.so.1.83.0*
libboost_random.a
libboost_random.so@
libboost_random.so.1.83.0*
libboost_regex.a
libboost_regex.so@
libboost_regex.so.1.83.0*
libboost_serialization.a
libboost_serialization.so@
libboost_serialization.so.1.83.0*
libboost_stacktrace_addr2line.a
libboost_stacktrace_addr2line.so@
libboost_stacktrace_addr2line.so.1.83.0*
libboost_stacktrace_backtrace.a
libboost_stacktrace_backtrace.so@
libboost_stacktrace_backtrace.so.1.83.0*
libboost_stacktrace_basic.a
libboost_stacktrace_basic.so@
libboost_stacktrace_basic.so.1.83.0*
libboost_stacktrace_noop.a
libboost_stacktrace_noop.so@
libboost_stacktrace_noop.so.1.83.0*
libboost_system.a
libboost_system.so@
libboost_system.so.1.83.0*
libboost_test_exec_monitor.a
libboost_thread.a
libboost_thread.so@
libboost_thread.so.1.83.0*
libboost_timer.a
libboost_timer.so@
libboost_timer.so.1.83.0*
libboost_type_erasure.a
libboost_type_erasure.so@
libboost_type_erasure.so.1.83.0*
libboost_unit_test_framework.a
libboost_unit_test_framework.so@
libboost_unit_test_framework.so.1.83.0*
libboost_url.a
libboost_url.so@
libboost_url.so.1.83.0*
libboost_wave.a
libboost_wave.so@
libboost_wave.so.1.83.0*
libboost_wserialization.a
libboost_wserialization.so@
libboost_wserialization.so.1.83.0*
node_modules/
python3.11/

Verify Boost System Libraries

The ldconfig command manages Linux system libraries. Here is the help message:

Shell

$ man ldconfig
ldconfig(8)             System Manager's Manual            ldconfig(8)

NAME
        ldconfig - configure dynamic linker run-time bindings

SYNOPSIS
        /sbin/ldconfig [-nNvVX] [-C cache] [-f conf] [-r root]
                      directory ...

        /sbin/ldconfig -l [-v] library ...

        /sbin/ldconfig -p

DESCRIPTION
        ldconfig creates the necessary links and cache to the most  re‐
        cent shared libraries found in the directories specified on the
        command line, in the file /etc/ld.so.conf, and in  the  trusted
        directories,  /lib  and /usr/lib.  On some 64-bit architectures
        such as x86-64, /lib and /usr/lib are the  trusted  directories
        for  32-bit libraries, while /lib64 and /usr/lib64 are used for
        64-bit libraries.

        The cache is used by the run-time linker, ld.so or ld-linux.so.
        ldconfig  checks  the  header and filenames of the libraries it
        encounters when determining which versions  should  have  their
        links  updated.   ldconfig  should normally be run by the supe‐
        ruser as it may require write permission on some root owned di‐
        rectories and files.

        ldconfig  will  look only at files that are named lib*.so* (for
        regular shared objects) or ld-*.so* (for the dynamic loader it‐
        self).   Other files will be ignored.  Also, ldconfig expects a
        certain pattern to how the symbolic links are set up, like this
        example, where the middle file (libfoo.so.1 here) is the SONAME
        for the library:

            libfoo.so -> libfoo.so.1 -> libfoo.so.1.12

        Failure to follow this pattern may result in compatibility  is‐
        sues after an upgrade.

OPTIONS
        -c fmt
        --format=fmt
              (Since  glibc 2.2) Use cache format fmt, which is one of
              old, new, or compat.  Since glibc 2.32, the  default  is
              new.  Before that, it was compat.

        -C cache
              Use cache instead of /etc/ld.so.cache.

        -f conf
              Use conf instead of /etc/ld.so.conf.

        -i
        --ignore-aux-cache
              (Since glibc 2.7) Ignore auxiliary cache file.

        -l     (Since  glibc  2.2)  Interpret  each operand as a libary
              name and configure its links.  Intended for use only  by
              experts.

        -n     Process  only  the  directories specified on the command
              line; don't process the trusted directories,  nor  those
              specified in /etc/ld.so.conf.  Implies -N.

        -N     Don't  rebuild  the cache.  Unless -X is also specified,
              links are still updated.

        -p
        --print-cache
              Print the lists of directories and  candidate  libraries
              stored in the current cache.

        -r root
              Change to and use root as the root directory.

        -v
        --verbose
              Verbose mode.  Print current version number, the name of
              each directory as it is scanned, and any links that  are
              created.  Overrides quiet mode.

        -V
        --version
              Print program version.

        -X     Don't  update  links.   Unless -N is also specified, the
              cache is still rebuilt.

FILES
        /lib/ld.so
              is the run-time linker/loader.
        /etc/ld.so.conf
              contains a list of directories, one per line,  in  which
              to search for libraries.
        /etc/ld.so.cache
              contains  an  ordered list of libraries found in the di‐
              rectories specified in /etc/ld.so.conf, as well as those
              found in the trusted directories.

SEE ALSO
        ldd(1), ld.so(8)

Linux man-pages 6.03          2023-01-07                   ldconfig(8)

The Boost installation procedure automatically causes the newly built Boost libraries to be added to the configured system libraries when bootstrap.sh is not invoked with a target path, for example by specifying the --with-python-root option.

To manually add the newly built Boost libraries to the system load path, use ldconfig. The following adds the newly compiled and installed Boost libraries in /usr/local/lib/ (and any other libraries that might happen to be in that directory) to the ld.so cache.

Shell

$ sudo ldconfig -n /usr/local/lib

We can verify that the library cache now contains the newly built Boost libraries by using ldconfig:

Shell

$ ldconfig -p | grep boost
libboost_wserialization.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_wserialization.so.1.83.0
libboost_wserialization.so (libc6,x86-64) => /usr/local/lib/libboost_wserialization.so
libboost_wave.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_wave.so.1.83.0
libboost_wave.so (libc6,x86-64) => /usr/local/lib/libboost_wave.so
libboost_url.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_url.so.1.83.0
libboost_url.so (libc6,x86-64) => /usr/local/lib/libboost_url.so
libboost_unit_test_framework.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_unit_test_framework.so.1.83.0
libboost_unit_test_framework.so (libc6,x86-64) => /usr/local/lib/libboost_unit_test_framework.so
libboost_type_erasure.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_type_erasure.so.1.83.0
libboost_type_erasure.so (libc6,x86-64) => /usr/local/lib/libboost_type_erasure.so
libboost_timer.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_timer.so.1.83.0
libboost_timer.so (libc6,x86-64) => /usr/local/lib/libboost_timer.so
libboost_thread.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_thread.so.1.83.0
libboost_thread.so (libc6,x86-64) => /usr/local/lib/libboost_thread.so
libboost_system.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_system.so.1.83.0
libboost_system.so (libc6,x86-64) => /usr/local/lib/libboost_system.so
libboost_stacktrace_noop.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_stacktrace_noop.so.1.83.0
libboost_stacktrace_noop.so (libc6,x86-64) => /usr/local/lib/libboost_stacktrace_noop.so
libboost_stacktrace_basic.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_stacktrace_basic.so.1.83.0
libboost_stacktrace_basic.so (libc6,x86-64) => /usr/local/lib/libboost_stacktrace_basic.so
libboost_stacktrace_backtrace.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_stacktrace_backtrace.so.1.83.0
libboost_stacktrace_backtrace.so (libc6,x86-64) => /usr/local/lib/libboost_stacktrace_backtrace.so
libboost_stacktrace_addr2line.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_stacktrace_addr2line.so.1.83.0
libboost_stacktrace_addr2line.so (libc6,x86-64) => /usr/local/lib/libboost_stacktrace_addr2line.so
libboost_serialization.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_serialization.so.1.83.0
libboost_serialization.so (libc6,x86-64) => /usr/local/lib/libboost_serialization.so
libboost_regex.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_regex.so.1.83.0
libboost_regex.so.1.74.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/libboost_regex.so.1.74.0
libboost_regex.so (libc6,x86-64) => /usr/local/lib/libboost_regex.so
libboost_random.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_random.so.1.83.0
libboost_random.so (libc6,x86-64) => /usr/local/lib/libboost_random.so
libboost_python311.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_python311.so.1.83.0
libboost_python311.so (libc6,x86-64) => /usr/local/lib/libboost_python311.so
libboost_program_options.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_program_options.so.1.83.0
libboost_program_options.so (libc6,x86-64) => /usr/local/lib/libboost_program_options.so
libboost_prg_exec_monitor.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_prg_exec_monitor.so.1.83.0
libboost_prg_exec_monitor.so (libc6,x86-64) => /usr/local/lib/libboost_prg_exec_monitor.so
libboost_nowide.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_nowide.so.1.83.0
libboost_nowide.so (libc6,x86-64) => /usr/local/lib/libboost_nowide.so
libboost_math_tr1l.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_math_tr1l.so.1.83.0
libboost_math_tr1l.so (libc6,x86-64) => /usr/local/lib/libboost_math_tr1l.so
libboost_math_tr1f.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_math_tr1f.so.1.83.0
libboost_math_tr1f.so (libc6,x86-64) => /usr/local/lib/libboost_math_tr1f.so
libboost_math_tr1.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_math_tr1.so.1.83.0
libboost_math_tr1.so (libc6,x86-64) => /usr/local/lib/libboost_math_tr1.so
libboost_math_c99l.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_math_c99l.so.1.83.0
libboost_math_c99l.so (libc6,x86-64) => /usr/local/lib/libboost_math_c99l.so
libboost_math_c99f.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_math_c99f.so.1.83.0
libboost_math_c99f.so (libc6,x86-64) => /usr/local/lib/libboost_math_c99f.so
libboost_math_c99.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_math_c99.so.1.83.0
libboost_math_c99.so (libc6,x86-64) => /usr/local/lib/libboost_math_c99.so
libboost_log_setup.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_log_setup.so.1.83.0
libboost_log_setup.so (libc6,x86-64) => /usr/local/lib/libboost_log_setup.so
libboost_log.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_log.so.1.83.0
libboost_log.so (libc6,x86-64) => /usr/local/lib/libboost_log.so
libboost_locale.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_locale.so.1.83.0
libboost_locale.so (libc6,x86-64) => /usr/local/lib/libboost_locale.so
libboost_json.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_json.so.1.83.0
libboost_json.so (libc6,x86-64) => /usr/local/lib/libboost_json.so
libboost_iostreams.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_iostreams.so.1.83.0
libboost_iostreams.so (libc6,x86-64) => /usr/local/lib/libboost_iostreams.so
libboost_graph.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_graph.so.1.83.0
libboost_graph.so (libc6,x86-64) => /usr/local/lib/libboost_graph.so
libboost_filesystem.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_filesystem.so.1.83.0
libboost_filesystem.so (libc6,x86-64) => /usr/local/lib/libboost_filesystem.so
libboost_fiber.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_fiber.so.1.83.0
libboost_fiber.so (libc6,x86-64) => /usr/local/lib/libboost_fiber.so
libboost_date_time.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_date_time.so.1.83.0
libboost_date_time.so (libc6,x86-64) => /usr/local/lib/libboost_date_time.so
libboost_coroutine.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_coroutine.so.1.83.0
libboost_coroutine.so (libc6,x86-64) => /usr/local/lib/libboost_coroutine.so
libboost_contract.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_contract.so.1.83.0
libboost_contract.so (libc6,x86-64) => /usr/local/lib/libboost_contract.so
libboost_context.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_context.so.1.83.0
libboost_context.so (libc6,x86-64) => /usr/local/lib/libboost_context.so
libboost_container.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_container.so.1.83.0
libboost_container.so (libc6,x86-64) => /usr/local/lib/libboost_container.so
libboost_chrono.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_chrono.so.1.83.0
libboost_chrono.so (libc6,x86-64) => /usr/local/lib/libboost_chrono.so
libboost_atomic.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_atomic.so.1.83.0
libboost_atomic.so (libc6,x86-64) => /usr/local/lib/libboost_atomic.so

As a further check, examine the full path of an arbitrary include file (src.hpp) and list the Boost libraries:

Shell

$ locate src.hpp
/usr/include/boost/asio/impl/src.hpp
/usr/include/boost/asio/ssl/impl/src.hpp
/usr/include/boost/beast/src.hpp 

$ find /usr/local/lib/ -iname *boost*
/usr/local/lib/libboost_math_tr1f.a
/usr/local/lib/libboost_log_setup.so
/usr/local/lib/libboost_system.so.1.83.0
/usr/local/lib/libboost_system.so
/usr/local/lib/libboost_serialization.a
/usr/local/lib/libboost_context.a
/usr/local/lib/libboost_nowide.so
/usr/local/lib/libboost_contract.so.1.83.0
/usr/local/lib/libboost_prg_exec_monitor.a
/usr/local/lib/libboost_stacktrace_backtrace.so
/usr/local/lib/libboost_iostreams.so
/usr/local/lib/libboost_math_c99.so.1.83.0
/usr/local/lib/libboost_stacktrace_addr2line.so
/usr/local/lib/libboost_locale.so.1.83.0
/usr/local/lib/libboost_math_c99l.so.1.83.0
/usr/local/lib/libboost_timer.a
/usr/local/lib/libboost_atomic.so.1.83.0
/usr/local/lib/libboost_url.so.1.83.0
/usr/local/lib/libboost_math_c99.a
/usr/local/lib/libboost_date_time.so
/usr/local/lib/libboost_fiber.a
/usr/local/lib/libboost_filesystem.so.1.83.0
/usr/local/lib/libboost_date_time.a
/usr/local/lib/libboost_log_setup.a
/usr/local/lib/libboost_type_erasure.so
/usr/local/lib/libboost_json.so.1.83.0
/usr/local/lib/libboost_wave.so.1.83.0
/usr/local/lib/libboost_unit_test_framework.so
/usr/local/lib/libboost_type_erasure.so.1.83.0
/usr/local/lib/libboost_random.so.1.83.0
/usr/local/lib/libboost_filesystem.so
/usr/local/lib/libboost_stacktrace_basic.a
/usr/local/lib/libboost_math_c99.so
/usr/local/lib/libboost_regex.a
/usr/local/lib/libboost_coroutine.so.1.83.0
/usr/local/lib/libboost_filesystem.a
/usr/local/lib/libboost_date_time.so.1.83.0
/usr/local/lib/libboost_wave.a
/usr/local/lib/libboost_regex.so
/usr/local/lib/libboost_container.a
/usr/local/lib/libboost_locale.a
/usr/local/lib/libboost_graph.so
/usr/local/lib/libboost_graph.so.1.83.0
/usr/local/lib/libboost_math_tr1f.so.1.83.0
/usr/local/lib/libboost_math_tr1.so.1.83.0
/usr/local/lib/libboost_program_options.so.1.83.0
/usr/local/lib/libboost_iostreams.so.1.83.0
/usr/local/lib/libboost_math_c99f.so.1.83.0
/usr/local/lib/libboost_nowide.so.1.83.0
/usr/local/lib/libboost_thread.so.1.83.0
/usr/local/lib/libboost_unit_test_framework.a
/usr/local/lib/cmake/boost_math_c99f-1.83.0
/usr/local/lib/cmake/boost_math_c99f-1.83.0/libboost_math_c99f-variant-shared.cmake
/usr/local/lib/cmake/boost_math_c99f-1.83.0/boost_math_c99f-config.cmake
/usr/local/lib/cmake/boost_math_c99f-1.83.0/libboost_math_c99f-variant-static.cmake
/usr/local/lib/cmake/boost_math_c99f-1.83.0/boost_math_c99f-config-version.cmake
/usr/local/lib/cmake/boost_test_exec_monitor-1.83.0
/usr/local/lib/cmake/boost_test_exec_monitor-1.83.0/libboost_test_exec_monitor-variant-static.cmake
/usr/local/lib/cmake/boost_test_exec_monitor-1.83.0/libboost_test_exec_monitor-variant-shared.cmake
/usr/local/lib/cmake/boost_test_exec_monitor-1.83.0/boost_test_exec_monitor-config.cmake
/usr/local/lib/cmake/boost_test_exec_monitor-1.83.0/boost_test_exec_monitor-config-version.cmake
/usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0
/usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0/libboost_stacktrace_backtrace-variant-shared.cmake
/usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0/boost_stacktrace_backtrace-config-version.cmake
/usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0/libboost_stacktrace_backtrace-variant-static.cmake
/usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0/boost_stacktrace_backtrace-config.cmake
/usr/local/lib/cmake/boost_wave-1.83.0
/usr/local/lib/cmake/boost_wave-1.83.0/libboost_wave-variant-shared.cmake
/usr/local/lib/cmake/boost_wave-1.83.0/boost_wave-config-version.cmake
/usr/local/lib/cmake/boost_wave-1.83.0/libboost_wave-variant-static.cmake
/usr/local/lib/cmake/boost_wave-1.83.0/boost_wave-config.cmake
/usr/local/lib/cmake/boost_log-1.83.0
/usr/local/lib/cmake/boost_log-1.83.0/libboost_log-variant-shared.cmake
/usr/local/lib/cmake/boost_log-1.83.0/boost_log-config.cmake
/usr/local/lib/cmake/boost_log-1.83.0/libboost_log-variant-static.cmake
/usr/local/lib/cmake/boost_log-1.83.0/boost_log-config-version.cmake
/usr/local/lib/cmake/boost_stacktrace_basic-1.83.0
/usr/local/lib/cmake/boost_stacktrace_basic-1.83.0/boost_stacktrace_basic-config.cmake
/usr/local/lib/cmake/boost_stacktrace_basic-1.83.0/boost_stacktrace_basic-config-version.cmake
/usr/local/lib/cmake/boost_stacktrace_basic-1.83.0/libboost_stacktrace_basic-variant-static.cmake
/usr/local/lib/cmake/boost_stacktrace_basic-1.83.0/libboost_stacktrace_basic-variant-shared.cmake
/usr/local/lib/cmake/boost_coroutine-1.83.0
/usr/local/lib/cmake/boost_coroutine-1.83.0/boost_coroutine-config-version.cmake
/usr/local/lib/cmake/boost_coroutine-1.83.0/boost_coroutine-config.cmake
/usr/local/lib/cmake/boost_coroutine-1.83.0/libboost_coroutine-variant-shared.cmake
/usr/local/lib/cmake/boost_coroutine-1.83.0/libboost_coroutine-variant-static.cmake
/usr/local/lib/cmake/boost_timer-1.83.0
/usr/local/lib/cmake/boost_timer-1.83.0/boost_timer-config-version.cmake
/usr/local/lib/cmake/boost_timer-1.83.0/boost_timer-config.cmake
/usr/local/lib/cmake/boost_timer-1.83.0/libboost_timer-variant-shared.cmake
/usr/local/lib/cmake/boost_timer-1.83.0/libboost_timer-variant-static.cmake
/usr/local/lib/cmake/boost_program_options-1.83.0
/usr/local/lib/cmake/boost_program_options-1.83.0/boost_program_options-config.cmake
/usr/local/lib/cmake/boost_program_options-1.83.0/libboost_program_options-variant-static.cmake
/usr/local/lib/cmake/boost_program_options-1.83.0/libboost_program_options-variant-shared.cmake
/usr/local/lib/cmake/boost_program_options-1.83.0/boost_program_options-config-version.cmake
/usr/local/lib/cmake/boost_math_tr1f-1.83.0
/usr/local/lib/cmake/boost_math_tr1f-1.83.0/libboost_math_tr1f-variant-static.cmake
/usr/local/lib/cmake/boost_math_tr1f-1.83.0/boost_math_tr1f-config.cmake
/usr/local/lib/cmake/boost_math_tr1f-1.83.0/boost_math_tr1f-config-version.cmake
/usr/local/lib/cmake/boost_math_tr1f-1.83.0/libboost_math_tr1f-variant-shared.cmake
/usr/local/lib/cmake/boost_date_time-1.83.0
/usr/local/lib/cmake/boost_date_time-1.83.0/libboost_date_time-variant-static.cmake
/usr/local/lib/cmake/boost_date_time-1.83.0/libboost_date_time-variant-shared.cmake
/usr/local/lib/cmake/boost_date_time-1.83.0/boost_date_time-config.cmake
/usr/local/lib/cmake/boost_date_time-1.83.0/boost_date_time-config-version.cmake
/usr/local/lib/cmake/boost_math_c99l-1.83.0
/usr/local/lib/cmake/boost_math_c99l-1.83.0/libboost_math_c99l-variant-static.cmake
/usr/local/lib/cmake/boost_math_c99l-1.83.0/libboost_math_c99l-variant-shared.cmake
/usr/local/lib/cmake/boost_math_c99l-1.83.0/boost_math_c99l-config.cmake
/usr/local/lib/cmake/boost_math_c99l-1.83.0/boost_math_c99l-config-version.cmake
/usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0
/usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0/libboost_prg_exec_monitor-variant-shared.cmake
/usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0/libboost_prg_exec_monitor-variant-static.cmake
/usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0/boost_prg_exec_monitor-config.cmake
/usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0/boost_prg_exec_monitor-config-version.cmake
/usr/local/lib/cmake/boost_wserialization-1.83.0
/usr/local/lib/cmake/boost_wserialization-1.83.0/libboost_wserialization-variant-shared.cmake
/usr/local/lib/cmake/boost_wserialization-1.83.0/boost_wserialization-config-version.cmake
/usr/local/lib/cmake/boost_wserialization-1.83.0/libboost_wserialization-variant-static.cmake
/usr/local/lib/cmake/boost_wserialization-1.83.0/boost_wserialization-config.cmake
/usr/local/lib/cmake/boost_serialization-1.83.0
/usr/local/lib/cmake/boost_serialization-1.83.0/boost_serialization-config.cmake
/usr/local/lib/cmake/boost_serialization-1.83.0/libboost_serialization-variant-static.cmake
/usr/local/lib/cmake/boost_serialization-1.83.0/boost_serialization-config-version.cmake
/usr/local/lib/cmake/boost_serialization-1.83.0/libboost_serialization-variant-shared.cmake
/usr/local/lib/cmake/boost_random-1.83.0
/usr/local/lib/cmake/boost_random-1.83.0/boost_random-config.cmake
/usr/local/lib/cmake/boost_random-1.83.0/libboost_random-variant-static.cmake
/usr/local/lib/cmake/boost_random-1.83.0/libboost_random-variant-shared.cmake
/usr/local/lib/cmake/boost_random-1.83.0/boost_random-config-version.cmake
/usr/local/lib/cmake/boost_log_setup-1.83.0
/usr/local/lib/cmake/boost_log_setup-1.83.0/boost_log_setup-config-version.cmake
/usr/local/lib/cmake/boost_log_setup-1.83.0/libboost_log_setup-variant-shared.cmake
/usr/local/lib/cmake/boost_log_setup-1.83.0/libboost_log_setup-variant-static.cmake
/usr/local/lib/cmake/boost_log_setup-1.83.0/boost_log_setup-config.cmake
/usr/local/lib/cmake/boost_exception-1.83.0
/usr/local/lib/cmake/boost_exception-1.83.0/boost_exception-config.cmake
/usr/local/lib/cmake/boost_exception-1.83.0/boost_exception-config-version.cmake
/usr/local/lib/cmake/boost_system-1.83.0
/usr/local/lib/cmake/boost_system-1.83.0/libboost_system-variant-shared.cmake
/usr/local/lib/cmake/boost_system-1.83.0/boost_system-config-version.cmake
/usr/local/lib/cmake/boost_system-1.83.0/libboost_system-variant-static.cmake
/usr/local/lib/cmake/boost_system-1.83.0/boost_system-config.cmake
/usr/local/lib/cmake/boost_python-1.83.0
/usr/local/lib/cmake/boost_python-1.83.0/boost_python-config.cmake
/usr/local/lib/cmake/boost_python-1.83.0/boost_python-config-version.cmake
/usr/local/lib/cmake/boost_thread-1.83.0
/usr/local/lib/cmake/boost_thread-1.83.0/libboost_thread-variant-shared.cmake
/usr/local/lib/cmake/boost_thread-1.83.0/boost_thread-config.cmake
/usr/local/lib/cmake/boost_thread-1.83.0/boost_thread-config-version.cmake
/usr/local/lib/cmake/boost_thread-1.83.0/libboost_thread-variant-static.cmake
/usr/local/lib/cmake/boost_json-1.83.0
/usr/local/lib/cmake/boost_json-1.83.0/libboost_json-variant-shared.cmake
/usr/local/lib/cmake/boost_json-1.83.0/libboost_json-variant-static.cmake
/usr/local/lib/cmake/boost_json-1.83.0/boost_json-config.cmake
/usr/local/lib/cmake/boost_json-1.83.0/boost_json-config-version.cmake
/usr/local/lib/cmake/boost_iostreams-1.83.0
/usr/local/lib/cmake/boost_iostreams-1.83.0/libboost_iostreams-variant-shared.cmake
/usr/local/lib/cmake/boost_iostreams-1.83.0/boost_iostreams-config.cmake
/usr/local/lib/cmake/boost_iostreams-1.83.0/libboost_iostreams-variant-static.cmake
/usr/local/lib/cmake/boost_iostreams-1.83.0/boost_iostreams-config-version.cmake
/usr/local/lib/cmake/boost_math_c99-1.83.0
/usr/local/lib/cmake/boost_math_c99-1.83.0/libboost_math_c99-variant-static.cmake
/usr/local/lib/cmake/boost_math_c99-1.83.0/boost_math_c99-config-version.cmake
/usr/local/lib/cmake/boost_math_c99-1.83.0/libboost_math_c99-variant-shared.cmake
/usr/local/lib/cmake/boost_math_c99-1.83.0/boost_math_c99-config.cmake
/usr/local/lib/cmake/boost_unit_test_framework-1.83.0
/usr/local/lib/cmake/boost_unit_test_framework-1.83.0/libboost_unit_test_framework-variant-static.cmake
/usr/local/lib/cmake/boost_unit_test_framework-1.83.0/boost_unit_test_framework-config-version.cmake
/usr/local/lib/cmake/boost_unit_test_framework-1.83.0/libboost_unit_test_framework-variant-shared.cmake
/usr/local/lib/cmake/boost_unit_test_framework-1.83.0/boost_unit_test_framework-config.cmake
/usr/local/lib/cmake/boost_filesystem-1.83.0
/usr/local/lib/cmake/boost_filesystem-1.83.0/libboost_filesystem-variant-static.cmake
/usr/local/lib/cmake/boost_filesystem-1.83.0/boost_filesystem-config.cmake
/usr/local/lib/cmake/boost_filesystem-1.83.0/boost_filesystem-config-version.cmake
/usr/local/lib/cmake/boost_filesystem-1.83.0/libboost_filesystem-variant-shared.cmake
/usr/local/lib/cmake/Boost-1.83.0
/usr/local/lib/cmake/Boost-1.83.0/BoostConfigVersion.cmake
/usr/local/lib/cmake/Boost-1.83.0/BoostConfig.cmake
/usr/local/lib/cmake/boost_headers-1.83.0
/usr/local/lib/cmake/boost_headers-1.83.0/boost_headers-config.cmake
/usr/local/lib/cmake/boost_headers-1.83.0/boost_headers-config-version.cmake
/usr/local/lib/cmake/boost_nowide-1.83.0
/usr/local/lib/cmake/boost_nowide-1.83.0/libboost_nowide-variant-static.cmake
/usr/local/lib/cmake/boost_nowide-1.83.0/libboost_nowide-variant-shared.cmake
/usr/local/lib/cmake/boost_nowide-1.83.0/boost_nowide-config.cmake
/usr/local/lib/cmake/boost_nowide-1.83.0/boost_nowide-config-version.cmake
/usr/local/lib/cmake/boost_atomic-1.83.0
/usr/local/lib/cmake/boost_atomic-1.83.0/boost_atomic-config-version.cmake
/usr/local/lib/cmake/boost_atomic-1.83.0/boost_atomic-config.cmake
/usr/local/lib/cmake/boost_atomic-1.83.0/libboost_atomic-variant-shared.cmake
/usr/local/lib/cmake/boost_atomic-1.83.0/libboost_atomic-variant-static.cmake
/usr/local/lib/cmake/boost_graph-1.83.0
/usr/local/lib/cmake/boost_graph-1.83.0/boost_graph-config-version.cmake
/usr/local/lib/cmake/boost_graph-1.83.0/libboost_graph-variant-static.cmake
/usr/local/lib/cmake/boost_graph-1.83.0/libboost_graph-variant-shared.cmake
/usr/local/lib/cmake/boost_graph-1.83.0/boost_graph-config.cmake
/usr/local/lib/cmake/boost_math_tr1-1.83.0
/usr/local/lib/cmake/boost_math_tr1-1.83.0/boost_math_tr1-config-version.cmake
/usr/local/lib/cmake/boost_math_tr1-1.83.0/libboost_math_tr1-variant-static.cmake
/usr/local/lib/cmake/boost_math_tr1-1.83.0/libboost_math_tr1-variant-shared.cmake
/usr/local/lib/cmake/boost_math_tr1-1.83.0/boost_math_tr1-config.cmake
/usr/local/lib/cmake/boost_graph_parallel-1.83.0
/usr/local/lib/cmake/boost_graph_parallel-1.83.0/boost_graph_parallel-config-version.cmake
/usr/local/lib/cmake/boost_graph_parallel-1.83.0/boost_graph_parallel-config.cmake
/usr/local/lib/cmake/boost_stacktrace_noop-1.83.0
/usr/local/lib/cmake/boost_stacktrace_noop-1.83.0/libboost_stacktrace_noop-variant-static.cmake
/usr/local/lib/cmake/boost_stacktrace_noop-1.83.0/boost_stacktrace_noop-config-version.cmake
/usr/local/lib/cmake/boost_stacktrace_noop-1.83.0/boost_stacktrace_noop-config.cmake
/usr/local/lib/cmake/boost_stacktrace_noop-1.83.0/libboost_stacktrace_noop-variant-shared.cmake
/usr/local/lib/cmake/boost_mpi-1.83.0
/usr/local/lib/cmake/boost_mpi-1.83.0/boost_mpi-config-version.cmake
/usr/local/lib/cmake/boost_mpi-1.83.0/boost_mpi-config.cmake
/usr/local/lib/cmake/boost_type_erasure-1.83.0
/usr/local/lib/cmake/boost_type_erasure-1.83.0/libboost_type_erasure-variant-shared.cmake
/usr/local/lib/cmake/boost_type_erasure-1.83.0/libboost_type_erasure-variant-static.cmake
/usr/local/lib/cmake/boost_type_erasure-1.83.0/boost_type_erasure-config-version.cmake
/usr/local/lib/cmake/boost_type_erasure-1.83.0/boost_type_erasure-config.cmake
/usr/local/lib/cmake/boost_chrono-1.83.0
/usr/local/lib/cmake/boost_chrono-1.83.0/boost_chrono-config-version.cmake
/usr/local/lib/cmake/boost_chrono-1.83.0/libboost_chrono-variant-static.cmake
/usr/local/lib/cmake/boost_chrono-1.83.0/libboost_chrono-variant-shared.cmake
/usr/local/lib/cmake/boost_chrono-1.83.0/boost_chrono-config.cmake
/usr/local/lib/cmake/boost_math_tr1l-1.83.0
/usr/local/lib/cmake/boost_math_tr1l-1.83.0/libboost_math_tr1l-variant-shared.cmake
/usr/local/lib/cmake/boost_math_tr1l-1.83.0/boost_math_tr1l-config.cmake
/usr/local/lib/cmake/boost_math_tr1l-1.83.0/boost_math_tr1l-config-version.cmake
/usr/local/lib/cmake/boost_math_tr1l-1.83.0/libboost_math_tr1l-variant-static.cmake
/usr/local/lib/cmake/boost_contract-1.83.0
/usr/local/lib/cmake/boost_contract-1.83.0/libboost_contract-variant-shared.cmake
/usr/local/lib/cmake/boost_contract-1.83.0/boost_contract-config-version.cmake
/usr/local/lib/cmake/boost_contract-1.83.0/boost_contract-config.cmake
/usr/local/lib/cmake/boost_contract-1.83.0/libboost_contract-variant-static.cmake
/usr/local/lib/cmake/boost_fiber-1.83.0
/usr/local/lib/cmake/boost_fiber-1.83.0/libboost_fiber-variant-static.cmake
/usr/local/lib/cmake/boost_fiber-1.83.0/boost_fiber-config-version.cmake
/usr/local/lib/cmake/boost_fiber-1.83.0/libboost_fiber-variant-shared.cmake
/usr/local/lib/cmake/boost_fiber-1.83.0/boost_fiber-config.cmake
/usr/local/lib/cmake/BoostDetectToolset-1.83.0.cmake
/usr/local/lib/cmake/boost_context-1.83.0
/usr/local/lib/cmake/boost_context-1.83.0/boost_context-config.cmake
/usr/local/lib/cmake/boost_context-1.83.0/libboost_context-variant-shared.cmake
/usr/local/lib/cmake/boost_context-1.83.0/libboost_context-variant-static.cmake
/usr/local/lib/cmake/boost_context-1.83.0/boost_context-config-version.cmake
/usr/local/lib/cmake/boost_container-1.83.0
/usr/local/lib/cmake/boost_container-1.83.0/libboost_container-variant-shared.cmake
/usr/local/lib/cmake/boost_container-1.83.0/boost_container-config-version.cmake
/usr/local/lib/cmake/boost_container-1.83.0/boost_container-config.cmake
/usr/local/lib/cmake/boost_container-1.83.0/libboost_container-variant-static.cmake
/usr/local/lib/cmake/boost_regex-1.83.0
/usr/local/lib/cmake/boost_regex-1.83.0/libboost_regex-variant-shared.cmake
/usr/local/lib/cmake/boost_regex-1.83.0/boost_regex-config.cmake
/usr/local/lib/cmake/boost_regex-1.83.0/libboost_regex-variant-static.cmake
/usr/local/lib/cmake/boost_regex-1.83.0/boost_regex-config-version.cmake
/usr/local/lib/cmake/boost_url-1.83.0
/usr/local/lib/cmake/boost_url-1.83.0/boost_url-config-version.cmake
/usr/local/lib/cmake/boost_url-1.83.0/libboost_url-variant-static.cmake
/usr/local/lib/cmake/boost_url-1.83.0/libboost_url-variant-shared.cmake
/usr/local/lib/cmake/boost_url-1.83.0/boost_url-config.cmake
/usr/local/lib/cmake/boost_locale-1.83.0
/usr/local/lib/cmake/boost_locale-1.83.0/libboost_locale-variant-static.cmake
/usr/local/lib/cmake/boost_locale-1.83.0/boost_locale-config.cmake
/usr/local/lib/cmake/boost_locale-1.83.0/libboost_locale-variant-shared.cmake
/usr/local/lib/cmake/boost_locale-1.83.0/boost_locale-config-version.cmake
/usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0
/usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0/libboost_stacktrace_addr2line-variant-static.cmake
/usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0/libboost_stacktrace_addr2line-variant-shared.cmake
/usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0/boost_stacktrace_addr2line-config-version.cmake
/usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0/boost_stacktrace_addr2line-config.cmake
/usr/local/lib/libboost_fiber.so.1.83.0
/usr/local/lib/libboost_chrono.a
/usr/local/lib/libboost_serialization.so
/usr/local/lib/libboost_fiber.so
/usr/local/lib/libboost_wserialization.a
/usr/local/lib/libboost_unit_test_framework.so.1.83.0
/usr/local/lib/libboost_stacktrace_noop.a
/usr/local/lib/libboost_iostreams.a
/usr/local/lib/libboost_math_c99f.a
/usr/local/lib/libboost_exception.a
/usr/local/lib/libboost_type_erasure.a
/usr/local/lib/libboost_thread.a
/usr/local/lib/libboost_chrono.so.1.83.0
/usr/local/lib/libboost_math_c99l.so
/usr/local/lib/libboost_log_setup.so.1.83.0
/usr/local/lib/libboost_stacktrace_addr2line.a
/usr/local/lib/libboost_container.so.1.83.0
/usr/local/lib/libboost_context.so
/usr/local/lib/libboost_program_options.so
/usr/local/lib/libboost_stacktrace_basic.so
/usr/local/lib/libboost_graph.a
/usr/local/lib/libboost_prg_exec_monitor.so.1.83.0
/usr/local/lib/libboost_atomic.a
/usr/local/lib/libboost_math_tr1l.a
/usr/local/lib/libboost_log.so.1.83.0
/usr/local/lib/libboost_random.a
/usr/local/lib/libboost_math_tr1f.so
/usr/local/lib/libboost_contract.a
/usr/local/lib/libboost_atomic.so
/usr/local/lib/libboost_serialization.so.1.83.0
/usr/local/lib/libboost_thread.so
/usr/local/lib/libboost_wave.so
/usr/local/lib/libboost_program_options.a
/usr/local/lib/libboost_context.so.1.83.0
/usr/local/lib/libboost_coroutine.a
/usr/local/lib/libboost_json.a
/usr/local/lib/libboost_stacktrace_noop.so.1.83.0
/usr/local/lib/libboost_wserialization.so.1.83.0
/usr/local/lib/libboost_math_tr1.a
/usr/local/lib/libboost_chrono.so
/usr/local/lib/libboost_stacktrace_backtrace.a
/usr/local/lib/libboost_stacktrace_addr2line.so.1.83.0
/usr/local/lib/libboost_stacktrace_basic.so.1.83.0
/usr/local/lib/libboost_math_c99l.a
/usr/local/lib/libboost_log.so
/usr/local/lib/libboost_log.a
/usr/local/lib/libboost_math_tr1l.so
/usr/local/lib/libboost_container.so
/usr/local/lib/libboost_coroutine.so
/usr/local/lib/libboost_prg_exec_monitor.so
/usr/local/lib/libboost_nowide.a
/usr/local/lib/libboost_url.so
/usr/local/lib/libboost_url.a
/usr/local/lib/libboost_math_c99f.so
/usr/local/lib/libboost_stacktrace_backtrace.so.1.83.0
/usr/local/lib/libboost_test_exec_monitor.a
/usr/local/lib/libboost_json.so
/usr/local/lib/libboost_wserialization.so
/usr/local/lib/libboost_system.a
/usr/local/lib/libboost_regex.so.1.83.0
/usr/local/lib/libboost_contract.so
/usr/local/lib/libboost_stacktrace_noop.so
/usr/local/lib/libboost_locale.so
/usr/local/lib/libboost_math_tr1.so
/usr/local/lib/libboost_timer.so.1.83.0
/usr/local/lib/libboost_random.so
/usr/local/lib/libboost_math_tr1l.so.1.83.0
/usr/local/lib/libboost_timer.so

Clean Up

Many Ubuntu/Debian packages are left behind by the Boost installation program. You can view them as follows:

Shell

$ apt --dry-run autoremove | grep -Po '^Remv \K[^ ]+'
libboost-mpi-python-dev
libboost-mpi-python1.74-dev
libboost-mpi-python1.74.0
mpi-default-bin
libboost-mpi-dev
libboost-mpi1.74-dev
mpi-default-dev
libopenmpi-dev
openmpi-bin
libcoarrays-openmpi-dev
libopenmpi3
libucx0
libfabric1
ibverbs-providers
libboost-atomic-dev
libboost-coroutine-dev
libboost-coroutine1.74-dev
libboost-fiber-dev
libboost-fiber1.74-dev
libboost-context1.74-dev
libboost-type-erasure-dev
libboost-type-erasure1.74-dev
libboost-thread1.74-dev
libboost-log-dev
libboost-log1.74-dev
libboost-atomic1.74-dev
libboost-atomic1.74.0
libboost-chrono-dev
libboost-timer-dev
libboost-timer1.74-dev
libboost-chrono1.74-dev
libboost-timer1.74.0
libboost-chrono1.74.0
libboost-container-dev
libboost-container1.74-dev
libboost-container1.74.0
libboost-context-dev
libboost-fiber1.74.0
libboost-coroutine1.74.0
libboost-context1.74.0
libboost-date-time-dev
libboost-date-time1.74-dev
libboost-date-time1.74.0
libboost-dev
libboost-exception-dev
libboost-exception1.74-dev
libboost-filesystem-dev
libboost-wave-dev
libboost-wave1.74-dev
libboost-filesystem1.74-dev
libboost-graph-dev
libboost-graph-parallel-dev
libboost-graph-parallel1.74-dev
libboost-graph-parallel1.74.0
libboost-graph1.74-dev
libboost-graph1.74.0
libboost-iostreams-dev
libboost-iostreams1.74-dev
libboost-locale-dev
libboost-locale1.74-dev
libboost-log1.74.0
libboost-math-dev
libboost-math1.74-dev
libboost-math1.74.0
libboost-mpi1.74.0
libboost-nowide-dev
libboost-nowide1.74-dev
libboost-nowide1.74.0
libboost-numpy-dev
libboost-numpy1.74-dev
libboost-numpy1.74.0
libboost-program-options-dev
libboost-program-options1.74-dev
libboost-program-options1.74.0
libboost-python-dev
libboost-python1.74-dev
libboost-python1.74.0
libboost-random-dev
libboost-random1.74-dev
libboost-random1.74.0
libboost-regex-dev
libboost-regex1.74-dev
libboost-serialization-dev
libboost-serialization1.74-dev
libboost-serialization1.74.0
libboost-stacktrace-dev
libboost-stacktrace1.74-dev
libboost-stacktrace1.74.0
libboost-system-dev
libboost-system1.74-dev
libboost-system1.74.0
libboost-test-dev
libboost-test1.74-dev
libboost-test1.74.0
libboost-thread-dev
libboost-tools-dev
libboost-type-erasure1.74.0
libboost-wave1.74.0
libboost1.74-dev
libboost1.74-tools-dev
libcaf-openmpi-3
libpmix-dev
libevent-dev
libevent-extra-2.1-7
libevent-openssl-2.1-7
libpmix2
libevent-pthreads-2.1-7
libhwloc-dev
libhwloc-plugins
libhwloc15
libibverbs-dev
librdmacm1
libibverbs1
libjs-jquery-ui
libmunge2
libnl-route-3-dev
libnl-3-dev
libnuma-dev
libpsm-infinipath1
libpsm2-2
openmpi-common

Remove the packages as follows:

Shell

$ yes | sudo apt autoremove
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  ibverbs-providers libboost-atomic-dev libboost-atomic1.74-dev libboost-atomic1.74.0 libboost-chrono-dev libboost-chrono1.74-dev libboost-chrono1.74.0
  libboost-container-dev libboost-container1.74-dev libboost-container1.74.0 libboost-context-dev libboost-context1.74-dev libboost-context1.74.0
  libboost-coroutine-dev libboost-coroutine1.74-dev libboost-coroutine1.74.0 libboost-date-time-dev libboost-date-time1.74-dev libboost-date-time1.74.0
  libboost-dev libboost-exception-dev libboost-exception1.74-dev libboost-fiber-dev libboost-fiber1.74-dev libboost-fiber1.74.0 libboost-filesystem-dev
  libboost-filesystem1.74-dev libboost-graph-dev libboost-graph-parallel-dev libboost-graph-parallel1.74-dev libboost-graph-parallel1.74.0
  libboost-graph1.74-dev libboost-graph1.74.0 libboost-iostreams-dev libboost-iostreams1.74-dev libboost-locale-dev libboost-locale1.74-dev
  libboost-log-dev libboost-log1.74-dev libboost-log1.74.0 libboost-math-dev libboost-math1.74-dev libboost-math1.74.0 libboost-mpi-dev
  libboost-mpi-python-dev libboost-mpi-python1.74-dev libboost-mpi-python1.74.0 libboost-mpi1.74-dev libboost-mpi1.74.0 libboost-nowide-dev
  libboost-nowide1.74-dev libboost-nowide1.74.0 libboost-numpy-dev libboost-numpy1.74-dev libboost-numpy1.74.0 libboost-program-options-dev
  libboost-program-options1.74-dev libboost-program-options1.74.0 libboost-python-dev libboost-python1.74-dev libboost-python1.74.0 libboost-random-dev
  libboost-random1.74-dev libboost-random1.74.0 libboost-regex-dev libboost-regex1.74-dev libboost-serialization-dev libboost-serialization1.74-dev
  libboost-serialization1.74.0 libboost-stacktrace-dev libboost-stacktrace1.74-dev libboost-stacktrace1.74.0 libboost-system-dev libboost-system1.74-dev
  libboost-system1.74.0 libboost-test-dev libboost-test1.74-dev libboost-test1.74.0 libboost-thread-dev libboost-thread1.74-dev libboost-timer-dev
  libboost-timer1.74-dev libboost-timer1.74.0 libboost-tools-dev libboost-type-erasure-dev libboost-type-erasure1.74-dev libboost-type-erasure1.74.0
  libboost-wave-dev libboost-wave1.74-dev libboost-wave1.74.0 libboost1.74-dev libboost1.74-tools-dev libcaf-openmpi-3 libcoarrays-openmpi-dev libevent-dev
  libevent-extra-2.1-7 libevent-openssl-2.1-7 libevent-pthreads-2.1-7 libfabric1 libhwloc-dev libhwloc-plugins libhwloc15 libibverbs-dev libibverbs1
  libjs-jquery-ui libmunge2 libnl-3-dev libnl-route-3-dev libnuma-dev libopenmpi-dev libopenmpi3 libpmix-dev libpmix2 libpsm-infinipath1 libpsm2-2
  librdmacm1 libucx0 mpi-default-bin mpi-default-dev openmpi-bin openmpi-common
0 upgraded, 0 newly installed, 121 to remove and 4 not upgraded.
After this operation, 413 MB disk space will be freed.
(Reading database ... 399799 files and directories currently installed.)
Removing libboost-mpi-python-dev (1.74.0.3ubuntu7) ...
Removing libboost-mpi-python1.74-dev (1.74.0-18.1ubuntu3) ...
Removing libboost-mpi-python1.74.0 (1.74.0-18.1ubuntu3) ...
Removing mpi-default-bin (1.14) ...
Removing libboost-mpi-dev (1.74.0.3ubuntu7) ...
Removing libboost-mpi1.74-dev (1.74.0-18.1ubuntu3) ...
Removing mpi-default-dev (1.14) ...
Removing libopenmpi-dev:amd64 (4.1.4-3ubuntu2) ...
Removing libcoarrays-openmpi-dev:amd64 (2.10.1-1) ...
update-alternatives: warning: alternative /usr/bin/caf.openmpi (part of link group caf) doesn't exist; removing from list of alternatives
update-alternatives: warning: /etc/alternatives/caf is dangling; it will be updated with best choice
Removing libboost-atomic-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-coroutine-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-coroutine1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-fiber-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-fiber1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-type-erasure-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-type-erasure1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-log-dev (1.74.0.3ubuntu7) ...
Removing libboost-log1.74-dev (1.74.0-18.1ubuntu3) ...
Removing libboost-chrono-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-timer-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-timer1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-timer1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-container-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-container1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-container1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-context-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-fiber1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-coroutine1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-date-time-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-exception-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-exception1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-filesystem-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-wave-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-wave1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-filesystem1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-graph-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-graph-parallel-dev (1.74.0.3ubuntu7) ...
  Removing libboost-graph-parallel1.74-dev (1.74.0-18.1ubuntu3) ...
  Removing libboost-graph-parallel1.74.0 (1.74.0-18.1ubuntu3) ...
  Removing libboost-graph1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-graph1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-iostreams-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-iostreams1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-locale-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-locale1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-log1.74.0 (1.74.0-18.1ubuntu3) ...
  Removing libboost-math-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-math1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-math1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-mpi1.74.0 (1.74.0-18.1ubuntu3) ...
  Removing libboost-nowide-dev (1.74.0.3ubuntu7) ...
  Removing libboost-nowide1.74-dev (1.74.0-18.1ubuntu3) ...
  Removing libboost-nowide1.74.0 (1.74.0-18.1ubuntu3) ...
  Removing libboost-numpy-dev (1.74.0.3ubuntu7) ...
  Removing libboost-numpy1.74-dev (1.74.0-18.1ubuntu3) ...
  Removing libboost-numpy1.74.0 (1.74.0-18.1ubuntu3) ...
  Removing libboost-program-options-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-program-options1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-program-options1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-python-dev (1.74.0.3ubuntu7) ...
  Removing libboost-python1.74-dev (1.74.0-18.1ubuntu3) ...
  Removing libboost-python1.74.0 (1.74.0-18.1ubuntu3) ...
  Removing libboost-random-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-random1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-random1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-regex-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-regex1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-serialization-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-stacktrace-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-stacktrace1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-stacktrace1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-system-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-test-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-test1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-test1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-thread-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-tools-dev (1.74.0.3ubuntu7) ...
  Removing libboost-type-erasure1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-wave1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost1.74-tools-dev (1.74.0-18.1ubuntu3) ...
  Removing libcaf-openmpi-3:amd64 (2.10.1-1) ...
  Removing libpmix-dev:amd64 (4.2.2-1) ...
  Removing libevent-dev (2.1.12-stable-8ubuntu3) ...
  Removing libevent-extra-2.1-7:amd64 (2.1.12-stable-8ubuntu3) ...
  Removing libevent-openssl-2.1-7:amd64 (2.1.12-stable-8ubuntu3) ...
  Removing libhwloc-dev:amd64 (2.9.0-1) ...
  Removing libibverbs-dev:amd64 (44.0-2) ...
  Removing libjs-jquery-ui (1.13.2+dfsg-1) ...
  Removing libnl-route-3-dev:amd64 (3.7.0-0.2) ...
  Removing libnl-3-dev:amd64 (3.7.0-0.2) ...
  Removing libnuma-dev:amd64 (2.0.16-1) ...
  Removing openmpi-bin (4.1.4-3ubuntu2) ...
  Removing libopenmpi3:amd64 (4.1.4-3ubuntu2) ...
  Removing libucx0:amd64 (1.13.1-1) ...
  Removing libfabric1:amd64 (1.17.0-3) ...
  Removing ibverbs-providers:amd64 (44.0-2) ...
  Removing libboost-context1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-thread1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-atomic1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-atomic1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-chrono1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-chrono1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-context1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-date-time1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-date-time1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-serialization1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-serialization1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-system1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-system1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libpmix2:amd64 (4.2.2-1) ...
  Removing libevent-pthreads-2.1-7:amd64 (2.1.12-stable-8ubuntu3) ...
  Removing libhwloc-plugins:amd64 (2.9.0-1) ...
  Removing libhwloc15:amd64 (2.9.0-1) ...
  Removing librdmacm1:amd64 (44.0-2) ...
  Removing libibverbs1:amd64 (44.0-2) ...
  Removing libmunge2 (0.5.15-2) ...
  Removing libpsm-infinipath1 (3.3+20.604758e7-6.2) ...
  update-alternatives: warning: alternative /usr/lib/libpsm1/libpsm_infinipath.so.1.16 (part of link group libpsm_infinipath.so.1) doesn't exist; removing from list of alternatives
  update-alternatives: warning: /etc/alternatives/libpsm_infinipath.so.1 is dangling; it will be updated with best choice
  Removing libpsm2-2 (11.2.185-2) ...
  Removing openmpi-common (4.1.4-3ubuntu2) ...
  Processing triggers for man-db (2.11.2-1) ...
  Processing triggers for libc-bin (2.37-0ubuntu2) ...
  /sbin/ldconfig.real: /usr/lib/wsl/lib/libcuda.so.1 is not a symbolic link

Go Forth and Get Boosted

😁

Boost usage as a C++ general-pupose library is widespread throughout a diverse range of industries and requirements.

PDF Manipulation

2023-09-08T00:00:00-04:00

Installation

The pdftk website discusses pdktk (a F/OSS command-line program), a Windows GUI, and mentions the book PDF Hacks.

An alternative free GUI is PDFTK Builder.

Install the command-line pdftk program on Ubuntu:

Shell

$ sudo apt install pdftk

This is the pdftk help text:

Shell

$ man pdftk
pdftk.pdftk-java(1)     General Commands Manual    pdftk.pdftk-java(1)

NAME
        pdftk - A handy tool for manipulating PDF

SYNOPSIS
        pdftk <input PDF files | - | PROMPT>
            [ input_pw <input PDF owner passwords | PROMPT> ]
            [ <operation> <operation arguments> ]
            [ output <output filename | - | PROMPT> ]
            [ encrypt_40bit | encrypt_128bit | encrypt_aes128 ]
            [ allow <permissions> ]
            [ owner_pw <owner password | PROMPT> ]
            [ user_pw <user password | PROMPT> ]
            [ flatten ] [ need_appearances ]
            [ compress | uncompress ]
            [ keep_first_id | keep_final_id ] [ drop_xfa ] [ drop_xmp
        ]
            [ replacement_font <font name> ]
            [ verbose ] [ dont_ask | do_ask ]
        Where:
            <operation> may be empty, or:
            [ cat | shuffle | burst | rotate |
              generate_fdf | fill_form |
              background | multibackground |
              stamp | multistamp |
              dump_data | dump_data_utf8 |
              dump_data_fields | dump_data_fields_utf8 |
              dump_data_annots |
              update_info | update_info_utf8 |
              attach_files | unpack_files ]

        For Complete Help: pdftk --help

DESCRIPTION
        If PDF is electronic paper, then pdftk is an electronic staple-
        remover, hole-punch, binder, secret-decoder-ring, and X-Ray-
        glasses.  Pdftk is a simple tool for doing everyday things with
        PDF documents.  Use it to:

        * Merge PDF Documents or Collate PDF Page Scans
        * Split PDF Pages into a New Document
        * Rotate PDF Documents or Pages
        * Decrypt Input as Necessary (Password Required)
        * Encrypt Output as Desired
        * Fill PDF Forms with X/FDF Data and/or Flatten Forms
        * Generate FDF Data Stencils from PDF Forms
        * Apply a Background Watermark or a Foreground Stamp
        * Report PDF Metrics, Bookmarks and Metadata
        * Add/Update PDF Metrics, Bookmarks or Metadata
        * Attach Files to PDF Pages or the PDF Document
        * Unpack PDF Attachments
        * Burst a PDF Document into Single Pages
        * Uncompress and Re-Compress Page Streams
        * Repair Corrupted PDF (Where Possible)

OPTIONS
        A summary of options is included below.

        --help, -h
              Show this summary of options.

        <input PDF files | - | PROMPT>
              A list of the input PDF files. If you plan to combine
              these PDFs (without using handles) then list files in
              the order you want them combined.  Use - to pass a sin‐
              gle PDF into pdftk via stdin.  Input files can be asso‐
              ciated with handles, where a handle is one or more up‐
              per-case letters:

              <input PDF handle>=<input PDF filename>

              Handles are often omitted.  They are useful when speci‐
              fying PDF passwords or page ranges, later.

              For example: A=input1.pdf QT=input2.pdf M=input3.pdf

        [input_pw <input PDF owner passwords | PROMPT>]
              Input PDF owner passwords, if necessary, are associated
              with files by using their handles:

              <input PDF handle>=<input PDF file owner password>

              If handles are not given, then passwords are associated
              with input files by order.

              Most pdftk features require that encrypted input PDF are
              accompanied by the ~owner~ password. If the input PDF
              has no owner password, then the user password must be
              given, instead.  If the input PDF has no passwords, then
              no password should be given.

              When running in do_ask mode, pdftk will prompt you for a
              password if the supplied password is incorrect or none
              was given.

        [<operation> <operation arguments>]
              Available operations are: cat, shuffle, burst, rotate,
              generate_fdf, fill_form, background, multibackground,
              stamp, multistamp, dump_data, dump_data_utf8,
              dump_data_fields, dump_data_fields_utf8, dump_data_an‐
              nots, update_info, update_info_utf8, attach_files, un‐
              pack_files. Some operations takes additional arguments,
              described below.

              If this optional argument is omitted, then pdftk runs in
              'filter' mode.  Filter mode takes only one PDF input and
              creates a new PDF after applying all of the output op‐
              tions, like encryption and compression.

          cat [<page ranges>]
                  Assembles (catenates) pages from input PDFs to create
                  a new PDF. Use cat to merge PDF pages or to split PDF
                  pages from documents. You can also use it to rotate
                  PDF pages. Page order in the new PDF is specified by
                  the order of the given page ranges. Page ranges are
                  described like this:

                  <input PDF handle>[<begin page number>[-<end page
                  number>[<qualifier>]]][<page rotation>]

                  Where the handle identifies one of the input PDF
                  files, and the beginning and ending page numbers are
                  one-based references to pages in the PDF file.  The
                  qualifier can be even, odd, or ~, and the page rota‐
                  tion can be north, south, east, west, left, right, or
                  down.

                  If a PDF handle is given but no pages are specified,
                  then the entire PDF is used. If no pages are speci‐
                  fied for any of the input PDFs, then the input PDFs'
                  bookmarks are also merged and included in the output.

                  If the handle is omitted from the page range, then
                  the pages are taken from the first input PDF.

                  The even qualifier causes pdftk to use only the even-
                  numbered PDF pages, so 1-6even yields pages 2, 4 and
                  6 in that order.  6-1even yields pages 6, 4 and 2 in
                  that order.

                  The odd qualifier works similarly to the even.

                  Pages can be subtracted from a page range using the ~
                  qualifier followed by a page range. For instance,
                  1-20~5-6 and 1-20~5~6 are equivalent to 1-4 7-20, and
                  ~5 yields all pages except page 5. Depending on your
                  shell, you may need to quote this argument because of
                  the ~ at the beginning.

                  The page rotation setting can cause pdftk to rotate
                  pages and documents.  Each option sets the page rota‐
                  tion as follows (in degrees): north: 0, east: 90,
                  south: 180, west: 270, left: -90, right: +90, down:
                  +180. left, right, and down make relative adjustments
                  to a page's rotation.

                  If no arguments are passed to cat, then pdftk com‐
                  bines all input PDFs in the order they were given to
                  create the output.

                  NOTES:
                  * <end page number> may be less than <begin page num‐
                  ber>.
                  * The keyword end may be used to reference the final
                  page of a document instead of a page number.
                  * Reference a single page by omitting the ending page
                  number.
                  * The handle may be used alone to represent the en‐
                  tire PDF document, e.g., B1-end is the same as B.
                  * You can reference page numbers in reverse order by
                  prefixing them with the letter r. For example, page
                  r1 is the last page of the document, r2 is the next-
                  to-last page of the document, and rend is the first
                  page of the document. You can use this prefix in
                  ranges, too, for example r3-r1 is the last three
                  pages of a PDF.

                  Page Range Examples without Handles:
                  1\-endeast – rotate entire document 90 degrees
                  5 11 20 – take single pages from input PDF
                  5-25oddwest – take odd pages in range, rotate 90 de‐
                  grees
                  6-1 – reverse pages in range from input PDF

                  Page Range Examples Using Handles:
                  Say A=in1.pdf B=in2.pdf, then:
                  A1-21 – take range from in1.pdf
                  Bend-1odd – take all odd pages from in2.pdf in re‐
                  verse order
                  A72 – take a single page from in1.pdf
                  A1-21 Beven A72 – assemble pages from both in1.pdf
                  and in2.pdf
                  Awest – rotate entire in1.pdf document 90 degrees
                  B – use all of in2.pdf
                  A2-30evenleft – take the even pages from the range,
                  remove 90 degrees from each page's rotation
                  A A – catenate in1.pdf with in1.pdf
                  Aevenwest Aoddeast – apply rotations to even pages,
                  odd pages from in1.pdf
                  Awest Bwest Bdown – catenate rotated documents

          shuffle [<page ranges>]
                  Collates pages from input PDFs to create a new PDF.
                  Works like the cat operation except that it takes one
                  page at a time from each page range to assemble the
                  output PDF.  If one range runs out of pages, it con‐
                  tinues with the remaining ranges.  Ranges can use all
                  of the features described above for cat, like reverse
                  page ranges, multiple ranges from a single PDF, and
                  page rotation.  This feature was designed to help
                  collate PDF pages after scanning paper documents.

          burst  Splits a single input PDF document into individual
                  pages. Also creates a report named doc_data.txt which
                  is the same as the output from dump_data.  The output
                  section can contain a printf-styled format string to
                  name these pages.  For example, if you want pages
                  named page_01.pdf, page_02.pdf, etc., pass output
                  page_%02d.pdf to pdftk. If the pattern is omitted,
                  then a default pattern g_%04d.pdf is appended and
                  produces pages named pg_0001.pdf, pg_0002.pdf, etc.
                  Encryption can be applied to the output by appending
                  output options such as owner_pw, e.g.:

                  pdftk in.pdf burst owner_pw foopass

          rotate [<page ranges>]
                  Takes a single input PDF and rotates just the speci‐
                  fied pages.  All other pages remain unchanged.  The
                  page order remains unchanged.  Specify the pages to
                  rotate using the same notation as you would with cat,
                  except you omit the pages that you aren't rotating:

                  [<begin page number>[-<end page number>[<quali‐
                  fier>]]][<page rotation>]

                  The qualifier can be even or odd, and the page rota‐
                  tion can be north, south, east, west, left, right, or
                  down.

                  Each option sets the page rotation as follows (in de‐
                  grees): north: 0, east: 90, south: 180, west: 270,
                  left: -90, right: +90, down: +180. left, right, and
                  down make relative adjustments to a page's rotation.

                  The given order of the pages doesn't change the page
                  order in the output.

          generate_fdf
                  Reads a single input PDF file and generates an FDF
                  file suitable for fill_form out of it to the given
                  output filename or (if no output is given) to stdout.
                  Does not create a new PDF.

          fill_form <FDF data filename | XFDF data filename | - |
          PROMPT>
                  Fills the single input PDF's form fields with the
                  data from an FDF file, XFDF file or stdin. Enter the
                  data filename after fill_form, or use - to pass the
                  data via stdin, like so:

                  pdftk form.pdf fill_form data.fdf output
                  form.filled.pdf

                  If the input FDF file includes Rich Text formatted
                  data in addition to plain text, then the Rich Text
                  data is packed into the form fields as well as the
                  plain text.  Pdftk also sets a flag that cues
                  Reader/Acrobat to generate new field appearances
                  based on the Rich Text data.  So when the user opens
                  the PDF, the viewer will create the Rich Text appear‐
                  ance on the spot.  If the user's PDF viewer does not
                  support Rich Text, then the user will see the plain
                  text data instead.  If you flatten this form before
                  Acrobat has a chance to create (and save) new field
                  appearances, then the plain text field data is what
                  you'll see.

                  Also see the flatten, need_appearances, and replace‐
                  ment_font options.

          background <background PDF filename | - | PROMPT>
                  Applies a PDF watermark to the background of a single
                  input PDF.  Pass the background PDF's filename after
                  background like so:

                  pdftk in.pdf background back.pdf output out.pdf

                  Pdftk uses only the first page from the background
                  PDF and applies it to every page of the input PDF.
                  This page is scaled and rotated as needed to fit the
                  input page.  You can use - to pass a background PDF
                  into pdftk via stdin.

                  If the input PDF does not have a transparent back‐
                  ground (such as a PDF created from page scans) then
                  the resulting background won't be visible – use the
                  stamp operation instead.

          multibackground <background PDF filename | - | PROMPT>
                  Same as the background operation, but applies each
                  page of the background PDF to the corresponding page
                  of the input PDF.  If the input PDF has more pages
                  than the stamp PDF, then the final stamp page is re‐
                  peated across these remaining pages in the input PDF.

          stamp <stamp PDF filename | - | PROMPT>
                  This behaves just like the background operation ex‐
                  cept it overlays the stamp PDF page on top of the in‐
                  put PDF document's pages.  This works best if the
                  stamp PDF page has a transparent background.

          multistamp <stamp PDF filename | - | PROMPT>
                  Same as the stamp operation, but applies each page of
                  the background PDF to the corresponding page of the
                  input PDF.  If the input PDF has more pages than the
                  stamp PDF, then the final stamp page is repeated
                  across these remaining pages in the input PDF.

          dump_data
                  Reads a single input PDF file and reports its meta‐
                  data, bookmarks (a/k/a outlines), page metrics (me‐
                  dia, rotation and labels), data embedded by STAMPtk
                  (see STAMPtk's embed option) and other data to the
                  given output filename or (if no output is given) to
                  stdout.  Non-ASCII characters are encoded as XML nu‐
                  merical entities.  Does not create a new PDF.

          dump_data_utf8
                  Same as dump_data except that the output is encoded
                  as UTF-8.

          dump_data_fields
                  Reads a single input PDF file and reports form field
                  statistics to the given output filename or (if no
                  output is given) to stdout. Non-ASCII characters are
                  encoded as XML numerical entities. Does not create a
                  new PDF.

          dump_data_fields_utf8
                  Same as dump_data_fields except that the output is
                  encoded as UTF-8.

          dump_data_annots
                  This operation currently reports only link annota‐
                  tions.  Reads a single input PDF file and reports an‐
                  notation information to the given output filename or
                  (if no output is given) to stdout. Non-ASCII charac‐
                  ters are encoded as XML numerical entities. Does not
                  create a new PDF.

          update_info <info data filename | - | PROMPT>
                  Changes the bookmarks, page labels, page sizes, page
                  rotations, and metadata in a single PDF's Info dic‐
                  tionary to match the input data file. The input data
                  file uses the same syntax as the output from
                  dump_data. Non-ASCII characters should be encoded as
                  XML numerical entities.

                  This operation does not change the metadata stored in
                  the PDF's XMP stream, if it has one. (For this reason
                  you should include a ModDate entry in your updated
                  info with a current date/timestamp, format: D:YYYYM‐
                  MDDHHmmSS, e.g. D:201307241346 – omitted data after
                  YYYY revert to default values.)

                  For example:

                  pdftk in.pdf update_info in.info output out.pdf

          update_info_utf8 <info data filename | - | PROMPT>
                  Same as update_info except that the input is encoded
                  as UTF-8.

          attach_files <attachment filenames | PROMPT> [to_page <page
          number | PROMPT> | relation <relationship>]
                  Packs arbitrary files into a PDF using PDF's file at‐
                  tachment features. More than one attachment may be
                  listed after attach_files. Attachments are added at
                  the document level unless the optional to_page option
                  is given, in which case the files are attached to the
                  given page number (the first page is 1, the final
                  page is end). Attachments at the document level may
                  be tagged with a relationship among Source, Data, Al‐
                  ternative, Supplement, and Unspecified (default).

                  For example:

                  pdftk in.pdf attach_files table1.html table2.html
                  to_page 6 output out.pdf

                  pdftk in.pdf attach_files in.tex relation Source out‐
                  put out.pdf

          unpack_files
                  Copies all of the attachments from the input PDF into
                  the current folder or to an output directory given
                  after output. For example:

                  pdftk report.pdf unpack_files output ~/atts/

                  or, interactively:

                  pdftk report.pdf unpack_files output PROMPT

        [output <output filename | - | PROMPT>]
              The output PDF filename may not be set to the name of an
              input filename. Use - to output to stdout.  When using
              the dump_data operation, use output to set the name of
              the output data file. When using the unpack_files opera‐
              tion, use output to set the name of an output directory.
              When using the burst operation, you can use output to
              control the resulting PDF page filenames (described
              above).

        [encrypt_40bit | encrypt_128bit | encrypt_aes128]
              If an output PDF user or owner password is given, the
              output PDF encryption algorithm defaults to AES-128. The
              weaker RC4 40-bit and RC4 128-bit algorithms can be cho‐
              sen by specifying encrypt_40bit or encrypt_128bit (dis‐
              couraged).

        [allow <permissions>]
              Permissions are applied to the output PDF only if an en‐
              cryption strength is specified or an owner or user pass‐
              word is given.  If permissions are not specified, they
              default to 'none,' which means all of the following fea‐
              tures are disabled.

              The permissions section may include one or more of the
              following features:

              Printing
                      Top Quality Printing

              DegradedPrinting
                      Lower Quality Printing

              ModifyContents
                      Also allows Assembly

              Assembly

              CopyContents
                      Also allows ScreenReaders

              ScreenReaders

              ModifyAnnotations
                      Also allows FillIn

              FillIn

              AllFeatures
                      Allows the user to perform all of the above, and
                      top quality printing.

        [owner_pw <owner password | PROMPT>]

        [user_pw <user password | PROMPT>]
              If an encryption strength is given but no passwords are
              supplied, then the owner and user passwords remain
              empty, which means that the resulting PDF may be opened
              and its security parameters altered by anybody.

        [compress | uncompress]
              These are only useful when you want to edit PDF code in
              a text editor like vim or emacs.  Remove PDF page stream
              compression by applying the uncompress filter. Use the
              compress filter to restore compression.

        [flatten]
              Use this option to merge an input PDF's interactive form
              fields (and their data) with the PDF's pages. Only one
              input PDF may be given. Sometimes used with the
              fill_form operation.

        [need_appearances]
              Sets a flag that cues Reader/Acrobat to generate new
              field appearances based on the form field values.  Use
              this when filling a form with non-ASCII text to ensure
              the best presentation in Adobe Reader or Acrobat.  It
              won't work when combined with the flatten option.

        [replacement_font <font name>]
              Use the specified font to display text in form fields.
              This option is useful when filling a form with non-ASCII
              text that is not supported by the fonts included in the
              input PDF. font name may be either the file name or the
              family name of a font, but using a file name is more re‐
              liable. Currently only TrueType fonts with Unicode text
              are supported.

        [keep_first_id | keep_final_id]
              When combining pages from multiple PDFs, use one of
              these options to copy the document ID from either the
              first or final input document into the new output PDF.
              Otherwise pdftk creates a new document ID for the output
              PDF. When no operation is given, pdftk always uses the
              ID from the (single) input PDF.

        [drop_xfa]
              If your input PDF is a form created using Acrobat 7 or
              Adobe Designer, then it probably has XFA data.  Filling
              such a form using pdftk yields a PDF with data that
              fails to display in Acrobat 7 (and 6?).  The workaround
              solution is to remove the form's XFA data, either before
              you fill the form using pdftk or at the time you fill
              the form. Using this option causes pdftk to omit the XFA
              data from the output PDF form.

              This option is only useful when running pdftk on a sin‐
              gle input PDF.  When assembling a PDF from multiple in‐
              puts using pdftk, any XFA data in the input is automati‐
              cally omitted.

        [drop_xmp]
              Many PDFs store document metadata using both an Info
              dictionary (old school) and an XMP stream (new school).
              Pdftk's update_info operation can update the Info dic‐
              tionary, but not the XMP stream.  The proper remedy for
              this is to include a ModDate entry in your updated info
              with a current date/timestamp. The date/timestamp format
              is: D:YYYYMMDDHHmmSS, e.g. D:201307241346 – omitted data
              after YYYY revert to default values. This newer ModDate
              should cue PDF viewers that the Info metadata is more
              current than the XMP data.

              Alternatively, you might prefer to remove the XMP stream
              from the PDF altogether – that's what this option does.
              Note that objects inside the PDF might have their own,
              separate XMP metadata streams, and that drop_xmp does
              not remove those.  It only removes the PDF's document-
              level XMP stream.

        [verbose]
              By default, pdftk runs quietly. Append verbose to the
              end and it will speak up.

        [dont_ask | do_ask]
              Depending on the compile-time settings (see
              ASK_ABOUT_WARNINGS), pdftk might prompt you for further
              input when it encounters a problem, such as a bad pass‐
              word. Override this default behavior by adding dont_ask
              (so pdftk won't ask you what to do) or do_ask (so pdftk
              will ask you what to do).

              When running in dont_ask mode, pdftk will over-write
              files with its output without notice.

EXAMPLES
        Collate scanned pages
          pdftk A=even.pdf B=odd.pdf shuffle A B output collated.pdf
          or if odd.pdf is in reverse order:
          pdftk A=even.pdf B=odd.pdf shuffle A Bend-1 output col‐
          lated.pdf

        The following examples use actual passwords as command line pa‐
        rameters, which is discouraged (see the SECURITY CONSIDERATIONS
        section).

        Decrypt a PDF
          pdftk secured.pdf input_pw foopass output unsecured.pdf

        Encrypt a PDF using AES-128 (the default), withhold all permis‐
        sions (the default)
          pdftk 1.pdf output 1.128.pdf owner_pw foopass

        Same as above, except password 'baz' must also be used to open
        output PDF
          pdftk 1.pdf output 1.128.pdf owner_pw foo user_pw baz

        Same as above, except printing is allowed (once the PDF is
        open)
          pdftk 1.pdf output 1.128.pdf owner_pw foo user_pw baz allow
          printing

        Apply RCA 40-bit encryption to output, revoking all permissions
        (the default). Set the owner PW to 'foopass'.
          pdftk 1.pdf 2.pdf cat output 3.pdf encrypt_40bit owner_pw
          foopass

        Join two files, one of which requires the password 'foopass'.
        The output is not encrypted.
          pdftk A=secured.pdf 2.pdf input_pw A=foopass cat output 3.pdf

        Join in1.pdf and in2.pdf into a new PDF, out1.pdf
          pdftk in1.pdf in2.pdf cat output out1.pdf
          or (using handles):
          pdftk A=in1.pdf B=in2.pdf cat A B output out1.pdf
          or (using wildcards):
          pdftk *.pdf cat output combined.pdf

        Remove page 13 from in1.pdf to create out1.pdf
          pdftk in.pdf cat 1-12 14-end output out1.pdf
          or:
          pdftk A=in1.pdf cat A1-12 A14-end output out1.pdf

        Uncompress PDF page streams for editing the PDF in a text edi‐
        tor (e.g., vim, emacs)
          pdftk doc.pdf output doc.unc.pdf uncompress

        Repair a PDF's corrupted XREF table and stream lengths, if pos‐
        sible
          pdftk broken.pdf output fixed.pdf

        Burst a single PDF document into pages and dump its data to
        doc_data.txt
          pdftk in.pdf burst

        Burst a single PDF document into encrypted pages. Allow low-
        quality printing
          pdftk in.pdf burst owner_pw foopass allow DegradedPrinting

        Write a report on PDF document metadata and bookmarks to re‐
        port.txt
          pdftk in.pdf dump_data output report.txt

        Rotate the first PDF page to 90 degrees clockwise
          pdftk in.pdf cat 1east 2-end output out.pdf

        Rotate an entire PDF document to 180 degrees
          pdftk in.pdf cat 1-endsouth output out.pdf

NOTES
        This is a port of pdftk to java. See
        https://gitlab.com/pdftk-java/pdftk
        The original program can be found at www.pdftk.com

AUTHOR
        Original author of pdftk is Sid Steward (sid.steward at pdflabs
        dot com).

SECURITY CONSIDERATIONS
        Passing a password as a command line parameter is insecure be‐
        cause it can get saved into the shell's history and be accessi‐
        ble by other users via /proc. Use the keyword PROMPT and input
        any passwords via standard input instead.

                            December 7, 2020        pdftk.pdftk-java(1)

OCR

https://tools.pdf24.org/en/ocr-pdf provides fast, free OCR, without ads.

This website works well, and offers a total of 30 PDF operations. The other operations of https://tools.pdf24.org/ include merging, splitting, compressing, editing, signing, redacting, watermarking, locking, unlocking, etc.

PdfTk Operations

Rotate page 2 only

Given my_file.pdf, a 2-page document:

Do not transform page 1 (keep it as-is)
Rotate page 2 only
Save as my_file_fixed.pdf

Shell

$ pdftk my_file.pdf cat 1 2south output my_file_fixed.pdf

Merge PDFs

Concatenate file_a.pdf, file_b.pdf and file_c.pdf, then save as big_file.pdf:

Shell

$ pdftk file_a.pdf file_b.pdf file_c.pdf cat output big_file.pdf

Assemble Book Excerpt

If you need to create multiple excerpts of a book, one way to proceed is to:

Create a PDF containing 3 pages: the front cover, the page containing the edition notice, and the back cover. Let's call this PDF file book_wrapper.pdf.
Scan each of the excerpts into separate PDFs. Let's call these PDF files book_excerpt_raw_1.pdf, book_excerpt_raw_2.pdf, etc.
Wrap the each excerpt within the front cover and edition notice, and the back cover. To do that, concatenate the first 2 pages of book_wrapper.pdf, book_excerpt_raw_N.pdf and the last page of book_wrapper.pdf, then save as book_excerpt_N.pdf. Following is an example of how to do that for 3 excerpts:

Shell

$ pdftk book_wrapper.pdf 1-2 \
  book_excerpt_raw_1.pdf \
  book_wrapper.pdf 3 \
  cat output book_excerpt_1.pdf

$ pdftk book_wrapper.pdf 1-2 \
  book_excerpt_raw_2.pdf \
  book_wrapper.pdf 3 \
  cat output book_excerpt_2.pdf

$ pdftk book_wrapper.pdf 1-2 \
  book_excerpt_raw_3.pdf \
  book_wrapper.pdf 3 \
  cat output book_excerpt_3.pdf

Cut a large PDF into 2 Parts

Cut bigfile.pdf into two parts:

bigfile_part1.pdf (containing pages 1-150)
bigfile_part2.pdf (containing pages 151-350)

Shell

$ pdftk A=bigfile.pdf cat 1-150   output bigfile_part1.pdf

$ pdftk A=bigfile.pdf cat 151-end output bigfile_part2.pdf

Delete a Page

Delete page 69 from bigfile.pdf and save as smaller.pdf:

Shell

$ pdftk A=bigfile.pdf cat ~69 output smaller.pdf

Delete Several Pages

Delete pages 69-96 from bigfile.pdf and save as smaller.pdf:

Shell

$ pdftk A=bigfile.pdf cat ~69-96 output smaller.pdf

Remove Password

Assuming protected.pdf is encrypted with password myPwd, make an unencrypted copy and save as output.pdf:

Shell

$ pdftk protected.pdf input_pw myPwd output output.pdf

You might need to enclose the password within single quotes.

Pytest and Visual Studio Code

2023-08-20T00:00:00-04:00

Python has several well-known testing frameworks. Popular choices for functional and unit testing include pytest, the Robot framework, and unittest. Lettuce and Behave are popular for behavior-driven testing.

I decided to use pytest for a recent Python project.

I found that pytest’s test discovery mechanism was finicky, and demanded that Python packages and modules be set up before tests could be run. This is in sharp contrast to writing unit tests for Ruby, for example, using rspec, because Ruby modules are so much more flexible, forgiving and malleable.

However, getting pytest to work in Visual Studio Code was nearly impossible. Microsoft's documentation leaves a lot unsaid. I share what I learned, and how I made things work in this article.

Pytest

You can run pytest from the command line 3 ways.

Pytest Command

The most common way to run pytest is to type the pytest command, like the following, which runs all tests:

Shell

$ pytest

Pytest Module

The second way to run pytest is to invoke the pytest Python module, as follows:

Shell

$ python -m pytest

Running pytest as a Python module is nearly the same as running the pytest command, except that running it as a module adds the current directory to sys.path, which is standard python behavior, and can be helpful.

I had best results running pytest tests from Visual Studio Code when pytest was invoked as a module.

Calling Pytest from Python

Your code can call pytest. Lots of interesting and advanced development setups could be crafted using this approach. Baby steps.

Help Message

The help message for pytest is:

Shell

$ pytest -h
usage: pytest [options] [file_or_dir] [file_or_dir] [...]

positional arguments:
  file_or_dir

general:
  -k EXPRESSION         Only run tests which match the given substring
                        expression. An expression is a Python evaluatable
                        expression where all names are substring-matched against
                        test names and their parent classes. Example: -k
                        'test_method or test_other' matches all test functions
                        and classes whose name contains 'test_method' or
                        'test_other', while -k 'not test_method' matches those
                        that don't contain 'test_method' in their names. -k 'not
                        test_method and not test_other' will eliminate the
                        matches. Additionally keywords are matched to classes
                        and functions containing extra names in their
                        'extra_keyword_matches' set, as well as functions which
                        have names assigned directly to them. The matching is
                        case-insensitive.
  -m MARKEXPR           Only run tests matching given mark expression. For
                        example: -m 'mark1 and not mark2'.
  --markers             show markers (builtin, plugin and per-project ones).
  -x, --exitfirst       Exit instantly on first error or failed test
  --fixtures, --funcargs
                        Show available fixtures, sorted by plugin appearance
                        (fixtures with leading '_' are only shown with '-v')
  --fixtures-per-test   Show fixtures per test
  --pdb                 Start the interactive Python debugger on errors or
                        KeyboardInterrupt
  --pdbcls=modulename:classname
                        Specify a custom interactive Python debugger for use
                        with --pdb.For example:
                        --pdbcls=IPython.terminal.debugger:TerminalPdb
  --trace               Immediately break when running each test
  --capture=method      Per-test capturing method: one of fd|sys|no|tee-sys
  -s                    Shortcut for --capture=no
  --runxfail            Report the results of xfail tests as if they were not
                        marked
  --lf, --last-failed   Rerun only the tests that failed at the last run (or all
                        if none failed)
  --ff, --failed-first  Run all tests, but run the last failures first. This may
                        re-order tests and thus lead to repeated fixture
                        setup/teardown.
  --nf, --new-first     Run tests from new files first, then the rest of the
                        tests sorted by file mtime
  --cache-show=[CACHESHOW]
                        Show cache contents, don't perform collection or tests.
                        Optional argument: glob (default: '*').
  --cache-clear         Remove all cache contents at start of test run
  --lfnf={all,none}, --last-failed-no-failures={all,none}
                        Which tests to run with no previously (known) failures
  --sw, --stepwise      Exit on test failure and continue from last failing test
                        next time
  --sw-skip, --stepwise-skip
                        Ignore the first failing test but stop on the next
                        failing test. Implicitly enables --stepwise.

Reporting:
  --durations=N         Show N slowest setup/test durations (N=0 for all)
  --durations-min=N     Minimal duration in seconds for inclusion in slowest
                        list. Default: 0.005.
  -v, --verbose         Increase verbosity
  --no-header           Disable header
  --no-summary          Disable summary
  -q, --quiet           Decrease verbosity
  --verbosity=VERBOSE   Set verbosity. Default: 0.
  -r chars              Show extra test summary info as specified by chars:
                        (f)ailed, (E)rror, (s)kipped, (x)failed, (X)passed,
                        (p)assed, (P)assed with output, (a)ll except passed
                        (p/P), or (A)ll. (w)arnings are enabled by default (see
                        --disable-warnings), 'N' can be used to reset the list.
                        (default: 'fE').
  --disable-warnings, --disable-pytest-warnings
                        Disable warnings summary
  -l, --showlocals      Show locals in tracebacks (disabled by default)
  --no-showlocals       Hide locals in tracebacks (negate --showlocals passed
                        through addopts)
  --tb=style            Traceback print mode (auto/long/short/line/native/no)
  --show-capture={no,stdout,stderr,log,all}
                        Controls how captured stdout/stderr/log is shown on
                        failed tests. Default: all.
  --full-trace          Don't cut any tracebacks (default is to cut)
  --color=color         Color terminal output (yes/no/auto)
  --code-highlight={yes,no}
                        Whether code should be highlighted (only if --color is
                        also enabled). Default: yes.
  --pastebin=mode       Send failed|all info to bpaste.net pastebin service
  --junit-xml=path      Create junit-xml style report file at given path
  --junit-prefix=str    Prepend prefix to classnames in junit-xml output

pytest-warnings:
  -W PYTHONWARNINGS, --pythonwarnings=PYTHONWARNINGS
                        Set which warnings to report, see -W option of Python
                        itself
  --maxfail=num         Exit after first num failures or errors
  --strict-config       Any warnings encountered while parsing the `pytest`
                        section of the configuration file raise errors
  --strict-markers      Markers not registered in the `markers` section of the
                        configuration file raise errors
  --strict              (Deprecated) alias to --strict-markers
  -c FILE, --config-file=FILE
                        Load configuration from `FILE` instead of trying to
                        locate one of the implicit configuration files.
  --continue-on-collection-errors
                        Force test execution even if collection errors occur
  --rootdir=ROOTDIR     Define root directory for tests. Can be relative path:
                        'root_dir', './root_dir', 'root_dir/another_dir/';
                        absolute path: '/home/user/root_dir'; path with
                        variables: '$HOME/root_dir'.

collection:
  --collect-only, --co  Only collect tests, don't execute them
  --pyargs              Try to interpret all arguments as Python packages
  --ignore=path         Ignore path during collection (multi-allowed)
  --ignore-glob=path    Ignore path pattern during collection (multi-allowed)
  --deselect=nodeid_prefix
                        Deselect item (via node id prefix) during collection
                        (multi-allowed)
  --confcutdir=dir      Only load conftest.py's relative to specified dir
  --noconftest          Don't load any conftest.py files
  --keep-duplicates     Keep duplicate tests
  --collect-in-virtualenv
                        Don't ignore tests in a local virtualenv directory
  --import-mode={prepend,append,importlib}
                        Prepend/append to sys.path when importing test modules
                        and conftest files. Default: prepend.
  --doctest-modules     Run doctests in all .py modules
  --doctest-report={none,cdiff,ndiff,udiff,only_first_failure}
                        Choose another output format for diffs on doctest
                        failure
  --doctest-glob=pat    Doctests file matching pattern, default: test*.txt
  --doctest-ignore-import-errors
                        Ignore doctest ImportErrors
  --doctest-continue-on-failure
                        For a given doctest, continue to run after the first
                        failure

test session debugging and configuration:
  --basetemp=dir        Base temporary directory for this test run. (Warning:
                        this directory is removed if it exists.)
  -V, --version         Display pytest version and information about plugins.
                        When given twice, also display information about
                        plugins.
  -h, --help            Show help message and configuration info
  -p name               Early-load given plugin module name or entry point
                        (multi-allowed). To avoid loading of plugins, use the
                        `no:` prefix, e.g. `no:doctest`.
  --trace-config        Trace considerations of conftest.py files
  --debug=[DEBUG_FILE_NAME]
                        Store internal tracing debug information in this log
                        file. This file is opened with 'w' and truncated as a
                        result, care advised. Default: pytestdebug.log.
  -o OVERRIDE_INI, --override-ini=OVERRIDE_INI
                        Override ini option with "option=value" style, e.g. `-o
                        xfail_strict=True -o cache_dir=cache`.
  --assert=MODE         Control assertion debugging tools.
                        'plain' performs no assertion debugging.
                        'rewrite' (the default) rewrites assert statements in
                        test modules on import to provide assert expression
                        information.
  --setup-only          Only setup fixtures, do not execute tests
  --setup-show          Show setup of fixtures while executing tests
  --setup-plan          Show what fixtures and tests would be executed but don't
                        execute anything

logging:
  --log-level=LEVEL     Level of messages to catch/display. Not set by default,
                        so it depends on the root/parent log handler's effective
                        level, where it is "WARNING" by default.
  --log-format=LOG_FORMAT
                        Log format used by the logging module
  --log-date-format=LOG_DATE_FORMAT
                        Log date format used by the logging module
  --log-cli-level=LOG_CLI_LEVEL
                        CLI logging level
  --log-cli-format=LOG_CLI_FORMAT
                        Log format used by the logging module
  --log-cli-date-format=LOG_CLI_DATE_FORMAT
                        Log date format used by the logging module
  --log-file=LOG_FILE   Path to a file when logging will be written to
  --log-file-level=LOG_FILE_LEVEL
                        Log file logging level
  --log-file-format=LOG_FILE_FORMAT
                        Log format used by the logging module
  --log-file-date-format=LOG_FILE_DATE_FORMAT
                        Log date format used by the logging module
  --log-auto-indent=LOG_AUTO_INDENT
                        Auto-indent multiline messages passed to the logging
                        module. Accepts true|on, false|off or an integer.
  --log-disable=LOGGER_DISABLE
                        Disable a logger by name. Can be passed multiple times.

[pytest] ini-options in the first pytest.ini|tox.ini|setup.cfg|pyproject.toml file found:

  markers (linelist):   Markers for test functions
  empty_parameter_set_mark (string):
                        Default marker for empty parametersets
  norecursedirs (args): Directory patterns to avoid for recursion
  testpaths (args):     Directories to search for tests when no files or
                        directories are given on the command line
  filterwarnings (linelist):
                        Each line specifies a pattern for
                        warnings.filterwarnings. Processed after
                        -W/--pythonwarnings.
  usefixtures (args):   List of default fixtures to be used with this project
  python_files (args):  Glob-style file patterns for Python test module
                        discovery
  python_classes (args):
                        Prefixes or glob names for Python test class discovery
  python_functions (args):
                        Prefixes or glob names for Python test function and
                        method discovery
  disable_test_id_escaping_and_forfeit_all_rights_to_community_support (bool):
                        Disable string escape non-ASCII characters, might cause
                        unwanted side effects(use at your own risk)
  console_output_style (string):
                        Console output: "classic", or with additional progress
                        information ("progress" (percentage) | "count" |
                        "progress-even-when-capture-no" (forces progress even
                        when capture=no)
  xfail_strict (bool):  Default for the strict parameter of xfail markers when
                        not given explicitly (default: False)
  tmp_path_retention_count (string):
                        How many sessions should we keep the `tmp_path`
                        directories, according to `tmp_path_retention_policy`.
  tmp_path_retention_policy (string):
                        Controls which directories created by the `tmp_path`
                        fixture are kept around, based on test outcome.
                        (all/failed/none)
  enable_assertion_pass_hook (bool):
                        Enables the pytest_assertion_pass hook. Make sure to
                        delete any previously generated pyc cache files.
  junit_suite_name (string):
                        Test suite name for JUnit report
  junit_logging (string):
                        Write captured log messages to JUnit report: one of
                        no|log|system-out|system-err|out-err|all
  junit_log_passing_tests (bool):
                        Capture log information for passing tests to JUnit
                        report:
  junit_duration_report (string):
                        Duration time to report: one of total|call
  junit_family (string):
                        Emit XML for schema: one of legacy|xunit1|xunit2
  doctest_optionflags (args):
                        Option flags for doctests
  doctest_encoding (string):
                        Encoding used for doctest files
  cache_dir (string):   Cache directory path
  log_level (string):   Default value for --log-level
  log_format (string):  Default value for --log-format
  log_date_format (string):
                        Default value for --log-date-format
  log_cli (bool):       Enable log display during test run (also known as "live
                        logging")
  log_cli_level (string):
                        Default value for --log-cli-level
  log_cli_format (string):
                        Default value for --log-cli-format
  log_cli_date_format (string):
                        Default value for --log-cli-date-format
  log_file (string):    Default value for --log-file
  log_file_level (string):
                        Default value for --log-file-level
  log_file_format (string):
                        Default value for --log-file-format
  log_file_date_format (string):
                        Default value for --log-file-date-format
  log_auto_indent (string):
                        Default value for --log-auto-indent
  pythonpath (paths):   Add paths to sys.path
  faulthandler_timeout (string):
                        Dump the traceback of all threads if a test takes more
                        than TIMEOUT seconds to finish
  addopts (args):       Extra command line options
  minversion (string):  Minimally required pytest version
  required_plugins (args):
                        Plugins that must be present for pytest to run

Environment variables:
  PYTEST_ADDOPTS           Extra command line options
  PYTEST_PLUGINS           Comma-separated plugins to load during startup
  PYTEST_DISABLE_PLUGIN_AUTOLOAD Set to disable plugin auto-loading
  PYTEST_DEBUG             Set to enable debug tracing of pytest's internals


to see available markers type: pytest --markers
to see available fixtures type: pytest --fixtures
(shown according to specified file_or_dir or current dir if not specified; fixtures with leading '_' are only shown with the '-v' option

Helpful Options

It is handy to run new and failing tests, but to skip previously passing tests. The ‑‑nf and ‑‑lf options, respectively, invoke that mode of behavior:

Shell

$ pytest --nf --lf

Setup

Configuration

Pytest needed to know that the source code for my Python project is stored in the src directory. I created pyproject.toml as follows, which configured pytest:

pyproject.toml

[tool.pytest.ini_options]
addopts = [
    "--import-mode=importlib",
]
pythonpath = "src"

The way in which pytest discovers tests is complex. TL;DR: specify the newest and best mechanism for test discovery by using the ‑‑import-mode=importlib option as shown above.

To always run pytest with the ‑‑nf and ‑‑lf options, add those options to pyproject.toml:

pyproject.toml

[tool.pytest.ini_options]
addopts = [
    "--import-mode=importlib",
    "--lf",
    "--nf",
]
pythonpath = "src"

Test Directories

Flexibility generally adds complexity. Pytest allows for a lot of flexibility regarding the location of where tests can reside. Unfortunately, Python’s package and module mechanism in combination with the pytest discovery mechanism for tests can lead to a frustrating experience when setting up a new project.

You can read about good practices for pytest. Those recommendations are intimidating for first-time pytest programmers. I did not follow all of them.

I found the simplest approach was to create a subdirectory for the bulk of the Python project code. If a file called __init__.py is created in that directory (use touch for this) then the subdirectory becomes a submodule of your Python program. For this type of pytest setup, each submodule needs a tests directory.

Here is how you could use touch to define an imaginary subdirectory/submodule called my_submodule:

Shell

$ mkdir -p src/my_submodule/

$ touch src/my_submodule/__init__.py

My project structure was as follows. Tree docs are here.

pytest project structure

$ tree -I .git -I __pycache__
.
├── README.md
├── dl
├── dl.config
├── pyproject.toml
├── requirements.txt
└── src
    ├── __main__.py
    └── dl
        ├── __init__.py
        ├── argument_parse.py
        ├── dl_config.py
        ├── media_file.py
        ├── remote.py
        ├── tests
        │   ├── __init__.py
        │   ├── test_dl_config.py
        │   └── test_uti.py
        └── util.py

3 directories, 15 files

src/dl/tests/__init__.py was an empty marker file, but the ___init__.py file in the parent module defined imports for the submodule:

src/dl/__init__.py

from dl.argument_parse import ArgParse
from dl.dl_config import DLConfig
from dl.media_file import MediaFile
from dl.remote import Remote
from dl.util import *

Sample Test

Here is a sample pytest file, containing 2 unit tests. It is simple and clear.

src/dl/tests/test_util.py

import pytest
from pathlib import Path
from dl import samba_mount, samba_parse

class TestUtil:
    def test_samba_parse(self):
        remote_drive, local_path = samba_parse('e:/media/renders')
        assert remote_drive=='e'
        assert local_path=='/media/renders'

    def test_samba_mount(self):
        samba_root = samba_mount('bear', 'e', False)
        assert samba_root==Path('/mnt/bear/e')
        assert samba_root.is_mount()

Secret Sauce

If you have been paying attention, you should realize that one option for running new and failing tests is to type the following from the command line. We will use this method of invocation in the next section to integrate with Visual Studio Code.

Shell

$ python -m pytest --nf --lf

Visual Studio Code Integration

Visual Studio Code has a Testing explorer and a Run and Debug explorer. I find the existance of two very similar explorers annoying, because you often want to debug a test. Yes, you can do that by pushing the right button in the Testing Explorer ... if it works. This did not work for me.

Shopify Ruby, the latest and greatest Ruby extension for Visual Studio Code, uses the Run and Debug explorer to run and debug tests as well as entire programs. This makes sense to me.

Microsoft Pylance (for Python) follows a different convention, and uses the Run and Debug explorer to run and debug Python programs. You must use the Testing explorer for running or debugging Python unit tests. Annoying.

In the Testing explorer, I was able to see my dl module, but no tests were discovered. This was really annoying, and this is where more documentation from Microsoft would have been helpful. After wasting lots of time trying to solve this problem, I discovered a solution.

Invoke Pytest As a Module from VSCode

Run and Debug configurations that invoke the pytest module work perfectly. Here is an example .vscode/launch.json that demonstrates this:

.vscode/launch.json

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "PyTest All",
      "type": "python",
      "request": "launch",
      "module": "pytest",
      "justMyCode": true
    },
    {
      "args": ["--nf", "--lf"],
      "name": "PyTest New and Failing",
      "type": "python",
      "request": "launch",
      "module": "pytest",
      "justMyCode": true
    },
  ]
}

😁

You can set breakpoints in your code and debug your tests this way.

C and C++ Online and On Ubuntu

2023-05-12T00:00:00-04:00

Lattice C

K&R Book

Borland Turbo C

Recently I decided to brush up on my C programming.

I first discovered C during a visit in 1978 to the Computer Science department of University of California, Berkeley, where I also encounted UNIX for the first time.

From about 1983 to 1985 I was the unofficial Canadian distributor of the Lattice C compiler. I used that compiler for most of the programming work I was doing on PCs at the time. FedEx Express Canada did not come into being until 1987, UPS was even worse with the Canadian border than it is now, and Canada Post took forever to bring packages across the border. I bought in bulk at a discount, and sold at list price. Distribution is all about time and place.

I also sold the first edition of the K&R book (“The C Programming Language”) via mail order, and I provided technical support, which brought in interesting consulting work.

The Borland Turbo C compiler, released in 1990, was the original IDE. It was eventually replaced by Borland C++ compiler, which also compiled C. In 2005 I became the product marketing manager for most of Borland’s compilers. One of my responsibilities was to give free copies to book authors and members of the trade press. Borland C++ shipped in a massive box that was responsible for a lot of dead trees.

The C language has evolved since K&R days. The current and previous iterations are well summarized on cppreference.com, which also provides an online C/C++ IDE for sample code.

Revision History

1978: K&R book first published.

1989: C89, also known as ANSI X3. 159-1989, or ANSI C.

1995: NA1 (aka C94 and C95) added support for wide-characters, wide-strings, and international keyboard layouts.

1999: C99 added restricted pointers, variable-length arrays, flexible array members, complex numbers support, type-generic math, long long int, extended identifiers, hexadecimal floating-point constants, compound literals, designated initializers, single line comments, extended integer type, the ability to mix declarations and code, variadic macros, new math functions, inline functions, boolean types, _Pragma and standard pragmas, and VA_COPY. C99 removed implicit function declaration.

2011: C11 added concurrency support, type-generic expressions, alignment-related facilities, static assertion, Unicode support, floating-point characteristic macros, no-return functions, anonymous structures and unions, bound-checking, and reentrancy functions.

2018: C18 addressed defects in C11 without introducing new language features.

2023: The most recent publicly available draft of the next release, C23, was released on April 1, 2023. Unfortunately, that document was written for insiders, and is very difficult to read. Happily, JeanHeyd Meneide, the Project Editor for ISO/IEC JTC1 SC22 WG14 - Programming Languages, C, wrote an easy-to-read summary of C23.

Linux Is Written in C

It has been reported that Linux was originally written in C89, but Linux moved to C11 in 2022. I am curious as to why C18 is not used, since C18 just contains bug fixes for C11. Most of the Linux kernel code is written using the GNU extensions of GCC.

Online IDEs

I pioneered online programming with SMX Explorer in 1994, JSP Explorer in 2000, and supported most available languages in 2002 with Zamples.com.

Now it seems that every day another online programming option becomes available. Many are free:

Coliru is used by cppreference.com; GitHub project
GitHub Codespaces is tightly integrated with Microsoft Azure.
JetBrains Space
OnlineGDB claims to be to first Online IDE, but the WayBack Machine shows that this site was created in 2016.
Programiz
TutorialsPoint CodingGround
Replit
OneCompiler
... and many more ... even more!

Installing C/C++ on WSL/Ubuntu

Online programming is fine for learning, but I prefer to be independent and self-reliant whenever practical for doing actual work.

Command-Line Toolchain

To program in C or C++ on Ubuntu using the traditional Linux command-line toolchain, install the build-essential Debian meta-package on Ubuntu. It includes the GNU Compiler Collection (GCC), which is a collection of compilers and libraries for C, C++, Objective-C, Fortran, Ada, Go, and D programming languages. It also includes make and libc6.1-dev, the GNU C development libraries and header files.

You probably should also install gdb, the GNU Project Debugger, and the manual pages about using GNU/Linux for development.

Shell

$ sudo apt install build-essential gdb manpages-dev

Now you are able to compile and run C programs at the command line:

Shell

$ mkdir test && cd test

$ cat >test.c <<EOF
#include <stdio.h>
int main(void) {
  printf("Hello, World!\n");
}
EOF

$ gcc test.c -o test && ./test
Hello, World!

Visual Studio Code

Visual Studio Code is a high-quality (and free) IDE. To work with C and C++ in Visual Studio Code, install the Microsoft C/C++ extension.

Don't Poke the Bear

2023-03-03T00:00:00-05:00

To the developers of the excellent ‘’ GitHub project

The name of this project is problematic, and you publicly dismissed the issue when it was brought to your attention. While ignorance of the law is no defense; demonstrably flaunting another party’s rights never has positive consequences.

The LEGO Group has not taken legal action yet, but the laws of most countries (including the US) are that if a trademark is not defended, then the trademark is at risk of being lost.

You should expect to receive a legal notice at some point. Not because the trademark owners are evil, but because they have a duty to their company, and the well-being of their employees, and a duty to their shareholders.

Your disregard for the LEGO Group's rights constituted a real and present danger to them, their employees and shareholders. They are compelled to address threats against their well-being.

CC BY-NC-SA 2.0' class="imgImg rounded shadow" src="/blog/images/bearpokelego.png" style='width: 100%; ' title='‘lego bear’ by angie
CC BY-NC-SA 2.0' />

‘lego bear’ by angie
CC BY-NC-SA 2.0

Think

Not only must a trademark holder defend their trademark, they must be seen to defend it. That means taking action that is publicly visible. For example, they might include GitHub and maybe even Microsoft in the legal action against you.

No-one would have reason to defend you

The LEGO Group would likely include GitHub/Microsoft in the case, even though they know that Section 230 would probably be found to apply (in due course), and GitHub/Microsoft would not be found to be liable – after they spent several million dollars in a successful legal defense.

It is easy to understand the motivation for the LEGO Group deliberately ‘losing’ at the attempt to ensnare GitHub/Microsoft in this court case. You would have caused GitHub/Microsoft an unnecessary and pointless waste of time and resources, and it would be shown that you knew better. Remember, you have already been publicly been warned that there would likely be consequences – yet you chose to ignore the warning.

This would likely motivate GitHub/Microsoft to protect themselves against similar misguided behavior of other GitHub users. Restrictive policies and procedures would likely get implemented, and the entire world would know that the resulting degraded GitHub experience was your fault. You would be vilified by every other GitHub user. Not a great thing for your resume.

... And GitHub/Microsoft might ask you to pay their legal expenses.

Do You Have Families?

Please understand that because you are acting as an informal collection of individuals, everyone who ever contributed to this project is exposed to signficant financial risk. The LEGO Group would probably not attempt to narrow down the guilty individuals – identifying the guilty and assigning punishment would be the outcomes of the court case. Instead, they would likely include every contributor in the litigation; that means every committer.

You are risking the financial well-being of your families ... just because you like a cute play on words?

Rational people would not expose themselves to significant risk without any prospect of signficant benefit, and when not faced with dire necessity.

If you presently balk at paying a lawyer to advise you on this matter, that cost will be as nothing compared to the world of hurt you are likely to find yourselves in if you do not change the name of your project.

From a technical aspect, you have done excellent work on your project, and you are freely providing the world a valuable benefit. Unfortunately, you trampled the rights of another in doing so... and for no reason. Please act on this before the inevitable negative consequences arise. Do not poke the bear because you are curious to see what it will do.

The wheels of corporate law grind slowly, and huge amounts of money are spent. According to the path you are now on, the committers of this project may be subjected to immense financial and mental burdens. Broken families and suicides may result.

Please heed this warning. If nothing bad has ever happened to you yet in life, rest assured that Lachesis, Atropos and Clotho will inevitably make themselves known. Do not let arrogance and pride ruin you.

I am trying to help you people.
Because you have done good work,
and it would be a shame to see you suffer.

Disclaimer

I have no relationship with the LEGO Group, or any individuals associated with them. Please consider this as a notice freely offered for public service.

Although I have experience working as an expert witness in the US Federal legal system as a software expert, I am not a lawyer and I have no legal training. However, I have seen corporate legal machinery at work many times, up close. I have spent thousands of hours working with corporate attorneys in the US and Europe. My clients have included Adobe, Amazon/AWS, Apple, and SAP.

Letsencrypt/ACME Wildcard SSL Certificates by Lego

2023-03-02T00:00:00-05:00

Letsencrypt certificates are only valid for 90 days. It is common practice to renew them every 60 days, a task that one should automate when the website is first published.

I wrote about setting up wildcard SSL certificates with Nginx 9 months ago. That post included a demonstration of how to create an SSL certificate manually, but did not discuss how to maintain it.

This blog post discusses how to automatically generate and maintain the SSL certificate using lego, a Letsencrypt/ACME client and library written in Go that has become popular since I last wrote about this topic.

Lego handles many moving parts transparently. It is much simpler than using DNS delegates with acme-dns.

We’ll start by briefly discussing some background information.

Letsencrypt’s Certbot and Wildcard SSL Certificates

You must prove to Letsencrypt that you control the DNS for a domain before it issues a wildcard SSL certificate for that domain. Letsencrypt’s certbot currently uses the DNS-01 challenge for this purpose.

DNS-01 Challenge

The DNS-01 challenge requires that DNS TXT records for the domain be created with specific values as part of the authentication mechanism. Several of these TXT records can co-exist among the DNS records for a domain. The name of the TXT records is always _acme-challenge.

When certbot requests that a new wildcard SSL certificate be created by Letsencrypt, it sends DNS-01 TXT queries to the primary DNS. The Letsencrypt service verifies that the required _acme-challenge TXT records are available with the correct values.

A Technical Deep Dive: Securing the Automation of ACME DNS Challenge Validation, published by the Electronic Frontier Foundation, explains how this works in detail.

Certbot plugins have been created for various DNS providers to make this process easier. Lego is a higher-level program that makes the entire process even easier.

Agenda

The high-level outline of the remainder of this blog is:

Install go language support.
Install lego.
Generate the wildcard SSL certificate.
Provide the certificate to your production web server.
Restart the webserver and verify the new certificate is served.
Add a new entry to crontab to automate the SSL certificate generation.

Install Go Language Support

Lego is written in Go. Ensure that Go language support is installed. For Ubuntu Linux (and the default WSL distro), type:

Shell

$ yes | sudo apt install golang-go
The following NEW packages will be installed:
  golang-1.18-go golang-1.18-src golang-go golang-src
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 82.2 MB of archives.
After this operation, 436 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 golang-1.18-src all 1.18.1-1ubuntu1 [16.2 MB]
Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 golang-1.18-go amd64 1.18.1-1ubuntu1 [66.0 MB]
Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 golang-src all 2:1.18~0ubuntu2 [4438 B]
Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 golang-go amd64 2:1.18~0ubuntu2 [41.8 kB]
Fetched 82.2 MB in 2s (33.6 MB/s)
Selecting previously unselected package golang-1.18-src.
(Reading database ... 173763 files and directories currently installed.)
Preparing to unpack .../golang-1.18-src_1.18.1-1ubuntu1_all.deb ...
Unpacking golang-1.18-src (1.18.1-1ubuntu1) ...
Selecting previously unselected package golang-1.18-go.
Preparing to unpack .../golang-1.18-go_1.18.1-1ubuntu1_amd64.deb ...
Unpacking golang-1.18-go (1.18.1-1ubuntu1) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../golang-src_2%3a1.18~0ubuntu2_all.deb ...
Unpacking golang-src (2:1.18~0ubuntu2) ...
Selecting previously unselected package golang-go:amd64.
Preparing to unpack .../golang-go_2%3a1.18~0ubuntu2_amd64.deb ...
Unpacking golang-go:amd64 (2:1.18~0ubuntu2) ...
Setting up golang-1.18-src (1.18.1-1ubuntu1) ...
Setting up golang-src (2:1.18~0ubuntu2) ...
Setting up golang-1.18-go (1.18.1-1ubuntu1) ...
Setting up golang-go:amd64 (2:1.18~0ubuntu2) ...
Processing triggers for man-db (2.10.2-1) ...
Scanning processes...
Scanning processor microcode...
Scanning linux images...

Failed to retrieve available kernel versions.

Failed to check for processor microcode upgrades.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

View the version of go that was installed:

Shell

$ go version
go version go1.19.2 linux/amd64

Installing Lego

The official lego installation instructions are here. They do not mention Ubuntu instructions, but the lego package for Ubuntu can be installed in the usual way:

Shell

$ yes | sudo apt install lego
Reading package lists... Done
  Building dependency tree... Done
  Reading state information... Done
  The following NEW packages will be installed:
    lego
  0 upgraded, 1 newly installed, 0 to remove and 7 not upgraded.
  Need to get 5322 kB of archives.
  After this operation, 19.7 MB of additional disk space will be used.
  Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 lego amd64 4.1.3-3ubuntu1.22.04.1 [5322 kB]
  Fetched 5322 kB in 1s (10.2 MB/s)
  Selecting previously unselected package lego.
  (Reading database ... 86044 files and directories currently installed.)
  Preparing to unpack .../lego_4.1.3-3ubuntu1.22.04.1_amd64.deb ...
  Unpacking lego (4.1.3-3ubuntu1.22.04.1) ...
  Setting up lego (4.1.3-3ubuntu1.22.04.1) ...

Here is the lego help message:

Shell

$ lego -h
NAME:
lego - Let's Encrypt client written in Go

USAGE:
lego [global options] command [command options] [arguments...]

VERSION:
dev

COMMANDS:
run      Register an account, then create and install a certificate
revoke   Revoke a certificate
renew    Renew a certificate
dnshelp  Shows additional help for the '--dns' global option
list     Display certificates and accounts information.
help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
--domains value, -d value    Add a domain to the process. Can be specified multiple times.
--server value, -s value     CA hostname (and optionally :port). The server certificate must be trusted in order to avoid further modifications to the client. (default: "https://acme-v02.api.letsencrypt.org/directory")
--accept-tos, -a             By setting this flag to true you indicate that you accept the current Let's Encrypt terms of service.
--email value, -m value      Email used for registration and recovery contact.
--csr value, -c value        Certificate signing request filename, if an external CSR is to be used.
--eab                        Use External Account Binding for account registration. Requires --kid and --hmac.
--kid value                  Key identifier from External CA. Used for External Account Binding.
--hmac value                 MAC key from External CA. Should be in Base64 URL Encoding without padding format. Used for External Account Binding.
--key-type value, -k value   Key type to use for private keys. Supported: rsa2048, rsa4096, rsa8192, ec256, ec384. (default: "ec256")
--filename value             (deprecated) Filename of the generated certificate.
--path value                 Directory to use for storing the data. (default: "/mnt/_/work/lego/.lego") [$LEGO_PATH]
--http                       Use the HTTP challenge to solve challenges. Can be mixed with other types of challenges.
--http.port value            Set the port and interface to use for HTTP based challenges to listen on.Supported: interface:port or :port. (default: ":80")
--http.proxy-header value    Validate against this HTTP header when solving HTTP based challenges behind a reverse proxy. (default: "Host")
--http.webroot value         Set the webroot folder to use for HTTP based challenges to write directly in a file in .well-known/acme-challenge. This disables the built-in server and expects the given directory to be publicly served with access to .well-known/acme-challenge
--http.memcached-host value  Set the memcached host(s) to use for HTTP based challenges. Challenges will be written to all specified hosts.
--tls                        Use the TLS challenge to solve challenges. Can be mixed with other types of challenges.
--tls.port value             Set the port and interface to use for TLS based challenges to listen on. Supported: interface:port or :port. (default: ":443")
--dns value                  Solve a DNS challenge using the specified provider. Can be mixed with other types of challenges. Run 'lego dnshelp' for help on usage.
--dns.disable-cp             By setting this flag to true, disables the need to wait the propagation of the TXT record to all authoritative name servers.
--dns.resolvers value        Set the resolvers to use for performing recursive DNS queries. Supported: host:port. The default is to use the system resolvers, or Google's DNS resolvers if the system's cannot be determined.
--http-timeout value         Set the HTTP timeout value to a specific value in seconds. (default: 0)
--dns-timeout value          Set the DNS timeout value to a specific value in seconds. Used only when performing authoritative name servers queries. (default: 10)
--pem                        Generate a .pem file by concatenating the .key and .crt files together.
--cert.timeout value         Set the certificate timeout value to a specific value in seconds. Only used when obtaining certificates. (default: 30)
--help, -h                   show help
--version, -v                print the version

Generating the Wildcard SSL Certificate

/etc/environment

My projects are defined in a directory tree, which is pointed to by an environment variable called work. This is helpful when working across many machines, each of which has a different directory layout.

I define the work environment variable in the system-wide /etc/environment file, where it affects all users, all scripts, and crontab entries:

/etc/environment

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
work=/mnt/_/work
sites=/mnt/_/www

$work is used in the script shown below, and that script will be launched via crontab, which is why /etc/environment is used instead of $HOME/.bashrc.

WSL does not run systemd services, and WSL2 only runs them if they are enabled. You can easily convert WSL instances into WSL2 instances.

Unless systemd starts automatically at boot time, /etc/environment will not be processed.

Read Microsoft’s announcement to learn more: Systemd support is now available in WSL!

Generation

When working with the lego CLI with DNS authentication, a directory is needed to hold all the files that are required. I created a directory for this purpose at $work/lego:

Shell

$ mkdir $work/lego

$ cd $work/lego

The lego --dns namecheap option connects to Namecheap, which is my DNS provider, and uses the Namecheap DNS for the DNS-01 challenge. Lego supports many other DNS providers.

Namecheap Users

The Namecheap API documentation is here. For some reason, that page reloads to a different page, which is annoying. Press the Esc key right after the page opens, so you can read it.

Please also read the FAQ.
Enable the Namecheap API here.
You could enable the Namecheap sandbox API here.

Here is a short script that displays your current public IP address:

whatismyip

#!/bin/bash

dig +short myip.opendns.com @resolver1.opendns.com

If your modem goes offline, Namecheap’s DNS will assign a new IP address the next time the modem reconnects. That will cause the scripts shown below to stop working until you manually provide the new address to the Namecheap API. Annoying! Be sure to plug your modem into a UPS to maintain the IP address even during short outages, power brownouts and voltage dropouts. A sine-wave UPS is strongly preferred, since modems are sensitive to low-quality power.

The Namecheap API needs 2 environment variables for authentication (NAMECHEAP_API_USER and NAMECHEAP_API_KEY). Here is how I defined those environment variables so they were available for the lego process launched by sudo:

Shell

$ export NAMECHEAP_API_USER=asdf

$ export NAMECHEAP_API_KEY=asdfasdf

To generate a wildcard certificate, you must specify the --domains option twice: once with the domain name, and once for all subdomains, like this:

Shell

$ lego \
  --accept-tos \
  --dns namecheap \
  --domains="scalacourses.com" \
  --domains="*.scalacourses.com" \
  --email="mslinn@scalacourses.com" \
  run
2023/03/01 13:40:03 [INFO] [*.scalacourses.com] acme: Obtaining bundled SAN certificate
2023/03/01 13:40:03 [INFO] [*.scalacourses.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/207409481036
2023/03/01 13:40:03 [INFO] [*.scalacourses.com] acme: use dns-01 solver
2023/03/01 13:40:03 [INFO] [*.scalacourses.com] acme: Preparing to solve DNS-01
2023/03/01 13:40:04 [INFO] [*.scalacourses.com] acme: Trying to solve DNS-01
2023/03/01 13:40:04 [INFO] [*.scalacourses.com] acme: Checking DNS record propagation using [172.19.176.1:53]
2023/03/01 13:40:19 [INFO] Wait for propagation [timeout: 1h0m0s, interval: 15s]
2023/03/01 13:40:20 [INFO] [*.scalacourses.com] acme: Waiting for DNS record propagation.
2023/03/01 13:40:35 [INFO] [*.scalacourses.com] acme: Waiting for DNS record propagation.
2023/03/01 13:40:56 [INFO] [*.scalacourses.com] The server validated our request
2023/03/01 13:40:56 [INFO] [*.scalacourses.com] acme: Cleaning DNS-01 challenge
2023/03/01 13:40:56 [INFO] [*.scalacourses.com] acme: Validations succeeded; requesting certificates
2023/03/01 13:40:57 [INFO] [*.scalacourses.com] Server responded with a certificate.

The wildcard SSL certificate and associated files are generated into the .lego/certificates directory.

Shell

$ ls -alF .lego/certificates
total 12
drwx------ 1 mslinn mslinn 4096 Mar  1 13:40 ./
drwx------ 1 mslinn mslinn 4096 Mar  1 13:15 ../
-rw------- 1 mslinn mslinn 5325 Mar  1 13:40 scalacourses.com.crt
-rw------- 1 mslinn mslinn 3751 Mar  1 13:40 scalacourses.com.issuer.crt
-rw------- 1 mslinn mslinn  239 Mar  1 13:40 scalacourses.com.json
-rw------- 1 mslinn mslinn  227 Mar  1 13:40 scalacourses.com.key

scalacourses.com.crt is the server certificate that nginx needs, already combined with the CA certificate.
scalacourses.com.key is the private key for the server certificate.
scalacourses.com.issuer.crt is the issuing Certificate Authority's certificate.

scalacourses.com.json contains JSON encoded meta information. It looked like this:

$work/lego/.lego/certificates/scalacourses.com.json

{
  "domain": "*.scalacourses.com",
  "certUrl": "https://acme-v02.api.letsencrypt.org/acme/cert/3939493849340275024024",
  "certStableUrl": "https://acme-v02.api.letsencrypt.org/acme/cert/738378373923492384932874932"
}

Using the value for certStableUrl in the above JSON file, we can view the generated (and combined) certificate that was saved on letsencrypt.org.

Shell

$ wget -O aw.crt \
https://acme-v02.api.letsencrypt.org/acme/cert/03b099f52b4841db17e035c5c2b390f0219b

certGenScalaCourses Script

60 days from now, when the certificate should be renewed, the last parameter passed to lego in the above command line should be changed from run to renew. Here is a bash script that you can edit for this purpose; at the end of this article this script is incorporated into a new crontab entry:

$work/lego/certGenScalaCourses

#!/bin/bash

export NAMECHEAP_API_USER=asdf
export NAMECHEAP_API_KEY=asdfasdf

ACTION=run
if [ -f .lego/certificates/scalacourses.com.crt ]; then
  ACTION=renew
fi

cd $work/lego

lego \
  --accept-tos \
  --dns namecheap \
  --domains="scalacourses.com" \
  --domains="*.scalacourses.com" \
  --email="mslinn@scalacourses.com" \
  "$ACTION"

sudo systemctl reload nginx

Viewing a Certificate

For Ubuntu 22.10, the filetype associations are defined in a hierarchy of files called mimeapps.list:

Shell

$ locate mimeapps.list
~/.config/mimeapps.list
~/.local/share/applications/mimeapps.list
/snap/core/14447/usr/share/applications/mimeapps.list
/snap/core/14784/usr/share/applications/mimeapps.list
/snap/core18/2679/usr/share/applications/mimeapps.list
/snap/core18/2697/usr/share/applications/mimeapps.list
/snap/core20/1778/usr/share/applications/mimeapps.list
/snap/core20/1822/usr/share/applications/mimeapps.list
/snap/core22/509/usr/share/applications/mimeapps.list
/snap/core22/522/usr/share/applications/mimeapps.list
/usr/share/gdm/greeter/applications/mimeapps.list

/snap/core22/509/usr/share/applications/mimeapps.list contains the default association for *.crt files:

/snap/core22/509/usr/share/applications/mimeapps.list

[Default Applications]
x-scheme-handler/http=xdg-open.desktop
x-scheme-handler/https=xdg-open.desktop
x-scheme-handler/mailto=xdg-open.desktop
x-scheme-handler/help=xdg-open.desktop

The above associates a program called xdg-open with CRT files.

Double-clicking on a crt file displayed by an Ubuntu file manager such as Nautilus causes xdg-open to open the file. You can also use the command line to open the file in xdg-open. For example, typing the following causes aw.crt to open:

Shell

$ xdg-open aw.crt

This is what xdg-open displays:

As you can see, this is a combined certificate, containing not just one certificate, but an entire certificate chain. Clicking on any of the red > Details buttons causes more information to be displayed about the corresponding certificate in the chain.

Warning

If you forget to include the domain, and just specify the wildcard for subdomains, the certificate will not be valid for the domain. For example, if above I had only specified one --domains option, like this:

--domains="scalacourses.com"

... instead of

--domains="scalacourses.com" \
--domains="*.scalacourses.com"

... then that would be an error. You can notice your error three ways:

The name of the generated certificate will start with an underscore (_), for example _.scalacourses.com.crt, instead of being named scalacourses.com.crt.
When you examine the certificate, the domain listed will show the subdomain wildcard, as shown below, with a leading underscore asterisk (*).
When you open the certificate, the section entitled Subject Alternative Names will just contain the subdomain wildcard, like this:

Subject Alternative Names
DNS: *.scalacourses.com

Or just the domain, without a wildcard:

Subject Alternative Names
DNS: scalacourses.com

Instead of both being present, like this:

Subject Alternative Names
DNS: *.scalacourses.com DNS: scalacourses.com

Provide the certificate to Nginx

Previously, before working with lego, I used raw certbot to generate the SSL wildcard certificates. That process generated files in ~/.certbot/scalacourses.com/config/live/scalacourses.com:

fullchain.pem (the ssl_certificate)
privkey.pem (the ssl_certificate_key)

From /etc/nginx/sites-enabled/scalacourses.com

ssl_certificate     /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/fullchain.pem;
ssl_certificate_key /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/privkey.pem;

Now I need to modify /etc/nginx/sites-enabled/scalacourses.com to reference the files generated by lego:

scalacourses.com.crt (the ssl_certificate)
scalacourses.com.key (the ssl_certificate_key)

If the webserver had access to the directory containing the new wildcard certificate, it would be simplest to provide the full path to the certificate and its key, like this:

Modified portion of /etc/nginx/sites-enabled/scalacourses.com

ssl_certificate     /mnt/_/work/lego/.lego/certificates/scalacourses.com.crt;
ssl_certificate_key /mnt/_/work/lego/.lego/certificates/scalacourses.com.key;

Restart Nginx

I tested the nginx configuration:

Shell

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Then I reloaded nginx:

Shell

$ sudo systemctl reload nginx

The new SSL certificate needs to be checked to make sure it was installed properly. I just visually examine the start and expire dates:

Shell

$ curl -Lvs https://www.scalacourses.com \
  2>&1 1>/dev/null | \
  grep '\(start\|expire\) date:'
*  start date: Mar  1 17:40:56 2023 GMT
*  expire date: May 30 17:40:55 2023 GMT

Add a New Entry to Crontab

Automating the regeneration of the SSL wildcard certificate is easy. Edit your crontab:

Shell

$ crontab -e

Add the following reference to the certGenScalaCourses script that I gave you earlier to crontab:

crontab Entry

# Runs every 2 months / 60 days (more or less)
30 0 1 */2 * $work/lego/certGenScalaCourses

The above only works because we defined the work environment variable in /etc/environment.

HTML Hyphens

2023-01-23T00:00:00-05:00

Web browsers now have built-in automatic hyphenation support, but it is turned off by default for compatibity with older browsers. You can enable hyphenation with two words of CSS, like this:

body {
  hyphens: auto;
}

W3 CSS3 Hyphenation Standard

Hyphenation occurs when the line breaks at a valid hyphenation opportunity, which is a type of soft wrap opportunity that exists within a word where hyphenation is allowed.

In CSS, hyphenation opportunities are controlled with the hyphens property. CSS Text Level 3 does not define the exact rules for hyphenation; however, UAs are strongly encouraged to optimize their choice of break points and to chose language-appropriate hyphenation points.

Possible hyphens values: none | manual | auto
Initial value: manual

– W3 CSS3 Standard

However, the W3 CSS3 hyphenation standard also says:

Correct automatic hyphenation requires a hyphenation resource appropriate to the language of the text being broken. The UA must therefore only automatically hyphenate text for which the content language is known and for which it has an appropriate hyphenation resource.

Authors should correctly tag their content’s language (e.g. using the HTML lang attribute or XML xml:lang attribute) in order to obtain correct automatic hyphenation.

– W3 CSS3 Standard

Let’s put this into practice now.

Your Website Stylesheet

You may not want certain passages to be hyphenated. Defining a CSS style that disables automatic hyphenation for portions of a document is easy; you can still use  in those portions of the document to manually define optional hyphenation points.

Here is a suggestion for your stylesheet, so it has the hyphenation support you need. Note that hypenation for kbd tags is completely disabled.

body, .hyphen {
  hyphens: auto;
}

.nohyphen {
  hyphens: manual;
}

kbd {
  hyphens: none;
}

Complete Example

The following example enables automatic hyphenation of the entire HTML body, according to US English rules. The HTML document contains nested tags, some of which disable or enable automatic hyphenation. One line demonstrates manual hyphenation, through the use of  entities:

index.html

<html lang="en-US">
  <head>
    <style>
      body, .hyphen {
        hyphens: auto;
      }

      .nohyphen {
        hyphens: manual;
      }
    </style>
  </head>
  <body>
    <p>This element is automatically hyphenated.</p>
    <p class="nohyphen">This element is not hyphenated.</p>
    <ol class="nohyphen">
      <li>This elem&shy;ent is manual&shy;ly hyphen&shy;ated.</li>
      <li class="hyphen">This element is automatically hyphenated.</li>
    </ol>
  </body>
</html>

😁

Easy!

Alternative Manual Hyphenation

Invisible hypenation is sometimes desirable. For example, if I have a long directory path, like $HOME/.rbenv/versions/$RUBY_VERSION/lib/ruby/gems/$RUBY_VERSION/$GEM_NAME-$GEM_VERSION I might want to tell the web browser where the string could be broken.

You have two options — they are actually different syntaxes for the same zero-width space.

Zero-Width Space HTML Entity

 is the HTML entity for a unicode character called the zero-width space (ZWSP).

For example, $HOME/.rbenv/versions/$RUBY_VERSION/lib/ruby/gems/$RUBY_VERSION/$GEM_NAME-$GEM_VERSION

renders as:

$HOME/.rbenv/versions/$RUBY_VERSION/lib/ruby/gems/$RUBY_VERSION/$GEM_NAME-$GEM_VERSION

Line Break Opportunity Tag

You could also use the line break opportunity tag,  .

For example, $HOME/.rbenv/versions/$RUBY_VERSION/lib/ruby/gems/$RUBY_VERSION/$GEM_NAME-$GEM_VERSION

renders as:

$HOME/.rbenv/versions/$RUBY_VERSION/lib/ruby/gems/$RUBY_VERSION/$GEM_NAME-$GEM_VERSION

Both Options Yield the Same Rendered Text

I prefer to use . YMMV.

Non-Breaking Hyphen

The non-breaking hyphen HTML entity (‑) displays a hyphen character that does not break.

For CSS, use content: "\2011" to produce a non-breaking hyphen.

A Curmudgeon’s Social Networking

2023-01-07T00:00:00-05:00

I am an unapologetic curmudgeon. I am not here to please anyone else, I am only here to earnestly be myself to the best of my ability.

If we do not continuously demonstrate that we endeavor to treat each other right, then we should be dismissed and forgotten.

Bullshit walks; money drives away, unsatisfied.

Fairness and equity are essential in all things without reservation. Otherwise, go away, life is short and you are just noise.

Twitter is now a circus. I don't need or want another circus in my communication. Do not look for me on Twitter any longer.

If someone connects with me on Facebook or LinkedIn, I expect them to interact with me in person, on the phone, and to want to engage with me. Failing that, I disconnect from them without hesitation.

Yes, please connect with me. Tell me your truth. Listen to me. I will listen to you.

Action Over Empty Words

Even better, we might endeavor to do things together. Imagining, conceptualizing, and building something real with others gives me joy. I want to share that joy.

People who talk about what ‘others’ should do, but are unwilling to act themselves, are a waste of time. They irritate me. Only approach me if you are an action-oriented individual.

The most difficult thing is the decision to act, the rest is merely tenacity. – Amelia Earhart

Peace and love. σ

Working With Volumes and Directories Under Ubuntu

2022-12-01T00:00:00-05:00

I've been reorganizing directories across several volumes on an Ubuntu server. This blog post documents how I used the commands I found to be most useful: rsync, ncdu, and fdupes. The latter two programs are not installed by default. You can install them this way:

Shell

$ yes | sudo apt install fdupes ncdu

Display Directory Sizes

Shell

$ ncdu /directory

Find Duplicate Files

This displays duplicate files and interactively asks which duplicate files to delete.

Shell

$ fdupes -r1Sd /directory

Compare Two Folders For Missing Files

This tip was inspired by an answer on unix.stackexchange.com.

Dry Run

The -n option displays the names of missing files in the destination directory, but makes no changes to the destination file system. If that option is not specified, then the missing files are copied.

Shell

$ sync -ri --ignore-existing -n /srcDir/ /destDir

There are a few things to note:

The first directory is the source, and it must end with a slash (/).
The second directory is the target, and it must NOT end with a slash.

Copy Missing Files

Simply do not provide the -n (dry run) option:

Shell

$ rsync -ri --ignore-existing /srcDir/ /destDir

JiraCLI, a Feature-rich Interactive Jira Command Line

2022-08-12T00:00:00-04:00

When first released in 2002, Jira was merely a bug tracker. Since then, it has gained features and is now used as a project management tool. Jira only provides a web user interface.

JiraCLI, an independent F/OSS project, provides command-line explorers for Jira issues, epics, and sprints.

Installation

JiraCLI is a Go program, so it is quick and easy to install on Ubuntu once Go is installed.

Shell

$ yes | sudo apt install golang-go

$ go install github.com/ankitpokhrel/jira-cli/cmd/jira@latest
go: downloading github.com/ankitpokhrel/jira-cli v1.0.0
go: downloading github.com/kr/text v0.2.0
go: downloading github.com/spf13/cobra v1.5.0
go: downloading github.com/spf13/viper v1.12.0
go: downloading github.com/zalando/go-keyring v0.2.1
go: downloading github.com/briandowns/spinner v1.18.1
go: downloading github.com/fatih/color v1.13.0
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/AlecAivazis/survey/v2 v2.3.5
go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go: downloading github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8
go: downloading github.com/cpuguy83/go-md2man/v2 v2.0.2
go: downloading github.com/spf13/pflag v1.0.5
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/atotto/clipboard v0.1.4
go: downloading github.com/charmbracelet/glamour v0.5.0
go: downloading github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d
go: downloading github.com/cli/safeexec v1.0.0
go: downloading github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51
go: downloading github.com/kentaro-m/blackfriday-confluence v0.0.0-20220126124413-8e85477b49b3
go: downloading github.com/russross/blackfriday/v2 v2.1.0
go: downloading github.com/mattn/go-colorable v0.1.12
go: downloading github.com/mattn/go-isatty v0.0.14
go: downloading github.com/gdamore/tcell/v2 v2.5.1
go: downloading github.com/rivo/tview v0.0.0-20220610163003-691f46d6f500
go: downloading github.com/fsnotify/fsnotify v1.5.4
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading github.com/spf13/afero v1.8.2
go: downloading github.com/spf13/cast v1.5.0
go: downloading github.com/spf13/jwalterweatherman v1.1.0
go: downloading github.com/godbus/dbus/v5 v5.1.0
go: downloading golang.org/x/text v0.3.7
go: downloading golang.org/x/term v0.0.0-20220526004731-065cf7ba2467
go: downloading golang.org/x/sys v0.0.0-20220615213510-4f61da869c0c
go: downloading github.com/subosito/gotenv v1.4.0
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading gopkg.in/ini.v1 v1.66.6
go: downloading github.com/magiconair/properties v1.8.6
go: downloading github.com/pelletier/go-toml/v2 v2.0.2
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/gdamore/encoding v1.0.0
go: downloading github.com/pelletier/go-toml v1.9.5
go: downloading github.com/lucasb-eyer/go-colorful v1.2.0
go: downloading github.com/mattn/go-runewidth v0.0.13
go: downloading github.com/muesli/termenv v0.12.0
go: downloading github.com/yuin/goldmark v1.4.12
go: downloading github.com/yuin/goldmark-emoji v1.0.1
go: downloading github.com/rivo/uniseg v0.2.0
go: downloading github.com/alecthomas/chroma v0.10.0
go: downloading github.com/microcosm-cc/bluemonday v1.0.18
go: downloading github.com/muesli/reflow v0.3.0
go: downloading github.com/olekukonko/tablewriter v0.0.5
go: downloading github.com/aymerick/douceur v0.2.0
go: downloading golang.org/x/net v0.0.0-20220621193019-9d032be2e588
go: downloading github.com/dlclark/regexp2 v1.4.0
go: downloading github.com/gorilla/css v1.0.0

Add $HOME/go/bin to the PATH so jira can be found:

Shell

$ echo 'PATH=$HOME/go/bin:$PATH' >> ~/.bashrc

$ source ~/.bashrc

$ which jira
/home/mslinn/go/bin/jira

Usage

This is the jira help message:

Shell

$ jira
Interactive Jira CLI.

USAGE
jira [flags]

MAIN COMMANDS
board       Board manages Jira boards in a project
epic        Epic manage epics in a project
issue       Issue manage issues in a project
open        Open issue in a browser
project     Project manages Jira projects
sprint      Sprint manage sprints in a project board

OTHER COMMANDS
completion  Output shell completion code for the specified shell (bash or zsh)
help        Help about any command
init        Init initializes jira config
man         Help generate man(7) pages for Jira CLI.
me          Displays configured jira user
version     Print the app version information

FLAGS
-c, --config string    Config file (default is /home/mslinn/.config/.jira/.config.yml)
--debug            Turn on debug output
-h, --help             help for jira
-p, --project string   Jira project to look into (defaults to /home/mslinn/.config/.jira/.config.yml)

LEARN MORE
Use 'jira <command> <subcommand> --help' for more information about a command.

You need an Jira API token before you can use JiraCLI. Get it from here.

You need to configure the program before you use it. Set the JIRA_API_TOKEN environment variable before running jira init. If you do not, then you will get an error like Received unexpected response '401 Unauthorized' from jira. in the next step.

Shell

$ export JIRA_API_TOKEN=asdfasdfasdf

$ jira init
? Installation type: Cloud
? Link to Jira server: https://xxx.atlassian.net/
? Login email: mslinn@mslinn.com
⠼ Verifying login details...
? Default project: My
? Default board:  [Use arrows to move, type to filter, ? for more help]
> [Search...]
  ----------
  My Scrum Board
  None

  ✓ Configuration generated: /home/mslinn/.config/.jira/.config.yml

This is the configuration file that was generated:

/home/mslinn/.config/.jira/.config.yml

board:
    id: 350
    name: My Scrum Board
    type: scrum
epic:
    name: customfield_10009
    link: customfield_10008
installation: Cloud
issue:
    fields:
        custom:
            - name: Epic Link
              key: customfield_10008
              schema:
                datatype: any
            - name: Epic Name
              key: customfield_10009
              schema:
                datatype: string
    types:
        - id: "3"
          name: Task
          handle: Task
          subtask: false
        - id: "5"
          name: Sub-task
          handle: Sub-task
          subtask: true
        - id: "7"
          name: Story
          handle: Story
          subtask: false
        - id: "1"
          name: Bug
          handle: Bug
          subtask: false
        - id: "6"
          name: Epic
          handle: Epic
          subtask: false
login: mslinn@mslinn.com
project:
    key: My
    type: classic
server: https://xxx.atlassian.net

Example Commands

Help for the jira issue subcommand:

Shell

$ jira issue -h
Issue manage issues in a given project. See available commands below.

  USAGE
  jira issue [flags]

  MAIN COMMANDS
  assign      Assign issue to a user
  clone       Clone duplicates an issue
  comment     Manage issue comments
  create      Create an issue in a project
  delete      Delete an issue
  edit        Edit an issue in a project
  link        Link connects two issues
  list        List lists issues in a project
  move        Transition an issue to a given state
  unlink      Unlink disconnects two issues from each other
  view        View displays contents of an issue
  worklog     Manage issue worklog

  FLAGS
  -h, --help   help for issue

  INHERITED FLAGS
  -c, --config string    Config file (default is /home/mslinn/.config/.jira/.config.yml)
  --debug            Turn on debug output
  -p, --project string   Jira project to look into (defaults to /home/mslinn/.config/.jira/.config.yml)

  ALIASES
  issues

  LEARN MORE
  Use 'jira <command> <subcommand> --help' for more information about a command.

Help for the jira issue list sub-subcommand:

Shell

$ jira issue list -h
List lists issues in a given project.

You can combine different flags to create a unique query. For instance,

# Issues that are of high priority, is in progress, was created this month, and has given labels
jira issue list -yHigh -s"In Progress" --created month -lbackend -l"high prio"

Issues are displayed in an interactive list view by default. You can use a --plain flag
to display output in a plain text mode. A --no-headers flag will hide the table headers
in plain view. A --no-truncate flag will display all available fields in plain mode.

USAGE
jira issue list [flags]

FLAGS
-t, --type string             Filter issues by type
-R, --resolution string       Filter issues by resolution type
-s, --status string           Filter issues by status
-y, --priority string         Filter issues by priority
-r, --reporter string         Filter issues by reporter (email or display name)
-a, --assignee string         Filter issues by assignee (email or display name)
-C, --component string        Filter issues by component
-l, --label stringArray       Filter issues by label
-P, --parent string           Filter issues by parent
--history                 Issues you accessed recently
-w, --watching                Issues you are watching
--created string          Filter issues by created date
Accepts: today, week, month, year, or a date in yyyy-mm-dd and yyyy/mm/dd format,
or a period format using w = weeks, d = days, h = hours, m = minutes. eg: -10d
Created filter will have precedence over created-after and created-before filter
--updated string          Filter issues by updated date
Accepts: today, week, month, year, or a date in yyyy-mm-dd and yyyy/mm/dd format,
or a period format using w = weeks, d = days, h = hours, m = minutes. eg: -10d
Updated filter will have precedence over updated-after and updated-before filter
--created-after string    Filter by issues created after certain date
--updated-after string    Filter by issues updated after certain date
--created-before string   Filter by issues created before certain date
--updated-before string   Filter by issues updated before certain date
-q, --jql string              Run a raw JQL query in a given project context
--order-by string         Field to order the list with (default "created")
--reverse                 Reverse the display order (default "DESC")
--paginate string         Paginate the result. Max 100 at a time, format: <from>:<limit> where <from> is optional (default "0:100")
--plain                   Display output in plain mode
--no-headers              Don't display table headers in plain mode. Works only with --plain
--no-truncate             Show all available columns in plain mode. Works only with --plain
--columns string          Comma separated list of columns to display in the plain mode.
Accepts: TYPE, KEY, SUMMARY, STATUS, ASSIGNEE, REPORTER, PRIORITY, RESOLUTION, CREATED, UPDATED
-h, --help                    help for list

INHERITED FLAGS
-c, --config string    Config file (default is /home/mslinn/.config/.jira/.config.yml)
--debug            Turn on debug output
-p, --project string   Jira project to look into (defaults to /home/mslinn/.config/.jira/.config.yml)

EXAMPLES
$ jira issue list

# Limit list to 20 items
$ jira issue list --paginate 20

# Get 50 items starting from 10
$ jira issue list --paginate 10:50

# List issues in a plain table view without headers
$ jira issue list --plain --no-headers

# List some columns of the issue in a plain table view
$ jira issue list --plain --columns key,assignee,status

# List issues in a plain table view and show all fields
$ jira issue list --plain --no-truncate

# List issues of type "Epic" in status "Done"
$ jira issue list -tEpic -sDone

# List issues in status other than "Open" and is assigned to no one
$ jira issue list -s~Open -ax

# List issues from all projects
$ jira issue list -q"project IS NOT EMPTY"

ALIASES
lists
ls

LEARN MORE
Use 'jira <command> <subcommand> --help' for more information about a command.

Issues Updated on a Certain Date

Shell

$ jira issue list --updated 2021-11-17

Issues Selected by Complex Criteria

The following command will yield a list of high-priority issues, created this month, with status To Do, that are assigned to you, and have the label backend.

Shell

$ jira issue list -yHigh -s"To Do" --created month -lbackend -a$(jira me)

Output might look something like the following, which was redacted:

Issue Ids Only

The --plain and --no-headers options are useful for driving scripts.

Shell

$ jira issue list --updated 2021-11-17 --plain --no-headers --columns KEY

View Issue

The following returns the available information about an issue, in a plain-text format.

Shell

$ jira issue view ISSUE-1 --comments 9999 --plain

Verdict

JiraCLI is useful project! 😁

ImageMagick Slicing on Ubuntu/WSL

2022-07-28T00:00:00-04:00

Lawyers like the Microsoft Office software suite; so when I am working on a court case as an expert, I endeavor to provide my clients with Word documents that contain necessary information. I like working in WSL/WSL2 because I can use Windows programs and Ubuntu programs together effectively.

Grab Image, Then Slice

Recently, I used SnagIt, a Windows program, to capture large web pages as single images. Some of these images were quite tall.

The CSS for the web pages made some content invisible on the printed page. Yes, I could have injected CSS using a Chrome plugin like My Style to ensure that all content will be printed, like this:

Injected Style

@media print {
  * { display: initial; visibility: visible; }
}

I decided to use screen grabs, which would guarantee that the contents of my report would exactly match what had been displayed on the screen, without injecting anything into the web pages.

ImageMagick is preinstalled on Ubuntu Desktop. I used ImageMagick to slice the image captures into smaller page-sized images, so they could be inserted into a Word document.

The Computer Worked Hard

Grabbing such large web pages was a lot of work for my desktop computer. The only programs active during the screen grab process were the Google Chrome browser and SnagIt. I found that 10GB RAM and 30% of the GPU capability (an NVidia GTX 1660 Super) was used.

The screen grab failed if I did not start scrolling from the top of the web page; while it is possible to scrub up and down smaller web pages in order to grab portions of interest, this fails for large pages.

I also found that scrolling too fast caused the screen grabbing process to fail. Clicking and holding the bottom scroll arrowhead at the bottom right of the screen seemed to result in a smooth and optimal scrolling speed. This meant that grabbing large web pages took a few minutes as the page slowly scrolled downward.

Setting Up the Conversion

The Word documents I usually work with are formatted for North American standards. This means one-inch margins on letter-sized paper (8.5" x 11"), which gives a working area of 6.5" x 9", yielding an aspect ratio of 0.72.

The tall captured images needed to be sliced into rectangles that fit efficiently into Word documents. The computations are as follows.

Determine the width of a screen grab and save it into W. The ImageMagick identify command does not provide a newline after its output, however I have inserted one for readability:
Shell
```
$ identify -ping -format '%w' ../IMG2005.png
1536 

$ export W="$( identify -ping -format '%w' ../IMG2005.png )"
```

Determine the height of a screen grab and save it into H:

Shell

$ identify -ping -format '%h' ../IMG2005.png

$ export H="$( identify -ping -format '%h' ../IMG2005.png )"

The width can be divided by the aspect ratio to obtain the desired height of each slice so they can be inserted optimally into the Word documents. I used the bc calculator provided with Bash to divide W / ASPECT_RATIO. The H2 integer variable contains the computed height for the images.
Shell
```
$ export ASPECT_RATIO=0.72

$ export H2="$( echo "scale=0 ; $W / $ASPECT_RATIO" | bc )"
```
Now the image called IMG2005.png can be sliced using ImageMagick’s convert command. The slices are stored into a subdirectory called slices, with file names like IMG2005-1.jpg, IMG2005-2.jpg, etc.
Shell
```
$ convert IMG2005.png -crop ${W}x${H2} \
  -quality 100% -scene 0 slices/IMG2005-%d.jpg
```

Automating the Conversion

I wrote the following bash script, which incorporates the above computations. It slices all the images in a directory and saves the results to a second directory.

sliceImages

#!/bin/bash

function help {
  if [ "$1" ]; then echo "Error: $1"; fi
  echo "
$(basename $0): slice all images in the given directory and place them into a specified directory,
which will be created if required.
"
  exit 1
}

function setup {
  export ASPECT_RATIO=0.72
  export W="$( identify -ping -format '%w' "$1" )"
  export H="$( identify -ping -format '%h' "$1" )"
  export H2="$( echo "scale=0 ; $W / $ASPECT_RATIO" | bc )"
}

function convert1 {
  FULLNAME=$(basename -- "$1")
  FILENAME="${FULLNAME%.*}"
  FILETYPE="${FULLNAME##*.}"

  convert "$1" \
    -crop "${W}x${H2}" \
    -quality 100% \
    -scene 0 \
    "$DIR_OUTPUT/$FILENAME-%d.png"
}


if [ -z "$1" ]; then help "No directory path for images to be converted was provided."; fi
export DIR_INPUT="$( realpath $1 )"

if [ -z "$2" ]; then help "No directory path for the image slices to be saved into was provided."; fi
export DIR_OUTPUT="$( realpath $2 )"

mkdir -p "$DIR_OUTPUT"

find $DIR_INPUT -type f -exec file --mime-type {} \+ | awk -F: '{if ($2 ~/image\//) print $1}' |
  while read FILE; do
    setup "$FILE"
    echo "Slicing $FILE into ${W}x${H2} pixels"
    convert1 "$FILE"
  done

Overcoming ImageMagick Processing Limits

Some of the web pages that I needed to grab were quite long, which resulted in those images requiring more computational resources than the default Imagemagick configuration allows. This caused errors such as the following to appear:

convert-im6.q16: no images defined `/mnt/c/images/slices/IMG1466-%d.png' @ error/convert.c/ConvertImageCommand/3229.
convert-im6.q16: cache resources exhausted `/mnt/c/images/IMG1091.png' @ error/cache.c/OpenPixelCache/4095.

Imagemagick defines computational resources limits in /etc/ImageMagick-6/policy.xml. The default maximum memory is 256 KB, the default maximum allowable height is 16,000 pixels (16KP), and the default maximum area is 128M pixels. These values are defined by the following entries:

/etc/ImageMagick-6/policy.xml

<policy domain="resource" name="memory" value="256MiB"/>
<policy domain="resource" name="height" value="16KP"/>
<policy domain="resource" name="area" value="128MP"/>

I changed the maximum memory limit to 2 GB RAM, the maximum height limit to 10,000,000 pixels (10MP), and the maximum area limit to 2G pixels with these entries:

/etc/ImageMagick-6/policy.xml

<policy domain="resource" name="memory" value="2GiB"/>
<policy domain="resource" name="height" value="10MP"/>
<policy domain="resource" name="area" value="2GP"/>

Alternatively, I could have simply commented out the limits, as shown in highlighted text below.

/etc/ImageMagick-6/policy.xml

<!--
<policy domain="resource" name="memory" value="256MiB"/>
<policy domain="resource" name="height" value="16KP"/>
<policy domain="resource" name="area" value="128MP"/>
-->

The largest web page to be sliced was converted to a very tall image, which was 83,703 pixels high. It was sliced into 40 images.

Word Macro

A Word macro is also needed to insert the images into the currently open Word document in alphabetical order. I modified this one.

Microsoft Word Macro

Sub insertImages()
    Dim intResult As Integer
    Dim strPath As String
    Dim strFolderPath As String
    Dim objFSO As Object
    Dim objFolder As Object
    Dim objFile As Object
    Dim i As Integer

    intResult = Application.FileDialog(msoFileDialogFolderPicker).Show
    'Check if user canceled the dialog
    If intResult <> 0 Then
        'dispaly message box
        strFolderPath = Application.FileDialog(msoFileDialogFolderPicker).SelectedItems(1)
        'Create an instance of the FileSystemObject
        Set objFSO = CreateObject("Scripting.FileSystemObject")
        'Get the folder object
        Set objFolder = objFSO.GetFolder(strFolderPath)
        i = 1
        'loops through each file in the directory and prints their names and path
        For Each objFile In objFolder.Files
            'get file path
            strPath = objFile.Path
            'insert the image
            Selection.InlineShapes.AddPicture FileName:= _
               strPath, LinkToFile:=False, _
               SaveWithDocument:=True
        Next objFile
    End If
End Sub

Done!

😁

Thanks to the above automation, I was able to deliver the Word documents containing the sliced web pages to my client soon after they were requested.

Uncomplicated Firewall on Ubuntu

2022-07-16T00:00:00-04:00

All websites, especially ecommerce sites, need to be secure. A properly set up firewall is an essential component for a secure server.

Ubuntu 22.04 uses the Uncomplicated Firewall ufw firewall frontend by default. Ufw has been provided for Ubuntu since v8.04 (Hardy Heron).

Quick Setup

Enable ufw as follows:

Shell

$ sudo ufw enable
Firewall is active and enabled on system startup

Enable the ufw application profiles for ssh and nginx (HTTP/HTTPS) like this:

Shell

$ sudo ufw allow OpenSSH
Output
  Rule added
  Rule added (v6) 

$ sudo ufw allow 'Nginx Full'
Output
  Rule added
  Rule added (v6)

Diving Deeper

The following is mostly true:

The default firewall on Ubuntu 22.04 Jammy Jellyfish is ufw, which is short for “uncomplicated firewall.” Ufw is a frontend for the typical Linux iptables commands, but it is developed in such a way that basic firewall tasks can be performed without the knowledge of iptables.

Additionally, ufw can be managed from a graphical interface. In this tutorial, you will learn how to enable and disable the ufw firewall on Ubuntu 22.04 Jammy Jellyfish from both command line and GUI.

– From How to enable/disable firewall on Ubuntu 22.04 LTS Jammy Jellyfish Linux

The above makes no mention of how Ubuntu 22.04 replaced iptables with nftables, as described below.

nftables as the default firewall backend

Firewalling on Linux consists of two components – the firewall mechanism within the Linux kernel, and the tools used to configure this from userspace. The Linux kernel has traditionally supported two different subsystems for firewall policies – iptables / xtables and the newer nftables.

Nftables brings significant benefits both in terms of performance and flexibility when creating and deploying firewall rules, particularly for dual stack IPv4/IPv6 systems.

The traditional iptables userspace management tool now configures the nftables kernel backend, whilst the new nft userspace tool is also present to allow the creation of more flexible rules not supported by the traditional iptables paradigm.

– From What’s new in Security for Ubuntu 22.04 LTS?

Digital Ocean has a good ufw tutorial.

Using Nginx As a Reverse Proxy With SSL

2022-07-08T00:00:00-04:00

ScalaCourses.com

Recently I migrated ScalaCourses.com from AWS EC2/S3/CloudFront to a server in my apartment, which has fiber optic internet service. The server’s motherboard is an ASUS Sabertooth x79 with an Intel i7 4820, 32 GB DDR3 RAM, and a 4TB SATA SSD. This motherboard is unable to boot from NVMe drives, so SATA is necessary. At the time of this writing, the server runs Ubuntu 22.04.

The backup server is currently being set up. I am repurposing an old Hackintosh as a fallback to the Ubuntu server. That motherboard is another ASUS Sabertooth x79, with 32 GB DDR3 RAM and two 2 TB SATA drives.

Why Am I Doing This?

Once again, I control my hardware, my software, my network, and all ancillary services. After several years of using PaaS vendor servers, I am now reverting to running ScalaCourses.com on my own hardware and software, using my own network.

No longer will I subject myself to the unlimited financial liability that current PaaS vendors expose their customers to.

Definitions

A proxy is a person or process serving as an authorized agent or substitute for another. In computer science, a more specific term is forward proxy.

A proxy server is a server process that acts as an intermediary between a client requesting a resource, and the process that provides the resource.

A reverse proxy is a process that sits in front of other processes, and forwards client requests to them. The term forwarding process is similar to reverse proxy.

The definitions for proxy, forward proxy and reverse proxy all sound identical. The key difference between a reverse proxy and a forward proxy is that a forward proxy enables computers isolated on a private network to connect to the public internet, while a reverse proxy enables computers on the internet to access a private subnet.

Dealing With Details

The old Pound v2.8-2 reverse proxy that was the front end for the old Play Framework app that runs ScalaCourses.com is no longer viable, and the new version 3 of Pound is incomplete. Depending on configuration, reverse proxies can provide extra security from external attacks on a website, decrypt https requests to http, and act as stream editors for the content.

This site still uses AWS CloudFront, for the moment. I am researching alternatives, with the goal of closing my AWS account... unless they introduce a way to cap financial liability before I fully migrate off AWS.

Apache httpd vs. Nginx

Two popular reverse proxies are Apache mod_proxy_http2 and nginx. Anything Pound can do, both nginx and Apache httpd/2 can do. While they both work well, Apache httpd has an older code base, in fact, many of my websites ran on Apache httpd at the turn of the century.

Nginx is relatively newer than Apache httpd. It is performant, well supported, well documented, and widely available. I decided to use nginx as the reverse proxy because I had found that nginx worked well on previous projects.

Installing nginx and the Letsencrypt software is easy on Debian distros such as Ubuntu:

Shell

$ sudo apt install nginx certbot python3-certbot-nginx

Verifying the Nginx Build

When acting as a proxy server, nginx requires the http_sub_module to translate HTTP content. This allows the website content to be stream edited, so various links and paths that are not translated properly can be fixed up.

The nginx package provided by Ubuntu includes the ‑‑with‑http_sub_module option. You can verify that your instance of nginx was built with the ‑‑with‑http_sub_module option as follows:

Shell

$ nginx -V
nginx version: nginx/1.18.0 (Ubuntu)
built with OpenSSL 3.0.2 15 Mar 2022
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -ffile-prefix-map=/build/nginx-9P0wNJ/nginx-1.18.0=.
-flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat
-Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions
-flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx
--conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log
--error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid
--modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy
--http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat
--with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module
--with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module
--with-threads --add-dynamic-module=/build/nginx-9P0wNJ/nginx-1.18.0/debian/modules/http-geoip2
--with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module
--with-http_sub_module

Nginx SSL Configuration

Two files are needed for Letsencrypt to be able to create SSL certificates for nginx:

The contents of /etc/letsencrypt/options-ssl-nginx.conf need to be stored into /etc/letsencrypt/options-ssl-nginx.conf. I later discovered this file was also available locally at /usr/lib/python3/dist-packages/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf.
The contents of /etc/letsencrypt/ssl-dhparams.pem need to be stored into /etc/letsencrypt/ssl-dhparams.pem. I later discovered that this file was also available locally at /usr/lib/python3/dist-packages/certbot/ssl-dhparams.pem.

Making a Wildcard SSL Certificate

I knew an SSL wildcard certificate would be needed, so I made one using Letsencrypt. Please read Creating and Renewing Letsencrypt Wildcard SSL Certificates for details.

Update – 8 months after writing this blog post, I wrote about a better way to generate wildcard SSL certificates: Wildcard SSL Certificates for Let's Encrypt with Lego.

Defining the Nginx Website Reverse Proxy

Please see my blog post entitled Cross-Origin Resource Sharing (CORS) for a discussion of how to configure servers whose content needs to be proxied.

I saved the following configuration in a new file called /etc/nginx/sites-available/scalacourses.com. Note that a single server block answers on ports 80 and 443, using IPv4 and IPv6, for SSL and non-SSL requests, for the apex domain (scalacourses.com) and the www.scalacourses.com subdomain. No redirects are used.

Shell

server {
  listen 80;
  listen [::]:80;
  listen 443 ssl;
  listen [::]:443 ssl;

  server_name scalacourses.com www.scalacourses.com;

  ssl_certificate         /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/fullchain.pem;
  ssl_certificate_key     /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/privkey.pem;
  ssl_trusted_certificate /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/chain.pem;
  include /etc/letsencrypt/options-ssl-nginx.conf;
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

  root /var/www/html;
  index index.html;  # This gets served if the proxied website is down

  location / {
    proxy_pass http://localhost:9000/;
    proxy_set_header Accept-Encoding "";  # sub_filter requires this

    sub_filter 'https://localhost:9000/authenticate/' 'https://www.scalacourses.com/authenticate/';
    sub_filter_once off;
  }
}

I disabled the default site:

Shell

$ sudo rm /etc/nginx/sites-enabled/default

I enabled the new scalacourses.com site:

Shell

$ sudo ln /etc/nginx/sites-{available,enabled}/scalacourses.com

The nginx configuration was tested for syntax:

Shell

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

The nginx configuration was reloaded:

Shell

$ sudo systemctl reload nginx

Flush DNS Cache

Ensure that DNS requests receive up-to-date values by flushing the DNS cache. The following works on Ubuntu, but not when running on WSL/WSL2.

Shell

$ sudo resolvectl flush-caches

The following works on Windows 10:

Command and PowerShell consoles

C:\Users\Mike Slinn> ipconfig /flushdns

Verifying nginx Works

I verified that nginx was listening on ports 80 and 443. I highlighted the nginx process number – you might need to scroll the output below to the right to see it.

Shell

$ sudo netstat -tulpn | grep ':\(443\|80\)'
tcp    0    0 0.0.0.0:80       0.0.0.0:*       LISTEN   -
tcp6   0    0 :::80            :::*            LISTEN   -
tcp    0    0 0.0.0.0:443      0.0.0.0:*       LISTEN   87487/nginx: master

The executable for process 87487 can be found by:

Shell

$ ls -l /proc/87487/exe
lrwxrwxrwx 1 mslinn mslinn 0 Jul  7 09:13 /proc/5166/exe -> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java*

If more detail is desired, get it from the process list. Enclosing a character of the process id within square brackets is an old trick for only showing the desired process.

Shell

$ ps aux | grep [8]7487
mslinn      87487  2.4  1.3 6596476 439456 ?      Sl   09:13   0:20 java -Xms1024m -Xmx1024m -Dhttp.port=9000 /path/to/jar

If there is any problem getting things to work, it is often helpful to monitor the logs as you click on the web pages:

Shell

$ sudo tail -f /var/log/nginx/*.log

Finishing Up

Make nginx start each time the system starts as follows.

Shell

$ sudo systemctl enable nginx
Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable nginx 

$ sudo update-rc.d nginx defaults

Performance Test

KeyCDN offfers free website performance statistics. The result column labeled TTFB means “time to first byte”, which is the length of time required for the website content to begin being received by a user's web browser.

I am in Montreal, Canada. Response time for TTFB reported by KeyCDN varies, from a minimum of 68ms in New York, to a maximum of 815ms in Bangalore. This range of response times is typical for websites that have a centralized process. The speed of light is finite, after all.

Free Availability Monitoring

HetrixTools offers free availability monitoring for up to 10 websites. It was quick and easy to set up.

We Are Live!

ScalaCourses.com now serves Scala students from its newly refurbished server! 😁

Trialing mslinn.com on Linode Storage

2022-07-01T00:00:00-04:00

This is another post in my ongoing saga of moving off AWS, which does not integrate security with real-time billing. I recently learned this the hard way: when my AWS account was hijacked, in less than 15 minutes, a huge bill was incurred.

“Pay-as-you-go” is shorthand for “there is nothing you can do to limit your financial liability”

The world of pain that I experienced after the breach was inflicted by broken and wasteful AWS remedial processes, and an ineffective AWS management structure. This type of issue only is enabled because of deficiencies in the AWS architecture. Those AWS architectural deficiencies feel like the result of an exploitive mindset:

Unlimited financial liability is our customer’s problem, not ours – as a result, exploits are quite profitable for us.

– From a mythical retrospective discussion at an AWS offsite.

At this moment that feature of setting limits does not exist, Azure is not able to safeguard customers from unlimited financial liability.

– From an email sent to me from Microsoft Azure support staff on 2022-06-22.

Demand Limits to Financial Liability From PaaS Vendors

PaaS vendors currently provide accounts with all services ready to go, without limit. That is good for the vendor's bottom line, but highly dangerous for their customers.

All PaaS customers should demand the ability for themselves to be able to set firm limits on budgeted expenses, along with the ability to deny all services not explicitly authorized.

You can buy insurance against losses resulting from various calamities, but you cannot limit your financial liability with PaaS vendors.

Yet.

Website Hosting Market

Recently, I have spent a lot of time looking at options for hosting websites. I found the following types of products:

	WordPress	General Web Server	VM	S3 Compatible
CDN	No	No	No	Yes
Speed	Slow to Medium	Slow to Medium	Slow to Fast	Fast
Reliability	Fair	Fair	Depends on you	Good
Financial Liability	Fixed monthly cost	Fixed monthly cost	Depends	Unlimited
Storage	Small	Small	Depends	Unlimited
CLI & API	No	No	Depends	Yes

This website requires about 220 GB of storage. It has numerous images, in 2 versions: webp and png. So for this website, ‘small’ means less than 250 GB storage.

I decided to give the S3-compatible product Linode Storage a try.

Why Linode?

First, the positive: Linode is a pioneer in virtual computing. It has been on my short list of places for hosting my pet projects for many years. Prices have always been quite competitive, products solid, with good support ... and smart people have answered the phone when I have called. Being able to easily speak with a capable human provides huge value.

On the other hand, as I shall demonstrate in this blog post, Linode’s documentation presents as a significant barrier to customers considering adopting their services. I had to work a lot harder than I should have to get my evaluation website up and running. Hopefully, this document is complete enough, so others can follow along and host their static websites on Linode Storage.

Acquired by Akamai

Linode was acquired by Akamai 3 months ago. Akamai is the original CDN, and their network is gigantic.

Linode does not yet offer a CDN product, but the person I spoke to at Linode when I started writing this post suggested that a CDN product from Linode based on Akamai’s network might be available soon. He also said that they were planning to working on a mechanism for limiting financial liability to their customers in 2022.

This was music to my financially risk-averse ears!

The remainder of this blog post will take you through all the steps necessary to:

Install and configure software tools on a WSL / WSL2 / Ubuntu computer
Make an S3-compatible bucket and set it up to hold a website
Upload the website, and easily handle mimetype issues
Generate and install a free 4096-bit SSL wildcard certificate
Get a security rating from Qualys / SSL Labs

Trialing Linode Storage

Installing s3cmd

I installed the recommended S3-compatible command-line program s3cmd on WSL2 / Ubuntu like this:

Shell

$ yes | sudo apt install s3cmd
Reading package lists... Done
  Building dependency tree... Done
  Reading state information... Done
  The following additional packages will be installed:
    python3-magic
  The following NEW packages will be installed:
    python3-magic s3cmd
  0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
  Need to get 133 kB of archives.
  After this operation, 584 kB of additional disk space will be used.
  Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-magic all 2:0.4.24-2 [12.6 kB]
  Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 s3cmd all 2.2.0-1 [120 kB]
  Fetched 133 kB in 0s (278 kB/s)
  Selecting previously unselected package python3-magic.
  (Reading database ... 169821 files and directories currently installed.)
  Preparing to unpack .../python3-magic_2%3a0.4.24-2_all.deb ...
  Unpacking python3-magic (2:0.4.24-2) ...
  Selecting previously unselected package s3cmd.
  Preparing to unpack .../archives/s3cmd_2.2.0-1_all.deb ...
  Unpacking s3cmd (2.2.0-1) ...
  Setting up python3-magic (2:0.4.24-2) ...
  Setting up s3cmd (2.2.0-1) ...
  Processing triggers for man-db (2.10.2-1) ...
  Scanning processes...
  Scanning processor microcode...
  Scanning linux images...

  Failed to retrieve available kernel versions.

  Failed to check for processor microcode upgrades.

  No services need to be restarted.

  No containers need to be restarted.

  No user sessions are running outdated binaries.

  No VM guests are running outdated hypervisor (qemu) binaries on this host.

Here is the s3cmd help message:

Shell

$ s3cmd
Usage: s3cmd [options] COMMAND [parameters]

S3cmd is a tool for managing objects in Amazon S3 storage. It allows for
making and removing "buckets" and uploading, downloading and removing
"objects" from these buckets.

Options:
-h, --help            show this help message and exit
--configure           Invoke interactive (re)configuration tool. Optionally
use as '--configure s3://some-bucket' to test access
to a specific bucket instead of attempting to list
them all.
-c FILE, --config=FILE
Config file name. Defaults to $HOME/.s3cfg
--dump-config         Dump current configuration after parsing config files
and command line options and exit.
--access_key=ACCESS_KEY
AWS Access Key
--secret_key=SECRET_KEY
AWS Secret Key
--access_token=ACCESS_TOKEN
AWS Access Token
-n, --dry-run         Only show what should be uploaded or downloaded but
don't actually do it. May still perform S3 requests to
get bucket listings and other information though (only
for file transfer commands)
-s, --ssl             Use HTTPS connection when communicating with S3.
(default)
--no-ssl              Don't use HTTPS.
-e, --encrypt         Encrypt files before uploading to S3.
--no-encrypt          Don't encrypt files.
-f, --force           Force overwrite and other dangerous operations.
--continue            Continue getting a partially downloaded file (only for
[get] command).
--continue-put        Continue uploading partially uploaded files or
multipart upload parts.  Restarts parts/files that
don't have matching size and md5.  Skips files/parts
that do.  Note: md5sum checks are not always
sufficient to check (part) file equality.  Enable this
at your own risk.
--upload-id=UPLOAD_ID
UploadId for Multipart Upload, in case you want
continue an existing upload (equivalent to --continue-
put) and there are multiple partial uploads.  Use
s3cmd multipart [URI] to see what UploadIds are
associated with the given URI.
--skip-existing       Skip over files that exist at the destination (only
for [get] and [sync] commands).
-r, --recursive       Recursive upload, download or removal.
--check-md5           Check MD5 sums when comparing files for [sync].
(default)
--no-check-md5        Do not check MD5 sums when comparing files for [sync].
Only size will be compared. May significantly speed up
transfer but may also miss some changed files.
-P, --acl-public      Store objects with ACL allowing read for anyone.
--acl-private         Store objects with default ACL allowing access for you
only.
--acl-grant=PERMISSION:EMAIL or USER_CANONICAL_ID
Grant stated permission to a given amazon user.
Permission is one of: read, write, read_acp,
write_acp, full_control, all
--acl-revoke=PERMISSION:USER_CANONICAL_ID
Revoke stated permission for a given amazon user.
Permission is one of: read, write, read_acp,
write_acp, full_control, all
-D NUM, --restore-days=NUM
Number of days to keep restored file available (only
for 'restore' command). Default is 1 day.
--restore-priority=RESTORE_PRIORITY
Priority for restoring files from S3 Glacier (only for
'restore' command). Choices available: bulk, standard,
expedited
--delete-removed      Delete destination objects with no corresponding
source file [sync]
--no-delete-removed   Don't delete destination objects [sync]
--delete-after        Perform deletes AFTER new uploads when delete-removed
is enabled [sync]
--delay-updates       *OBSOLETE* Put all updated files into place at end
[sync]
--max-delete=NUM      Do not delete more than NUM files. [del] and [sync]
--limit=NUM           Limit number of objects returned in the response body
(only for [ls] and [la] commands)
--add-destination=ADDITIONAL_DESTINATIONS
Additional destination for parallel uploads, in
addition to last arg.  May be repeated.
--delete-after-fetch  Delete remote objects after fetching to local file
(only for [get] and [sync] commands).
-p, --preserve        Preserve filesystem attributes (mode, ownership,
timestamps). Default for [sync] command.
--no-preserve         Don't store FS attributes
--exclude=GLOB        Filenames and paths matching GLOB will be excluded
from sync
--exclude-from=FILE   Read --exclude GLOBs from FILE
--rexclude=REGEXP     Filenames and paths matching REGEXP (regular
expression) will be excluded from sync
--rexclude-from=FILE  Read --rexclude REGEXPs from FILE
--include=GLOB        Filenames and paths matching GLOB will be included
even if previously excluded by one of
--(r)exclude(-from) patterns
--include-from=FILE   Read --include GLOBs from FILE
--rinclude=REGEXP     Same as --include but uses REGEXP (regular expression)
instead of GLOB
--rinclude-from=FILE  Read --rinclude REGEXPs from FILE
--files-from=FILE     Read list of source-file names from FILE. Use - to
read from stdin.
--region=REGION, --bucket-location=REGION
Region to create bucket in. As of now the regions are:
us-east-1, us-west-1, us-west-2, eu-west-1, eu-
central-1, ap-northeast-1, ap-southeast-1, ap-
southeast-2, sa-east-1
--host=HOSTNAME       HOSTNAME:PORT for S3 endpoint (default:
s3.amazonaws.com, alternatives such as s3-eu-
west-1.amazonaws.com). You should also set --host-
bucket.
--host-bucket=HOST_BUCKET
DNS-style bucket+hostname:port template for accessing
a bucket (default: %(bucket)s.s3.amazonaws.com)
--reduced-redundancy, --rr
Store object with 'Reduced redundancy'. Lower per-GB
price. [put, cp, mv]
--no-reduced-redundancy, --no-rr
Store object without 'Reduced redundancy'. Higher per-
GB price. [put, cp, mv]
--storage-class=CLASS
Store object with specified CLASS (STANDARD,
STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER
or DEEP_ARCHIVE). [put, cp, mv]
--access-logging-target-prefix=LOG_TARGET_PREFIX
Target prefix for access logs (S3 URI) (for [cfmodify]
and [accesslog] commands)
--no-access-logging   Disable access logging (for [cfmodify] and [accesslog]
commands)
--default-mime-type=DEFAULT_MIME_TYPE
Default MIME-type for stored objects. Application
default is binary/octet-stream.
-M, --guess-mime-type
Guess MIME-type of files by their extension or mime
magic. Fall back to default MIME-Type as specified by
--default-mime-type option
--no-guess-mime-type  Don't guess MIME-type and use the default type
instead.
--no-mime-magic       Don't use mime magic when guessing MIME-type.
-m MIME/TYPE, --mime-type=MIME/TYPE
Force MIME-type. Override both --default-mime-type and
--guess-mime-type.
--add-header=NAME:VALUE
Add a given HTTP header to the upload request. Can be
used multiple times. For instance set 'Expires' or
'Cache-Control' headers (or both) using this option.
--remove-header=NAME  Remove a given HTTP header.  Can be used multiple
times.  For instance, remove 'Expires' or 'Cache-
Control' headers (or both) using this option. [modify]
--server-side-encryption
Specifies that server-side encryption will be used
when putting objects. [put, sync, cp, modify]
--server-side-encryption-kms-id=KMS_KEY
Specifies the key id used for server-side encryption
with AWS KMS-Managed Keys (SSE-KMS) when putting
objects. [put, sync, cp, modify]
--encoding=ENCODING   Override autodetected terminal and filesystem encoding
(character set). Autodetected: UTF-8
--add-encoding-exts=EXTENSIONs
Add encoding to these comma delimited extensions i.e.
(css,js,html) when uploading to S3 )
--verbatim            Use the S3 name as given on the command line. No pre-
processing, encoding, etc. Use with caution!
--disable-multipart   Disable multipart upload on files bigger than
--multipart-chunk-size-mb
--multipart-chunk-size-mb=SIZE
Size of each chunk of a multipart upload. Files bigger
than SIZE are automatically uploaded as multithreaded-
multipart, smaller files are uploaded using the
traditional method. SIZE is in Mega-Bytes, default
chunk size is 15MB, minimum allowed chunk size is 5MB,
maximum is 5GB.
--list-md5            Include MD5 sums in bucket listings (only for 'ls'
command).
-H, --human-readable-sizes
Print sizes in human readable form (eg 1kB instead of
1234).
--ws-index=WEBSITE_INDEX
Name of index-document (only for [ws-create] command)
--ws-error=WEBSITE_ERROR
Name of error-document (only for [ws-create] command)
--expiry-date=EXPIRY_DATE
Indicates when the expiration rule takes effect. (only
for [expire] command)
--expiry-days=EXPIRY_DAYS
Indicates the number of days after object creation the
expiration rule takes effect. (only for [expire]
command)
--expiry-prefix=EXPIRY_PREFIX
Identifying one or more objects with the prefix to
which the expiration rule applies. (only for [expire]
command)
--progress            Display progress meter (default on TTY).
--no-progress         Don't display progress meter (default on non-TTY).
--stats               Give some file-transfer stats.
--enable              Enable given CloudFront distribution (only for
[cfmodify] command)
--disable             Disable given CloudFront distribution (only for
[cfmodify] command)
--cf-invalidate       Invalidate the uploaded filed in CloudFront. Also see
[cfinval] command.
--cf-invalidate-default-index
When using Custom Origin and S3 static website,
invalidate the default index file.
--cf-no-invalidate-default-index-root
When using Custom Origin and S3 static website, don't
invalidate the path to the default index file.
--cf-add-cname=CNAME  Add given CNAME to a CloudFront distribution (only for
[cfcreate] and [cfmodify] commands)
--cf-remove-cname=CNAME
Remove given CNAME from a CloudFront distribution
(only for [cfmodify] command)
--cf-comment=COMMENT  Set COMMENT for a given CloudFront distribution (only
for [cfcreate] and [cfmodify] commands)
--cf-default-root-object=DEFAULT_ROOT_OBJECT
Set the default root object to return when no object
is specified in the URL. Use a relative path, i.e.
default/index.html instead of /default/index.html or
s3://bucket/default/index.html (only for [cfcreate]
and [cfmodify] commands)
-v, --verbose         Enable verbose output.
-d, --debug           Enable debug output.
--version             Show s3cmd version (2.2.0) and exit.
-F, --follow-symlinks
Follow symbolic links as if they are regular files
--cache-file=FILE     Cache FILE containing local source MD5 values
-q, --quiet           Silence output on stdout
--ca-certs=CA_CERTS_FILE
Path to SSL CA certificate FILE (instead of system
default)
--ssl-cert=SSL_CLIENT_CERT_FILE
Path to client own SSL certificate CRT_FILE
--ssl-key=SSL_CLIENT_KEY_FILE
Path to client own SSL certificate private key
KEY_FILE
--check-certificate   Check SSL certificate validity
--no-check-certificate
Do not check SSL certificate validity
--check-hostname      Check SSL certificate hostname validity
--no-check-hostname   Do not check SSL certificate hostname validity
--signature-v2        Use AWS Signature version 2 instead of newer signature
methods. Helpful for S3-like systems that don't have
AWS Signature v4 yet.
--limit-rate=LIMITRATE
Limit the upload or download speed to amount bytes per
second.  Amount may be expressed in bytes, kilobytes
with the k suffix, or megabytes with the m suffix
--no-connection-pooling
Disable connection re-use
--requester-pays      Set the REQUESTER PAYS flag for operations
-l, --long-listing    Produce long listing [ls]
--stop-on-error       stop if error in transfer
--content-disposition=CONTENT_DISPOSITION
Provide a Content-Disposition for signed URLs, e.g.,
"inline; filename=myvideo.mp4"
--content-type=CONTENT_TYPE
Provide a Content-Type for signed URLs, e.g.,
"video/mp4"

Commands:
Make bucket
s3cmd mb s3://BUCKET
Remove bucket
s3cmd rb s3://BUCKET
List objects or buckets
s3cmd ls [s3://BUCKET[/PREFIX]]
List all object in all buckets
s3cmd la
Put file into bucket
s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX]
Get file from bucket
s3cmd get s3://BUCKET/OBJECT LOCAL_FILE
Delete file from bucket
s3cmd del s3://BUCKET/OBJECT
Delete file from bucket (alias for del)
s3cmd rm s3://BUCKET/OBJECT
Restore file from Glacier storage
s3cmd restore s3://BUCKET/OBJECT
Synchronize a directory tree to S3 (checks files freshness using size and md5 checksum, unless overridden by options, see below)
s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR or s3://BUCKET[/PREFIX] s3://BUCKET[/PREFIX]
Disk usage by buckets
s3cmd du [s3://BUCKET[/PREFIX]]
Get various information about Buckets or Files
s3cmd info s3://BUCKET[/OBJECT]
Copy object
s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
Modify object metadata
s3cmd modify s3://BUCKET1/OBJECT
Move object
s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
Modify Access control list for Bucket or Files
s3cmd setacl s3://BUCKET[/OBJECT]
Modify Bucket Policy
s3cmd setpolicy FILE s3://BUCKET
Delete Bucket Policy
s3cmd delpolicy s3://BUCKET
Modify Bucket CORS
s3cmd setcors FILE s3://BUCKET
Delete Bucket CORS
s3cmd delcors s3://BUCKET
Modify Bucket Requester Pays policy
s3cmd payer s3://BUCKET
Show multipart uploads
s3cmd multipart s3://BUCKET [Id]
Abort a multipart upload
s3cmd abortmp s3://BUCKET/OBJECT Id
List parts of a multipart upload
s3cmd listmp s3://BUCKET/OBJECT Id
Enable/disable bucket access logging
s3cmd accesslog s3://BUCKET
Sign arbitrary string using the secret key
s3cmd sign STRING-TO-SIGN
Sign an S3 URL to provide limited public access with expiry
s3cmd signurl s3://BUCKET/OBJECT <expiry_epoch|+expiry_offset>
Fix invalid file names in a bucket
s3cmd fixbucket s3://BUCKET[/PREFIX]
Create Website from bucket
s3cmd ws-create s3://BUCKET
Delete Website
s3cmd ws-delete s3://BUCKET
Info about Website
s3cmd ws-info s3://BUCKET
Set or delete expiration rule for the bucket
s3cmd expire s3://BUCKET
Upload a lifecycle policy for the bucket
s3cmd setlifecycle FILE s3://BUCKET
Get a lifecycle policy for the bucket
s3cmd getlifecycle s3://BUCKET
Remove a lifecycle policy for the bucket
s3cmd dellifecycle s3://BUCKET
List CloudFront distribution points
s3cmd cflist
Display CloudFront distribution point parameters
s3cmd cfinfo [cf://DIST_ID]
Create CloudFront distribution point
s3cmd cfcreate s3://BUCKET
Delete CloudFront distribution point
s3cmd cfdelete cf://DIST_ID
Change CloudFront distribution point parameters
s3cmd cfmodify cf://DIST_ID
Display CloudFront invalidation request(s) status
s3cmd cfinvalinfo cf://DIST_ID[/INVAL_ID]

For more information, updates and news, visit the s3cmd website:
http://s3tools.org

I went to the Linode signup page and signed up with my email. Using third parties for authentication means they track you more easily, and introduces an unnecessary dependency.

The email signup procedure requires a mobile phone for SMS-based MFA. I would rather use a TOTP authenticator app instead of SMS for MFA.

I want to ensure that I limit my financial liability. One way of protecting myself is to limit the maximum amount that can be charged to the payment mechanism. Because PayPal has no maximum transaction limit, it is not a suitable choice for a service with unlimited financial liability. However, credit cards can have transaction limits set, and Google Pay has the following limits:

If you set up your Google Pay balance to make contactless payments, there are some transaction limits:

Maximum single transaction amount: $2,000 USD.
Daily maximum total transaction amount: $2,500 USD.
Up to 15 transactions per day.
Additional limits on the dollar amount or frequency of transactions may be imposed in accordance with the Google Pay Terms of Service. – From Google Pay Help

Here are additional limits for Google Pay. Because of prior good experiences with how credit card processors handled fraud, I decided to use a credit card with a low limit instead of Google Pay.

Generating Linode Access Keys

I followed the directions:

Logged into the Cloud Manager.
Selected the Object Storage menu item in the sidebar and clicked on the Access Keys tab.
Clicked on the Create Access Key button, which displays the Create Access Key panel.
Typed in the name mslinn as the label for the new access key.
Clicked the Submit button to create the access key.
The new access key and its secret key are displayed. This is the only time that the secret key is visible. I stored the access key and the secret key in my password manager, lastpass.com.

Configuring s3cmd

s3cmd is a Python program that works with all S3-compatible APIs, such as those provided by AWS, Linode Storage, Google Cloud Storage and DreamHost DreamObjects. I intend to use its sync subcommand to synchronize the www.mslinn.com bucket contents in Linode Storage with the most recently generated version of this website by Jekyll.

s3cmd needs to be configured for Linode before it can be used. Some configuration settings for Linode are non-obvious. The following should work for most users, with the caution that the access key and secret key are of course unique for every user.

Shell

$ s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
  Refer to user manual for detailed description of all options.

  Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
  Access Key [asdfasdf]: asdfasdfasdf
  Secret Key [asdfasdfasdf]: asdfasdfasdf
  Default Region [US]:

  Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
  S3 Endpoint [s3.amazonaws.com]: us-east-1.linodeobjects.com

  Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
  if the target S3 system supports dns based buckets.
  DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: %(bucket)s.us-east-1.linodeobjects.com

  Encryption password is used to protect your files from reading
  by unauthorized persons while in transfer to S3
  Encryption password:
  Path to GPG program [/usr/bin/gpg]:

  When using secure HTTPS protocol all communication with Amazon S3
  servers is protected from 3rd party eavesdropping. This method is
  slower than plain HTTP, and can only be proxied with Python 2.7 or newer
  Use HTTPS protocol [Yes]:

  On some networks all internet access must go through a HTTP proxy.
  Try setting it here if you can't connect to S3 directly
  HTTP Proxy server name:

  New settings:
  Access Key: asdfasdfasdf
Secret Key: asdfasdfasdf
Default Region: US
  S3 Endpoint: https://us-east-1.linodeobjects.com
DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.us-east-1.linodeobjects.com
Encryption password:
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

  Test access with supplied credentials? [Y/n] n
  Save settings? [y/N] y
  Configuration saved to '/home/mslinn/.s3cfg'

I was unable to successfully test access. Two different errors appeared at different times:

Please wait, attempting to list all buckets...
ERROR: Test failed: 403 (SignatureDoesNotMatch)
ERROR: Test failed: [Errno -2] Name or service not known

Eventually I saved the configuration without testing as shown above. This created a file called $HOME/.s3cfg:

~/.s3cfg

[default]
access_key = asdfasdfasdf
access_token =
add_encoding_exts =
add_headers =
bucket_location = US
ca_certs_file =
cache_file =
check_ssl_certificate = True
check_ssl_hostname = True
cloudfront_host = cloudfront.amazonaws.com
connection_max_age = 5
connection_pooling = True
content_disposition =
content_type =
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encoding = UTF-8
encrypt = False
expiry_date =
expiry_days =
expiry_prefix =
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = us-east-1.linodeobjects.com
host_bucket = %(bucket)s.us-east-1.linodeobjects.com
human_readablesizes = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
kms_key =
limit = -1
limitrate = 0
list_md5 = False
log_target_prefix =
long_listing = False
max_delete = -1
mime_type =
multipart_chunksize_mb = 15
multipart_copy_chunksize_mb = 1024
multipart_max_chunks = 10000
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
public_url_use_https = False
put_continue = False
recursive = False
recv_chunk = 65536
reduced_redundancy = False
requester_pays = False
restore_days = 1
restore_priority = Standard
secret_key = asdfasdfasdf
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
signurl_use_https = False
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 300
ssl_client_cert_file =
ssl_client_key_file =
stats = False
stop_on_error = False
storage_class =
throttle_max = 100
upload_id =
urlencoding_mode = normal
use_http_expect = False
use_https = True
use_mime_magic = True
verbosity = INFO
website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
website_error =
website_index = index.html

According to the docs (Access Buckets and Files through URLs), the configuration file needs website_endpoint: http://%(bucket)s.website-[cluster-url]/. However, this is an error; instead of cluster-url, which might be https://us-east-1.linodeobjects.com, cluster-id should be used, which for me was simply us-east-1. Thus the endpoint for me, and everyone with a bucket at us-east-1 in Newark, New Jersey should be:
http://%(bucket)s.website-us-east-1.linodeobjects.com/

After editing the file to manually change the value of website_endpoint, I tried to list the buckets, and that worked:

Shell

$ s3cmd ls
2022-07-01 17:32  s3://mslinn

linode-cli

Next I tried linode-cli.

Shell

$ pip install linode-cli
Collecting linode-cli
Downloading linode_cli-5.21.0-py2.py3-none-any.whl (204 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 204.2/204.2 kB 4.8 MB/s eta 0:00:00
Collecting terminaltables
Downloading terminaltables-3.1.10-py2.py3-none-any.whl (15 kB)
Collecting PyYAML
Downloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 682.2/682.2 kB 16.8 MB/s eta 0:00:00
Requirement already satisfied: requests in /home/mslinn/venv/default/lib/python3.10/site-packages (from linode-cli) (2.28.0)
Requirement already satisfied: charset-normalizer~=2.0.0 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests->linode-cli) (2.0.12)
Requirement already satisfied: idna<4,>=2.5 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests->linode-cli) (3.3)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests->linode-cli) (1.26.9)
Requirement already satisfied: certifi>=2017.4.17 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests->linode-cli) (2022.6.15)
Installing collected packages: terminaltables, PyYAML, linode-cli
Successfully installed PyYAML-6.0 linode-cli-5.21.0 terminaltables-3.1.10

The documentation fails to mention that the first time linode-cli is executed, and every time it is invoked with the configure subcommand, it attempts to open a web browser. WSL and WSL2 will not respond appropriately unless an X client is already running, for example GWSL.

Shell

$ linode-cli configure

Welcome to the Linode CLI.
This will walk you through some initial setup.

The CLI will use its web-based authentication to log you in.
If you prefer to supply a Personal Access Token, use
linode-cli configure --token

Press enter to continue.
This will open a browser and proceed with authentication.
A browser should open directing you to this URL to authenticate:

https://login.linode.com/oauth/authorize?client_id=asdfasdfasdf&response_type=token&scopes=*&redirect_uri=http://localhost:45789

If you are not automatically directed there, please copy/paste the link into your browser
to continue..


Configuring mslinn


Default Region for operations.  Choices are:
1 - ap-west
2 - ca-central
3 - ap-southeast
4 - us-central
5 - us-west
6 - us-southeast
7 - us-east
8 - eu-west
9 - ap-south
10 - eu-central
11 - ap-northeast

Default Region (Optional): 7

Default Type of Linode to deploy.  Choices are:
1 - g6-nanode-1
2 - g6-standard-1
3 - g6-standard-2
4 - g6-standard-4
5 - g6-standard-6
6 - g6-standard-8
7 - g6-standard-16
8 - g6-standard-20
9 - g6-standard-24
10 - g6-standard-32
11 - g7-highmem-1
12 - g7-highmem-2
13 - g7-highmem-4
14 - g7-highmem-8
15 - g7-highmem-16
16 - g6-dedicated-2
17 - g6-dedicated-4
18 - g6-dedicated-8
19 - g6-dedicated-16
20 - g6-dedicated-32
21 - g6-dedicated-48
22 - g6-dedicated-50
23 - g6-dedicated-56
24 - g6-dedicated-64
25 - g1-gpu-rtx6000-1
26 - g1-gpu-rtx6000-2
27 - g1-gpu-rtx6000-3
28 - g1-gpu-rtx6000-4

Default Type of Linode (Optional):

Default Image to deploy to new Linodes.  Choices are:
1 - linode/almalinux8
2 - linode/almalinux9
3 - linode/alpine3.12
4 - linode/alpine3.13
5 - linode/alpine3.14
6 - linode/alpine3.15
7 - linode/alpine3.16
8 - linode/arch
9 - linode/centos7
10 - linode/centos-stream8
11 - linode/centos-stream9
12 - linode/debian10
13 - linode/debian11
14 - linode/debian9
15 - linode/fedora34
16 - linode/fedora35
17 - linode/fedora36
18 - linode/gentoo
19 - linode/kali
20 - linode/debian11-kube-v1.20.15
21 - linode/debian9-kube-v1.20.7
22 - linode/debian9-kube-v1.21.1
23 - linode/debian11-kube-v1.21.12
24 - linode/debian9-kube-v1.22.2
25 - linode/debian11-kube-v1.22.9
26 - linode/debian11-kube-v1.23.6
27 - linode/opensuse15.3
28 - linode/opensuse15.4
29 - linode/rocky8
30 - linode/slackware14.2
31 - linode/slackware15.0
32 - linode/ubuntu16.04lts
33 - linode/ubuntu18.04
34 - linode/ubuntu20.04
35 - linode/ubuntu21.10
36 - linode/ubuntu22.04
37 - linode/centos8
38 - linode/slackware14.1
39 - linode/ubuntu21.04

Default Image (Optional):
Active user is now mslinn

Config written to /home/mslinn/.config/linode-cli

linode-cli configuration created this file:

~/.config/linode-cli

[DEFAULT]
default-user = mslinn

[mslinn]
token = asdfasdfasdfasdfasdfasdf
region = us-east

Right after I did the above, I was surprised to get the following email from Linode:

Hello,

This is a notification to inform you that a device has been trusted to skip authentication on your Linode Account (mslinn). The request came from the following IP address: 70.53.179.203.

The device will not be prompted for a username or password for 30 days.

If this action did not originate from you, we recommend logging in and changing your password immediately. Also, as an extra layer of security, we highly recommend enabling two-factor authentication on your account. Please see the link below for more information on how to enable two-factor authentication:

Two-Factor Authentication

If you have any questions or concerns, please do not hesitate to contact us 24/7 by opening a support ticket from within the Linode Manager, giving us a call, or emailing support@linode.com.

Contact

Thank you,
The Linode.com Team

Contact Us

I began to suspect that Linode’s documentation had major deficiencies at this point. Instead of merely configuring a command-line client, my IP was whitelisted unexpectedly. I have nothing polite to say about this.

After poking around a bit, I discovered my API Tokens page on Linode. It seems that the linode-cli created a personal access token with a label containing the name of the computer used: Linode CLI @ camille.

Clicking on the View Scopes button above displays the permissions granted to the token:

This personal access token is all-powerful. The linode-cli quietly created a personal access token with all permissions enabled, and did not warn the user, and did not say how to reduce or limit the permissions.

Security is enhanced when the minimum permission necessary is provided to accomplish necessary tasks. Instead, this token silently maximizes the financial risk to the person or entity who pays for the account.

Creating the Website Bucket

The name of the bucket must exactly match the name of the subdomain that the website is served from. If you try to serve content from a misnamed bucket, you will not be able to apply an SSL certificate; instead, the certificate will be issued by linodeobjects.com.

I wanted to test using the subdomain linode.mslinn.com, and then if all was well, switch www.mslinn.com over to Linode Storage. Because buckets cannot be renamed, I had to make 2 buckets with identical contents, one called linode.mslinn.com and the other called www.mslinn.com. If I decide to stay with Linode, I will delete the bucket I am using for testing, called linode.mslinn.com, and run this website from the www.mslinn.com bucket.

I created a website-enabled bucket called www.mslinn.com as follows:

Shell

$ s3cmd mb --acl-public s3://www.mslinn.com
Bucket 's3://www.mslinn.com/' created 

$ s3cmd ls
2022-07-01 17:32  s3://www.mslinn.com 

$ s3cmd ws-create \
  --ws-index=index.html \
  --ws-error=404.html \
  s3://www.mslinn.com
Bucket 's3://www.mslinn.com/': website configuration created.

The s3cmd ws-info subcommand displays the Linode Object Storage bucket URL as the Website endpoint:

Shell

$ s3cmd ws-info s3://www.mslinn.com
Bucket s3://www.mslinn.com/: Website configuration
  Website endpoint: http://www.mslinn.com.website-us-east-1.linodeobjects.com/
  Index document:   index.html
  Error document:   404.html

Defining a CNAME for the Bucket

Although Linode provides a DNS Manager, while I am testing out Linode I wanted to continue using Namecheap for DNS, so I navigated to ap.www.namecheap.com/Domains/DomainControlPanel/mslinn.com/advancedns and created CNAMEs like this:

linode.mslinn.com  CNAME  linode.mslinn.com.website-us-east-1.linodeobjects.com
www.mslinn.com     CNAME  www.mslinn.com.website-us-east-1.linodeobjects.com

Warning: if the CNAME does not point to a URL with the word website in it, for example www.mslinn.com.us-east-1.linodeobjects.com, the default index file and the error file will not automatically be served when expected.

Making a Free SSL Certificate

Qualys / SSL Labs rates the SSL security provided by AWS CloudFront using the AWS-generated SSL certificates with a “B” grade.

Lets see if Linode Object Storage can host a more secure website when provided with a 4096-bit certificate generated by certbot/letsencrypt. The conclusion of this post will reveal the answer.

Please read Creating and Renewing Letsencrypt Wildcard SSL Certificates for details.

Linode DNS Authentication

Eventually, if I stick with Linode, I could use the certbot DNS plugin for Linode, but I will not use it right now because I am trialing Linode. Use of this plugin requires a configuration file containing Linode API credentials, obtained from your Linode account’s Applications & API Tokens page. I stored the credentials in ~/.certbot/linode.ini:

~/.certbot/linode.ini

# Linode API credentials used by Certbot
dns_linode_key = 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ64
dns_linode_version = [|3|4]

Here is how to create an SSL certificate certbot using Linode DNS authentication:

Shell

$ certbot certonly \
  --dns-linode \
  --dns-linode-credentials ~/.certbot/linode.ini \
  --rsa-key-size 4096 \
  -d mslinn.com \
  -d *.mslinn.com \
  --config-dir ~/.certbot/mslinn.com/config \
  --logs-dir ~/.certbot/mslinn.com/logs \
  --work-dir ~/.certbot/mslinn.com/work

Installing the Certificate

Now I created 2 certificates: one with the full chain of responsibility (fullchain.pem) and one containing my private key (privkey.pem). I used bash environment variables called CERT_DIR, CERT and KEY to make the incantation convenient to type in. CERT and KEY contain the contents of the files fullchain.pem and privkey.pem, respectively. As you can see, the result was a valid (wildcard) SSL certificate.

Shell

$ CERT_DIR=/home/mslinn/.certbot/mslinn.com/config/live/mslinn.com

$ CERT="$( cat $CERT_DIR/fullchain.pem )"

$ KEY="$( cat $CERT_DIR/privkey.pem )"

$ linode-cli object-storage ssl-upload \
  us-east-1 www.mslinn.com \
  --certificate "$CERT" \
  --private_key "$KEY"
┌──────┐
│ ssl  │
├──────┤
│ True │
└──────┘

Syncing the Website Bucket

I wrote a bash script to build this Jekyll-generated website. Jekyll places the generated website in a directory called _site. I synchronized the contents of the freshly generated mslinn.com website, stored in _site/, with the bucket as follows. Note the trailing slash on _site/, it is significant.

Shell

$ s3cmd sync \
  --acl-public --delete-removed --guess-mime-type --quiet \
  _site/ \
  s3://www.mslinn.com
INFO: No cache file found, creating it.
INFO: Compiling list of local files...
INFO: Running stat() and reading/calculating MD5 values on 2157 files, this may take some time...
INFO: [1000/2157]
INFO: [2000/2157]
INFO: Retrieving list of remote files for s3://www.mslinn.com/_site ...
INFO: Found 2157 local files, 0 remote files
INFO: Verifying attributes...
INFO: Summary: 2137 local files to upload, 20 files to remote copy, 0 remote files to delete
... lots more output ...
Done. Uploaded 536630474 bytes in 153.6 seconds, 3.33 MB/s.

Originally, I did not provide the --acl-public option to s3cmd sync. That meant the files in the bucket were all private – the opposite of what is required for a website. Here is the incantation for making all the files in the bucket publicly readable:

Shell

$ s3cmd setacl s3://www.mslinn.com/ \
  --acl-public --recursive --quiet

Linode’s S3 implementation faithfully mirrors a problem that AWS S3 has. CSS files are automatically assigned the MIME type text/plain, instead of text/css.

Shell

$ curl -sI http://linode.mslinn.com/assets/css/style.css | \
  grep Content-Type
Content-Type: text/plain

I re-uploaded the CSS files with the proper mime type using this incantation:

Shell

$ cd _site

$ find . -name *.css -exec \
  s3cmd put -P --mime-type=text/css {} s3://www.mslinn.com/{} \;

It should be possible to provide a MIME-type mapping somehow! Linode, if you are reading this, please provide an extension to the S3 functionality, so customers could go beyond AWS's bugs.

The Result

Success! The same content can be fetched with 3 different URLs:

https://linode.mslinn.com uses the free wildcard certificate that was created above.
http://linode.mslinn.com.website-us-east-1.linodeobjects.com works without an SSL certificate.
https://linode.mslinn.com.website-us-east-1.linodeobjects.com uses a certificate from us-east-1.linodeobjects.com, which shows as an error, as it should. The error would be prevented if the certificate was a wildcard certificate issued by linodeobjects.com instead.

Qualys / SSL Labs rated the site on Linode Storage with the Let’s Encrypt SSL certificate: and gave it the same rating as when the site was hosted on AWS S3 / CloudFront.

BTW, you can test the date range of a certificate with this incantation:

Shell

$ curl https://linode.mslinn.com -vI --stderr - | grep 'date:'
*  start date: Nov 25 17:46:07 2022 GMT
*  expire date: Feb 23 17:46:06 2023 GMT

The site feels quite responsive. 😁

Considering Microsoft Azure for Static Websites

2022-06-20T00:00:00-04:00

After realizing that AWS does not integrate security with real-time billing, and being presented with a huge bill after my account was hijacked, I decided to see if Microsoft Azure offered me better financial security.

In particular, I am looking for two things:

A more secure authentication mechanism – If AWS credentials are compromised, they can be used from anywhere in the world.
Restricting scope and scale of authorized services – With AWS, if an IAM user has a role that allows EC2 instances to be launched, any number of any size EC2 instances can be launched. There is no way to cap the number or type of EC2 instances, and AWS real-time billing is not integrated with the authentication mechanism. Furthermore, AWS has a peculiar set of busy work tasks for victims of account hijacking that appears to be targeted more towards increasing support revenue than address root issues. This means that the bad guys have no limit to the financial damage they can incur on their victims.

For PaaS vendors such as AWS, Azure, Digital Ocean, Cloudflare, ScaleWay, etc: “pay-as-you-go” is shorthand for “there is nothing you can do to limit your financial liability”.

I set out to discover if Azure also suffers from these same problems.

Azure Roles and Policies

Policies

Azure policies can be used to define the desired behavior for your organization's Windows VMs and Linux VMs. By using policies, an organization can enforce various conventions and rules throughout the enterprise. Enforcement of the desired behavior can help mitigate risk while contributing to the success of the organization.

Azure role-based access control

Using Azure role-based access control (Azure RBAC), you can segregate duties within your team and grant only the amount of access to users on your VM that they need to perform their jobs. Instead of giving everybody unrestricted permissions on the VM, you can allow only certain actions. You can configure access control for the VM in the Azure portal, using the Azure CLI, or Azure PowerShell.

– From Azure Virtual Machines Documentation

Ekran Systems publishes a good description of various flavors of RBAC, and the more complex ABAC. Okta, Inc. also publishes a good overview of RBAC and ABAC. I have no connection with either company.

I soon discovered the following passage in the Azure documentation, which appeared to exactly match the second item in the wish list at the top of this article:

Azure RBAC focuses on managing user actions at different scopes. If control of an action is required, then Azure RBAC is the correct tool to use. Even if an individual has access to perform an action, if the result is a non-compliant resource, Azure Policy still blocks the create or update.

The combination of Azure RBAC and Azure Policy provides full scope control in Azure. – From Azure documentation

AWS RBAC does not provide functionality equivalent to that provided by Microsoft Azure RBAC plus Azure Policy, even when combined with other AWS functionality.

No Limits On Other Azure Services

The financial limitations that Azure allows you to impose are only available for virtual machines. Users are subject to unlimited financial liability from other Azure services. It seems I have ~~two~~three choices:

Figure out how to set up RBAC and accept that other services might run up a huge bill.
Look for a fixed-price hosting service
Give Azure Spending Limit a try

Before I sign off today, I encountered a blocking issue with Azure that I'd like to mention.

Active Directory?

I want to update my website by running a command-line command that runs something analogous to rsync. I spent quite some time looking for how this might be done with Azure.

Before long, it seemed that Active Directory was the best way forward, however setting it up is daunting. Apparently a service principal is better suited for scripts that run on-premises, like my home office. I got the impression that I need to register an app; unsure what app might be required for hosting a web site on Azure Blob Storage with Azure CDN.

These Active Directory docs look like a lot of abstract, generalized information that is way more complex than I need. I found a streamlined article for my use case. Not sure what this means: “Only storage accounts created with the Azure Resource Manager deployment model support Azure AD authorization.” This streamlined document is not very streamlined. For me, this issue is a significant barrier to adoption.

Update: Azure Spending Limit

I just bumped into an article entitled Azure spending limit. Perhaps this might be useful. I tried to find out more from Microsoft, but at first they said I would have to subscribe to an expensive support plan before anyone would speak to me. That is not how I wish to proceed. Later, I opened a ticket inquiring about Azure customers facing unlimited financial liability.

Update 2022-06-29

The ticket was escalated twice, then Microsoft sent me the following official response (emphasis is mine):

Correct, at this moment that feature of setting limits does not exist, Azure is not able to safeguard customers from unlimited financial liability.

We are a support team that we provide help with current account or billing issues, in addition to that, I would like to mention that Azure products are constantly being updated, also Azure is constantly improving to offer better services to our customers. For this reason, I would like to invite you to leave your feedback in the Azure feedback forum (General Feedback · Community (azure.com)), so your feedback is forwarded through the right channel and to the Microsoft internal resources that work on it.

I responded by indicating that I would close my Azure account.

While the practice of imposing unlimited financial liability on customers is common today for PaaS vendors, it is unnecessary and predatory. If in the future I need to use a product or service that incurs unlimited financial liability for a client, I will require the client to accept the liability.

Creating and Renewing Letsencrypt Wildcard SSL Certificates

2022-06-15T00:00:00-04:00

The process of making wildcard SSL certificates can be hard to get right at first. However, I use wildcard SSL certificates for most of my internet domains, so it is important to me. This blog post distills the essence of what I have found to be important when creating these certificates.

Update – 9 months after writing this blog post, I wrote about a better way to generate wildcard SSL certificates: Wildcard SSL Certificates for Letsencrypt with Lego.

Making a Free Wildcard SSL Certificate

Certbot powers EFF’s Letsencrypt capability. Source code is on GitHub.

Install certbot on Debian distros such as Ubuntu as follows:

Shell

$ sudo apt install certbot
Preparing to unpack .../06-python3-zope.event_4.4-3_all.deb ...
  Unpacking python3-zope.event (4.4-3) ...
  Selecting previously unselected package python3-zope.component.
  Preparing to unpack .../07-python3-zope.component_4.3.0-3_all.deb ...
  Unpacking python3-zope.component (4.3.0-3) ...
  Selecting previously unselected package python3-certbot.
  Preparing to unpack .../08-python3-certbot_1.21.0-1build1_all.deb ...
  Unpacking python3-certbot (1.21.0-1build1) ...
  Selecting previously unselected package python3-icu.
  Preparing to unpack .../09-python3-icu_2.8.1-0ubuntu2_amd64.deb ...
  Unpacking python3-icu (2.8.1-0ubuntu2) ...
  Selecting previously unselected package certbot.
  Preparing to unpack .../10-certbot_1.21.0-1build1_all.deb ...
  Unpacking certbot (1.21.0-1build1) ...
  Setting up python3-configargparse (1.5.3-1) ...
  Setting up python3-requests-toolbelt (0.9.1-1) ...
  Setting up python3-parsedatetime (2.6-2) ...
  Setting up python3-icu (2.8.1-0ubuntu2) ...
  Setting up python3-zope.event (4.4-3) ...
  Setting up python3-zope.hookable (5.1.0-1build1) ...
  Setting up python3-josepy (1.10.0-1) ...
  Setting up python3-zope.component (4.3.0-3) ...
  Setting up python3-acme (1.21.0-1) ...
  Setting up python3-certbot (1.21.0-1build1) ...
  Setting up certbot (1.21.0-1build1) ...
  Created symlink /etc/systemd/system/timers.target.wants/certbot.timer → /lib/systemd/system/certbot.timer.
  Processing triggers for man-db (2.10.2-1) ...
  Scanning processes...
  Scanning processor microcode...
  Scanning linux images...

  Failed to retrieve available kernel versions.

  Failed to check for processor microcode upgrades.

  No services need to be restarted.

  No containers need to be restarted.

  No user sessions are running outdated binaries.

  No VM guests are running outdated hypervisor (qemu) binaries on this host.

certbot has 2 subcommands of interest: certonly (used when creating a certificate for the first time), and renew (used when updating a pre-existing certificate). The creation process requires the user to do things before it can complete, so it can only be run interactively.

The files and directories creating by the process of creating a new SSL certificate should not be deleted. Contrary to what the Letsencrypt certbot documentation says, I have found that these files and directories allow the renewal process to proceed without requiring user interaction, so it can be scripted.

Certbot Help Messages

Here is the help message for the certbot certonly subcommand:

Shell

$ certbot certonly --help
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...

  Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
  it will attempt to use a webserver both for obtaining and installing the
  certificate. The most common SUBCOMMANDS and flags are:

  obtain, install, and renew certificates:
  (default) run   Obtain & install a certificate in your current webserver
  certonly        Obtain or renew a certificate, but do not install it
  renew           Renew all previously obtained certificates that are near
  expiry
  enhance         Add security enhancements to your existing configuration
  -d DOMAINS       Comma-separated list of domains to obtain a certificate for

  (the certbot apache plugin is not installed)
  --standalone      Run a standalone webserver for authentication
  (the certbot nginx plugin is not installed)
  --webroot         Place files in a server's webroot folder for authentication
  --manual          Obtain certificates interactively, or using shell script
  hooks

  -n               Run non-interactively
  --test-cert       Obtain a test certificate from a staging server
  --dry-run         Test "renew" or "certonly" without saving any certificates
  to disk

  manage certificates:
  certificates    Display information about certificates you have from Certbot
  revoke          Revoke a certificate (supply --cert-name or --cert-path)
  delete          Delete a certificate (supply --cert-name)

  manage your account:
  register        Create an ACME account
  unregister      Deactivate an ACME account
  update_account  Update an ACME account
  --agree-tos       Agree to the ACME server's Subscriber Agreement
  -m EMAIL         Email address for important account notifications

  More detailed help:

  -h, --help [TOPIC]    print this message, or detailed help on a topic;
  the available TOPICS are:

  all, automation, commands, paths, security, testing, or any of the
  subcommands or plugins (certonly, renew, install, register, nginx,
  apache, standalone, webroot, etc.)
  -h all                print a detailed help page including all topics
  --version             print the version number
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Here is the help message for the certbot certonly subcommand:

Shell

$ certbot renew --help
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...

  Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
  it will attempt to use a webserver both for obtaining and installing the
  certificate. The most common SUBCOMMANDS and flags are:

  obtain, install, and renew certificates:
  (default) run   Obtain & install a certificate in your current webserver
  certonly        Obtain or renew a certificate, but do not install it
  renew           Renew all previously obtained certificates that are near
  expiry
  enhance         Add security enhancements to your existing configuration
  -d DOMAINS       Comma-separated list of domains to obtain a certificate for

  (the certbot apache plugin is not installed)
  --standalone      Run a standalone webserver for authentication
  (the certbot nginx plugin is not installed)
  --webroot         Place files in a server's webroot folder for authentication
  --manual          Obtain certificates interactively, or using shell script
  hooks

  -n               Run non-interactively
  --test-cert       Obtain a test certificate from a staging server
  --dry-run         Test "renew" or "certonly" without saving any certificates
  to disk

  manage certificates:
  certificates    Display information about certificates you have from Certbot
  revoke          Revoke a certificate (supply --cert-name or --cert-path)
  delete          Delete a certificate (supply --cert-name)

  manage your account:
  register        Create an ACME account
  unregister      Deactivate an ACME account
  update_account  Update an ACME account
  show_account    Display account details
  --agree-tos       Agree to the ACME server's Subscriber Agreement
  -m EMAIL         Email address for important account notifications

  More detailed help:

  -h, --help [TOPIC]    print this message, or detailed help on a topic;
  the available TOPICS are:

  all, automation, commands, paths, security, testing, or any of the
  subcommands or plugins (certonly, renew, install, register, nginx,
  apache, standalone, webroot, etc.)
  -h all                print a detailed help page including all topics
  --version             print the version number
  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Complete documentation for certbot is here.

Generic DNS Authentication

I used certbot to create a special file within the website as follows, without using sudo. I want a wildcard SSL certificate, which requires certbot to use DNS authentication.

BTW, the following options include two -d options, one for the domain apex (mslinn.com) and one for the subdomains (*.mslinn.com)

Shell

$ certbot certonly \
  --manual \
  --agree-tos \
  --preferred-challenges dns-01 \
  --rsa-key-size 4096 \
  -d mslinn.com -d *.mslinn.com \
  --config-dir ~/.certbot/mslinn.com/config \
  --logs-dir ~/.certbot/mslinn.com/logs \
  --work-dir ~/.certbot/mslinn.com/work
Saving debug log to /home/mslinn/.certbot/mslinn.com/logs/letsencrypt.log
Enter email address (used for urgent renewal and security notices)
(Enter 'c' to cancel):  mslinn@mslinn.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: n
Account registered.
Requesting a certificate for mslinn.com and *.mslinn.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please deploy a DNS TXT record under the name:

_acme-challenge.mslinn.com.

with the following value:

n6o3qMw5N7qxAzV4uNQePKxJjaw0f-Bo32CybWr-lhE

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue
(This must be set up in addition to the previous challenges; do not remove,
replace, or undo the previous challenge tasks yet. Note that you might be
asked to create multiple distinct TXT records with the same name. This is
permitted by DNS standards.)

Before continuing, verify the TXT record has been deployed. Depending on the DNS
provider, this may take some time, from a few seconds to multiple minutes. You can
check if it has finished deploying with aid of online tools, such as the Google
Admin Toolbox: https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.mslinn.com.
Look for one or more bolded line(s) below the line ';ANSWER'. It should show the
value(s) you’ve just added.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue
Successfully received certificate.
Certificate is saved at: /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/fullchain.pem
Key is saved at:         /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/privkey.pem
This certificate expires on 2022-09-29.
These files will be updated when the certificate renews.

NEXT STEPS:
- This certificate will not be renewed automatically.
Autorenewal of --manual certificates requires the use of an authentication hook script
(--manual-auth-hook) but one was not provided.
To renew this certificate, repeat this same certbot command before the certificate’s expiry date.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let’s Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

I made a bash script (~/.local/bin/sslCert) to create SSL certificates, which performs the same steps as the above, but it works for any site, and can renew certificates as well as create them:

~/.local/bin/sslCert

#!/bin/bash

function help {
  echo "Creates or updates an wildcard SSL certificate

Usage: $(basename $0) DOMAIN

Example: $(basename $0) scalacourses.com

The relevant files are stored in ~/.certbot/DOMAIN/

Creating a new certificate requires an interactive user session;
updating a certificate can be done in a cron job.

Currently, the renew verb is capable of either renewing all installed
certificates that are due to be renewed or renewing a single certificate
specified by its name. If you would like to renew specific certificates
by their domains, use the certonly command instead.

The renew verb may provide other options for selecting certificates to
renew in the future.

Ask for help or search for solutions at https://community.letsencrypt.org
See the logfile /home/mslinn/.certbot/ancientwarmth.com/logs/letsencrypt.log
or re-run Certbot with -v for more details.


BTW, you can test the date range of the certificate with this incantation:
  curl https://domain.com -vI --stderr - | grep 'date:'
"
  exit 1
}

if [ -z "$1" ]; then help; fi

DOMAIN="$1"
if [ -d "$HOME/.certbot/$DOMAIN" ]; then # Renew existing certificate
  CMD=certonly # renew
  OPTIONS=--force-renew
else # Make new certificate (interactively)
  CMD=certonly
  unset OPTIONS
fi
certbot "$CMD" $OPTIONS \
  --agree-tos \
  --config-dir "$HOME/.certbot/$DOMAIN/config" \
  -d "$DOMAIN" -d "*.$DOMAIN" \
  --email mslinn@mslinn.com \
  --logs-dir "$HOME/.certbot/$DOMAIN/logs" \
  --manual \
  --preferred-challenges dns-01 \
  --rsa-key-size 4096 \
  --work-dir "$HOME/.certbot/$DOMAIN/work"

Here is the help message for the script:

Shell

$ sslCert
Creates or updates an wildcard SSL certificate

Usage: sslCert DOMAIN

Example: sslCert scalacourses.com

The relevant files are stored in ~/.certbot/DOMAIN/

Creating a new certificate requires an interactive user session;
updating a certificate can be done in a cron job.

The following crontab entry causes the script to run every 60 days.

crontab

0 0 1 */2 * .local/bin/sslCert mslinn.com

The next time the script was run, output looked like:

Shell

$ sslCert mslinn.com
Saving debug log to /home/mslinn/.certbot/mslinn.com/logs/letsencrypt.log
Renewing an existing certificate for mslinn.com and *.mslinn.com

Successfully received certificate.
Certificate is saved at: /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/fullchain.pem
Key is saved at:         /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/privkey.pem
This certificate expires on 2022-12-13.
These files will be updated when the certificate renews.

NEXT STEPS:
- This certificate will not be renewed automatically.
  Autorenewal of --manual certificates requires the use of an authentication hook script (--manual-auth-hook)
  but one was not provided. To renew this certificate, repeat this same certbot command before the
  certificate's expiry date.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
* Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
* Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

😁

Considering Cloudflare R2 for Static Websites

2022-06-09T00:00:00-04:00

After deciding to close my AWS account, I started looking for alternatives to host my static websites. Without getting into my selection criteria, my short list of possible hosting companies was Microsoft Azure, Cloudflare, Digital Ocean, Linode, and Netlify. Many other options exist.

Cloudflare has a free tier that has no time limit. It includes an S3 work-alike called R2, SSL and a built-in world-wide CDN that works automatically. 250 GB of storage and 1 TB/month of transfer are provided at no charge, forever. There are also no ingress or egress charges. Cloudflare’s edge network now spans 275 cities around the world, with nearly all Internet users within 50 milliseconds of a Cloudflare server.

This blog post documents my experience with Cloudflare. If you don't care about details, and want to know my verdict on Cloudflare R2, skip to the end.

For PaaS vendors such as AWS, Azure, Digital Ocean, Cloudflare, ScaleWay, etc.: “pay-as-you-go” is shorthand for “there is nothing you can do to limit your financial liability”.

This is what I did

After creating an account, I followed the directions at R2 get started guide.

Wrangler

The directions told me to set up Wrangler v2, a command-line interface for transferring files to R2.

Shell

$ npm install -g wrangler
npm WARN config global `--global`, `--local` are deprecated. Use `--location=global` instead.
npm WARN deprecated rollup-plugin-inject@3.0.2: This package has been deprecated and is no longer maintained. Please use @rollup/plugin-inject.

added 56 packages, and audited 57 packages in 8s

10 high severity vulnerabilities

To address issues that do not require attention, run:
npm audit fix

To address all issues, run:
npm audit fix --force

Run `npm audit` for details.

The above messages do not inspire confidence. Wrangler is written using Node. I believe that Node has more security issues than any other computer language, particularly in package management. The OWasp recommendations do not address the fundamental security vulnerabilities in the Node package management infrastructure.

RClone

I looked for alternatives to Wrangler and found RClone. RClone is a command-line program to manage files on cloud storage. It has many subcommands, including two types of sync. Although the RClone documentation does not mention Cloudflare, the Cloudflare docs described how to set up RClone.

Limits

Platform limits are important. Not only are the technical limits important for defining inputs and outputs, users should be particularly interested in spend limits, so they are not subject to unlimited financial liability.

Workers Paid plan is separate from any other Cloudflare plan (Free, Professional, Business) you may have. If you are an Enterprise customer, reach out to your account team to confirm pricing details.

Only requests that hit a Worker will count against your limits and your bill.

– From Cloudflare Workers Pricing Fine Print

Setup

Caching

Cache control.

Pages

> Cloudflare Pages is a JAMstack platform for frontend developers to collaborate and deploy websites.

– From pages.cloudflare.com

Cloudflare Pages supports Jekyll sites. However, it looks like Cloudflare Pages builds the site in the cloud. While this might be a useful mechanism for many, my Jekyll builds need to access my local machine, and use my Jekyll plugins.

Cloudflare Workers Sites suits my use case. The Start From Existing documentation looks appropriate, except it is written for using Wrangler, which I view as a security threat.

The Verdict: No to Cloudflare

CloudFlare does not offer a spend limit for accounts on paid plans. This is unacceptable. I tried to remove my credit card, but found I could not. I then deleted my user account, and saw:

It could take up to 12 months to delete your information completely.

There is no way to frame this as an example of how Cloudflare is looking out for the best interests of their customers.

Upgrading PostgreSQL Ubuntu

2022-05-28T00:00:00-04:00

This blog post describes my recommended steps to follow when upgrading a Postgres database forced upon you when upgrading Ubuntu, and the new OS includes a new release of Postgres.

I use PostgreSQL as the database for ScalaCourses.com. The site is served from an Ubuntu instance, and there is a backup instance.

When upgrading to Ubuntu 22.04, PostgreSQL upgraded from v13 to v14; I used pg_upgrade, which required temporarily swapping the ports that Postgres used.

However, I used pg_upgradecluster when upgrading to Ubuntu 23.04; and PostgreSQL upgraded from v14 to v15. Unfortunately, the documented procedure to use pg_upgradecluster has no consideration for systemd, and this makes the upgrade slightly more awkward than it might otherwise be.

While both approaches have issues, I found that it is easier and faster to upgrade a Postgres installation using pg_upgradecluster than using pg_upgrade.

Upgrade Backup System First

One of the benefits of a backup system is that you can work through upgrade problems on the backup, before doing the same with the live instance.

This article suggests to upgrade the entire system to Ubuntu 23.04, then upgrade the active Postgres database cluster. The alternative would be to add a new PPA for Postgres, and just update that program before upgrading the entire system; this merely substitutes one set of potential issues for another. The article also demonstrates renaming the extra database cluster that the Postgres upgrade creates, but that just creates cruft since it is not required for anything.

I decided to delete the new database cluster instead, since upgrading the old database cluster is all that is required. As you will read in the Upgrade Procedure section, this is also the advice given during the upgrade procedure.

I decided to first upgrade the Ubuntu OS, then migrate the Postgres database cluster.

Upgrade Removed NVidia Support

After running do-release upgrade on the production system, the system worked fine, except the console had no graphics. I fixed it by automatically installing all video drivers:

Shell

$ sudo ubuntu-drivers autoinstall

Examining the Postgres Databases

pg_lsclusters displays the available PostgreSQL database clusters. Prior to upgrading to Postgres 15, I had never deleted the old versions. They had quietly accumulated over the 11 years that I had been running ScalaCourses:

Shell

$ pg_lsclusters
Ver Cluster Port Status Owner    Data directory               Log file
9.5 main    5431 online postgres /var/lib/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log
9.6 main    5433 online postgres /var/lib/postgresql/9.6/main /var/log/postgresql/postgresql-9.6-main.log
10  main    5434 online postgres /var/lib/postgresql/10/main  /var/log/postgresql/postgresql-10-main.log
12  main    5432 online postgres /var/lib/postgresql/12/main  /var/log/postgresql/postgresql-12-main.log
13  main    5435 online postgres /var/lib/postgresql/13/main  /var/log/postgresql/postgresql-13-main.log
14  main    5436 online postgres /var/lib/postgresql/14/main  /var/log/postgresql/postgresql-14-main.log

Listing the databases in the active cluster showed an unnecessary database:

Shell

$ sudo -iu postgres psql -l
List of databases
     Name     |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
--------------+----------+----------+-------------+-------------+-----------------------
 bix_dev      | postgres | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 |
 postgres     | postgres | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 |
 scalacourses | postgres | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 |
 template0    | postgres | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 | =c/postgres          +
              |          |          |             |             | postgres=CTc/postgres
 template1    | postgres | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 | =c/postgres          +
              |          |          |             |             | postgres=CTc/postgres
(5 rows)

I do not care about the bix_dev database, so I decided to delete it. The database that I care about is called scalacourses.

Shell

$ sudo -iu postgres psql -c 'drop database bix_dev;'
psql (14.7 (Ubuntu 14.7-0ubuntu0.22.10.1))
Type "help" for help.

DROP DATABASE

Upgrade Procedure

Ubuntu is normally upgraded by running do-release-upgrade. Near the end of that process, the following message appeared.

At this point postgresql-15 and postgresql-client-15 had just been installed. The message did not reflect the true state of the system, which caused me some consternation for a moment.

The PostgreSQL version 14 is obsolete, but the server or client packages are still installed. Please install the latest packages (postgresql-15 and postgresql-client-15) and upgrade the existing clusters with pg_upgradecluster (see manpage).

Please be aware that the installation of postgresql-15 will automatically create a default cluster 15/main. If you want to upgrade the 14/main cluster, you need to remove the already existing 15 cluster (pg_dropcluster --stop 15 main, see manpage for details).

The old server and client packages are no longer supported. After the existing clusters are upgraded, the postgresql-14 and postgresql-client-14 packages should be removed.

Please see /usr/share/doc/postgresql-common/README.Debian.gz for details.

Following is /usr/share/doc/postgresql-common/README.Debian.gz, which was referenced by the above message. Most of that file is uninteresting, however two sections are important, and we will look at them.

/usr/share/doc/postgresql-common/README.Debian.gz

PostgreSQL for Debian
=====================

PostgreSQL is a fully featured object-relational database management system. It
supports a large part of the SQL standard and is designed to be extensible by
users in many aspects. Its features include ACID transactions, foreign keys,
views, sequences, subqueries, triggers, outer joins, multiversion concurrency
control, and user-defined types and functions.

Since the on-disk data format of all major PostgreSQL versions (like 9.6,
11, etc.) is incompatible to each other, Debian's PostgreSQL packaging
architecture is designed to maintain clusters of different major versions in
parallel.

This postgresql-common package provides the common infrastructure and all
frontend programs that users and administrators use. The version specific
server and client programs are shipped in postgresql-*-<version> packages.

For a detailed description of the architecture, please see

/usr/share/doc/postgresql-common/README.md.gz

First steps for the impatient
-----------------------------
Eventually you will not get around reading at least some parts of the manual,
but if you want to get straight into playing SQL, here are the steps to create
a database user and a database for the Unix user 'joe':

1. Install a database server with the major version of your choice
('postgresql-XY', e. g. 'postgresql-11').  Preferably the latest
version, which you can get by installing the metapackage
'postgresql'. This will automatically create a default cluster
'main' with the database superuser 'postgres'.

2. Get a shell for the database superuser 'postgres'. If your system
has an active root user, use su:

# su -s /bin/bash postgres

If your system uses sudo to get administrative rights, use sudo instead:

joe$ sudo -u postgres bash

3. In this postgres shell, create a database user with the same name as your
Unix login:

$ createuser -DRS joe

For details about the options, see createuser(1).

4. Create a database "joework" which is owned by "joe":

$ createdb -O joe joework

For details about the options, see createdb(1).

5. Exit the postgres shell.

6. As user joe, you should now be able to connect to your database with

$ psql joework

Cluster management
------------------
For managing clusters, the following commands are provided (each with its own
manual page):

pg_createcluster  - Create a new cluster or integrate an existing one into
the postgresql-common architecture.
pg_dropcluster    - Completely remove a cluster.
pg_ctlcluster     - Control the server process of a cluster (start, stop,
restart).
pg_lsclusters     - Show a list of all existing clusters and their status.
pg_upgradecluster - Migrate a cluster from one major version to another one.
pg_renamecluster  - Rename a cluster.

Please note that you can of course also use the upstream tools for
creating clusters, such as initdb(1). However, please note that in
this case you cannot expect *any* of above pg_* tools to work, since
they use different configuration settings (SSL, data directories,
etc.) and file locations (e. g.
/etc/postgresql/11/main/postgresql.conf). If in doubt, then do *not*
use initdb, but only pg_createcluster. Since merely installing
postgresql-NN will already set up a default cluster which is ready to
work, most people do not need to bother about initdb or
pg_createcluster at all.

Port assignment
---------------
Please note that the pg_* tools automatically manage the server ports
unless you specify them manually. The first cluster which is ever
created (by any major version) will run on the default port 5432, and
each new cluster will use the next higher free one.

E. g. if you first install "postgresql-11" on a clean system, the
default 11/main cluster will run on port 5432. If you then create
another 11 cluster, or install the "postgresql-12" package, that new
one will run on 5433.

Please use "pg_lsclusters" for displaying the cluster <-> port
mapping, and please have a look at the pg_createcluster manpage (the
--port option) for details.

Default clusters and upgrading
------------------------------
When installing a postgresql-NN package from scratch, a default
cluster 'main' will automatically be created. This operation is
equivalent to doing 'pg_createcluster NN main --start'.

Due to this default cluster, an immediate attempt to upgrade an
earlier 'main' cluster to a new version will fail and you need to
remove the newer default cluster first. E. g., if you have
postgresql-9.6 installed and want to upgrade to 11, you first install
postgresql-11:

apt-get install postgresql-11

Then drop the default 11 cluster that was just created:

pg_dropcluster 11 main --stop

And then upgrade the 9.6 cluster to the latest installed version (e. g. 11):

pg_upgradecluster 9.6 main

SSL
---
The PostgreSQL server packages support SSL, which provides encrypted and
authenticated network communication. SSL should be used if you have an
untrusted network between a database server and a client and these exchange
security sensitive data like passwords or confidential database contents.

When a cluster is created with pg_createcluster, SSL support will automatically
be enabled. postgresql-common makes use of the 'snakeoil' SSL certificate that
is generated by the ssl-cert package, so that SSL works out of the box
(ssl_cert_file, ssl_key_file). In addition, if /etc/postgresql-common/root.crt
exists, it will be used as CA certificate file (ssl_ca_file).

/etc/postgresql-common/root.crt is a dummy file by default, so that
client-side authentication is not performed. To enable it, you should
add some root certificates to it. A reasonable choice is to just
symlink the file to /etc/ssl/certs/ssl-cert-snakeoil.pem; in this
case, client certificates need to be signed by the snakeoil
certificate, which might be desirable in many cases. See

/usr/share/doc/postgresql-doc-11/html/ssl-tcp.html

for details (in package postgresql-doc).

Further documentation
---------------------
All commands shipped by postgresql-common have detailed manpages. See
postgresql-common(7) for the documentation of the database client program
wrapping, and user_clusters(5) and postgresqlrc(5) for the cluster
configuration.

The documentation of the database server and client functions, SQL commands,
modules, etc.  documented is shipped in the per-version packages
postgresql-doc-<version>.

The Cluster management section in the above file provides some background information for our purposes, so I repeat it here:

Cluster management

For managing clusters, the following commands are provided (each with
its own manual page):

pg_createcluster  - Create a new cluster or integrate an existing one
                    into the postgresql-common architecture.
pg_dropcluster    - Completely remove a cluster.
pg_ctlcluster     - Control the server process of a cluster
                    (start, stop, restart).
pg_lsclusters     - Show a list of all existing clusters and their status.
pg_upgradecluster - Migrate a cluster from one major version to another one.
pg_renamecluster  - Rename a cluster.

Please note that you can of course also use the upstream tools for
creating clusters, such as initdb(1). However, please note that in
this case you cannot expect *any* of above pg_* tools to work, since
they use different configuration settings (SSL, data directories,
etc.) and file locations (e. g.
/etc/postgresql/11/main/postgresql.conf). If in doubt, then do *not*
use initdb, but only pg_createcluster. Since merely installing
postgresql-NN will already set up a default cluster which is ready to
work, most people do not need to bother about initdb or
pg_createcluster at all.

The Default clusters and upgrading section is more important, however it contained errors. I fixed the errors in the following instructions.

Default clusters and upgrading

When installing a postgresql-NN package from scratch, a default
cluster 'main' will automatically be created. This operation is
equivalent to doing 'pg_createcluster NN main --start'.

Due to this default cluster, an immediate attempt to upgrade an
earlier 'main' cluster to a new version will fail and you need to
remove the newer default cluster first. E. g., if you have
postgresql-9.6 installed and want to upgrade to 11, you first install
postgresql-11:

sudo apt install postgresql-11

Then drop the default 11 cluster that was just created:

sudo -iu postgres pg_dropcluster 11 main --stop

And then upgrade the 9.6 cluster to the latest installed version (e. g. 11):

sudo -iu postgres pg_upgradecluster 9.6 main

Systemd can significantly affect the upgrade process, and unfortunately it is completely ignored by the above instructions. I muddled through, as you will see.

Transcript

The following is a transcript of what I actually did:

Shell

$ sudo -iu postgres pg_dropcluster --stop 15 main
Warning: stopping the cluster using pg_ctlcluster will mark the systemd
unit as failed. Consider using systemctl:
  sudo systemctl stop postgresql@15-main
Warning: systemd was not informed about the removed cluster yet. Operations like
"service postgresql start" might fail. To fix, run:
  sudo systemctl daemon-reload 

$ sudo -iu postgres pg_upgradecluster 14 main
Stopping old cluster...
Warning: stopping the cluster using pg_ctlcluster will mark the systemd unit as failed.
Consider using systemctl:
  sudo systemctl stop postgresql@14-main
Restarting old cluster with restricted connections...
Notice: extra pg_ctl/postgres options given, bypassing systemctl for start operation
Creating new PostgreSQL cluster 15/main ...
/usr/lib/postgresql/15/bin/initdb -D /var/lib/postgresql/15/main --auth-local peer --auth-host scram-sha-256 --no-instructions --encoding UTF8 --lc-collate en_CA.UTF-8 --lc-ctype en_CA.UTF-8
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_CA.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/15/main ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... America/Toronto
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Warning: systemd does not know about the new cluster yet. Operations like
"service postgresql start" will not handle it. To fix, run:
  sudo systemctl daemon-reload

Copying old configuration files...
Copying old start.conf...
Copying old pg_ctl.conf...
Starting new cluster...
Notice: extra pg_ctl/postgres options given, bypassing systemctl for start operation
Roles, databases, schemas, ACLs...
 set_config
------------

(1 row)

set_config
------------

(1 row)

set_config
------------

(1 row)

set_config
------------

(1 row)

Fixing hardcoded library paths for stored procedures...
Upgrading database template1...
Analyzing database template1...
Fixing hardcoded library paths for stored procedures...
Upgrading database postgres...
Analyzing database postgres...
Fixing hardcoded library paths for stored procedures...
Upgrading database scalacourses...
Analyzing database scalacourses...
Stopping target cluster...
Stopping old cluster...
Disabling automatic startup of old cluster...
Starting upgraded cluster on port 5432...
Warning: the cluster will not be running as a systemd service.
Consider using systemctl:
  sudo systemctl start postgresql@15-main

Success. Please check that the upgraded cluster works. If it does,
you can remove the old cluster with
     pg_dropcluster 14 main

Ver Cluster Port Status Owner    Data directory              Log file
14  main    5433 down   postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log
Ver Cluster Port Status Owner    Data directory              Log file
15  main    5432 online postgres /var/lib/postgresql/15/main /var/log/postgresql/postgresql-15-main.log

When I attempted to following the above instructions for systemd, errors appeared:

Shell

$ sudo systemctl daemon-reload

$ sudo systemctl start postgresql@15-main
Job for postgresql@15-main.service failed because the service did not take the steps required by its unit configuration.
See "systemctl status postgresql@15-main.service" and "journalctl -xeu postgresql@15-main.service" for details.

Great, an obscure failure message. Others have fussed with this problem. I expected that rebooting would be much faster than trying to fix it, and that did prove to be the case:

Shell

$ sudo reboot

After rebooting, the upgraded Postgres cluster worked fine:

Shell

$ pg_lsclusters
Ver Cluster Port Status Owner    Data directory              Log file
14  main    5433 down   postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log
15  main    5432 online postgres /var/lib/postgresql/15/main /var/log/postgresql/postgresql-15-main.log

$ sudo -iu postgres psql -l
List of databases
    Name     |    Owner     | Encoding |   Collate   |    Ctype    | ICU Locale | Locale Provider |   Access privileges
--------------+--------------+----------+-------------+-------------+------------+-----------------+-----------------------
postgres     | postgres     | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 |            | libc            |
scalacourses | scalacourses | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 |            | libc            |
template0    | postgres     | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 |            | libc            | =c/postgres          +
              |              |          |             |             |            |                 | postgres=CTc/postgres
template1    | postgres     | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 |            | libc            | =c/postgres          +
              |              |          |             |             |            |                 | postgres=CTc/postgres
 (4 rows)

Here is how I deleted the older database clusters:

Shell

$ for X in 9.5 9.6 10 12 13; do
  sudo -iu postgres pg_dropcluster --stop $X main
end

😁

All done!

Montréal International vs. Bill 96

2022-05-27T00:00:00-04:00

Last night I attended the 25th Annual General Meeting of Montréal International, an energetic organization dedicated to attracting foreign business to establish a presence in the greater Montréal region. My key takeaways were:

I spoke with about 20 Montréal International employees; all of them were bilingual or trilingual, intelligent, motivated and well-informed. Their talent and commitment is palpable. Clearly, the hiring process is successful.
The event was marred by long-winded, self-congratulatory speeches by senior management, and former senior managers who were invited to a panel discussion. This lack of discipline caused the event to significantly run overtime.
The proceedings were conducted entirely in French. For an organization that calls itself “Montréal International”, this is unforgivable.
I asked every employee I spoke with if Quebec’s new Bill 96 impacted their goals. All of them expressed grave concern, and said that their mandate to attract foreign business had become extremely difficult as a result. However, not one mention was made by anyone on stage about Bill 96.

Montréal International’s mandate is severely impacted by Quebec's Bill 96. If any organization has an intrinsic mandate to demonstrate public leadership towards repealing or modifying Bill 96 so it respects basic human rights, it would be Montréal International. Instead, Montréal International’s leadership, which consists entirely of old white French-speaking alpha males, is complicit in their silence, and by their actions.

Limit Your Financial Vulnerability From AWS Account Hijacking

2022-05-26T00:00:00-04:00

I am a free agent, independent and critical. You can buy my time as a software expert, but not my opinion. Just because Amazon / AWS hired me in the past to help them against a patent troll does not mean they own me. Quite the contrary.

In the time you take a quick shower, your AWS account can be highjacked and tens of thousands of dollars in fees can be incurred. EC2 is the service that provides the greatest financial risk. You can take steps to limit your liability, and this blog post shows you how.

Out-of-Control Cloud Costs

Out-of-control costs are causing cloud customers to reduce or eliminate their cloud use. The cloud backlash has begun: Why big data is pulling compute back on premises – Thomas Robinson, published by TechCrunch 2023-03-20.

Why we’re leaving the cloud, by David Heinemeier Hansson, creator of Ruby on Rails and CTO of 37signals. Mr. Hansson also wrote We stand to save $7m over five years from our cloud exit.

AWS IAM Users and Roles Have No Budget Limitations

AWS customers are exposed to unlimited financial liability

I encountered two main issues when attempting to secure against AWS account hijacking:

AWS does not provide a mechanism to guard against launching expensive services. For example, if an IAM user has a role that allows EC2 instances to be launched, then that user can launch an unlimited number of EC2 instances of any size.

The most expensive Amazon EC2 is currently the p4de.24xlarge, which costs $40.96 USD per hour on-demand in the US East (N. Virginia) region. The instance comes with 96 vCPUs, 1152 GiB of RAM and eight 1TB NVMe SSDs, and has eight NVIDIA A100 Tensor Core GPUs.

Using stolen credentials that allow EC2 instances to be launched, an attacker using scripts could spin up an armada of p4de.24xlarge instances and incur eye-popping costs in minutes.
AWS Budgets provides a convenient way to shut down pre-designated services when a total budgetary amount is exceeded. However, the AWS AMI security model is not integrated into real-time cost monitoring. Thus, there is no way to automatically shut down newly launched service instances that exceed a pre-authorized budgetary amount.

Shared Security Responsibility

The AWS Customer Agreement and Shared Responsibility Model describe how AWS shares responsibility for security with users. If you follow the AWS security recommendations, described next, you will not be held accountable for extra charges if your AWS account is hijacked.

The AWS Security Blog published a nice overview, entitled Getting Started: Follow Security Best Practices as You Configure Your AWS Resources. Some of the same information is also provided in Security best practices in IAM.

Even though the AWS instructions do not prevent bad guys taking over your account, follow them anyway so you won't be held liable for the costs incurred if your AWS account is hijacked.

AWS Security Recommendations

Security needs to have depth. Any single measure can fail, and given enough scale, all measures will eventually fail. Layer your security measures so that one failure, no matter how grave, will not be fatal.

AWS recommends the following. I have highlighted what AWS personnel have described to me as being the quickest and easiest path; this blog post walks through those steps to the extent that I have been able.

Set up at least two of the following services to monitor cost and usage:
2. .
3. CloudTrail User Guide.
4. Web Application Firewall (WAF).
5. Trusted Advisor. I found that Trusted Advisor was somewhat easier to use than CloudWatch.
For more information about managing your AWS cost and usage, see the AWS Cost Management User Guide.
Set up at least one of the following security best practices:
1. .
2. AWS Security Hub.
3. Amazon GuardDuty.

Do The Easiest Things First

I usually prefer to do the easiest things first. Not everyone agrees. Doing even one highlighted item above provides a significant security improvement, so why wait?

The best is the enemy of the good.

– Voltaire

Better a diamond with a flaw than a pebble without.

– Confucius

Striving to better, oft we mar what’s well.

– Shakespeare

Root Credentials

If a bad guy has your root credentials, they can change budget limits. For greatest security, delete your root credentials by selecting Access Keys (access key ID and secret access key) as shown in the image below. However, if you do so, many things become impossible.

Enabling MFA

Yubikey NFC

The easiest highlighted item above is to enable multifactor authentication (MFA).

I use a virtual MFA device, enabled for my root account Google Authenticator for Android and iOS. Google Authenticator features a RFC 6238 standards-based TOTP (time-based one-time password) algorithm.

I use a YubiKey to provide MFA for the AMI user ID that I use to log into the AWS console for everyday work.

Only use root credentials when absolutely necessary.

Trusted Advisor

After adding MFA to root credentials, I found that using AWS Trusted Advisor was the next easiest thing to try to make my AWS account secure.

Once you visit the Trusted Advisor Console, and agree to enable Trusted Advisor, IAM permissions are added to the AWS IAM user that you are logged in as.

Usage seems straightforward; however, I found the information for securing S3 buckets puzzling at first:

The column labeled ACL Allows List means that buckets flagged with Yes have an ACL that allows objects to be listed. AWS recommends that S3 buckets not allow objects to be listed by the public. To correct this:

Click on the bucket name.
Click on the Permissions tab in the page that opens next.
Scroll down to Access Control List.
Click on the Edit button.
Disable the items marked in red, as shown below.

Referring back to the previous screenshot, the Policy Allows Access column flags all S3 buckets used to serve webpages as insecure. To serve HTML pages and assets stored in an S3 bucket, the permissions must be set to allow objects to be read by everyone. Trusted Advisor flags these buckets as insecure, which does not make sense to me.

Perhaps there is a better security policy for when webpages are served via CloudFront, but as is often the case with AWS documentation, it is difficult to piece together how to optimally configure S3 permissions with CloudFront options.

Best Practices Evolve With Technology

The AWS documentation is written as if the products described have always existed in their present state. In the course of researching this article, I discovered that AWS provides revision histories for their services. For example, the revision history of S3 is documented in the AWS S3 User Guide, and the CloudFront revision history is documented in the AWS CloudFront Developer Guide. Revision histories are useful for experienced technologists to review periodically, so they can keep up with best practices as they evolve.

I started using AWS S3 to serve websites in 2013, and at that time, CloudFront did not exist yet. As the two services matured, best practices, which were at first unknown, evolved and interactions between them became more complex. In particular, CloudFront http/https promotion, how origins are specified, and S3 permissions, all interact in ways that were not well documented for several years. This was a source of security problems and unwanted downtime for me.

I have 15 buckets that are used to serve web pages. On my AWS Support Center console, Trusted Advisor shows that “You have a yellow check affecting 15 of 24 resources”; however, the Trusted Advisor console displays those items as red triangles with exclamation marks.

This is confusing, and became frustrating when an AWS security support technician suggested I disregard anything that seemed annoying. For an IT/security person, that advice is the quickest way to hell that I know.

Rough Spots That Need Love

While researching this article I found:

Under Origin Settings, for Origin Domain Name, choose the Amazon S3 bucket that you created previously. For S3 bucket access, select Yes, use OAI (bucket can restrict access to only CloudFront). For the Origin access identity, you can choose from the list, or choose Create new OAI (both will work). For Bucket policy, select Yes, update the bucket policy.

– From Use an Amazon CloudFront distribution to serve a static website

However, I found conflicting information near the top of this article:

Important

If you use an Amazon S3 bucket configured as a website endpoint, you must set it up with CloudFront as a custom origin. You can’t use the origin access identity feature described in this topic. However, you can restrict access to content on a custom origin by setting up custom headers and configuring your origin to require them. For more information, see Restricting access to files on custom origins.

– From Restricting access to Amazon S3 content by using an origin access identity (OAI)

As if this is not confusing enough, Using various origins with CloudFront distributions states that in order for Amazon S3 redirects to work, a non-standard format for the origin must be used:

http://bucket-name.s3-website-region.amazonaws.com

As I discussed in a previous blog post, AWS S3 buckets support two types of redirects, but there is no indication of which type might or might not work with the non-standard origin format.

Also, there is no mention of whether https would work or not, or if these disparate instructions even work together. As usual, there is lots of uncertainty about how things actually work on AWS, and for how long before things change.

As per standard operating procedure with AWS, one would have to muck about extensively to determine which of the conflicting statements above is generally prevalent, and under what circumstances. I expect that one article is newer than the other, and it likely supercedes the older article somewhat to some degree under as-yet unknown circumstances. If I get lucky, the Trusted Advisor warnings will go away. Trusted Advisor should reference whichever article is currently correct.

Update 2022-05-31

I received the following email from Trusted Advisor. Unfortunately, the contents of the email were different from what I found on the Trusted Advisor console. Perhaps this is because a higher-level, and more expensive, service contract would be required to see those messages. If so, then this is not the way to gain happy customers. The cost of the higher service level would be 4 times more than my total AWS costs at present, clearly not something that I would want to do.

Dear AWS customer:

AWS Trusted Advisor currently shows alerts for 3 checks (1 red and 2 yellow) and $0 of potential monthly savings based on your usage.

Here is a summary of status changes for this week:

The alert severity of these checks has improved:

From red to yellow:
   Amazon S3 Bucket Permissions

From red to green:
   VPC
   Security Groups - Specific Ports Unrestricted

From yellow to green:
   VPC Internet Gateways

The alert status of 5 checks has not changed. For details and recommendations, visit AWS Trusted Advisor.

Sign in to the Trusted Advisor console to view your check results.

Best,

AWS Support
Was this report helpful? Send us your feedback.

To add insult to injury, the final "Send us your feedback" just links to a generic form for AWS sales, where one can pound sand.

There is abundant evidence to demonstrate that whereas amazon.com is consumer-centric, AWS marches to a very different drummer. Perhaps small businesses are not presently viewed by AWS as a significant growth opportunity. My personal experience is that the current risk/value proposition provided by AWS is no longer attractive for that demographic.

Update 2022-06-03

The credit card I used to pay for my AWS usage had a very high limit. In retrospect that was unwise.

AWS took weeks to agree to refund the charge on my credit card, and then they said it would take another week for the reversed charges to appear. This meant that if I only paid for the other charges on that card, the month-to-month credit card interest on the fraudulent AWS charges would have been problematic.

Just before the credit card billing period ended, I called Visa and asked what my options were. Apparently I am not the first person who had this problem with AWS. The answer was immediate: report the charges as fraud. OK, that made sense. This meant my card was immediately cancelled, the charges were forgiven by Visa, and a new card with a new number would arrive in about a week.

Managing Costs With AWS Budgets

Short-lived hijacking is the most serious financial threat, and AWS Budgets was seemingly not designed to guard against that. Ticking off this item will enable AWS to absolve you of financial responsibility for being hacked, but you should not expect that it will do much to slow down an attacker.

AWS Budgets are easy to set up in a passive manner, but they are clumsy to work with and not effective for preventing new services from being launched that would exceed the budget. The cost limiting functionality is not automatic, it must be set up manually. AWS documentation does not provide detailed how-to instructions for common scenarios. Tens of thousands of words of documentation must be digested before the full capabilities can be harnessed.

Another significant issue is that only three types of actions are supported:

IAM Policy – I am unclear on how to set this up so the bad guys cannot launch new services, and I am not certain this is possible.
Service Control Policy – Hopefully this could prevent new services from being launched that would exceed the budget; however, my head exploded when I researched this.

Attempting to use this option to prevent new, expensive services from being launched would seem to introduce lots of complexity. Perhaps AWS did not consider this use case, or deemed it unimportant.
Automate Instances to Stop for EC2 or RDS – If you run a t3.nano instance, for example, bad guys can only incur a certain amount of financial damage. Instead of shutting down the EC2 or RDS instances that I want to run, my main concern is to prevent bad guys from launching expensive new services. This option is therefore unsuitable for my use case.

If AWS Budgets provides an easy way of preventing new services to be launched that would exceed the budget, I could not find it after several hours of reading.

Passively Monitoring Budget Spend

Simply follow the directions and accept the defaults. Daily budgets do not support enabling forecasted alerts, or daily budget planning, so select Monthly Budgets. There is no need to set up SNS alerts or AWS Chatbot alerts, email notification works just as well for most users.

Attaching Actions

Following are my notes from my attempt to enable cost limiting. I was unable to attach an action that prevented expensive new services from being started, and it is unclear if resolving the problem will, in fact, provide the desired benefit. Perhaps someone reading these notes will be able to tell me about an approach that might work for a reasonable amount of effort.

Because AWS will absolve you of financial liability if you create a Budget without any actions, don't stress about the pointless exercise involved in making a Budget that cannot protect you. Just do it and give AWS more time to figure out a better way of preventing account hijacking.

You can stop reading this article now, unless the details interest you.

For PaaS vendors such as AWS, Azure, Digital Ocean, Cloudflare, ScaleWay, etc: “pay-as-you-go” is shorthand for “there is nothing you can do to limit your financial liability”.

When you get to the step labeled Attach actions - Optional, choose Add Action.
You must choose between using an existing IAM role for AWS Budgets to use, or you can create a new IAM role expressly for this purpose. IAM roles are free. I like the idea of a dedicated IAM role, so I clicked on manually create an IAM role.
A new browser tab opened, and I clicked on the blue Create role button.
Now a confusing page appeared:
I selected the default Trusted entity type, AWS Service, then from the pulldown menu labeled Use cases for other AWS services: I selected Budgets.
A new radio button appeared underneath the pull-down, also labeled Budgets, and I clicked on that. This does not win any usability awards.
I clicked on the button labeled Next.
Managed policy name: AWSBudgetsActionsRolePolicyForResourceAdministrationWithSSM

This managed policy is focused on specific actions that AWS Budgets takes on your behalf when completing a specific action. This policy gives AWS Budgets broad permission to control AWS resources. For example, starts and stops Amazon EC2 or Amazon RDS instances by running AWS Systems Manager (SSM) scripts.
– From Using identity-based policies (IAM policies) for AWS Cost Management

On the page, I applied AWSBudgetsActionsRolePolicyForResourceAdministrationWithSSM, then I clicked on Next.
On the final page, I named the IAM role Budget and clicked on Create role.
Back in the Billing Management Console, I refreshed the list of available IAM roles, then selected the new role called Budget.
I selected Service Control Policy from the pull-down that appeared next, labeled Which action type should be applied when the budget threshold has been exceeded?. I have no idea how to proceed, or even if going in this direction might provide the desired results.

Microsoft Clarity Lets Me Watch You Click and Scroll

2022-03-31T00:00:00-04:00

I spent a fair bit of time designing the plugins for this Jekyll-powered website for SEO. These plugins are published as open source; click on the word Jekyll in the sentence above this paragraph to learn more. You are welcome!

I do not use Google Analytics because it slows down page load time dramatically. I do use Google Search Console, however, and recently started trying out Microsoft Bing Webmaster Tools.

Microsoft Clarity and Hotjar

The day before April Fool's Day in 2022, I stumbled across Microsoft Clarity, a free product that I knew nothing about. I was curious to know what benefit it might provide.

I learned that one of the things it can do is provide videos of actual user sessions as they interact with the website. Those videos are fantastic.

As a writer of a lot of online content, I have spent hours watching you, dear readers. Individually yet anonymously. Watching people read teaches you a lot about what is important to them. As a writer, knowing your audience allows you to be more relevant. BTW, I do not know who or where my readers are, just the country they are in.

Update 2023-07-10

I no longer need to watch the videos.
Microsoft Clarity's AI generates summaries.
Incredibly actionable information.

Today I noticed the insights tab, visible when viewing an entry for a ‘hit’. Seems like Microsoft has employed AI to generate the following impressive summary of the user's behavior. I added punctuation, a light edit and a whimsical pad of notepaper.

Visited URL matches regex: ^https://www\.mslinn\.com/softwareexpert/index\.html(\?.*)?$
Last 7 days

Session insights
Some key takeaways from this session are:

The user was interested in the software expert witness and computer expert services offered by Mike Slinn, as they visited the homepage from Google and spent about four minutes there.

The user was also curious about the technology expert article series, especially the ones related to Git and version control systems. They visited the article index page multiple times and clicked on several articles, spending about 15 minutes in total on this topic.

The user was most engaged with the article on libgit2, a library that provides low-level access to Git operations. They spent about one and a half minutes on this article and clicked on a link to git-fame, a tool that shows the contribution statistics of a Git repository.

The user frequently resized their browser window, which may indicate that they were using a mobile device or adjusting their screen for better readability. The user also switched between tabs or applications often, as indicated by the page hidden and page visible events. This may suggest that they were multitasking or comparing information from different sources.

01:47 / 47:31

Update 2023-07-18

Today I stumbled across the Clarity Live plugin for the Google Chrome web browser. There is no such plugin for Firefox, sadly.

View instant heatmaps right on your live site and watch recent session recordings for any page you are on with our extension.

GDPR & CCPA ready
No sampling
Built on open source

– From Microsoft Clarity

Check out this video!

This is one of the first user sessions that Microsoft Clarity recorded for this website.

As you can see, Microsoft Clarity lets me watch movies of users clicking and scrolling through my website; spooky yet very informative.

Clarity is open source. Here is the GitHub project.

I have since learned that Hotjar is similar to Microsoft Clarity.

Online Behavior Matches Real-World Behavior

The user in the above video read a bit about my experience as a software expert witness, then straightaway downloaded my resume. They were on the website for just over one minute. Although they did call me, their online behavior showed a lack of urgency, and their ‘real-world’ behavior mirrored what I saw in the movie.

A week later another party visited my site, and spent 80 minutes carefully reading 3 pages, among others. Their ‘real-world’ behavior also matched their online behavior, in that they exhibited a sense of urgency towards engaging a software expert.

Tracking Downloads And Other Behavior

Microsoft Clarity does not consider a download as a click. It does not even notice downloads. For me, downloads are what I care about most. If you never download my resume, you probably are not a candidate for hiring me as a software expert. I want to track downloads, not clicks. Clicks are nice, but there is a direct relationship between resume downloads and signing contracts with legal firms who represent new clients.

AWS CloudTrail and CloudWatch can provide download details and much more. For example, user IP addresses and geographic location can be captured when a monitored event occurs.

Many Websites Perform Surveillance

Wired Magazine published an article on a similar type of surveillance (keyloggers) on May 11, 2022. I paraphrased two sentences from that article:

1.8% of websites studied gathered an EU user's email address without their consent, and a staggering 2.95% logged a US user's email.
For US users, 8.4% of sites may have been leaking data to Meta, Facebook’s parent company, and 7.4% of sites may be impacted for EU users.

– From Thousands of Popular Websites See What You Type—Before You Hit Submit

Yours truly does nothing of the sort.

Looking Ahead

I would very much like to receive a continuous data feed of the above AI-generated per-visit summary, in real time as it becomes available. An email stream would be a quick way to start. A streaming api for data would be a logical next step.

Make a Visual Studio Code Extension

2022-03-01T00:00:00-05:00

I want to be able to move pages in my Jekyll-powered website without breaking links from the outside. The jekyll-redirect-from Jekyll plugin generates little HTML files that contain http-meta-refresh client-side redirects as desired. I wanted an easy way to inject the names of those redirect pages into the Jekyll front matter. Since I usually author my website using Visual Studio Code, a custom Visual Studio Code extension seems like a good way to create the redirects.

Extension Requirements

All the extension has to do is discover the URL path component of the page currently being edited, and write an entry into the front matter. For example, I have set up Jekyll such that given a page at collections/_posts/2022/2022-02-21-jekyll-debugging.html, it would deploy to /blog/2022/02/21/jekyll-debugging.html. The required front matter would be:

redirect_from:
  - /blog/2022/02/21/jekyll-debugging.html

With that in mind, if I want to move a published post to another location without breaking it, the extension should:

Present a new menu option when a right-click on a file in the sidebar, or a right-click on a file name tab in the editor.
When the menu item is selected, obtain the relative path to the file within the project directory (for example: collections/_posts/2022/2022-02-21-jekyll-debugging.html)
Convert the relative path to the deployed relative path (for example, /blog/2022/02/21/jekyll-debugging.html)
Write the entry into the front matter of the file. For example:
```
redirect_from:
  - /blog/2022/02/21/jekyll-debugging.html
```

Background

The Microsoft documentation describes how to write a Visual Studio Code extension.

Setup

Visual Studio Code extensions are written in JavaScript (actually, node.js) or TypeScript. Ensure that a version of node.js has been installed:

Shell

$ which node
/home/mslinn/.nvm/versions/node/v17.3.1/bin/node

Ensure you have set up your global package manager for node.js, then type:

Shell

$ npm install -g yo generator-code
  npm WARN deprecated har-validator@5.1.5: this library is no longer supported
  npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
  npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142

  added 872 packages, and audited 873 packages in 57s

  15 vulnerabilities (13 moderate, 2 high)

  To address issues that do not require attention, run:
    npm audit fix

  To address all issues (including breaking changes), run:
    npm audit fix --force

  Run `npm audit` for details.

The yo module is the culprit; lets address its vulnerabilities:

Shell

$ npm install -g npm-check-updates
added 270 packages, and audited 271 packages in 9s

found 0 vulnerabilities

Generating the Skeleton

I decided to use TypeScript instead of JavaScript for the extension. TypeScript code converts to JavaScript, however it uses type inference and integrates better with some editors.

Shell

$ mkdir redirect_generator

$ cd redirect_generator/

$ $ yo code
? ==========================================================================
We’re constantly looking for ways to make yo better!
May we anonymously report usage statistics to improve the tool over time?
More info: https://github.com/yeoman/insight & http://yeoman.io
==========================================================================  No


     _-----_     ╭──────────────────────────╮
    |       |    │   Welcome to the Visual  │
    |--(o)--|    │   Studio Code Extension  │
   `---------´   │        generator!        │
    ( _´U`_ )    ╰──────────────────────────╯
    /___A___\   /
     |  ~  |
   __'.___.'__
 ´   `  |° ´ Y `

? What type of extension do you want to create?  New Extension (TypeScript)
? What's the name of your extension?  redirect_generator
? What's the identifier of your extension?  redirect-generator
? What's the description of your extension? Injects the URL of a redirect page into Jekyll front matter
? Initialize a git repository? Yes
? Bundle the source code with webpack? No
? Which package manager to use? npm
Writing in /mnt/f/work/jekyll/redirect_generator/redirect-generator...
    create redirect-generator/.vscode/extensions.json
    create redirect-generator/.vscode/launch.json
    create redirect-generator/.vscode/settings.json
    create redirect-generator/.vscode/tasks.json
    create redirect-generator/package.json
    create redirect-generator/tsconfig.json
    create redirect-generator/.vscodeignore
    create redirect-generator/vsc-extension-quickstart.md
    create redirect-generator/README.md
    create redirect-generator/CHANGELOG.md
    create redirect-generator/src/extension.ts
    create redirect-generator/src/test/runTest.ts
    create redirect-generator/src/test/suite/extension.test.ts
    create redirect-generator/src/test/suite/index.ts
    create redirect-generator/.eslintrc.json

Changes to package.json were detected.

Running npm install for you to install the required dependencies.

added 203 packages, and audited 204 packages in 29s

found 0 vulnerabilities

Your extension redirect-generator has been created!

To start editing with Visual Studio Code, use the following commands:

     code redirect-generator

Open vsc-extension-quickstart.md inside the new extension for further instructions
on how to modify, test and publish your extension.

For more information, also visit http://code.visualstudio.com and follow us @code.


? Do you want to open the new folder with Visual Studio Code? Open with `code`

     _-----_     ╭───────────────────────╮
    |       |    │      Bye from us!     │
    |--(o)--|    │       Chat soon.      │
   `---------´   │      Yeoman team      │
    ( _´U`_ )    │    http://yeoman.io   │
    /___A___\   /╰───────────────────────╯
     |  ~  |
   __'.___.'__
 ´   `  |° ´ Y `

The instructions in the web page assume the happy path is the only possibility. Instead:

Do not work on the extension in a workspace that has any other projects in it.
The online instructions assume that TypeScript was used, not JavaScript, so the file types are assumed to be .ts.

The instructions in vsc-extension-quickstart.md do not make assumptions about TypeScript.

Debugging the Extension

I found the procedure awkward at first:

Press F5 to activate the first defined launcher
Once the new debug VSCode instance settles down, click in it and type Ctrl-Shift-P, then type redirect.
Any breakpoints in the original VSCode instance will trigger.
Once the breakpoint is cleared, a message saying Add redirecct from redirect_generator! will appear in the debugged instance of VSCode.

Extension Anatomy

I moved on to the next part of the tutorial, Extension Anatomy.

Node.js, NVM, NPM and Yarn

2022-03-01T00:00:00-05:00

Node.js, a JavaScript runtime, is not my favorite programming environment. Originally released 14 years ago by Ryan Dahl, a software engineer at Google, it’s historical disregard for security and stability is problematic.

However, node.js has significant traction, especially amongst Millennial software technologists, and is used by many Ruby and Scala projects for HTML asset pipelines.

I put together these notes for installing and maintaining node.js on Ubuntu/WSL using a virtualized version manager.

Why Virtualize Node.js?

It is better to use virtualized user- and project-specific node.js instances, instead of working with a system-wide installation of node.js. This allows you to install and upgrade node.js packages without using supervisor privileges. Also, virtualized instances allows you to work on many different independent node.js projects at the same time, without package version collisions.

Docker is over-sold. It adds unnecessary complexity to software projects. Instead of virtualizing the entire software environment, as docker attempts to do, virtualizing the programming environment with node.js nvm, Python venv, or Ruby rbenv are much easier and more productive approaches.

I think docker has been pushed hard in the media because it is a gateway technology to PaSS. This is a trend that PaSS vendors like AWS and Azure want to encourage, but customers are pushing back.

Nvm

Nvm, the Node Version Manager, makes it easy to install multiple virtualized node.js instances, and to easily switch between them. Nvm retains a unique set of installed packages for each node.js instance. The node.js version in each instance is distinct.

Nvm is installed and updated as follows:

Shell

$ curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14926  100 14926    0     0  84806      0 --:--:-- --:--:-- --:--:-- 85291
=> nvm is already installed in /home/mslinn/.nvm, trying to update using git
=> => Compressing and cleaning up git repository

=> nvm source string already in /home/mslinn/.bashrc
=> bash_completion source string already in /home/mslinn/.bashrc
=> Close and reopen your terminal to start using nvm or run the following to use it now:

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion

View the currently installed versions of node.js:

Shell

$ nvm list
->       system
iojs -> N/A (default)
node -> stable (-> N/A) (default)
unstable -> N/A (default)

View the very long list of available versions of node.js like this:

Shell

$ nvm list-remote
          v17.7.0
        v17.7.1
        v17.7.2
        v17.8.0
        v17.9.0
        v17.9.1
        v18.0.0
        v18.1.0
        v18.2.0
        v18.3.0
        v18.4.0
        v18.5.0
        v18.6.0
        v18.7.0
        v18.8.0
        v18.9.0
        v18.9.1
       v18.10.0
       v18.11.0
       v18.12.0   (LTS: Hydrogen)
       v18.12.1   (LTS: Hydrogen)
       v18.13.0   (Latest LTS: Hydrogen)
        v19.0.0
        v19.0.1
        v19.1.0
        v19.2.0
        v19.3.0
        v19.4.0
        v19.5.0

Using Nvm to Install Node.js

Install the latest release of node.js using nvm:

Shell

$ nvm install node
Downloading and installing node v19.5.0...
Downloading https://nodejs.org/dist/v19.5.0/node-v19.5.0-linux-x64.tar.xz...
##################################################################### 100.0%
Computing checksum with sha256sum
Checksums matched!
Now using node v19.5.0 (npm v9.3.1)
Creating default alias: default -> node (-> v19.5.0)

In the above example, node is an alias for “the latest version of node.js”. To install a specific version of node.js, for example 18.13.0:

Shell

$ nvm install 18.13.0

Yarn

Needle and thread under an electron microscope

Whether you use nvm as your node.js virtualization mechanism, you also need a node.js package manager to install and maintain project dependencies.

Yarn supposedly stands for Yet Another Resource Negotiator, and it is a package manager like npm, described next. It was developed by Facebook and is now open-source. Yarn was developed to address npm’s performance and security issues.

The name actually suggests the major advantage yarn has over its predecessor, npm: multi-threading instead of single-threading.

Yarn generates a yarn.lock file, which helps easy merges. The merges are predictable. Yarn caches every package it has downloaded, so it never needs to download the same package again. It also does almost everything concurrently to maximize resource utilization. This means faster installs. Yarn guarantees that any installation that works on one system will work exactly the same on another system, unlike npm.

The notes for the yarn package state that Yarn was inspired by Bundler and Cargo, and is nearly command-line compatible with npm.

Yarn introduces the zero-install concept, which means that a project should be able to be used as soon as it is cloned. It uses Plug’n’Play to resolve dependencies via the yarn cache folder and not from node_modules. The cache folder is by default stored within your project folder, in .yarn/cache.

Installing Yarn

To install yarn on Ubuntu:

Shell

$ curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | \
  sudo apt-key add -

$ echo "deb https://dl.yarnpkg.com/debian/ stable main" | \
  sudo tee /etc/apt/sources.list.d/yarn.list

$ sudo apt update && sudo apt install yarn

Configuring a Project With Yarn

The yarn init command initiates an interactive session that creates a package.json file.

Shell

$ cd /path/to/my/node/project

$ yarn init
yarn init v1.22.19
question name (node): blah
question version (1.0.0): 0.1.0
question description: Just a little song I wrote
question entry point (index.js):
question repository url: https://github.com/mslinn/blah
question author: Mike Slinn
question license (MIT):
question private:
success Saved package.json
Done in 48.29s.

The above dialog creates package.json in the current directory:

package.json

{
  "name": "blah",
  "version": "0.1.0",
  "description": "Just a little song I wrote",
  "main": "index.js",
  "repository": "https://github.com/mslinn/blah",
  "author": "Mike Slinn",
  "license": "MIT"
}

Adding a Dependency to a Project

Yarn stores dependencies locally. If the proper version is present locally, it is fetched from the disk during a yarn add command, otherwise it is downloaded. This is the general format of the command to add a dependency to a node.js project using yarn:

Shell

$ yarn add package_name

To add a specific version of a package:

Shell

$ yarn add package_name@version_number

To install a global package, the general formats are:

Shell

$ yarn global add package_name

$ yarn global add package_name@version_number

Yarn and Git

The official Yarn docs say to add the following to .gitignore for git projects that use yarn:

.gitignore

.yarn/*
!.yarn/cache
!.yarn/patches
!.yarn/plugins
!.yarn/releases
!.yarn/sdks
!.yarn/versions

Installing Dependencies

To install all the dependencies in a node.js project, type yarn or yarn install:

shell – Updating All Dependencies With Yarn

$ cd /path/to/my/project

$ yarn

Updating Dependencies

To upgrade all the dependencies in a node.js project, use yarn upgrade:

shell – Updating All Dependencies With Yarn

$ cd /path/to/my/project

$ yarn upgrade

Npm

Npm is the original package manager for the family of JavaScript programming languages, and it is still the default package manager for node.js. This will likely change as yarn matures. Npm helps install libraries, plugins, frameworks and applications. It consists of a command-line client, also called npm, and an online database of public and paid-for private packages called the npm registry.

If you are using yarn then you do not need to use npm.

Npm fetches dependencies from the npm registry for every npm install command.

Npm generates a package-lock.json file. The layout of this file is a trade-off between determinism and simplicity. The same node_modules/ folder will be generated by every version of npm. Every dependency will have a version number associated with it in the package-lock file.

Npm vs. Yarn

When using npm install, dependencies are installed sequentially, one after another. The output logs in the terminal are informative but a bit hard to read. To install the packages with Yarn, run the yarn command. Yarn installs packages in parallel, which is one of the reasons it is quicker than npm.

For more information, see Difference between npm and yarn.

Install A Global Package

To install a global package, the command template for npm is:

Shell

$ npm install -g package_name[@version_number]

For example:

Shell

$ npm install -g broken-link-checker
npm WARN deprecated calmcard@0.1.1: no longer maintained
npm WARN deprecated nopter@0.3.0: try optionator
npm WARN deprecated uuid@2.0.3: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated urlobj@0.0.11: use universal-url, minurl, relateurl, url-relation

added 104 packages, and audited 105 packages in 5s

Fun With Python Enums

2022-02-10T00:00:00-05:00

This blog post demonstrates how to define additional properties for Python 3 enums. Defining an additional property in a Python enum can provide a simple way to provide string values. The concept is then expanded to demonstrate composition, an important concept for functional programming. This post concludes with a demonstration of dynamic dispatch in Python, by further extending an enum.

Adding Properties to Python Enums

Searching for python enum string value yields some complex and arcane ways to approach the problem.

Below is a short example of a Python enum that demonstrates a simple way to provide lower-case string values for enum constants: a new property, to_s, is defined. This property provides the string representation that is required. You could define other properties and methods to suit the needs of other projects.

cad_enums.py

"""Defines enums"""

from enum import Enum, auto

class EntityType(Enum):
    """Types of entities"""
    SITE = auto()
    GROUP = auto()
    COURSE = auto()
    SECTION = auto()
    LECTURE = auto()

    @property
    def to_s(self) -> str:
        """:return: lower-case name of this instance"""
        return self.name.lower()

Adding the following to the bottom of the program allows us to demonstrate it:

cad_enums.py (part 2)

if __name__ == "__main__":  # Just for demonstration
    print("Specifying individual values:")
    print(f"  {EntityType.SITE.value}: {EntityType.SITE.to_s}")
    print(f"  {EntityType.GROUP.value}: {EntityType.GROUP.to_s}")
    print(f"  {EntityType.COURSE.value}: {EntityType.COURSE.to_s}")
    print(f"  {EntityType.SECTION.value}: {EntityType.SECTION.to_s}")
    print(f"  {EntityType.LECTURE.value}: {EntityType.LECTURE.to_s}")

    print("\nIterating through all values:")
    for entity_type in EntityType:
        print(f"  {entity_type.value}: {entity_type.to_s}")

Running the program produces this output:

Shell

$ cad_enums.py
Specifying individual values:
  1: site
  2: group
  3: course
  4: section
  5: lecture

Iterating through all values:
  1: site
  2: group
  3: course
  4: section
  5: lecture

😁

Easy!

Constructing Enums

Enum constructors work the same as other Python class constructors. There are several ways to make a new instance of a Python enum. Let's try two ways by using the Python interpreter. Throughout this blog post I've inserted a blank line between Python interpreter prompts for readability.

Python

$ python
Python 3.9.7 (default, Sep 10 2021, 14:59:43)
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> from cad_enums import EntityType

>>> # Specify the desired enum constant value symbolically
>>> gtype = EntityType.GROUP
>>> print(gtype)
EntityType.GROUP 

>>> # Specify the desired enum constant value numerically
>>> stype = EntityType(1)
>>> print(stype)
EntityType.SITE

Enum Ordering

A program I am working on needs to obtain the parent EntityType. By 'parent' I mean the EntityType with the next lowest numeric value. For example, the parent of EntityType.GROUP is EntityType.SITE. We can obtain a parent enum by computing its numeric value by adding the following method to the EntityType class definition.

Python

@property
def parent(self) -> 'EntityType':
    """:return: entity type of parent; site has no parent"""
    return EntityType(max(self.value - 1, 1))

The return type above is enclosed in quotes ('EntityType') to keep Python's type checker happy, because this is a forward reference. This is a forward reference because the type is referenced before it is fully compiled.

The complete enum class definition is now:

cad_enums.py

"""Defines enums"""

from enum import Enum, auto

class EntityType(Enum):
    """Types of entities"""
    SITE = auto()
    GROUP = auto()
    COURSE = auto()
    SECTION = auto()
    LECTURE = auto()

    @property
    def to_s(self) -> str:
        """:return: lower-case name of this instance"""
        return self.name.lower()

    @property
    def parent(self) -> 'EntityType':
        """:return: entity type of parent; site has no parent"""
        return EntityType(max(self.value - 1, 1))

Lets try out the new parent property in the Python interpreter.

Python

>>> EntityType.LECTURE.parent
<EntityType.SECTION: 4> 

>>> EntityType.SECTION.parent
<EntityType.COURSE: 3> 

>>> EntityType.COURSE.parent
<EntityType.GROUP: 2> 

>>> EntityType.GROUP.parent
<EntityType.SITE: 1> 

>>> EntityType.SITE.parent
<EntityType.SITE: 1>

Enum Composition

Like methods and properties in all other Python classes, enum methods and properties compose if they return an instance of the class. Composition is also known as method chaining, and also can apply to class properties. Composition is an essential practice of a functional programming style.

The parent property returns an instance of the EntityType enum class, so it can be composed with any other property or method of that class, for example the to_s property shown earlier.

Python

>>> EntityType.LECTURE.parent.to_s
'section' 

>>> EntityType.SECTION.parent.to_s
'course' 

>>> EntityType.COURSE.parent.to_s
'group' 

>>> EntityType.GROUP.parent.to_s
'site' 

>>> EntityType.SITE.parent.to_s
'site'

Dynamic Dispatch

The Python documentation might lead someone to assume that writing dynamic dispatch code is more complex than it actually is.

To summarize the documentation, all Python classes, methods and instances are callable. Callable functions have type Callable[[InputArg1Type, InputArg2Type], ReturnType]. If you do not want any type checking, write Callable[..., Any]. However, this is not very helpful information for dynamic dispatch. Fortunately, working with Callable is very simple.

You can pass around any Python class, constructor, function or method, and later provide it with the usual arguments. Invocation just works.

Let me show you how easy it is to write dynamic dispatch code in Python, let's construct one of five classes, depending on the value of an enum. First, we need a class definition for each enum value:

Python

# pylint: disable=too-few-public-methods

class BaseClass():
    """Demo only"""

class TestLecture(BaseClass):
    """This constructor has type Callable[[int, str], TestLecture]"""
    def __init__(self, id_: int, action: str):
        print(f"CadLecture constructor called with id {id_} and action {action}")

class TestSection(BaseClass):
    """This constructor has type Callable[[int, str], TestSection]"""
    def __init__(self, id_: int, action: str):
        print(f"CadSection constructor called with id {id_} and action {action}")

class TestCourse(BaseClass):
    """This constructor has type Callable[[int, str], TestCourse]"""
    def __init__(self, id_: int, action: str):
        print(f"CadCourse constructor called with id {id_} and action {action}")

class TestGroup(BaseClass):
    """This constructor has type Callable[[int, str], TestGroup]"""
    def __init__(self, id_: int, action: str):
        print(f"CadGroup constructor called with id {id_} and action {action}")

class TestSite(BaseClass):
    """This constructor has type Callable[[int, str], TestSite]"""
    def __init__(self, id_: int, action: str):
        print(f"CadSite constructor called with id {id_} and action {action}")

Now lets add another method, called construct, to EntityType that invokes the appropriate constructor according to the value of an EntityType instance:

Python

@property
def construct(self) -> Callable:
    """:return: the appropriate Callable for each enum value"""
    if self == EntityType.LECTURE:
        return TestLecture
    if self == EntityType.SECTION:
        return TestSection
    if self == EntityType.COURSE:
        return TestCourse
    if self == EntityType.GROUP:
        return TestGroup
    return TestSite

Using named arguments makes your code resistant to problems that might sneak in due to parameters changing over time.

I favor using named arguments at all times; it avoids many problems. As code evolves, arguments might be added or removed, or even reordered.

Let's test out dynamic dispatch in the Python interpreter. A class specific to each EntityType value is constructed by invoking the appropriate Callable and passing it named arguments id_ and action.

Python

>>> EntityType.LECTURE.construct(id_=55, action="gimme_lecture")
TestLecture constructor called with id 55 and action gimme_lecture
<entity_types.TestLecture object at 0x7f9aac690070> 

>>> EntityType.SECTION.construct(id_=13, action="gimme_section")
TestSection constructor called with id 13 and action gimme_section
<entity_types.TestSection object at 0x7f9aac5c1730> 

>>> EntityType.COURSE.construct(id_=40, action="gimme_course")
TestCourse constructor called with id 40 and action gimme_course
<entity_types.TestCourse object at 0x7f9aac6900a0> 

>>> EntityType.GROUP.construct(id_=103, action="gimme_group")
TestGroup constructor called with id 103 and action gimme_group
<entity_types.TestGroup object at 0x7f9aac4c6b20> 

>>> EntityType.SITE.construct(id_=1, action="gimme_site")
TestSite constructor called with id 1 and action gimme_site
<entity_types.TestSite object at 0x7f9aac5c1730>

Because these factory methods return the newly created instance of the desired type, the string representation is printed on the console after the method finishes outputting its processing results, for example: <entity_types.TestLecture object at 0x7f9aac690070>.

Using enums to construct class instances and/or invoke methods (aka dynamic dispatch) is super powerful. It rather resembles generics, actually, even though Python’s support for generics is still in its infancy.

The complete Python program discussed in this post is here.

Linking Directories on NTFS and Ext4 Volumes

2022-02-07T00:00:00-05:00

Sometimes I need to insert some code into a program that depends on the type of format that a drive volume has. For example, today I need to either make a Windows junction to connect two directories on NTFS volumes. On the other hand, if one or both directories were on other types of volumes, I would have to connect the directories using a Linux symlink.

All of the bash scripts shown in this blog post are meant to run in a Bash shell running on WSL or WSL2.

Use Windows Junctions When Possible

When working on WSL, Windows junctions are more desirable than Linux hard links and symlinks because junctions are visible in Windows and also in WSL. Unlike Linux hard links, which only work within a single volume, Windows junctions can span two volumes. Linux symlinks are only visible from WSL; a symlinked directory only appears as a useless file when viewed from Windows.

Both directories need to be on NTFS volumes to make a Windows junction between them. Junctions are permitted within a single NTFS volume, or between two NTFS volumes. Linux symlinks can be used on all volume types, but only work properly when viewed from Linux.

Windows junctions are shown with a small arrow icon in Windows File Manager. In the image above, the curriculum directory is a junction.

Determining a Volume Type

I wrote the volumeType bash function to obtain the type of the volume that contains a file or directory. Linux ext4 volumes have partition type ext4. NTFS volumes have partition type 9p.

Shell

function volumeType {
  # Usually returns volume types ext4 or 9p (for NTFS)
  df -Th "$1" | tail -n 1 | awk '{print $2}'
}

Here are examples of using volumeType:

Shell

$ volumeType /mnt/c  # Under WSL/WSL2 this is usually NTFS
9p 

$ volumeType /       # For Ubuntu this defaults to ext4
ext4

All of the remaining scripts on this page either return a value (indicating true), or they do not return anything (indicating false).

Two more bash functions test if a file or directory is part of an NTFS or ext4 volume:

Shell

function isNTFS {
  if [ "$( volumeType "$1" )" == 9p ]; then echo yes; fi
}

function isExt4 {
  if [ "$( volumeType "$1" )" == ext4 ]; then echo yes; fi
}

Here are examples of using isNTFS and isExt4:

Shell

$ isNTFS /mnt/c
yes 

$ isExt4 /mnt/c

$ isNTFS /

$ isExt4 /
yes

Windows Junctions

The bothOnNTFS bash function indicates if both of the paths passed to it are on NTFS volumes.

Shell

function bothOnNTFS {
  if [ "$( isNTFS "$1" )" ] && [ "$( isNTFS "$2" )" ]; then echo yes; fi
}

Let's try out bothOnNTFS.

$ bothOnNTFS /mnt/c /mnt/f
yes 

$ bothOnNTFS /mnt/c /

bothOnNTFS lets us decide how to connect two directories. If they are both on NTFS volumes, we can connect them using a Windows junction; otherwise we'll need to symlink them.

Connecting Via a Windows Junction or Linux Symlink

We could either make a Windows junction using the mklink command, or we could make a Linux symlink using the ln -s command. Notice how the order of parameters between mklink is the reverse of the order of the Linux ln command.

Shell

if [ $( bothOnNTFS "$cadenzaCurriculum" . ) ]; then
    WINDOWS_PATH="$( wslpath -w "$cadenzaCurriculum/site_$TITLE" )"
    echo "Making Windows junction from $cadenzaCurriculum/site_$TITLE to curriculum/"
    cmd.exe /C mklink /j curriculum "$WINDOWS_PATH"
else
  echo "Symlinking $cadenzaCurriculum/site_$TITLE to curriculum/"
  ln -s "$cadenzaCurriculum/site_$TITLE" curriculum
fi

When the above code ran it produced:

Making Windows junction from /mnt/f/work/cadenzaHome/cadenzaCurriculum/site_ScalaCourses.com to curriculum/
Junction created for curriculum <<===>> F:\work\cadenzaHome\cadenzaCurriculum\site_ScalaCourses.com

Windows Junction for Windows Home in Ubuntu

Neither Linux nor Java appreciate spaces in directories. Spaces in home directories are by nature problematic. If you have a space in your home directory on Windows, you can use the PowerShell New-Item commandlet to create an equivalent junction to a new directory node without spaces in the name.

The following uses the PowerShell New-Item commandlet to create a new directory called C:\Users\mslinn, accessible from WSL as /mnt/c/mslinn, which points to the same directory as C:\Users\Mike Slinn.

PowerShell

PS C:\Users\Mike Slinn> ni "C:\Users\mslinn" -i SymbolicLink -ta "C:\Users\Mike Slinn"

Handcrafted Dynamic DNS for AWS Route53 and Namecheap

2022-01-30T00:00:00-05:00

Now that I have fiber-optic internet service in my apartment, with 500 GB/s upload and download, I thought I would save money by hosting my scalacourses.com website on an Ubuntu server that runs here, instead of AWS. My home IP address is quite stable, and only changes when the fiber modem boots up. The modem is branded as a Bell Home Hub 4000, but I believe it is actually made by Arris (formerly known as Motorola).

It makes little sense to pay the commercial cost of dedicated dynamic DNS services (typically $55 USD / year) when it is so easy to automate, and the operational cost is less than one cent per year.

I wrote two little scripts that automatically check my public IP address, and modifies the DNS record for my home IP address whenever the IP address changes. One script is for sites that use DNS provided by AWS Route53, and the other is for DNS provided by Namecheap.

The approach shown here could be used for all DNS servers that have a command-line interface. This was originally written for AWS Route53, but when I moved off AWS I made a version for Namecheap. Alternative DNS providers include Azure DNS, Cloudflare DNS, DNSMadeEasy, DNSimple, Google Cloud DNS, and UltraDNS.

Forwarding HTTP Requests

I added some entries to the modem so incoming HTTP traffic on ports 80 and 443 would be forwarded to ports 9000 and 9443 on my home server, gojira.

Using the Scripts

The version for AWS Route53 is called dynamicDnsAws, and the version for Namecheap is called dynamicDnsNamecheap.

The scripts save the IP address to a file, and periodically compare the saved value to the current value. Then the scripts modify the DNS record for a specified subdomain whenever the value of the public IP address changes.

Using the AWS Script

Here is the help information for the script:

Shell

$ dynamicDnsAws
dynamicDnsAws - Maintains a dynamic DNS record in AWS Route53

Saves data in '/home/mslinn/.dynamicDnsAws'

Syntax:
  dynamicDnsAws [OPTIONS] SUB_DOMAIN DOMAIN

OPTIONS:
  -v Verbose mode

Example usage:
  dynamicDnsAws www scalacourses.com
  dynamicDnsAws -v www scalacourses.com

Here is a sample usage:

Shell

$ dynamicDnsAws www scalacourses.com
{
    "ChangeInfo": {
        "Id": "/change/C075751811HI18SH4L8L0",
        "Status": "PENDING",
        "SubmittedAt": "2022-01-30T21:10:09.261Z",
        "Comment": "UPSERT a record for www.scalacourses.com"
    }
}

Using the Namecheap Script

Here is the help information for the script:

Shell

$ dynamicDnsNamecheap
dynamicDnsNamecheap - Maintains two Namecheap dynamic DNS records

  Saves data in '/home/mslinn/.dynamicDns'

  Syntax:
    dynamicDnsNamecheap [OPTIONS] DOMAIN PASSWORD

  OPTIONS:
    -v Verbose mode

  Example usage:
    dynamicDnsNamecheap mydomain.com  asdfasdfasdfasdfasdf
    dynamicDnsNamecheap -v mydomain.com asdfasdfasdfasdf

Here is sample usage:

Shell

$ dynamicDnsNamecheap scalacourses.com asdfasdfasdfasdfasdf
<?xml version="1.0" encoding="utf-16"?>
<interface-response>
  <Command>SETDNSHOST</Command>
  <Language>eng</Language>
  <IP>142.126.4.220</IP>
  <ErrCount>0</ErrCount>
  <errors />
  <ResponseCount>0</ResponseCount>
  <responses />
  <Done>true</Done>
  <debug><![CDATA[]]></debug>
</interface-response><?xml version="1.0" encoding="utf-16"?>
<interface-response>
  <Command>SETDNSHOST</Command>
  <Language>eng</Language>
  <IP>142.126.4.220</IP>
  <ErrCount>0</ErrCount>
  <errors />
  <ResponseCount>0</ResponseCount>
  <responses />
  <Done>true</Done>
  <debug><![CDATA[]]></debug>
</interface-response>

Invoking the Scripts from Crontab

A personal crontab can be modified by typing:

Shell

$ crontab -e

I pasted in the following into crontab on my Ubuntu server, running at home. These lines invoke the dynamicDnsNamecheap script via crontab every 5 minutes.

Shell

*/5 * * * * /path/to/dynamicDnsNamecheap my_domain.com asdfasdfasdfasdfasdf

One the above is saved, crontab will run the script every 5 minutes.

Script Source Codes

Here are the bash scripts:

dynamicDnsAws

#!/bin/bash

# Author: Mike Slinn mslinn@mslinn.com
# Written 2022-01-30

export SAVE_FILE_NAME="$HOME/.dynamicDns"

function help {
  echo "$( basename $0 ) - Maintains a dynamic DNS record in AWS Route53

Saves data in '$SAVE_FILE_NAME'

Syntax:
  $( basename $0) [OPTIONS] SUB_DOMAIN DOMAIN

OPTIONS:
  -v Verbose mode

Example usage:
  $( basename $0) my_subdomain mydomain.com
  $( basename $0) -v my_subdomain mydomain.com

"  
  exit 1
}

function upsert {
  export HOSTED_ZONES="$(
    aws route53 list-hosted-zones
  )"

  export HOSTED_ZONE_RECORD="$(
    jq -r ".HostedZones[] | select(.Name == \"$DOMAIN.\")" <<< "$HOSTED_ZONES"
  )"

  export HOSTED_ZONE_RECORD_ID="$(
    jq -r .Id <<< "$HOSTED_ZONE_RECORD"
  )"

  aws route53 change-resource-record-sets \
    --hosted-zone-id "$HOSTED_ZONE_RECORD_ID" \
    --change-batch "{
      \"Comment\": \"UPSERT a record for $SUBDOMAIN.$DOMAIN\",
      \"Changes\": [{
      \"Action\": \"UPSERT\",
        \"ResourceRecordSet\": {
          \"Name\": \"$SUBDOMAIN.$DOMAIN\",
          \"Type\": \"A\",
          \"TTL\": 300,
          \"ResourceRecords\": [{ \"Value\": \"$IP\"}]
        }
      }]
    }"

  echo "$IP" > "$SAVE_FILE_NAME"
}

if [ "$1" == -v ]; then
  export VERBOSE=true
  shift
fi

if [ -z "$2" ]; then help; fi

set -e 

export SUBDOMAIN="$1"
export DOMAIN="$2"
export IP="$( dig +short myip.opendns.com @resolver1.opendns.com )"

if [ ! -f "$SAVE_FILE_NAME" ]; then
  if [ "$VERBOSE" ]; then echo "Creating $SAVE_FILE_NAME"; fi
  upsert; 
elif [ $( cat "$SAVE_FILE_NAME" ) != "$IP" ]; then 
  if [ "$VERBOSE" ]; then 
    echo "Updating $SAVE_FILE_NAME"
    echo "'$IP' was not equal to '$( cat "$SAVE_FILE_NAME" )'"
  fi
  upsert; 
else
  if [ "$VERBOSE" ]; then echo "No change necessary for $SAVE_FILE_NAME"; fi
fi

dynamicDnsNamecheap

#!/bin/bash

# Author: Mike Slinn mslinn@mslinn.com
# Modified from AWS version (dynameicDnsAws) 2022-06-30
# See https://www.namecheap.com/support/knowledgebase/article.aspx/36/11/how-do-i-start-using-dynamic-dns/

export SAVE_FILE_NAME="$HOME/.dynamicDns"

function help {
  echo "$( basename $0 ) - Maintains two Namecheap dynamic DNS records

Saves data in '$SAVE_FILE_NAME'

Syntax:
  $( basename $0) [OPTIONS] DOMAIN PASSWORD

OPTIONS:
  -v Verbose mode

Example usage:
  $( basename $0) mydomain.com  asdfasdfasdfasdfasdf
  $( basename $0) -v mydomain.com asdfasdfasdfasdf

"
  exit 1
}

function upsert {
  curl "https://dynamicdns.park-your-domain.com/update?host=@&domain=$DOMAIN&password=$PASSWORD&ip=$IP"
  curl "https://dynamicdns.park-your-domain.com/update?host=www&domain=$DOMAIN&password=$PASSWORD&ip=$IP"
  echo "$IP" > "$SAVE_FILE_NAME"
  echo ""
}

if [ "$1" == -v ]; then
  export VERBOSE=true
  shift
fi

if [ -z "$2" ]; then help; fi

set -e

export DOMAIN="$1"
export PASSWORD="$2"
export IP="$( dig +short myip.opendns.com @resolver1.opendns.com )"

if [ ! -f "$SAVE_FILE_NAME" ]; then
  if [ "$VERBOSE" ]; then echo "Creating $SAVE_FILE_NAME"; fi
  upsert;
elif [ "$( cat "$SAVE_FILE_NAME" )" != "$IP" ]; then
  if [ "$VERBOSE" ]; then
    echo "Updating $SAVE_FILE_NAME"
    echo "'$IP' was not equal to '$( cat "$SAVE_FILE_NAME" )'"
  fi
  upsert;
else
  if [ "$VERBOSE" ]; then echo "No change necessary for $SAVE_FILE_NAME"; fi
fi

Windows Diskpart Cooperates With Diskmgmt

2022-01-14T00:00:00-05:00

Recently, I wanted to use some old hard drives as backup media. That meant scrubbing all the partitions off the drives, and installing new partitions, which of course would be empty.

Warning – Working with a command line program for system-level operations, without having backed up the system, is like walking on a tightrope without a net.

A mistake could inadvertently wipe out a different hard drive on the computer than you intended. Without the ability to restore the system disk from backup, your computer could become inoperable.

However, I was unable to repartition one of those old drives using the GUI-based Windows diskmgmt.msc disk manager. The drive had soft errors. For some reason, those errors made it impossible for diskmgmt.msc to scan the drive.

The more powerful Windows diskpart, a command line program, was able to get the job done.

This article ends by demonstrating how you can get benefit from the diskmgmt GUI even when you are using the diskpart command line interface. This is possible because every command you type into diskpart causes a Windows system event to be published, and because diskmgmt subscribes to those events, it is able to display the results as they happen.

Starting Diskpart

Run diskpart as administrator as follows:

Press the Windows key. Do not hold it down, just depress it once, as you would do for any other key, and let it go.
Type diskpart.
Use the mouse or arrow keys to select Run as administrator.

A window should open up labeled Microsoft DiskPart. Let's start by listing all the diskpart commands.

Microsoft DiskPart

Microsoft DiskPart version 10.0.19041.964

Copyright (C) Microsoft Corporation.
On computer: BEAR

DISKPART> help

Microsoft DiskPart version 10.0.19041.964

ACTIVE      - Mark the selected partition as active.
ADD         - Add a mirror to a simple volume.
ASSIGN      - Assign a drive letter or mount point to the selected volume.
ATTRIBUTES  - Manipulate volume or disk attributes.
ATTACH      - Attaches a virtual disk file.
AUTOMOUNT   - Enable and disable automatic mounting of basic volumes.
BREAK       - Break a mirror set.
CLEAN       - Clear the configuration information, or all information, off the
              disk.
COMPACT     - Attempts to reduce the physical size of the file.
CONVERT     - Convert between different disk formats.
CREATE      - Create a volume, partition or virtual disk.
DELETE      - Delete an object.
DETAIL      - Provide details about an object.
DETACH      - Detaches a virtual disk file.
EXIT        - Exit DiskPart.
EXTEND      - Extend a volume.
EXPAND      - Expands the maximum size available on a virtual disk.
FILESYSTEMS - Display current and supported file systems on the volume.
FORMAT      - Format the volume or partition.
GPT         - Assign attributes to the selected GPT partition.
HELP        - Display a list of commands.
IMPORT      - Import a disk group.
INACTIVE    - Mark the selected partition as inactive.
LIST        - Display a list of objects.
MERGE       - Merges a child disk with its parents.
ONLINE      - Online an object that is currently marked as offline.
OFFLINE     - Offline an object that is currently marked as online.
RECOVER     - Refreshes the state of all disks in the selected pack.
              Attempts recovery on disks in the invalid pack, and
              resynchronizes mirrored volumes and RAID5 volumes
              that have stale plex or parity data.
REM         - Does nothing. This is used to comment scripts.
REMOVE      - Remove a drive letter or mount point assignment.
REPAIR      - Repair a RAID-5 volume with a failed member.
RESCAN      - Rescan the computer looking for disks and volumes.
RETAIN      - Place a retained partition under a simple volume.
SAN         - Display or set the SAN policy for the currently booted OS.
SELECT      - Shift the focus to an object.
SETID       - Change the partition type.
SHRINK      - Reduce the size of the selected volume.
UNIQUEID    - Displays or sets the GUID partition table (GPT) identifier or
              master boot record (MBR) signature of a disk.

Listing Drives, Partitions and Volumes

Listing the drives is generally a good first step. Let's discover the command for that:

Microsoft DiskPart

DISKPART> list

Microsoft DiskPart version 10.0.19041.964

DISK        - Display a list of disks. For example, LIST DISK.
PARTITION   - Display a list of partitions on the selected disk.
              For example, LIST PARTITION.
VOLUME      - Display a list of volumes. For example, LIST VOLUME.
VDISK       - Displays a list of virtual disks.

OK, we can list disks, partitions, volumes and virtual disks. Let's list the disk drives.

Microsoft DiskPart

DISKPART> list disk

Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online         1863 GB  1024 KB        *
  Disk 1    Online          465 GB  1024 KB
  Disk 2    Online         1863 GB  1024 KB        *
  Disk 3    Online         1863 GB      0 B
* Disk 5    Online          931 GB   931 GB        *

NewerTech Voyager S3 caddy

At this point, I inserted the old 5.25" SATA drive that I wanted to repurpose into a NewerTech Voyager S3 caddy, connected to my computer via a USB 3 cable, and Windows automatically mounted it. The caddy also accepts 2.5" SATA drives, such as those commonly found in laptops.

Now I told diskpart to rescan the drives, and then I listed the volumes on all disks.

Microsoft DiskPart

DISKPART> rescan

Please wait while DiskPart scans your configuration...

DiskPart has finished scanning your configuration.

DISKPART> list volume

Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
  ----------  ---  -----------  -----  ----------  -------  ---------  --------
  Volume 0     D                       DVD-ROM         0 B  No Media
  Volume 1     C   BEAR_C       NTFS   Partition   1861 GB  Healthy    Boot
  Volume 2                      FAT32  Partition    100 MB  Healthy    System
  Volume 3                      NTFS   Partition    539 MB  Healthy    Hidden
  Volume 4                      NTFS   Partition    450 MB  Healthy    Hidden
  Volume 5     F   Work         NTFS   Partition   1863 GB  Healthy
  Volume 6     E   BEAR_E       NTFS   Partition   1863 GB  Healthy
  Volume 8                      FAT32  Partition    512 MB  Healthy    Hidden

Diskpart displayed the hidden volume (#8) in the drive in the caddy. This drive has readonly status set, which prevents its contents from being modified or deleted. That would be good if I wanted to use this drive as an archive, but instead I want to scrub it and write new information on it. The currently existing partitions on this drive cannot be erased until readonly is cleared. Lets remove readonly status from the drive now:

Microsoft DiskPart

DISKPART> select volume 8
Volume 8 is the selected volume.

DISKPART> attributes disk clear readonly
Disk attributes cleared successfully.

DISKPART> rescan
Please wait while DiskPart scans your configuration...
DiskPart has finished scanning your configuration.

DISKPART> list volume
Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
  ----------  ---  -----------  -----  ----------  -------  ---------  --------
  Volume 0     D                       DVD-ROM         0 B  No Media
  Volume 1     C   BEAR_C       NTFS   Partition   1861 GB  Healthy    Boot
  Volume 2                      FAT32  Partition    100 MB  Healthy    System
  Volume 3                      NTFS   Partition    539 MB  Healthy    Hidden
  Volume 4                      NTFS   Partition    450 MB  Healthy    Hidden
  Volume 5     F   Work         NTFS   Partition   1863 GB  Healthy
  Volume 6     E   BEAR_E       NTFS   Partition   1863 GB  Healthy
  Volume 8                      FAT32  Partition    512 MB  Healthy    Hidden

Wiping the Drive

Now it is time to wipe the drive clean, which removes all partitions.

Microsoft DiskPart

DISKPART> select disk 5
Disk 5 is now the selected disk.

DISKPART> list disk
Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online         1863 GB  1024 KB        *
  Disk 1    Online          465 GB  1024 KB
  Disk 2    Online         1863 GB  1024 KB        *
  Disk 3    Online         1863 GB      0 B
* Disk 5    Online          931 GB   931 GB        *

DISKPART> clean
DiskPart succeeded in cleaning the disk.

Archiving a Drive

If instead of wiping the drive, I wanted to archive the drive, I would want to set the read-only status. To do that, first select the drive as before, then type:

Microsoft DiskPart

DISKPART> list disk
# Output as shown above 

DISKPART> select disk N
Disk N is now the selected disk. 

DISKPART> attributes disk set readonly

Now the drive's contents could not accidently be erased or modified.

Create A New Partition

We need to create a new partition that spans the entire disk on the now-empty selected drive. The create command can do that. Let's look at the help before using the command:

Microsoft DiskPart

DISKPART> create
Microsoft DiskPart version 10.0.19041.964

PARTITION   - Create a partition.
VOLUME      - Create a volume.
VDISK       - Creates a virtual disk file.

DISKPART> create partition

Microsoft DiskPart version 10.0.19041.964

EFI         - Create an EFI system partition.
EXTENDED    - Create an extended partition.
LOGICAL     - Create a logical drive.
MSR         - Create a Microsoft Reserved partition.
PRIMARY     - Create a primary partition.

To create a new partition that spans the entire disk on the now-empty selected drive and make it active:

Microsoft DiskPart

DISKPART> create partition primary
DiskPart succeeded in creating the specified partition.

DISKPART> select partition 1
Partition 1 is now the selected partition.

DISKPART> active
DiskPart marked the current partition as active.

Format A Volume

Let's format the entire selected drive as one volume. Like the clean command, the format command operates on the currently selected disk. First, let's look at the help:

Microsoft DiskPart

DISKPART> help format

Formats the specified volume for use with Windows.

Syntax:  FORMAT [[FS=] [REVISION=] | RECOMMENDED] [LABEL=<"label">]
                [UNIT=] [QUICK] [COMPRESS] [OVERRIDE] [DUPLICATE] [NOWAIT]
                [NOERR]

    FS=     Specifies the type of file system. If no file system is given,
                the default file system displayed by the FILESYSTEMS command is
                used.

    REVISION=

                Specifies the file system revision (if applicable).

    RECOMMENDED If specified, use the recommended file system and revision
                instead of the default if a recommendation exists. The
                recommended file system (if one exists) is displayed by the
                FILESYSTEMS command.

    LABEL=<"label">

                Specifies the volume label.

    UNIT=    Overrides the default allocation unit size. Default settings
                are strongly recommended for general use. The default
                allocation unit size for a particular file system is displayed
                by the FILESYSTEMS command.

                NTFS compression is not supported for allocation unit sizes
                above 4096.

    QUICK       Performs a quick format.

    COMPRESS    NTFS only: Files created on the new volume will be compressed
                by default.

    OVERRIDE    Forces the file system to dismount first if necessary. All
                opened handles to the volume would no longer be valid.

    DUPLICATE   UDF Only: This flag applies to UDF format, version 2.5 or
                higher.
                This flag instructs the format operation to duplicate the file
                system meta-data to a second set of sectors on the disk. The
                duplicate meta-data is used by applications, for example repair
                or recovery applications. If the primary meta-data sectors are
                found to be corrupted, the file system meta-data will be read
                from the duplicate sectors.

    NOWAIT      Forces the command to return immediately while the format
                process is still in progress. If NOWAIT is not specified,
                DiskPart will display format progress in percentage.

    NOERR       For scripting only. When an error is encountered, DiskPart
                continues to process commands as if the error did not occur.
                Without the NOERR parameter, an error causes DiskPart to exit
                with an error code.

    A volume must be selected for this operation to succeed.

Examples:

    FORMAT FS=NTFS LABEL="New Volume" QUICK COMPRESS
    FORMAT RECOMMENDED OVERRIDE

Diskpart automatically chooses the optimal file system for the selected drive if you do not specify it. Choices include FAT, FAT32 and NTFS. To quick format the selected drive using the optimal file system:

Microsoft DiskPart

DISKPART> format quick

The default is to fully format the selected drive, which is what you want if the selected drive is suspect:

Microsoft DiskPart

DISKPART> format

You could specify multiple parameters, for example:

Microsoft DiskPart

DISKPART> format fs=ntfs label="My Drive" quick

Assigning a Drive Letter

Once the drive was formatted, I assigned the selected drive the letter G. You do not normally need to perform this step, unless you want to define a default letter for this drive.

Microsoft DiskPart

DISKPART> assign letter=G

Diskmgmt GUI Shows Progress

Having a GUI continuously report the current state of the drive maintenance you are performing using a command-line interface is a good practice.

The GUI-based Windows disk manager, diskmgmt.msc, can show the instantaneous progress of all diskpart commands, working on every drive, including creating and deleting partitions, volumes, formatting and much more. For example, formatting progress can be seen here:

Run diskmgmt.msc by:

Press the Windows key once and let go.
Type diskmgmt
Press Enter

Exit

To exit diskpart, either type Exit or press Ctrl-C.

Microsoft DiskPart

DISKPART> exit

The drive is ready for its next assignment!

WSL / WSL 2 Backup and Restore

2022-01-10T00:00:00-05:00

I needed to back up my WSL2 installation before reinstalling Windows 10, as one must do every 6 months if you work your machine like a developer. It had been years since I had last refreshed this machine, and it was now bluescreening several times a day. Reboots took forever.

This article presents the best WSL backup approaches I found

I wanted to retain my WSL2 instance. When refreshing Windows you must reinstall all your programs, and you lose all the WSL/WSL2 instances. The refresh decommissions them, then moves them into a hidden directory tree, along with the rest of the stuff stored in your old Windows 10 profile directory tree.

I also wanted to be able to replicate my WSL2 instance reliably and easily. It was very important to me to be able to administer WSL instances separately from my Windows 10 profile.

Location, Location, Location

Only recently has it become possible to work with WSL/WSL2 images on non-system drives. Most online documentation is now out of date in this regard. Running Windows off a different drive than the drive that a WSL/WSL2 client OS runs from can provide a dramatic performance boost for some applications.

Reinstalling Windows is less traumatic if the WSL/WSL2 image elsewhere in the filesystem

Dell’s Ignorance Causes Pain

I use Dell laptops, because I love their onsite warranty price and product features. However, sometimes it feels like Dell’s solution to every problem is to replace the motherboard.

When Windows 10 encounters a new motherboard, it takes anti-piracy measures, and refuses to do much of anything useful.

Refreshing Windows solves that problem, but doing that blows away the standard WSL/WSL2 image. Again. And again. And again!

Dell replaced the motherboard 4 times in 2 years for one of my laptops. They simply do not understand how replacing the motherboard creates long-term issues.

Test Your Backup

Backing up WSL/WSL2 can be problematic. Restoring it is even more delicate. Knowing this, I decided to backup and test the restoration process before refreshing Windows and potentially losing my working system.

I have tried several times before to make this work. As I said, it had been years since I allowed the OS in my main workstation to be refreshed. The reason I resisted doing proper maintenance was because I wanted to preserve my WSL2 Ubuntu image. Until today, I met with frustrating failures every time I attempted to use the standard tools. Today I succeeded, via new software, hence the publication of my notes. Who wants to read stories about all the things that do not work?

Following is my experience. I began by following the directions in How to back up a Windows Subsystem for Linux (WSL) distribution, published 18 Feb 2021 by Windows Central. It is a nice story, but I've tried this stuff on a variety of computers, and this is not Microsoft’s most robust code base. Read on and I will tell you of reality as I found it.

WSL Command-Line Options

First let's look at the command-line options for the wsl command.

cmd.exe

Microsoft Windows [Version 10.0.19044.1415]
(c) Microsoft Corporation. All rights reserved. 

C:\Users\Mike Slinn> wsl --help
Copyright (c) Microsoft Corporation. All rights reserved.
  For privacy information about this product please visit https://aka.ms/privacy.

  Usage: wsl.exe [Argument] [Options...] [CommandLine]

  Arguments for running Linux binaries:

  If no command line is provided, wsl.exe launches the default shell.

  --exec, -e <CommandLine>
  Execute the specified command without using the default Linux shell.

  --shell-type <Type>
  Execute the specified command with the provided shell type.

  Types:
  standard
  Execute the specified command using the default Linux shell.

  login
  Execute the specified command using the default Linux shell as a log-in shell.

  none
  Execute the specified command without using the default Linux shell.

  --
  Pass the remaining command line as-is.

  Options:
  --cd <Directory>
  Sets the specified directory as the current working directory.
  If ~ is used the Linux user's home path will be used. If the path begins
  with a / character, it will be interpreted as an absolute Linux path.
  Otherwise, the value must be an absolute Windows path.

  --distribution, -d <Distro>
  Run the specified distribution.

  --user, -u <UserName>
  Run as the specified user.

  --system
  Launches a shell for the system distribution.

  Arguments for managing Windows Subsystem for Linux:

  --help
  Display usage information.

  --debug-shell
  Open a WSL2 debug shell for diagnostics purposes.

  --install [Distro] [Options...]
  Install a Windows Subsystem for Linux distribution.
  For a list of valid distributions, use 'wsl.exe --list --online'.

  Options:
  --no-launch, -n
  Do not launch the distribution after install.

  --web-download
  Download the distribution from the internet instead of the Microsoft Store.

  --mount <Disk>
  Attaches and mounts a physical or virtual disk in all WSL 2 distributions.

  Options:
  --vhd
  Specifies that <Disk> refers to a virtual hard disk.

  --bare
  Attach the disk to WSL2, but don't mount it.

  --name <Name>
  Mount the disk using a customised name for the mount-point.

  --type <Type>
  Filesystem to use when mounting a disk, if not specified defaults to ext4.

  --options <Options>
  Additional mount options.

  --partition <Index>
  Index of the partition to mount, if not specified defaults to the whole disk.

  --set-default-version <Version>
  Changes the default install version for new distributions.

  --shutdown
  Immediately terminates all running distributions and the WSL 2
  lightweight utility virtual machine.

  --status
  Show the status of Windows Subsystem for Linux.

  --unmount [Disk]
  Unmounts and detaches a disk from all WSL2 distributions.
  Unmounts and detaches all disks if called without argument.

  --update
  Update the Windows Subsystem for Linux package.

  Options:
  --web-download
  Download the update from the internet instead of the Microsoft Store.

  --pre-release
  Download a pre-release version if available. Implies --web-download.

  --version, -v
  Display version information.

  Arguments for managing distributions in Windows Subsystem for Linux:

  --export <Distro> <FileName> [Options]
  Exports the distribution to a tar file.
  The filename can be - for standard output.

  Options:
  --vhd
  Specifies that the distribution should be exported as a .vhdx file.

  --import <Distro> <InstallLocation> <FileName> [Options]
  Imports the specified tar file as a new distribution.
  The filename can be - for standard input.

  Options:
  --version <Version>
  Specifies the version to use for the new distribution.

  --vhd
  Specifies that the provided file is a .vhdx file, not a tar file.
  This operation makes a copy of the .vhdx file at the specified install location.

  --import-in-place <Distro> <FileName>
  Imports the specified .vhdx file as a new distribution.
  This virtual hard disk must be formatted with the ext4 filesystem type.

  --list, -l [Options]
  Lists distributions.

  Options:
  --all
  List all distributions, including distributions that are
  currently being installed or uninstalled.

  --running
  List only distributions that are currently running.

  --quiet, -q
  Only show distribution names.

  --verbose, -v
  Show detailed information about all distributions.

  --online, -o
  Displays a list of available distributions for install with 'wsl.exe --install'.

  --set-default, -s <Distro>
  Sets the distribution as the default.

  --set-version <Distro> <Version>
  Changes the version of the specified distribution.

  --terminate, -t <Distro>
  Terminates the specified distribution.

  --unregister <Distro>
  Unregisters the distribution and deletes the root filesystem.

Backing Up With WSL

Now let's use the wsl command to back up the Ubuntu VM in WSL2.

cmd.exe

C:\Users\Mike Slinn> wsl --export Ubuntu ubuntuBear_2021-01-10.tar

Well, well, it backed up without any problem in about an hour! Color me ~~surprised~~happy. File size was about 50 GB.

Alright, let's try importing the image now. We'll locate the new image at f:\ubuntuBear. It used to be that WSL did not support moving or installing a distro to non-system drives. No longer!

cmd.exe

C:\Users\Mike Slinn> wsl --import Ubuntu f:\ubuntuBear ubuntuBear_2021-01-10.tar
A distribution with the supplied name already exists.

The error message A distribution with the supplied name already exists makes sense because I backed up a WSL2 instance called Ubuntu and instance names must be unique. Let's import under the name UbuntuBear:

cmd.exe

C:\Users\Mike Slinn> wsl --import UbuntuBear f:\ubuntuBear ubuntuBear_2021-01-10.tar
Unspecified error

Ahh, the dreaded Unspecified error that wsl import is infamous for. I found a few potential solutions, detailed below; I show the most desirable solution first.

Solution 1: Importing the VDHX File

Importing the original vhdx file directly instead of creating and importing the tar file has been reported to work by several people. This is a more desirable solution because it requires less work and is faster. Note that the ^ character is a DOS command-line continuation character, much like the Bash \ character.

cmd.exe

C:\Users\Mike Slinn> wsl --import Ubuntu-20.04-d^
  D:\WSL\Ubuntu-22.04\^
  %HOMEDRIVE%%HOMEPATH%\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu20.04onWindows_79rhkp1fndgsc\LocalState\ext4.vhdx^
  --vhd

Solution 2: Updating Linux Kernel

The following was written by ken-cdit:

Had the same issue. I exported two WSL 2 distros and then did a clean install of my Windows OS. "Unspecified error" came up when I tried to import them.

I fixed it by downloading the Linux kernel update package referred to at https://docs.microsoft.com/en-us/windows/wsl/install-win10 and then running the command wsl --set-default-version 2.

After this, my two distros imported without error.

I had the whole thing still in the console so here is a screencap...

Solution 3: LxRunOffline

Installing LxRunOffline

Google brought me to a potential solution, LxRunOffline, which provided me limited success.

I downloaded the binaries in LxRunOffline-v3.5.0-msvc.zip directly from GitHub into C:\Program Files\LxRunOffline.

Now add C:\Program Files\LxRunOffline to the Windows PATH:

cmd.exe

C:\Users\Mike Slinn> setx PATH "%PATH%;C:\Program Files\LxRunOffline"

SUCCESS: Specified value was saved. %}

I then ran the following in an administrative shell to register the DLL:

cmd.exe

C:\Users\Mike Slinn> regsvr32 "C:\Program Files\LxRunOffline\LxRunOfflineShellExt.dll"

LxRunOffline Help Info

Let's progressively discover how this command-line program can be used. First I'll just type the command name, which causes the program to list its top-level actions.

cmd.exe

C:\Users\Mike Slinn> LxRunOffline
[ERROR] No action is specified.

Supported actions are:
    l, list            List all installed distributions.
    gd, get-default    Get the default distribution, which is used by bash.exe.
    sd, set-default    Set the default distribution, which is used by bash.exe.
    i, install         Install a new distribution.
    ui, uninstall      Uninstall a distribution.
    rg, register       Register an existing installation directory.
    ur, unregister     Unregister a distribution but not delete the installation directory.
    m, move            Move a distribution to a new directory.
    d, duplicate       Duplicate an existing distribution in a new directory.
    e, export          Export a distribution"s filesystem to a .tar.gz file, which can be imported by the "install" command.
    r, run             Run a command in a distribution.
    di, get-dir        Get the installation directory of a distribution.
    gv, get-version    Get the filesystem version of a distribution.
    ge, get-env        Get the default environment variables of a distribution.
    se, set-env        Set the default environment variables of a distribution.
    ae, add-env        Add to the default environment variables of a distribution.
    re, remove-env     Remove from the default environment variables of a distribution.
    gu, get-uid        Get the UID of the default user of a distribution.
    su, set-uid        Set the UID of the default user of a distribution.
    gk, get-kernelcmd  Get the default kernel command line of a distribution.
    sk, set-kernelcmd  Set the default kernel command line of a distribution.
    gf, get-flags      Get some flags of a distribution. See https://docs.microsoft.com/en-us/previous-versions/windows/desktop/api/wslapi/ne-wslapi-wsl_distribution_flags for details.
    sf, set-flags      Set some flags of a distribution. See https://docs.microsoft.com/en-us/previous-versions/windows/desktop/api/wslapi/ne-wslapi-wsl_distribution_flags for details.
    s, shortcut        Create a shortcut to launch a distribution.
    ec, export-config  Export configuration of a distribution to an XML file.
    ic, import-config  Import configuration of a distribution from an XML file.
    sm, summary        Get general information of a distribution.
    version            Get version information about this LxRunOffline.exe.

Alright, let's get information about the i (install) option.

cmd.exe

C:\Users\Mike Slinn> LxRunOffline i
[ERROR] the option '-d' is required but missing

Options:
  -n arg                Name of the distribution
  -d arg                The directory to install the distribution into.
  -f arg                The tar file containing the root filesystem of the
                        distribution to be installed. If a file of the same
                        name with a .xml extension exists and "-c" isn"t
                        specified, that file will be imported as a config file.
  -r arg                The directory in the tar file to extract. This argument
                        is optional.
  -c arg                The config file to use. This argument is optional.
  -v arg (=2)           The version of filesystem to use, latest available one
                        if not specified.
  -s                    Create a shortcut for this distribution on Desktop.

I'll use the -d option to specify the directory to install into, as well as the -f and -n options.

LxRunOffline Import

I feel brave, let's try importing for real now:

cmd.exe

C:\Users\Mike Slinn> LxRunOffline i -d f:/ubuntuBear -n UbuntuBear -f ubuntuBear_2021-01-10.tar
[ERROR] The distro "UbuntuBear" already exists.

Some debris remains from the failed import (remember the Unspecified error a moment ago?). I will delete the b0rked UbuntuBear instance in a moment. Let's call this newly cloned linux instance UbuntuBear2.

cmd.exe

C:\Users\Mike Slinn> LxRunOffline i -d f:/ubuntuBear -n UbuntuBear2 -f ubuntuBear_2021-01-10.tar
[WARNING] Ignoring an unsupported file "var/lib/docker/volumes/backingFsBlockDev" of type 0060000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-17041-8893592-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10716-846285-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1899-75308257-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10151-8749190-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-18789-129803847-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-811-127664322-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12447-115564106-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-24480-65839207-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-23230-107496852-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-23230-107496852-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-24480-65839207-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-14211-5836039-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19778-110293600-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1969-34761241-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-17041-8893592-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1899-75308257-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-7642-17996-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10151-8749190-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19082-8772885-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12857-5829248-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-26692-70009588-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12447-115564106-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-22582-1181861-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-14211-5836039-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1969-34761241-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-7642-17996-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10716-846285-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-11424-57667328-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-28497-49814437-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-18789-129803847-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-2234-80045-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19082-8772885-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12857-5829248-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-22582-1181861-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-26692-70009588-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10990-5814132-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-811-127664322-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10990-5814132-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-11424-57667328-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-2234-80045-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19778-110293600-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-28497-49814437-in" of type 0010000.
[WARNING] Ignoring an unsupported file "dev/random" of type 0020000.
[WARNING] Ignoring an unsupported file "dev/tty" of type 0020000.
[WARNING] Ignoring an unsupported file "dev/full" of type 0020000.
[WARNING] Ignoring an unsupported file "dev/urandom" of type 0020000.
[WARNING] Ignoring an unsupported file "dev/ptmx" of type 0020000.
[WARNING] Ignoring an unsupported file "dev/zero" of type 0020000.
[WARNING] Ignoring an unsupported file "dev/console" of type 0020000.
[WARNING] Ignoring an unsupported file "dev/null" of type 0020000.
[WARNING] Ignoring an unsupported file "dev/mapper/control" of type 0020000.
[ERROR] Couldn"t create the file "\\?\f:\ubuntuBear\rootfs\home\mslinn\.atom\packages\markdown-preview-plus\spec\fixtures\subdir\�cc�nt�d.md".
Reason: The file exists.

Most of the warnings do not seem to be important. I do not care about Docker anyway. Also, now I know that temporary files and dev nodes should all be deleted before running this program. Even better, the program that created the tar should be modified to not attempt to replicate any temporary files.

I don't understand the problem with the atom package, but I'll try to delete all of the atom settings from the tar, along with the other problematic directories, and then retry.

Cleaning Up WSL

First let's see the VMs registered with WSL:

Shell

C:\Users\Mike Slinn> wsl -l
Windows Subsystem for Linux Distributions:
Ubuntu (Default)
UbuntuBear
UbuntuBear2

Let's delete the debris remaining from the failed UbuntuBear and UbuntuBear2 imports:

Shell

C:\Users\Mike Slinn> wsl --unregister UbuntuBear
Unregistering... 

C:\Users\Mike Slinn> wsl --unregister UbuntuBear2
Unregistering...

Attempting to delete the directory created by LxRunOffline hit a corrupted directory.

Shell

C:\Users\Mike Slinn> del /s /q f:\ubuntuBear
Deleted file - f:\ubuntuBear\rootfs\init
Deleted file - f:\ubuntuBear\rootfs\lib
Deleted file - f:\ubuntuBear\rootfs\lib32
Deleted file - f:\ubuntuBear\rootfs\lib64
Deleted file - f:\ubuntuBear\rootfs\libx32
Deleted file - f:\ubuntuBear\rootfs\sbin
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.wget-hsts
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.Xauthority
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zcompdump
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zcompdump-Bear-5.8
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zcompdump-localhost-5.8
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zshenv
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zshrc
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zshrc.pre-oh-my-zsh
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zsh_history
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\ancient_warmth_workspace.code-workspace
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\bear2.zip
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\bear3.zip
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\bearDirs.tar
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\dead.letter
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\django_bash_completion
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\jekyll_workspace.code-workspace
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\msp.txt
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\nodesource_setup.sh
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\package-lock.json
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\package.json
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\worldPeaceMusicCollective.png
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\worldPeaceMusicCollectiveBordered.png
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.vscode-server\extensions\wix.vscode-import-cost-2.15.0\package.json
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.vscode-server\extensions\wix.vscode-import-cost-2.15.0\pom.xml
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.vscode-server\extensions\wix.vscode-import-cost-2.15.0\README.md
The file or directory is corrupted and unreadable.

“The file or directory is corrupted and unreadable.” That needs to be dealt with right away! I fixed the directory errors in drive F like this:

cmd.exe

C:\Program Files> chkdsk /F F:
The type of the file system is NTFS.

Chkdsk cannot run because the volume is in use by another
process.  Chkdsk may run if this volume is dismounted first.
ALL OPENED HANDLES TO THIS VOLUME WOULD THEN BE INVALID. y

Chkdsk cannot dismount the volume because it is a system drive or
there is an active paging file on it.  Would you like to schedule
this volume to be checked the next time the system restarts? (Y/N) y

This volume will be checked the next time the system restarts.

I rebooted the system, and it fixed the errors on drive F:.

Pruning the tar

First lets back up the tar that we want to prune.

Shell

$ pushd '/mnt/c/Users/Mike Slinn/'

$ ls -alF *.tar
-rwxr--r-- 1 mslinn mslinn   524482560 Jan 10 18:45 'bear_ubuntu_2021-01-10.tar'*
-rwxr--r-- 1 mslinn mslinn 51464028160 Jan 10 21:11 'ubuntuBear_2021-01-10.tar'* 

$ cp ubuntuBear_2021-01-10.tar ubuntuBearPruned_2021-01-10.tar

Now lets try to prune out all the problematic, and unnecessary, files.

Shell

$ tar -f ubuntuBearPruned_2021-01-10.tar \
  --delete dev/* \
  --delete tmp/* \
  --delete home/mslinn/.atom/* \
  --delete var/lib/docker/* \
  --delete var/sitesUbuntu/* \
  --delete var/work/* \
  --delete var/tmp/*
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.fuseoverlayfs.opaque'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.fuseoverlayfs.opaque'
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: dev/*: Not found in archive
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: tmp/*: Not found in archive
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: home/mslinn/.atom/*: Not found in archive
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: var/lib/docker/*: Not found in archive
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: var/sitesUbuntu/*: Not found in archive
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: var/work/*: Not found in archive
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: var/tmp/*: Not found in archive
tar: Exiting with failure status due to previous errors

I think the error messages just indicate that the tar was made by BSD-TAR, while Ubuntu uses GNU-TAR. I installed bsdtar like this:

Shell

$ yes | sudo apt-get install libarchive-tools
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  libarchive-tools
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 57.1 kB of archives.
After this operation, 207 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu hirsute/universe amd64 libarchive-tools amd64 3.4.3-2 [57.1 kB]
Fetched 57.1 kB in 0s (167 kB/s)
Selecting previously unselected package libarchive-tools.
(Reading database ... 244498 files and directories currently installed.)
Preparing to unpack .../libarchive-tools_3.4.3-2_amd64.deb ...
Unpacking libarchive-tools (3.4.3-2) ...
Setting up libarchive-tools (3.4.3-2) ...
Processing triggers for man-db (2.9.4-2) ...

I tried again using bsdtar, which does not support GNU tar's --delete option. Unfortunately, I only got a 1KB file out, no matter what I did.

Shell

$ bsdtar -cvf ubuntuBearPruned2_2021-01-10.tar \
  --exclude 'dev/*' \
  --exclude 'tmp/*' \
  --exclude 'home/mslinn/.atom/*' \
  --exclude 'var/lib/docker/*' \
  --exclude 'var/sitesUbuntu/*' \
  --exclude 'var/work/*' \
  --exclude 'var/tmp/*' \
  @ubuntuBear_2021-01-10.tar

Maybe there is a bug. Maybe there is bad documentation. I do not think I made an error. Life is short. Bugs are rampant. Illegitimi non carborundum!

I give up on this direction. Let's try to win some other way!

Success Duplicating the Ubuntu Instance

Let’s try duplicating the Ubuntu instance with LxRunOffline. First the Ubuntu VM must be terminated.

cmd.exe

C:\Users\Mike Slinn> wsl -t Ubuntu

C:\Users\Mike Slinn> LxRunOffline duplicate -n Ubuntu -N UbuntuWsl2 -d f:\UbuntuWsl2

The above ran for a couple of hours and concluded without error. LxRunOffline duplicate automatically registers the new instance.

The F:\UbuntuWsl2directory was 239 GB. That's a fair-sized VM. The directory only contains one file, called ext4.vhdx.

Let's see the Ubuntu instances that are currently set:

cmd.exe

C:\Users\Mike Slinn> wsl --list
Windows Subsystem for Linux Distributions:
Ubuntu (Default)
UbuntuWsl2

😁

That is what I wanted to see! I started the new UbuntuWsl2 Ubuntu instance like this:

cmd.exe

C:\Users\Mike Slinn> wsl -d UbuntuWsl2

To set the default Ubuntu instance I typed:

cmd.exe

C:\Users\Mike Slinn> wsl --setdefault UbuntuWsl2

Let's verify that worked:

cmd.exe

C:\Users\Mike Slinn> wsl --list
Windows Subsystem for Linux Distributions:
UbuntuWsl2 (Default)
Ubuntu

I then deleted the huge tar files that I had created in earlier steps. I never needed them because I used LxRunOffline duplicate.

Ubuntu on 500GB USB3 SSD

I have a Sandisk Extreme 500GB USB3 SSD drive. This tiny, light, and very portable storage device should be perfect for holding my Ubuntu development system. How cool it would be to be able to drop it into a small pocket!

Better yet, the performance of portable USB3 SSD drives blows away traditional hard drives.

For example, assume that your SSD drive appears as drive X. Also assume that LxRunOffline.exe and LxRunOffline.dll have been copied to the root directory of the SSD drive. To register your portable Ubuntu you would simply type:

cmd.exe

C:\> X:LxRunOffline register X:\MyUbuntu -n UbuntuMSlinn

You can walk up to any recently updated Windows 10 machine, plug your tiny USB3 SSD drive into a USB3 or USB3.1 port, run the registration command, and boom! you are productive with great storage performance.

😁

Duplicating to SSD

Again, the Ubuntu VM must be terminated.

cmd.exe

C:\Users\Mike Slinn> wsl -t Ubuntu

C:\Users\Mike Slinn> LxRunOffline duplicate -n Ubuntu -N UbuntuWsl2Extreme -d I:\UbuntuWsl2Extreme

The above ran for a couple of hours and concluded without error. LxRunOffline duplicate automatically registers the new instance.

Let's see the Ubuntu instances that are currently set:

cmd.exe

C:\Users\Mike Slinn> wsl --list
Windows Subsystem for Linux Distributions:
Ubuntu (Default)
UbuntuWsl2
UbuntuWsl2Extreme

😁

LxRunOffline Update 2023-04-25

Arman Mazloumzadeh posted the following:

I finally resolved this amiss during the following steps (Windows 10.0.19045 Build 19045):

Running wsl.exe --import Ubuntu A:\wsl\Ubuntu A:\wsl\Ubuntu.tar = > Unspecified error

Running LxRunOffline.exe install -n Ubuntu -d A:\wsl\Ubuntu -f A:\wsl\Ubuntu.tar => [ERROR] Couldn't get the value "DistributionName" of the registry key "Software\Microsoft\Windows\CurrentVersion\Lxss\AppxInstallerCache"

Removing the Software\Microsoft\Windows\CurrentVersion\Lxss\AppxInstallerCache registry key
Running LxRunOffline.exe install -n Ubuntu -d A:\wsl\Ubuntu -f A:\wsl\Ubuntu.tar => [ERROR] Couldn't get the value "DistributionName" of the registry key "Software\Microsoft\Windows\CurrentVersion\Lxss\TryStoreWSL"

Removing the Software\Microsoft\Windows\CurrentVersion\Lxss\TryStoreWSL registry key
Running LxRunOffline.exe install -n Ubuntu -d A:\wsl\Ubuntu -f A:\wsl\Ubuntu.tar => ✔

Running ubuntu.exe config --default-user arman => ✔

done 🎉

Starting and Stopping Portable VMs

You can start the WSL VM called UbuntuMSlinn by using wsl -d:

cmd.exe

C:\> wsl -d UbuntuMSlinn

You can stop it by using wsl -t:

cmd.exe

C:\> wsl -t UbuntuMSlinn

Microsoft WSL Issue Reporting

In the course of writing this blog post I found the web page for reporting a WSL issue, so that Microsoft can look at it.

AI / ML System Behavior Reflects the Society That Produced It

2021-12-26T00:00:00-05:00

Artificial intelligence systems, including machine learning systems, are primarily software-driven. Like all software, AI and ML systems reflect the societies that produced them. Conway’s Law explains how the internal organization of software reflects the organization that created it.

I postulate that the behavior of AI & ML systems reflect the society that the systems are embedded in. Do you find the AI systems are instrusive or exploitive? Look in the collective mirror for the society it was created by.

If the populace is viewed as merely something for corporations to exploit, and this view is accepted at enough levels in society, that becomes the status quo. In my opinion, the USA in 2022 has definitely reached that point.

Conway’s Law

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.

– Melvin E. Conway, 1967

Allan Kelly’s Corollary

Organisational design is system design.

– Allan Kelly, 2005

The The Mirroring Hypothesis: Theory, Evidence and Exceptions by Lyra J. Colfer and Carliss Y. Baldwin was published by the Harvard Business School in 2016.

A large-scale system is one that stretches across time and space. A large organization similarly stretches across time and space. Conway’s Law ties the two. How we organize defines how we think collectively, and thus what we make collectively...

... To build software, first build a community...

Software is essentially about people, and the larger-scale we make it, the more that truth becomes visible. One program reflects how one person thinks. A large-scale application reflects how many people think together.

– Pieter Hintjes, Sex in Title, and Other Stories, Dec 16, 2013

Do you even care? If so, please be the change you want to see in the world.

We but mirror the world. All the tendencies present in the outer world are to be found in the world of our body. If we could change ourselves, the tendencies in the world would also change. As a man changes his own nature, so does the attitude of the world change towards him. This is the divine mystery supreme. A wonderful thing it is and the source of our happiness. We need not wait to see what others do. – From Mahatma Gandhi

Spring-Breezifier: Solving COVID-19 With HVAC

2021-12-22T00:00:00-05:00

When the COVID-19 pandemic began, the world was unprepared and no one knew how the virus was transmitted. As the medical profession lurched awkwardly into action, it gave advice that later turned out to be incorrect. As time went by, important details about how the virus is transmitted were revealed, and the previous medical advice had become politicized. The new advice has not yet been generally adopted as policy and is currently unknown to most people.

For every complex problem there is an answer that is clear, simple, and wrong.

– H. L. Mencken 1880-1956.

About This Article

This article's intent is to show how the COVID-19 pandemic could be dealt with simply, cheaply, and with a minimum of aggravation by raising awareness of the potential role of HVAC (heating, ventilation and air conditioning) equipment.

The author is an electrical engineer with no health-related or HVAC-related qualifications. However, he can read and write, ideas often come to mind when presented with new information.

After initially publishing this blog post, an old friend showed me a YouTube video made by a consulting civil engineer who specializes in this topic. You can skip to it if you are impatient and want to know in-depth technical specifics.

All the new information referenced in this article is generally available, from qualified and reputable sources, and this article just disseminates it.

This article concludes with actionable suggestions.

Update 2023-06-08

[The] federal agency has set a target - five air changes per hour - for how much rooms and buildings should be ventilated...

it’s easy to see the guidance only in the context of Covid-19, it will help with many other airborne hazards like wildfire smoke, allergens and other infectious diseases, such as the flu...

“If they had broadcast and implemented these changes at the beginning, there never would have been a pandemic”, said Kimberly Prather, an atmospheric chemist at the University of California at San Diego and the Scripps Institution of Oceanography.

– From CDC sets first target for indoor air ventilation to prevent spread of Covid-19

COVID-19 Is Primarily Transmitted Via Aerosols

The most significant bit of information about COVID-19, which was not generally accepted at the beginning of the pandemic, is that it is mostly transmitted via aerosols; that is, via small airborne droplets only a few microns in diameter. (One micron is a millionth of a meter, or one twenty-five thousandth of an inch). Aerosol particles can hang in the air for hours and travel hundreds of feet.

The Centers for Disease Control and Prevention (CDC) officially recently officially recognized that SARS-CoV-2 (the virus that causes COVID-19) is airborne, meaning it is highly transmissible through the air.

Exposure occurs in three principal ways:

inhalation of very fine respiratory droplets and aerosol particles
deposition of respiratory droplets and particles on exposed mucous membranes in the mouth, nose, or eye by direct splashes and sprays
touching mucous membranes with hands that have been soiled either directly by virus-containing respiratory fluids or indirectly by touching surfaces with virus on them.

People release respiratory fluids during exhalation (e.g., quiet breathing, speaking, singing, exercise, coughing, sneezing) in the form of droplets across a spectrum of sizes. These droplets carry virus and transmit infection. To stay healthy, avoid these droplets.

The largest droplets settle out of the air rapidly, within seconds to minutes. The smallest very fine droplets, and aerosol particles formed when these fine droplets rapidly dry, are small enough that they can remain suspended in the air for minutes to hours.

–From the CDC Scientific Brief: SARS-CoV-2 Transmission, Updated May 7, 2021.

The idea that physical distancing of 6 feet (2 meters) might help keep people healthy is an example of bad science from a study 80 years ago that was not debunked until recently.

Transmission of COVID-19 from inhalation of virus in the air can occur at distances greater than six feet. Particles from an infected person can move throughout an entire room or indoor space. The particles can also linger in the air after a person has left the room – they can remain airborne for hours in some cases.

–From the US Environmental Protection Agency (EPA): Indoor Air and Coronavirus (COVID-19), web page was updated December 15, 2021.

Distance between people is not a significant transmissibility factor. Imagine two people, back to back, one (downwind) very sick with COVID-19, and the other (healthy) upwind, and the wind is blowing at 20 mph (32 km/h). The sick person will not infect the healthy person, unless they exchange positions. Just control the air that is breathed, and all airborne viruses such as COVID-19 will be controled.

What we learned during the pandemic is that aerosols were one of the main drivers in spreading the COVID-19 virus and that their importance in the transmission of many other respiratory pathogens has been systematically underappreciated.

– Dr. Robert Schooley, Professor in the Department of Medicine at the University of California San Diego (November 22, 2021), quoted from COVID Gets Airborne – UC San Diego develops computer model to aid understanding of how viruses travel through the air

Controlling Virus Infiltration

We would do well to remember that the COVID-19 virus causes sickness. Without exposure to the virus people would not get sick. The risk that a person might catch the disease is directly related to the number of viruses inhaled per unit of time. For example, the more viruses someone inhales in an hour, the more likely they will get sick.

The amount of virus necessary to make a person sick is called the infectious dose. All the countermeasures that have been used (hand washing, physical distancing, masks, curfews, etc.) are intended to reduce the number of viruses people are exposed to per unit time. Pandemic policy treats countermeasures as proxies for virus transmission vectors. By not updating pandemic policy as new information becomes available, we loose the ability to control the spread of the virus.

HEPA Filters

HEPA filters on HVAC equipment are designed to remove these aerosols. They come in a huge variety of sizes, shapes and air flow capacities. If you could just breathe the pure air emitted from a fan that had a HEPA filter, you would not get sick from any airborne virus. Any number of people could gather safely, if each of them received an adequate supply of pure air in their face at all times.

HEPA is a type of pleated mechanical air filter. It is an acronym for “high efficiency particulate air filter” (as officially defined by the U.S. Dept. of Energy). This type of air filter can theoretically remove at least 99.97% of dust, pollen, mold, bacteria, and any airborne particles with a size of 0.3 microns (µm).

–US Environment Protection Agency

HEPA filters are inexpensive and are readily available. They are not new. In fact, HEPA filter technology was created in the 1940s by the US Army Chemical Corps and National Defense Research Committee as part of the Manhattan Project. HEPA technology was declassified after World War II and became available for commercial and personal use. HEPA filters play a key role in the research and development of modern pharmaceuticals, aerospace engineering, and computer chip manufacturing.

HEPA air cleaners, which remain little-used in Canadian hospitals, are a cheap and easy way to reduce risk from airborne pathogens.

–From Nature Magazine, October 6, 2021: Real-world data show that filters clean COVID-causing virus from air – An inexpensive type of portable filter efficiently screened SARS-CoV-2 and other disease-causing organisms from hospital air.

It is easy to attach a HEPA filter to a fan, or to replace an existing HVAC filter with a HEPA filter. Here is a very scientific approach to a home-made solution.

Dr. Jeffrey E. Terrell, director of the Michigan Sinus Center, demonstrates how to build an air purifier with a HEPA filter for about $25 with parts from your local hardware store.

One of the comments on the above video, from Rick Rude in 2014, was:

Better to pull air thru the filter, if you push air through the filter dust that builds up will get blown off and go back into the room. Also, if the fan is pulling air through, the filter will stick to the fan and won't need as much tape to hold it on. Always handle used filters with care because handling them rough will release concentrated dust back into your air. If possible, always take air cleaners or vacuum cleaners outside to change the filters or bags.

MERV Filters

MERV is a graduated standard; MERV 7 is the minimum standard for furnaces, while MERV 13 is the highest.

MERV 13 filters are more efficient at removing large particles from the air, while HEPA filters are more efficient at removing small particles from the air. MERV 13 filters can remove up to 99.97% of particles from the air, while HEPA filters can remove up to 99.99% of particles from the air.

How to Build a Corsi-Rosenthal Box.

How to Identify and Rectify Poorly Ventilated Indoor Spaces Using Engineering Controls

David Elfstrom, P. Eng., is a independent civil engineer based in Simcoe, Ontario who consults on the efficient use of energy in buildings. During the pandemic he has been drawing attention to the importance of ventilation, filtration, and overall indoor environmental quality. Earlier this year Mr. Elfstrom completed a ventilation and air pathways assessment of an apartment building in outbreak for a public health unit. This presentation was part of Passive Buildings Canada's 2021 Annual General Meeting.

Mr. Elfstrom also co-authored the Masks4Canada Room Ventilation/Filtration Guide and Tip Sheet.

N95, KF94, FFP2, and 9152 Masks and Their Cousins

If you need to move about in a crowd, or enter a space that contained people a few hours ago, you need to wear a properly fitting face mask that filters out the tiny aerosol particles that contain the COVID-19 virus. N95 masks and their cousins (KN95, KF94, etc.) do the job because they filter particles as small as 0.3 microns. This is similar to the particle size filtered by HEPA filters. Caution: KN95 is a self-reported test standard, and lacks strict government regulation by China, resulting in many underperforming and often flat-out fake masks.

The US National Personal Protective Technology Laboratory (NPPTL), which is part of the US National Institute for Occupational Safety and Health (NIOSH), which itself is part of the CDC, evaluated various masks and published the results as NPPTL Respirator Assessments to Support the COVID-19 Response.

The following types of masks do not reliably filter tiny aerosols, so they do not adequately protect you from COVID-19 variants such as Omicron:

Surgical masks (what most people wear these days)
Masks with activated charcoal
Vented masks
Cloth masks
Gaiters
Ill-fitting masks that do not cover and seal the mouth and nose

Inoculations and Pills

Inoculations, including boosters, have proven to be very helpful. Pills from Pfizer Inc. and Merck & Co. for people sick with COVID-19 will also be very helpful once they become available. However, pills will only help sick people get better, they will not prevent getting sick.

Until everyone in the entire world has somehow acquired an effective level of antibodies, COVID-19 will continue to mutate.

The Only Solution Available Today

Inoculations and masks are good, but they are not perfect preventative measures against COVID-19. Protection against infection is not 100%. At present, the only preventative measure against COVID-19 that would allow people to safely mingle in person would be to guarantee continuous streams of pure air directed individually at each person's face.

This measure would also prevent the spread of all other diseases and their variants that are primarily spread via aerosols, including coronoviruses, colds, flus, tuberculosis, MERS-CoV, measles, ebola and chickenpox. Pollen, dust and other particulates would also be removed, so athsma sufffers would feel relief from extended periods breathing pure air.

The solution to living with COVID-19 is simple, safe, inexpensive and is not disruptive:

Live your social life with a pure breeze in your face.

Life as we knew it before the pandemic started 2 years ago could resume, mostly.

HVAC Upgrades Are Key

The engineering group that encompasses HVAC is ASHRAE. ASHRAE was formed as the American Society of Heating, Refrigerating and Air-Conditioning Engineers by the merger in 1959 of American Society of Heating and Air-Conditioning Engineers (ASHAE), founded in 1894 and The American Society of Refrigerating Engineers (ASRE), founded in 1904.

ASHRAE offers Core Recommendations for Reducing Airborne Infectious Aerosol Exposure. MERV 13 or better levels of performance are recommended, and HEPA filters surpass that specifation. In fact, all HEPA filters have a rating of a MERV 17 or higher.

Entrepreneurs

This represents an opportunity for entrepreneurs.

Restaurants

Imagine a restaurant that stays open with near-normal seating capacity because each table is equipped with a fan and ducting that blows a gentle breeze of pure air directly into the face of every patron. The air would recirculate within the restaurant, so there would be no need to upgrade the existing HVAC system because the HEPA filters located at each table would continuously clean the air within the restaurant. Those filters would need to be cleaned and disinfected daily. The wait staff would all wear N95 or similar masks, however each cashier could instead enjoy their own gentle breeeze of pure air.

Cost to equip each seat in the restaurant could be less than $50. For do-it-yourself owners, the cost could approach $10 per seat. For example, a 100-seat restaurant might be able to retrofit 75 of those seats at a cost of less than $3750. They would never have to close again due to any pandemic caused by airborne aerosols... provided, of course, that local regulations recognized this approach.

Retail Stores and Office Buildings

Enclosing sales clerks in stores behind clear plastic walls is wrong because it decreases air circulation. Instead, each those staff members should have a dedicated ducted fan that blows a gentle breeze of pure air into their face at all times.

Similarly, queues of people awaiting their turn at checking out should have fans blowing pure air at their heads. The air would recirculate within the store, there would be no need to upgrade the existing HVAC system because the HEPA filters would continuously clean the air within the building.

Spring-Breezifier Is Offered Into the Public Domain

I offer this idea to the world; I am an idea machine so I cannot act on most of them. Just for fun, let's call this idea the Spring-Breezifier.

Designing and building Spring-Breezifiers seems like it might be a good high school or youth group project. HVAC installers in particular should be able to make short work of this idea; go ahead and make lots of money providing pure airflows by building custom Spring-Breezifiers for your customers, we all thank you!

If anyone would like to talk to me about their Spring-Breezifier project, I would be happy to speak with them.

Next Steps

Medical HVAC specialists could suggest the necessary cubic feet per minute of air required per person sufficient to guarantee a continuous stream of pure air, and provide guidelines for designing ductwork to direct that air effectively.
Build and test prototypes. Most of the effort will be in designing and building the ducting and/or the scaffolding. Adding inner baffles would raise the cost and weight, lower the noise and make the airflow less turbulent. Perhaps Spring-Breezifiers could be built entirely from off-the-shelf parts for some or even most installations. Large-format 3D printers for special circumstances might be helpful.
Propose the inclusion of pure air streams into public spaces as a matter of public health policy.

Disappointing Scala 3 Installation Experience

2021-05-19T00:00:00-04:00

I run ScalaCourses.com, an online Scala training web site. After years of hype, Scala 3 is now available.

Scala 3: Not Yet Ready for Production

Scala 3 was eight years in the making. You would never know that from the horrible installation process and the disappointing installation instructions.

It is going to take quite a while before Scala 3, which was known as Dotty before it was released, can be trusted in production. According to the Dotty GitHub project, the only published future milestone is v3.1.0, which has no due date. Given that Scala 3 uses an entirely new build process, and an entirely new (and nonstandard) installation process, and that the internals of the Scala compiler were almost completely replaced, I doubt that version will be stable enough for use on production projects.

Installation Choices

Installation cholices include:

Command-line Scala has limited use cases, but is nice to have around for occassional experimentation.
SBT (which features an enhanced Scala REPL) is very helpful for interactively developing code, as well as for building and testing.
IntelliJ provides the best Scala coding productivity and code quality.
VSCode has been playing catch-up but is not full-featured yet.

Installation Transcript

The following is the transcript of how I installed command-line Scala on Ubuntu 20.10 running under WSL2.

Remove Scala 2

This step is not required. Scala 2 and Scala 3 can easily co-exist on the same system because their names are different.

Shell

$ sudo apt remove scala
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  scala
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 666 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 236024 files and directories currently installed.)
Removing scala (2.13.4-400) ...
Processing triggers for man-db (2.9.3-2) ...

Install Coursier

Coursier is a multithreaded downloader for project dependencies, and it now also downloads Scala 3. SBT uses coursier internally.

Shell

$ curl -fLo cs https://git.io/coursier-cli-"$(uname | tr LD ld)"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   144  100   144    0     0    285      0 --:--:-- --:--:-- --:--:--  6000
100 57.1M  100 57.1M    0     0  3656k      0  0:00:15  0:00:15 --:--:-- 4092k 

$ mv cs ~/.local/bin/

$ chmod a+x ~/.local/bin/cs

$ cs install cs
https://repo1.maven.org/maven2/io/get-coursier/apps/maven-metadata.xml
  100.0% [##########] 1.8 KiB (8.5 KiB / s)
https://repo1.maven.org/maven2/io/get-coursier/coursier-cli_2.12/maven-metadata.xml
  No new update since 2021-03-23 14:35:16
Wrote cs
Warning: /home/mslinn/.local/share/coursier/bin is not in your PATH
To fix that, add the following line to ~/.bashrc

export PATH="$PATH:/home/mslinn/.local/share/coursier/bin" 

$  export PATH="$PATH:/home/mslinn/.local/share/coursier/bin"

Now let's test Coursier:

Shell

$ cs
Coursier 2.0.16
Usage: cs [options] [command] [command-options]

Available commands: bootstrap, channel, complete, fetch, get, install, java, java-home, launch, list, publish, resolve, setup, uninstall, update, search

Type  cs command --help  for help on an individual command

This is the Coursier help message for the install subcommand:

Shell

$ cs install --help
Command: install
Usage: cs install
--cache  <string?>
Cache directory (defaults to environment variable COURSIER_CACHE, or ~/.cache/coursier/v1 on Linux and ~/Library/Caches/Coursier/v1 on Mac)
--mode | -m  <offline|update-changing|update|missing|force>
Download mode (default: missing, that is fetch things missing from cache)
--ttl | -l  <duration>
TTL duration (e.g. "24 hours")
--parallel | -n  <int>
Maximum number of parallel downloads (default: 6)
--checksum  <checksum1,checksum2,...>
Checksum types to check - end with none to allow for no checksum validation if no checksum is available, example: SHA-256,SHA-1,none
--retry-count  <int>
Retry limit for Checksum error when fetching a file
--cache-file-artifacts | --cfa  <bool>
Flag that specifies if a local artifact should be cached.
--follow-http-to-https-redirect  <bool>
Whether to follow http to https redirections
--credentials  <host(realm) user:pass|host user:pass>
Credentials to be used when fetching metadata or artifacts. Specify multiple times to pass multiple credentials. Alternatively, use the COURSIER_CREDENTIALS environment variable
--credential-file  <string*>
Path to credential files to read credentials from
--use-env-credentials  <bool>
Whether to read credentials from COURSIER_CREDENTIALS (env) or coursier.credentials (Java property), along those passed with --credentials and --credential-file
--quiet | -q  <counter>
Quiet output
--verbose | -v  <counter>
Increase verbosity (specify several times to increase more)
--progress | -P  <bool>
Force display of progress bars
--log-changing  <bool>
Log changing artifacts
--log-channel-version | --log-index-version | --log-jvm-index-version  <bool>
Log app channel or JVM index version
--graalvm-home  <string?>
--graalvm-option  <string*>
--graalvm-default-version  <string?>
--install-dir | --dir  <string?>
--install-platform  <string?>
Platform for prebuilt binaries (e.g. "x86_64-pc-linux", "x86_64-apple-darwin", "x86_64-pc-win32")
--install-prefer-prebuilt  <bool>
--only-prebuilt  <bool>
Require prebuilt artifacts for native applications, don't try to build native executable ourselves
--repository | -r  <maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|ivy:$pattern>
Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots)
--default-repositories  <bool>
--proguarded  <bool?>
--channel  <org:name>
Channel for apps
--default-channels  <bool>
Add default channels
--contrib  <bool>
Add contrib channel
--file-channels  <bool>
Add channels read from the configuration directory
--jvm  <string?>
--jvm-dir  <string?>
--system-jvm  <bool?>
--local-only  <bool>
--update  <bool>
--jvm-index  <string?>
--repository | -r  <maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|scala-integration|scala-nightlies|ivy:$pattern|jitpack|clojars|jcenter|apache:$repo>
Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots)
--no-default  <bool>
Do not add default repositories (~/.ivy2/local, and Central)
--sbt-plugin-hack  <bool>
Modify names in Maven repository paths for sbt plugins
--drop-info-attr  <bool>
Drop module attributes starting with 'info.' - these are sometimes used by projects built with sbt
--channel  <org:name>
Channel for apps
--default-channels  <bool>
Add default channels
--contrib  <bool>
Add contrib channel
--file-channels  <bool>
Add channels read from the configuration directory
--repository | -r  <maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|scala-integration|scala-nightlies|ivy:$pattern|jitpack|clojars|jcenter|apache:$repo>
Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots)
--no-default  <bool>
Do not add default repositories (~/.ivy2/local, and Central)
--sbt-plugin-hack  <bool>
Modify names in Maven repository paths for sbt plugins
--drop-info-attr  <bool>
Drop module attributes starting with 'info.' - these are sometimes used by projects built with sbt
--channel  <org:name>
Channel for apps
--default-channels  <bool>
Add default channels
--contrib  <bool>
Add contrib channel
--file-channels  <bool>
Add channels read from the configuration directory
--env  <bool>
--disable-env | --disable  <bool>
--setup  <bool>
--user-home  <string?>
--add-channel  <string*>
(deprecated)
--force | -f  <bool>

Install Scala 3

The Scala 3 compiler and REPL are separate programs: scala3-compiler and scala3-repl.

Shell

$ cs install scala3-compiler
https://repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.0.0/scala3-compiler_3-3.0.0.pom
  100.0% [##########] 4.8 KiB (79.7 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.0.0/tasty-core_3-3.0.0.pom
  100.0% [##########] 3.5 KiB (69.5 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.0.0/scala3-library_3-3.0.0.pom
  100.0% [##########] 3.6 KiB (53.9 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.0.0/scala3-interfaces-3.0.0.pom
  100.0% [##########] 3.4 KiB (65.9 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.0.0/scala3-interfaces-3.0.0.jar
  100.0% [##########] 3.4 KiB (113.9 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.0.0/tasty-core_3-3.0.0.jar
  100.0% [##########] 71.9 KiB (192.7 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.0.0/scala3-library_3-3.0.0.jar
  100.0% [##########] 1.1 MiB (1.8 MiB / s)
https://repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.0.0/scala3-compiler_3-3.0.0.jar
  100.0% [##########] 14.7 MiB (3.7 MiB / s)
Wrote scala3-compiler 

$ cs install scala3-repl
https://repo1.maven.org/maven2/io/get-coursier/apps/maven-metadata.xml
  No new update since 2021-05-14 04:42:19
Wrote scala3-repl

Run Scala 3 REPL

The part is easy!

Shell

$ scala3-repl --version
Scala code runner version 3.0.0 -- Copyright 2002-2021, LAMP/EPFL 

$ scala3-repl
scala>

Easily Run Scala REPL With SBT

If you do not mind directories called project/ and target/ being created in your current directory, and you have already installed sbt, you can get a REPL powered by Scala 3 like this:

Shell

$ sbt "-Dsbt.version=1.5.2" ++3.0.0! console
[info] welcome to sbt 1.5.2 (Ubuntu Java 11.0.11)
  [info] loading global plugins from /home/mslinn/.sbt/1.0/plugins
  [info] loading project definition from /var/work/ancientWarmth/ancientWarmth/project
  [info] set current project to ancientwarmth (in build file:/var/work/ancientWarmth/ancientWarmth/)
  [info] Forcing Scala version to 3.0.0 on all projects.
  [info] Reapplying settings...
  [info] set current project to ancientwarmth (in build file:/var/work/ancientWarmth/ancientWarmth/)
  [info] Updating
  [info] Resolved  dependencies
  [info] Updating
  https://repo1.maven.org/maven2/org/scala-lang/scaladoc_3/3.0.0/scaladoc_3-3.0.0.pom
    100.0% [##########] 6.1 KiB (82.6 KiB / s)
  https://repo1.maven.org/maven2/org/scala-lang/scala3-tasty-inspector_3/3.0.0/scala3-tasty-inspector_3-3.0.0.pom
    100.0% [##########] 3.6 KiB (80.8 KiB / s)
  [info] Resolved  dependencies
  [info] Fetching artifacts of
  [info] Fetched artifacts of
  [info] Fetching artifacts of
  https://repo1.maven.org/maven2/org/scala-lang/scala3-tasty-inspector_3/3.0.0/scala3-tasty-inspector_3-3.0.0.jar
    100.0% [##########] 16.6 KiB (338.1 KiB / s)
  https://repo1.maven.org/maven2/org/scala-lang/scaladoc_3/3.0.0/scaladoc_3-3.0.0.jar
    100.0% [##########] 1.5 MiB (3.1 MiB / s)
  [info] Fetched artifacts of

  scala>

Thanks to @renghen for this tip.

ScalaCourses

If you want to learn how to work effectively with Scala for functional and object-oriented programming, ScalaCourses.com is your best option. The course material is suitable for Scala 2 and Scala 3. Visit ScalaCourses.com to learn how to become a proficient Scala programmer.

OCI / Docker / AWS Lambda / Django / Buildah / podman

2021-04-29T00:00:00-04:00

This blog post is a work in progress. Some of it may be incorrect, and some thoughts might lead nowhere. I am publicly posting it in this state so I can discuss it with others. This post will be improved as information becomes available.

Goal

As previously discussed, Buildah is a drop-in replacement for using docker build and a Dockerfile. Buildah’s build-using-dockerfile, or bud argument makes it behave just like docker build does.

The goal of this blog post is to use Buildah / podman to create an Open Container Initiative (OCI) container image with a Django app, including the Python 3.8 runtime installed. The Django app will start when the container is created. The code for the Django app will be stored on the local machine where its source code can be edited, and it will be mapped into the container from the host system. Changes made to the code from the host system will be immediately visible inside the container.

TODO

Background: AWS publishes Deploying Python with an AWS base image, but that does not discuss running or testing. Create a Lambda function with the console is a more complete article, but is focused on using the web browser console, using Docker, and Node.js. So many differences from the desired goal make the articles difficult to translate to AWS CLI, Buildah / podman and Python.

Talk about the AWS Lambda Runtime Interface Emulator, compare and contrast with the AWS Lambda Python Runtime Interface Client.

Compare these AWS Lambda Runtimes with other, equivalant runtimes.

OCI images are swapped in when AWS Lambda is invoked. Do larger images cost more to use? If so, discuss.

Deploy Python Lambda function with Container Image

Consider this Dockerfile, which launches a Python 3.8 command-line application in a manner compatible with AWS Lambda:

Dockerfile

FROM public.ecr.aws/lambda/python:3.8

COPY app.py ./
CMD ["app.handler"]

Following is a small Python app called app.py, which will be launched by the Dockerfile. The Python app can be run as an AWS Lambda program because it implements the handler entry point.

app.py

import sys

def handler(event, context):
    return f"Hello from AWS Lambda using Python {sys.version}!"

Build image

Buildah builds the image, just the same way that Docker would:

Shell

$ buildah bud -t hello .
STEP 1: FROM public.ecr.aws/lambda/python:3.8
Getting image source signatures
Copying blob 03ac043af787 skipped: already exists
Copying blob 420e64b38334 done
Copying blob ff259f25b075 done
Copying blob 3ff716981d54 done
Copying blob 6b6e623a48a8 done
Copying blob 9aa8f1e66d54 done
Copying config 67dc3a2a54 done
Writing manifest to image destination
Storing signatures
STEP 2: COPY app.py   ./
STEP 3: CMD ["app.handler"]
STEP 4: COMMIT hello
Getting image source signatures
Copying blob 683073d39306 skipped: already exists
Copying blob 658871a69e1f skipped: already exists
Copying blob 6fa16f35d11e skipped: already exists
Copying blob d6fa53d6caa6 skipped: already exists
Copying blob 61c062506436 skipped: already exists
Copying blob 1c1d66a5fd95 skipped: already exists
Copying blob 33af9dc6463a done
Copying config 98862dfd20 done
Writing manifest to image destination
Storing signatures
--> 98862dfd208
98862dfd2087152ee821553d6cb1c033e735af06e5f11c814bcc9300fb65584e

Test Lambda function Locally

Before calling the Lambda API from a local container, first run the container. Containers default to running in the foreground, but the -d option causes a container to be run as a background process. This container is given the name hello, the external HTTP endpoint at 9000 is mapped to internal port 8080, and the latest version of the hello lambda function is run in the container.

Shell

$ podman run \
  -d \
  --name hello \
  -p 9000:8080 \
  hello:latest
d4d296e4c91d01c98d312e3f79599dca53990d95218e94bbdfbbac6a43cde9e8

Call the local version of the Lambda API:

Shell

$ curl \
  -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" \
  -d '{}'
"Hello from AWS Lambda using Python 3.8.9 (default, Apr 20 2021, 13:58:54) \n[GCC 7.3.1 20180712 (Red Hat 7.3.1-12)]!"

Stop the container called hello.

Shell

$ podman stop hello
96cc1b1ed92368a1165d6a6ad0b1e5544d4ac751b64e94df33bf2322e6d7b30c

Create AWS ECR Repository

AWS provides a registry for OCI-compatible image repositories called the AWS Elastic Container Registry (ECR).

Shell

$ aws ecr create-repository help
CREATE-REPOSITORY() CREATE-REPOSITORY()

NAME
create-repository -

DESCRIPTION
Creates a repository. For more information, see Amazon ECR Repositories
in the Amazon Elastic Container Registry User Guide .

See also: AWS API Documentation

See 'aws help' for descriptions of global parameters.

SYNOPSIS
create-repository
--repository-name <value>
[--tags <value>]
[--image-tag-mutability <value>]
[--image-scanning-configuration <value>]
[--cli-input-json <value>]
[--generate-cli-skeleton <value>]

OPTIONS
--repository-name (string)
The name to use for the repository. The repository name may be spec-
ified on its own (such as nginx-web-app ) or it can be prepended
with a namespace to group the repository into a category (such as
project-a/nginx-web-app ).

--tags (list)
The metadata that you apply to the repository to help you categorize
and organize them. Each tag consists of a key and an optional value,
both of which you define. Tag keys can have a maximum character
length of 128 characters, and tag values can have a maximum length
of 256 characters.

(structure)
The metadata that you apply to a resource to help you categorize
and organize them. Each tag consists of a key and an optional
value, both of which you define. Tag keys can have a maximum
character length of 128 characters, and tag values can have a
maximum length of 256 characters.

Key -> (string)
One part of a key-value pair that make up a tag. A key is a
general label that acts like a category for more specific tag
values.

Value -> (string)
The optional part of a key-value pair that make up a tag. A
value acts as a descriptor within a tag category (key).

Shorthand Syntax:

Key=string,Value=string ...

JSON Syntax:

[
{
"Key": "string",
"Value": "string"
}
...
]

--image-tag-mutability (string)
The tag mutability setting for the repository. If this parameter is
omitted, the default setting of MUTABLE will be used which will al-
low image tags to be overwritten. If IMMUTABLE is specified, all im-
age tags within the repository will be immutable which will prevent
them from being overwritten.

Possible values:

o MUTABLE

o IMMUTABLE

--image-scanning-configuration (structure)
The image scanning configuration for the repository. This setting
determines whether images are scanned for known vulnerabilities af-
ter being pushed to the repository.

scanOnPush -> (boolean)
The setting that determines whether images are scanned after be-
ing pushed to a repository. If set to true , images will be
scanned after being pushed. If this parameter is not specified,
it will default to false and images will not be scanned unless a
scan is manually started with the StartImageScan API.

Shorthand Syntax:

scanOnPush=boolean

JSON Syntax:

{
"scanOnPush": true|false
}

--cli-input-json (string) Performs service operation based on the JSON
string provided. The JSON string follows the format provided by --gen-
erate-cli-skeleton. If other arguments are provided on the command
line, the CLI values will override the JSON-provided values. It is not
possible to pass arbitrary binary values using a JSON-provided value as
the string will be taken literally.

--generate-cli-skeleton (string) Prints a JSON skeleton to standard
output without sending an API request. If provided with no value or the
value input, prints a sample input JSON that can be used as an argument
for --cli-input-json. If provided with the value output, it validates
the command inputs and returns a sample output JSON for that command.

See 'aws help' for descriptions of global parameters.

EXAMPLES
Example 1: To create a repository

The following create-repository example creates a repository inside the
specified namespace in the default registry for an account.

aws ecr create-repository \
--repository-name project-a/nginx-web-app

Output:

{
"repository": {
"registryId": "123456789012",
"repositoryName": "sample-repo",
"repositoryArn": "arn:aws:ecr:us-west-2:123456789012:repository/project-a/nginx-web-app"
}
}

For more information, see Creating a Repository in the Amazon ECR User
Guide.

Example 2: To create a repository configured with image tag immutabil-
ity

The following create-repository example creates a repository configured
for tag immutability in the default registry for an account.

aws ecr create-repository \
--repository-name sample-repo \
--image-tag-mutability IMMUTABLE

Output:

{
"repository": {
"registryId": "123456789012",
"repositoryName": "sample-repo",
"repositoryArn": "arn:aws:ecr:us-west-2:123456789012:repository/sample-repo",
"imageTagMutability": "IMMUTABLE"
}
}

For more information, see Image Tag Mutability in the Amazon ECR User
Guide.

Example 3: To create a repository configured with a scanning configura-
tion

The following create-repository example creates a repository configured
to perform a vulnerability scan on image push in the default registry
for an account.

aws ecr create-repository \
--repository-name sample-repo \
--image-scanning-configuration scanOnPush=true

Output:

{
"repository": {
"registryId": "123456789012",
"repositoryName": "sample-repo",
"repositoryArn": "arn:aws:ecr:us-west-2:123456789012:repository/sample-repo",
"imageScanningConfiguration": {
"scanOnPush": true
}
}
}

For more information, see Image Scanning in the Amazon ECR User Guide.

OUTPUT
repository -> (structure)
The repository that was created.

repositoryArn -> (string)
The Amazon Resource Name (ARN) that identifies the repository.
The ARN contains the arn:aws:ecr namespace, followed by the re-
gion of the repository, AWS account ID of the repository owner,
repository namespace, and repository name. For example,
arn:aws:ecr:region:012345678910:repository/test .

registryId -> (string)
The AWS account ID associated with the registry that contains
the repository.

repositoryName -> (string)
The name of the repository.

repositoryUri -> (string)
The URI for the repository. You can use this URI for Docker push
or pull operations.

createdAt -> (timestamp)
The date and time, in JavaScript date format, when the reposi-
tory was created.

imageTagMutability -> (string)
The tag mutability setting for the repository.

imageScanningConfiguration -> (structure)
The image scanning configuration for a repository.

scanOnPush -> (boolean)
The setting that determines whether images are scanned after
being pushed to a repository. If set to true , images will be
scanned after being pushed. If this parameter is not speci-
fied, it will default to false and images will not be scanned
unless a scan is manually started with the StartImageScan
API.



CREATE-REPOSITORY()

The following creates an AWS ECR image repository in called hello within the test namespace. Images are scanned for known vulnerabilities after they are pushed to the repository.

Shell

$ aws ecr create-repository \
  --repository-name test/hello \
  --image-scanning-configuration scanOnPush=true
{
  "repository": {
      "repositoryArn": "arn:aws:ecr:us-east-1:031372724784:repository/test/hello",
      "registryId": "031372724784",
      "repositoryName": "test/hello",
      "repositoryUri": "031372724784.dkr.ecr.us-east-1.amazonaws.com/test/hello",
      "createdAt": 1620232146.0,
      "imageTagMutability": "MUTABLE",
      "imageScanningConfiguration": {
          "scanOnPush": true
      }
  }
}

Tag Image

podman tag – Assigns a new image name to an existing image. A full name refers to the entire image name, including the optional tag after the :. If there is no tag provided, then podman will default to latest for both the image and the target-name. – From man podman-tag.

Shell

$ IMAGE_NAME=hello

$ IMAGE_VERSION=0.1

$ podman tag $IMAGE_NAME:$IMAGE_VERSION \
  $REGISTRY/$IMAGE_NAME:$IMAGE_VERSION

$ podman images
REPOSITORY                                                  TAG              IMAGE ID      CREATED         SIZE
localhost/hello                                             0.1              98862dfd2087  39 minutes ago  622 MB
752246127823.dkr.ecr.us-east-1.amazonaws.com/hello          latest           98862dfd2087  39 minutes ago  622 MB
public.ecr.aws/lambda/python                                3.8              67dc3a2a54fb  25 hours ago    622 MB
752246127823.dkr.ecr.us-east-1.amazonaws.com/ancientwarmth  latest           5d18ea34fc30  28 hours ago    2.03 GB
localhost/ancientwarmth                                     latest           5d18ea34fc30  28 hours ago    2.03 GB
<none>                                                      <none>           40ef32b39cf4  5 days ago      622 MB
docker.io/library/amazonlinux                               latest           53ef897d731f  5 days ago      170 MB
docker.io/amazon/aws-lambda-python                          3.8              e12ea62c5582  9 days ago      622 MB
docker.io/library/alpine                                    latest           6dbb9cc54074  2 weeks ago     5.88 MB
docker.io/lambci/lambda                                     build-python3.8  714c659c9f6f  3 months ago    2.03 GB

Push Image to ECR

Podman will use the IAM credentials for the dev profile in ~/.aws/credentials to log into that AWS account:

~/.aws/credentials

[default]
aws_access_key_id = ********************
aws_secret_access_key = ****************************************
region = us-east-1

[dev]
aws_access_key_id = ********************
aws_secret_access_key = ****************************************
region = us-east-1

Shell

$ export AWS_PROFILE=dev

$ AWS_ACCOUNT="$( aws sts get-caller-identity \
  --query Account \
  --output text
)"

$ AWS_REGION="$( aws configure get region )"

$ REGISTRY="$AWS_ACCOUNT.dkr.ecr.$AWS_REGION.amazonaws.com"

$ aws ecr get-login-password \
    --region "$AWS_REGION" | \
  podman login \
    --password-stdin \
    --username AWS \
    "$REGISTRY"
Login Succeeded!

Now that podman is logged into AWS, use podman push the image to AWS ECR:

Shell

$ podman push test/$IMAGE_NAME \
  $REGISTRY/$IMAGE_NAME:$IMAGE_VERSION
Getting image source signatures
Copying blob 692590faf2d1 [--------------------------------------] 8.0b / 8.2MiB
Copying blob 397718cff58d [--------------------------------------] 8.0b / 206.2MiB
Copying blob 9ca787b1c91c [--------------------------------------] 8.0b / 93.1MiB
Copying blob ef26f5221b79 [--------------------------------------] 8.0b / 196.7MiB
Copying blob 0a3f69c27a89 [--------------------------------------] 8.0b / 316.4MiB
Copying blob 5b3cbb76df75 [--------------------------------------] 8.0b / 1.1GiB
Copying blob e9cad39831b0 [--------------------------------------] 8.0b / 3.5KiB
Error: Error copying image to the remote destination:
Error writing blob: Error initiating layer upload to /v2/ancientwarmth/blobs/uploads/ in
752246127823.dkr.ecr.us-east-1.amazonaws.com:
name unknown: The repository with name 'hello' does not exist in the registry with id '752246127823'

The results of an image scan for the new repository can be retrieved as follows:

Shell

$ aws ecr describe-image-scan-findings \
  --repository-name test/hello \
  --image-id imageTag=tag_name

Deploy Python Lambda function with Container Image

Podman can invoke the app using an OCI container with Amazon Linux 2 and Python 3.8:

Shell

$ podman container run -ti \
  public.ecr.aws/lambda/python:3.8 \
  blog/docker/podman/app.py
Trying to pull public.ecr.aws/lambda/python:3.8...
Getting image source signatures
Copying blob 1de4740de1c2 done
Copying blob 03ac043af787 done
Copying blob 2e2bb77ae2dc done
Copying blob 842c9dce67e8 done
Copying blob df513d38f4d9 done
Copying blob 031c6369fb2b done
Copying config e12ea62c55 done
Writing manifest to image destination
Storing signatures
time="2021-05-02T23:38:30.971" level=info msg="exec '/var/runtime/bootstrap' (cwd=/var/task, handler=)"

Docker, OCI Images, Buildah and podman

2021-04-28T00:00:00-04:00

There are many ways to create and run Docker-compatible images. Docker is probably the worst option, mostly because it runs as a daemon, and all *nix daemons run with root privileges. Also, the docker-ce package lists iptables as a dependency, which needs systemd to be running normally, and WSL2 only partially supports systemd.

A Comprehensive Container Runtime Comparison provides helpful background information and an interesting historical viewpoint.

Open Container Initiative (OCI)

The latest evolution of Docker-compatible images, OCI image format (not to be confused with Oracle Cloud Infrastructure), is compatible with:

Supported OCI formats include:

Docker containers schema 1
Docker containers schema 2
Pods (groups of containers)
Images
Volumes

Buildah, podman and skopeo

This blog post discusses 3 related open source projects from RedHat / IBM that provide an alternative to Docker: Buildah, podman and skopeo. These 3 projects share a common source code base, and are daemonless tools for managing Open Container Initiative (OCI) images.

Paraphrasing the reasons expressed in Podman and Buildah for Docker Users for using podman instead of Docker, wherever podman is mentioned, read “podman, Buildah and skopeo”:

The Podman approach is simply to directly interact with the image registry, with the container and image storage, and with the Linux kernel through the runC container runtime process (not a daemon).

Running Podman as a normal user means that Podman will, by default, store images and containers in the user’s home directory. Podman uses a repository in the user’s home directory: ~/.local/share/containers (instead of /var/lib/docker).

Despite the new locations for the local repositories, the images created by Docker and Podman are compatible with the OCI standard. Podman can push to and pull from popular container registries like Quay.io and Docker hub, as well as private registries.

Buildah vs. podman

Podman can build OCI containers interactively or in batch mode. You can either build using a Dockerfile using podman build (batch mode), or you can interactively run a container, make changes to the running image, and then podman commit those changes to a new image tag.

Buildah was written before podman. Some of Buildah's source code for creating and managing container images was ported to podman. The podman build command is a subset of Buildah’s functionality.

However, apparently the differences between the two programs are important:

Buildah builds OCI images. Confusingly, podman build can also be used to build Docker images also, but it’s incredibly slow and used up a lot of disk space by using the vfs storage driver by default. buildah bud (‘build using Dockerfile’) was much faster for me, and uses the overlay storage driver.

– From Goodbye Docker: Purging is Such Sweet Sorrow by Ian Miell.

podman

Podman supports developing, managing, and running OCI Containers on Linux systems, including WSL, without requiring root privilege.

shell Installation on Ubuntu

$ yes | sudo apt install buildah podman skopeo

Podman commands are very nearly the same as Docker’s.

Because podman is a drop-in replacement for docker, the following alias enables the docker command to invoke podman:

~/.bash_aliases

alias docker=podman

As described in How to Install and Use Podman on Ubuntu 20.04, I added 'registry.access.redhat.com' to the list of registries in /etc/containers/registries.conf. I also added gallery.ecr.aws and gcr.io.

/etc/containers/registries.conf

[registries.search]
registries = ['docker.io', 'gallery.ecr.aws', 'gcr.io', 'quay.io', 'registry.access.redhat.com']

podman Help

Shell

$ podman --help
Manage pods, containers and images

  Usage:
    podman [flags]
    podman [command]

  Available Commands:
    attach      Attach to a running container
    auto-update Auto update containers according to their auto-update policy
    build       Build an image using instructions from Containerfiles
    commit      Create new image based on the changed container
    container   Manage containers
    cp          Copy files/folders between a container and the local filesystem
    create      Create but do not start a container
    diff        Display the changes to the object's file system
    events      Show podman events
    exec        Run a process in a running container
    export      Export container's filesystem contents as a tar archive
    generate    Generate structured data based on containers and pods.
    healthcheck Manage health checks on containers
    help        Help about any command
    history     Show history of a specified image
    image       Manage images
    images      List images in local storage
    import      Import a tarball to create a filesystem image
    info        Display podman system information
    init        Initialize one or more containers
    inspect     Display the configuration of object denoted by ID
    kill        Kill one or more running containers with a specific signal
    load        Load an image from container archive
    login       Login to a container registry
    logout      Logout of a container registry
    logs        Fetch the logs of one or more containers
    manifest    Manipulate manifest lists and image indexes
    mount       Mount a working container's root filesystem
    network     Manage networks
    pause       Pause all the processes in one or more containers
    play        Play a pod and its containers from a structured file.
    pod         Manage pods
    port        List port mappings or a specific mapping for the container
    ps          List containers
    pull        Pull an image from a registry
    push        Push an image to a specified destination
    restart     Restart one or more containers
    rm          Remove one or more containers
    rmi         Removes one or more images from local storage
    run         Run a command in a new container
    save        Save image to an archive
    search      Search registry for image
    start       Start one or more containers
    stats       Display a live stream of container resource usage statistics
    stop        Stop one or more containers
    system      Manage podman
    tag         Add an additional name to a local image
    top         Display the running processes of a container
    unmount     Unmounts working container's root filesystem
    unpause     Unpause the processes in one or more containers
    unshare     Run a command in a modified user namespace
    untag       Remove a name from a local image
    version     Display the Podman Version Information
    volume      Manage volumes
    wait        Block on one or more containers

  Flags:
        --cgroup-manager string     Cgroup manager to use ("cgroupfs"|"systemd") (default "cgroupfs")
        --cni-config-dir string     Path of the configuration directory for CNI networks
        --conmon string             Path of the conmon binary
    -c, --connection string         Connection to use for remote Podman service
        --events-backend string     Events backend to use ("file"|"journald"|"none") (default "file")
        --help                      Help for podman
        --hooks-dir strings         Set the OCI hooks directory path (may be set multiple times) (default [/usr/share/containers/oci/hooks.d])
        --identity string           path to SSH identity file, (CONTAINER_SSHKEY)
        --log-level string          Log messages above specified level (debug, info, warn, error, fatal, panic) (default "error")
        --namespace string          Set the libpod namespace, used to create separate views of the containers and pods on the system
        --network-cmd-path string   Path to the command for configuring the network
    -r, --remote                    Access remote Podman service (default false)
        --root string               Path to the root directory in which data, including images, is stored
        --runroot string            Path to the 'run directory' where all state information is stored
        --runtime string            Path to the OCI-compatible binary used to run containers, default is /usr/bin/runc
        --storage-driver string     Select which storage driver is used to manage storage of images and containers (default is overlay)
        --storage-opt stringArray   Used to pass an option to the storage driver
        --syslog                    Output logging information to syslog as well as the console (default false)
        --tmpdir string             Path to the tmp directory for libpod state content.

                                    Note: use the environment variable 'TMPDIR' to change the temporary storage location for container images, '/var/tmp'.

        --url string                URL to access Podman service (CONTAINER_HOST) (default "unix:/home/mslinn/.docker/run/podman/podman.sock")
    -v, --version                   Version of Podman

  Use "podman [command] --help" for more information about a command.

podman info

Shell

$ podman info
host:
  arch: amd64
  buildahVersion: 1.15.2
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: unknown'
  cpus: 8
  distribution:
    distribution: ubuntu
    version: "20.10"
  eventLogger: file
  hostname: Bear
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.72-microsoft-standard-WSL2
  linkmode: dynamic
  memFree: 897724416
  memTotal: 6231638016
  ociRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc93
      commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
      spec: 1.0.2-dev
      go: go1.13.15
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    path: /home/mslinn/.docker/run/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.3.1
  swapFree: 0
  swapTotal: 0
  uptime: 306h 23m 6.37s (Approximately 12.75 days)
registries:
  search:
  - quay.io
  - docker.io
  - gallery.ecr.aws
  - registry.access.redhat.com
store:
  configFile: /home/mslinn/.config/containers/storage.conf
  containerStore:
    number: 8
    paused: 0
    running: 0
    stopped: 8
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.0.0
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/mslinn/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 4
  runRoot: /home/mslinn/.docker/run/containers
  volumePath: /home/mslinn/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Wed Dec 31 19:00:00 1969
  GitCommit: ""
  GoVersion: go1.14.7
  OsArch: linux/amd64
  Version: 2.0.6

podman container Help

Shell

$ man podman-container
podman-container(1)                                   General Commands Manual                                   podman-container(1)

NAME
       podman-container - Manage containers

SYNOPSIS
       podman container subcommand

DESCRIPTION
       The container command allows you to manage containers

COMMANDS
       ┌───────────┬────────────────────────────────┬───────────────────────────────────┐
       │Command    │ Man Page                       │ Description                       │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │attach     │ podman-attach(1)               │ Attach to a running container.    │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │checkpoint │ podman-container-checkpoint(1) │ Checkpoints one or more running   │
       │           │                                │ containers.                       │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │cleanup    │ podman-container-cleanup(1)    │ Cleanup the container's network   │
       │           │                                │ and mountpoints.                  │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │commit     │ podman-commit(1)               │ Create new image based on the     │
       │           │                                │ changed container.                │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │cp         │ podman-cp(1)                   │ Copy files/folders between a      │
       │           │                                │ container and the local           │
       │           │                                │ filesystem.                       │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │create     │ podman-create(1)               │ Create a new container.           │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │diff       │ podman-diff(1)                 │ Inspect changes on a container or │
       │           │                                │ image's filesystem.               │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │exec       │ podman-exec(1)                 │ Execute a command in a running    │
       │           │                                │ container.                        │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │exists     │ podman-container-exists(1)     │ Check if a container exists in    │
       │           │                                │ local storage                     │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │export     │ podman-export(1)               │ Export a container's filesystem   │
       │           │                                │ contents as a tar archive.        │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │init       │ podman-init(1)                 │ Initialize a container            │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │inspect    │ podman-inspect(1)              │ Display a container or image's    │
       │           │                                │ configuration.                    │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │kill       │ podman-kill(1)                 │ Kill the main process in one or   │
       │           │                                │ more containers.                  │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │list       │ podman-ps(1)                   │ List the containers on the        │
       │           │                                │ system.(alias ls)                 │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │logs       │ podman-logs(1)                 │ Display the logs of a container.  │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │mount      │ podman-mount(1)                │ Mount a working container's root  │
       │           │                                │ filesystem.                       │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │pause      │ podman-pause(1)                │ Pause one or more containers.     │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │port       │ podman-port(1)                 │ List port mappings for the        │
       │           │                                │ container.                        │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │prune      │ podman-container-prune(1)      │ Remove all stopped containers     │
       │           │                                │ from local storage.               │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │restart    │ podman-restart(1)              │ Restart one or more containers.   │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │restore    │ podman-container-restore(1)    │ Restores one or more containers   │
       │           │                                │ from a checkpoint.                │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │rm         │ podman-rm(1)                   │ Remove one or more containers.    │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │run        │ podman-run(1)                  │ Run a command in a container.     │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │runlabel   │ podman-container-runlabel(1)   │ Executes a command as described   │
       │           │                                │ by a container image label.       │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │start      │ podman-start(1)                │ Starts one or more containers.    │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │stats      │ podman-stats(1)                │ Display a live stream of one or   │
       │           │                                │ more container's resource usage   │
       │           │                                │ statistics.                       │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │stop       │ podman-stop(1)                 │ Stop one or more running          │
       │           │                                │ containers.                       │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │top        │ podman-top(1)                  │ Display the running processes of  │
       │           │                                │ a container.                      │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │unmount    │ podman-unmount(1)              │ Unmount a working container's     │
       │           │                                │ root filesystem.(Alias unmount)   │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │unpause    │ podman-unpause(1)              │ Unpause one or more containers.   │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │wait       │ podman-wait(1)                 │ Wait on one or more containers to │
       │           │                                │ stop and print their exit codes.  │
       └───────────┴────────────────────────────────┴───────────────────────────────────┘

SEE ALSO
       podman, podman-exec, podman-run

                                                                                                                podman-container(1)

Podman Run Help

Shell

$ podman container run --help
Run a command in a new container

Description:
  Runs a command in a new container from the given image

Usage:
  podman container run [flags] IMAGE [COMMAND [ARG...]]

Examples:
  podman container run imageID ls -alF /etc
        podman container run --network=host imageID dnf -y install java
        podman container run --volume /var/hostdir:/var/ctrdir -i -t fedora /bin/bash

Flags:
      --add-host strings                         Add a custom host-to-IP mapping (host:ip) (default [])
      --annotation strings                       Add annotations to container (key:value)
  -a, --attach strings                           Attach to STDIN, STDOUT or STDERR
      --authfile string                          Path of the authentication file. Use REGISTRY_AUTH_FILE environment variable to override
      --blkio-weight string                      Block IO weight (relative weight) accepts a weight value between 10 and 1000.
      --blkio-weight-device DEVICE_NAME:WEIGHT   Block IO weight (relative device weight, format: DEVICE_NAME:WEIGHT)
      --cap-add strings                          Add capabilities to the container
      --cap-drop strings                         Drop capabilities from the container
      --cgroup-parent string                     Optional parent cgroup for the container
      --cgroupns string                          cgroup namespace to use
      --cgroups string                           control container cgroup configuration ("enabled"|"disabled"|"no-conmon") (default "enabled")
      --cidfile string                           Write the container ID to the file
      --conmon-pidfile string                    Path to the file that will receive the PID of conmon
      --cpu-period uint                          Limit the CPU CFS (Completely Fair Scheduler) period
      --cpu-quota int                            Limit the CPU CFS (Completely Fair Scheduler) quota
      --cpu-rt-period uint                       Limit the CPU real-time period in microseconds
      --cpu-rt-runtime int                       Limit the CPU real-time runtime in microseconds
      --cpu-shares uint                          CPU shares (relative weight)
      --cpus float                               Number of CPUs. The default is 0.000 which means no limit
      --cpuset-cpus string                       CPUs in which to allow execution (0-3, 0,1)
      --cpuset-mems string                       Memory nodes (MEMs) in which to allow execution (0-3, 0,1). Only effective on NUMA systems.
  -d, --detach                                   Run container in background and print container ID
      --detach-keys [a-Z]                        Override the key sequence for detaching a container. Format is a single character [a-Z] or a comma separated sequence of `ctrl-<value>`, where `<value>` is one of: `a-cf`, `@`, `^`, `[`, `\`, `]`, `^` or `_` (default "ctrl-p,ctrl-q")
      --device strings                           Add a host device to the container
      --device-cgroup-rule strings               Add a rule to the cgroup allowed devices list
      --device-read-bps strings                  Limit read rate (bytes per second) from a device (e.g. --device-read-bps=/dev/sda:1mb)
      --device-read-iops strings                 Limit read rate (IO per second) from a device (e.g. --device-read-iops=/dev/sda:1000)
      --device-write-bps strings                 Limit write rate (bytes per second) to a device (e.g. --device-write-bps=/dev/sda:1mb)
      --device-write-iops strings                Limit write rate (IO per second) to a device (e.g. --device-write-iops=/dev/sda:1000)
      --disable-content-trust                    This is a Docker specific option and is a NOOP
      --dns strings                              Set custom DNS servers
      --dns-opt strings                          Set custom DNS options
      --dns-search strings                       Set custom DNS search domains
      --entrypoint string                        Overwrite the default ENTRYPOINT of the image
  -e, --env stringArray                          Set environment variables in container (default [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,TERM=xterm])
      --env-file strings                         Read in a file of environment variables
      --env-host                                 Use all current host environment variables in container
      --expose strings                           Expose a port or a range of ports
      --gidmap strings                           GID map to use for the user namespace
      --group-add strings                        Add additional groups to join
      --health-cmd string                        set a healthcheck command for the container ('none' disables the existing healthcheck)
      --health-interval string                   set an interval for the healthchecks (a value of disable results in no automatic timer setup) (default "30s")
      --health-retries uint                      the number of retries allowed before a healthcheck is considered to be unhealthy (default 3)
      --health-start-period string               the initialization time needed for a container to bootstrap (default "0s")
      --health-timeout string                    the maximum time allowed to complete the healthcheck before an interval is considered failed (default "30s")
  -h, --hostname string                          Set container hostname
      --http-proxy                               Set proxy environment variables in the container based on the host proxy vars (default true)
      --image-volume string                      Tells podman how to handle the builtin image volumes ("bind"|"tmpfs"|"ignore") (default "bind")
      --init                                     Run an init binary inside the container that forwards signals and reaps processes
      --init-path string                         Path to the container-init binary
  -i, --interactive                              Keep STDIN open even if not attached
      --ip string                                Specify a static IPv4 address for the container
      --ipc string                               IPC namespace to use
      --kernel-memory <number>[<unit>]           Kernel memory limit (format: <number>[<unit>], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes))
  -l, --label stringArray                        Set metadata on container
      --label-file strings                       Read in a line delimited file of labels
      --log-driver string                        Logging driver for the container
      --log-opt strings                          Logging driver options
      --mac-address string                       Container MAC address (e.g. 92:d0:c6:0a:29:33)
  -m, --memory <number>[<unit>]                  Memory limit (format: <number>[<unit>], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes))
      --memory-reservation <number>[<unit>]      Memory soft limit (format: <number>[<unit>], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes))
      --memory-swap string                       Swap limit equal to memory plus swap: '-1' to enable unlimited swap
      --memory-swappiness int                    Tune container memory swappiness (0 to 100, or -1 for system default) (default -1)
      --mount stringArray                        Attach a filesystem mount to the container
      --name string                              Assign a name to the container
      --network string                           Connect a container to a network (default "slirp4netns")
      --no-healthcheck                           Disable healthchecks on container
      --no-hosts                                 Do not create /etc/hosts within the container, instead use the version from the image
      --oom-kill-disable                         Disable OOM Killer
      --oom-score-adj int                        Tune the host's OOM preferences (-1000 to 1000)
      --pid string                               PID namespace to use
      --pids-limit int                           Tune container pids limit (set 0 for unlimited, -1 for server defaults)
      --pod string                               Run container in an existing pod
      --pod-id-file string                       Read the pod ID from the file
      --privileged                               Give extended privileges to container
  -p, --publish strings                          Publish a container's port, or a range of ports, to the host (default [])
  -P, --publish-all                              Publish all exposed ports to random ports on the host interface
      --pull string                              Pull image before creating ("always"|"missing"|"never") (default "missing")
  -q, --quiet                                    Suppress output information when pulling images
      --read-only                                Make containers root filesystem read-only
      --read-only-tmpfs                          When running containers in read-only mode mount a read-write tmpfs on /run, /tmp and /var/tmp (default true)
      --replace                                  If a container with the same name exists, replace it
      --restart string                           Restart policy to apply when a container exits ("always"|"no"|"on-failure")
      --rm                                       Remove container (and pod if created) after exit
      --rmi                                      Remove container image unless used by other containers
      --rootfs                                   The first argument is not an image but the rootfs to the exploded container
      --seccomp-policy string                    Policy for selecting a seccomp profile (experimental) (default "default")
      --security-opt stringArray                 Security Options
      --shm-size <number>[<unit>]                Size of /dev/shm (format: <number>[<unit>], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes)) (default "65536k")
      --sig-proxy                                Proxy received signals to the process (default true)
      --stop-signal string                       Signal to stop a container. Default is SIGTERM
      --stop-timeout uint                        Timeout (in seconds) to stop a container. Default is 10 (default 10)
      --subgidname string                        Name of range listed in /etc/subgid for use in user namespace
      --subuidname string                        Name of range listed in /etc/subuid for use in user namespace
      --sysctl strings                           Sysctl options
      --systemd string                           Run container in systemd mode ("true"|"false"|"always") (default "true")
      --tmpfs tmpfs                              Mount a temporary filesystem (tmpfs) into a container
  -t, --tty                                      Allocate a pseudo-TTY for container
      --uidmap strings                           UID map to use for the user namespace
      --ulimit strings                           Ulimit options
  -u, --user string                              Username or UID (format: <name|uid>[:<group|gid>])
      --userns string                            User namespace to use
      --uts string                               UTS namespace to use
  -v, --volume stringArray                       Bind mount a volume into the container
      --volumes-from strings                     Mount volumes from the specified container(s)
  -w, --workdir string                           Working directory inside the container

podman run

From Building and Deploying Lambdas from a Docker Container by Keith Gregory:

Shell

$ podman run \
  -it \
  --entrypoint /bin/bash \
  --rm \
  -v /tmp:/mnt \
  amazon/aws-lambda-python:3.8
Trying to pull quay.io/amazon/aws-lambda-python:3.8...
  Requesting bear token: invalid status code from registry 405 (Method Not Allowed)
Trying to pull docker.io/amazon/aws-lambda-python:3.8...
Getting image source signatures
Copying blob df513d38f4d9 skipped: already exists
Copying blob 2e2bb77ae2dc skipped: already exists
Copying blob 031c6369fb2b skipped: already exists
Copying blob 03ac043af787 skipped: already exists
Copying blob 842c9dce67e8 skipped: already exists
Copying blob 1de4740de1c2 [--------------------------------------] 0.0b / 0.0b
Copying config e12ea62c55 done
Writing manifest to image destination
Storing signatures
bash-4.2# pwd
/var/task

Cleaning Up a Container

podman container cleanup is a good command to know about.

Shell

$ podman container cleanup --help
Cleanup network and mountpoints of one or more containers

Description:

   podman container cleanup

   Cleans up mount points and network stacks on one or more containers
   from the host. The container name or ID can be used. This command is
   used internally when running containers, but can also be used if
   container cleanup has failed when a container exits.


Usage:
  podman container cleanup [options] CONTAINER [CONTAINER...]

Examples:
  podman container cleanup --latest
  podman container cleanup ctrID1 ctrID2 ctrID3
  podman container cleanup --all

Options:
  -a, --all           Cleans up all containers
      --exec string   Clean up the given exec session instead of the container
  -l, --latest        Act on the latest container podman is aware of
                      Not supported with the "--remote" flag
      --rm            After cleanup, remove the container entirely
      --rmi           After cleanup, remove the image entirely

Buildah

Buildah is a drop-in replacement for using docker build and a Dockerfile.

Where Buildah really shines is in its native commands, which you can use to interact with container builds. Rather than using build-using-dockerfile/bud for each build, Buildah has commands to actually interact with the temporary container created during the build process. (Docker uses temporary, or intermediate containers, too, but you don’t really interact with them while the image is being built.)

Unlike docker build, Buildah doesn’t commit changes to a layer automatically for every instruction in the Dockerfile – it builds everything from top to bottom, every time. On the positive side, this means non-cached builds (for example, those you would do with automation or build pipelines) end up being somewhat faster than their Docker build counterparts, especially if there are many instructions.

– From Getting started with Buildah., published by opensource.com

Some key Buildah subcommands:

buildah bud: Buildah’s build-using-dockerfile, or bud argument makes it behave just like docker build does.
buildah from: Build up a container root filesystem from an image or from scratch.
buildah config: Adjust defaults in the image's configuration blob.
buildah run: buildah run is for running commands that build a container image. This is similar to RUN in a Dockerfile, and unlike docker run.
buildah commit: Commit changes to the container to a new image.
buildah push: Push images to registries (such a Quay) or a local dockerd instance.

Buildah Help

Shell

$ buildah -h
A tool that facilitates building OCI images

Usage:
  buildah [flags]
  buildah [command]

Available Commands:
  add                    Add content to the container
  build-using-dockerfile Build an image using instructions in a Dockerfile
  commit                 Create an image from a working container
  config                 Update image configuration settings
  containers             List working containers and their base images
  copy                   Copy content into the container
  from                   Create a working container based on an image
  help                   Help about any command
  images                 List images in local storage
  info                   Display Buildah system information
  inspect                Inspect the configuration of a container or image
  login                  Login to a container registry
  logout                 Logout of a container registry
  manifest               Manipulate manifest lists and image indexes
  mount                  Mount a working container's root filesystem
  pull                   Pull an image from the specified location
  push                   Push an image to a specified destination
  rename                 Rename a container
  rm                     Remove one or more working containers
  rmi                    Remove one or more images from local storage
  run                    Run a command inside of the container
  tag                    Add an additional name to a local image
  umount                 Unmount the root file system of the specified working containers
  unshare                Run a command in a modified user namespace
  version                Display the Buildah version information

Flags:
  -h, --help                                 help for buildah
      --log-level string                     The log level to be used. Either "debug", "info", "warn" or "error". (default "error")
      --registries-conf string               path to registries.conf file (not usually used)
      --registries-conf-dir string           path to registries.conf.d directory (not usually used)
      --root string                          storage root dir (default "/var/lib/containers/storage")
      --runroot string                       storage state dir (default "/var/run/containers/storage")
      --storage-driver string                storage-driver
      --storage-opt strings                  storage driver option
      --userns-gid-map ctrID:hostID:length   default ctrID:hostID:length GID mapping to use
      --userns-uid-map ctrID:hostID:length   default ctrID:hostID:length UID mapping to use
  -v, --version                              version for buildah

Use "buildah [command] --help" for more information about a command.

Buildah / Dockerfile Compatibility

Buildah can create an image from a Dockerfile by typing:

Shell

$ buildah bud -t hello .

…instead of:

Shell

$ sudo docker build -t hello .

Buildah can create an image called hello from the Dockerfile and the Python app by typing:

Shell

$ buildah bud -t hello .
STEP 1: FROM public.ecr.aws/lambda/python:3.8
  Getting image source signatures
  Copying blob 1de4740de1c2 done
  Copying blob 2e2bb77ae2dc done
  Copying blob df513d38f4d9 done
  Copying blob 03ac043af787 done
  Copying blob 031c6369fb2b done
  Copying blob 842c9dce67e8 done
  Copying config e12ea62c55 done
  Writing manifest to image destination
  Storing signatures
  STEP 2: COPY app.py   ./
  STEP 3: CMD ["app.handler"]
  STEP 4: COMMIT hello
  Getting image source signatures
  Copying blob 109f575f8e6a skipped: already exists
  Copying blob ff64b4f854ad skipped: already exists
  Copying blob dd66ad8702f4 skipped: already exists
  Copying blob d6fa53d6caa6 skipped: already exists
  Copying blob 80166c3283e5 skipped: already exists
  Copying blob 61f74564c3aa skipped: already exists
  Copying blob d95ebdc79761 done
  Copying config 40ef32b39c done
  Writing manifest to image destination
  Storing signatures
  --> 40ef32b39cf
  40ef32b39cf4ffd3d2e4e3426bec4a5ea168524f7f3fcfe863a378abd9794270

Once the build is complete, the new image can be displayed with the buildah images command:

Shell

$ buildah images
REPOSITORY                     TAG      IMAGE ID       CREATED          SIZE
localhost/hello                latest   40ef32b39cf4   56 seconds ago   622 MB

The new image, tagged hello:latest, can be pushed to a remote image registry. This is easily accomplished with the buildah push command.

buildah push Help

Shell

$ man buildah-push
buildah-push(1)                                   General Commands Manual                                  buildah-push(1)

NAME
       buildah-push - Push an image from local storage to elsewhere.

SYNOPSIS
       buildah push [options] image [destination]

DESCRIPTION
       Pushes an image from local storage to a specified destination, decompressing and recompessing layers as needed.

imageID
       Image stored in local container/storage

DESTINATION
       The  DESTINATION  is a location to store container images. If omitted, the source image parameter will be reused as
       destination.

       The Image "DESTINATION" uses a "transport":"details" format. Multiple transports are supported:

       dir:path
         An existing local directory path storing the manifest, layer tarballs and signatures as individual files. This is
       a non-standardized format, primarily useful for debugging or noninvasive container inspection.

       docker://docker-reference
         An  image  in a registry implementing the "Docker Registry HTTP API V2". By default, uses the authorization state
       in $XDG\_RUNTIME\_DIR/containers/auth.json, which is set using (buildah login). If the authorization state  is  not
       found there, $HOME/.docker/config.json is checked, which is set using (docker login).
         If  docker-reference  does  not include a registry name, the image will be pushed to a registry running on local‐
       host.

       docker-archive:path[:docker-reference]
         An image is stored in the docker save formatted file.  docker-reference is only used when creating such  a  file,
       and it must not contain a digest.

       docker-daemon:docker-reference
         An image _dockerreference stored in the docker daemon internal storage. If _dockerreference does not begin with a
       valid registry name (a domain name containing "." or the reserved name "localhost") then the default registry  name
       "docker.io"  will be prepended. _dockerreference must contain either a tag or a digest. Alternatively, when reading
       images, the format can also be docker-daemon:algo:digest (an image ID).

       oci:path:tag
         An image tag in a directory compliant with "Open Container Image Layout Specification" at path.

       oci-archive:path:tag
         An image tag in a tar archive compliant with "Open Container Image Layout Specification" at path.

       If the transport part of DESTINATION is omitted, "docker://" is assumed.

OPTIONS
       --authfile path

       Path of the authentication file. Default is ${XDG_RUNTIME_DIR}/containers/auth.json, which is set using buildah lo‐
       gin.   If  the  authorization  state  is  not found there, $HOME/.docker/config.json is checked, which is set using
       docker login.

       --cert-dir path

       Use certificates at path (*.crt, *.cert, *.key) to connect to the  registry.   Default  certificates  directory  is
       /etc/containers/certs.d.

       --creds creds

       The [username[:password]] to use to authenticate with the registry if required.  If one or both values are not sup‐
       plied, a command line prompt will appear and the value can be entered.  The password is entered without echo.

       --digestfile Digestfile

       After copying the image, write the digest of the resulting image to the file.

       --disable-compression, -D

       Don't compress copies of filesystem layers which will be pushed.

       --encryption-key key

       The [protocol:keyfile] specifies the encryption protocol, which can be JWE  (RFC7516),  PGP  (RFC4880),  and  PKCS7
       (RFC2315) and the key material required for image encryption. For instance, jwe:/path/to/key.pem or pgp:admin@exam‐
       ple.com or pkcs7:/path/to/x509-file.

       --format, -f

       Manifest Type (oci, v2s1, or v2s2) to use when saving image to directory using the  'dir:'  transport  (default  is
       manifest type of source)

       --quiet, -q

       When writing the output image, suppress progress output.

       --remove-signatures

       Don't copy signatures when pushing images.

       --sign-by fingerprint

       Sign the pushed image using the GPG key that matches the specified fingerprint.

       --tls-verify bool-value

       Require HTTPS and verify certificates when talking to container registries (defaults to true)

EXAMPLE
       This example pushes the image specified by the imageID to a local directory in docker format.

       # buildah push imageID dir:/path/to/image

       This example pushes the image specified by the imageID to a local directory in oci format.

       # buildah push imageID oci:/path/to/layout:image:tag

       This example pushes the image specified by the imageID to a tar archive in oci format.

       # buildah push imageID oci-archive:/path/to/archive:image:tag

       This example pushes the image specified by the imageID to a container registry named registry.example.com.

       # buildah push imageID docker://registry.example.com/repository:tag

       This example pushes the image specified by the imageID to a container registry named registry.example.com and saves
       the digest in the specified digestfile.

       # buildah push --digestfile=/tmp/mydigest imageID docker://registry.example.com/repository:tag

       This example works like docker push, assuming registry.example.com/my_image is a local image.

       # buildah push registry.example.com/my_image

       This example pushes the image specified by the imageID to a private container registry  named  registry.example.com
       with authentication from /tmp/auths/myauths.json.

       # buildah push --authfile /tmp/auths/myauths.json imageID docker://registry.example.com/repository:tag

       This example pushes the image specified by the imageID and puts into the local docker container store.

       # buildah push imageID docker-daemon:image:tag

       This example pushes the image specified by the imageID and puts it into the registry on the localhost while turning
       off tls verification.
        # buildah push --tls-verify=false imageID docker://localhost:5000/my-imageID

       This example pushes the image specified by the imageID and puts it into the registry on the localhost using creden‐
       tials and certificates for authentication.
        #   buildah   push   --cert-dir      /auth  --tls-verify=true  --creds=username:password  imageID  docker://local‐
       host:5000/my-imageID

ENVIRONMENT
       BUILD_REGISTRY_SOURCES

       BUILD_REGISTRY_SOURCES, if set, is treated as a JSON object which contains lists of registry names under  the  keys
       insecureRegistries, blockedRegistries, and allowedRegistries.

       When pushing an image to a registry, if the portion of the destination image name that corresponds to a registry is
       compared to the items in the blockedRegistries list, and if it matches any of them, the push attempt is denied.  If
       there are registries in the allowedRegistries list, and the portion of the name that corresponds to the registry is
       not in the list, the push attempt is denied.

       TMPDIR The TMPDIR environment variable allows the user to specify where temporary files are  stored  while  pulling
       and pushing images.  Defaults to '/var/tmp'.

FILES
       registries.conf (/etc/containers/registries.conf)

       registries.conf  is the configuration file which specifies which container registries should be consulted when com‐
       pleting image names which do not include a registry or domain portion.

       policy.json (/etc/containers/policy.json)

       Signature policy file.  This defines the trust policy for container images.  Controls  which  container  registries
       can be used for image, and whether or not the tool should trust the images.

SEE ALSO
       buildah(1), buildah-login(1), containers-policy.json(5), docker-login(1), containers-registries.conf(5)

buildah                                                  June 2017                                         buildah-push(1)

Shell

$ buildah run \
  --entrypoint /var/lang/bin/pip \
  --rm \
  --user "$(id -u):$(id -g)" \
  -v "$(pwd):/mnt" \
  amazon/aws-lambda-python:3.8 \
  install --target /mnt/build --upgrade psycopg2-binary

How To

The following was inspired by Recap: images and containers from Docker for beginners. The equivalent commands for Docker alternatives are shown.

Check software version

$ docker -v
Docker version 20.10.2, build 20.10.2-0ubuntu1~20.10.1

$ buildah -v
buildah version 1.15.2 (image-spec 1.0.1, runtime-spec 1.0.2-dev)

$ podman -v
podman version 2.0.6

Download the Amazon Linux 2 image

AWS Lambda functions run under Amazon Linux.

Each of these 3 commands does a very similar task, downloading a specific image. Docker uses different subdirectories for images than Buildah and podman do.

$ sudo docker pull amazonlinux
Using default tag: latest
latest: Pulling from library/amazonlinux
3c2c91c7c431: Pull complete
Digest: sha256:06b9e2433e4e563e1d75bc8c71d32b76dc49a2841e9253746eefc8ca40b80b5e
Status: Downloaded newer image for amazonlinux:latest
docker.io/library/amazonlinux:latest

Buildah works without complaint.

$ buildah pull amazonlinux
53ef897d731f9a5673c083d0e86d7911f85d6e63bb2be2346b17bdbacdc58637

podman seems to hiccup and then complete successfully.

$ podman pull amazonlinux
Trying to pull quay.io/amazonlinux...
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n"
Trying to pull docker.io/library/amazonlinux...
Getting image source signatures
Copying blob 3c2c91c7c431 [--------------------------------------] 0.0b / 0.0b
Copying config 53ef897d73 done
Writing manifest to image destination
Storing signatures
53ef897d731f9a5673c083d0e86d7911f85d6e63bb2be2346b17bdbacdc58637

Run a Bash Command in an OCI Container

Again, Docker must be run as root for this operation, this represents an unnecessary security risk.

$ sudo docker container run amazonlinux echo 'Hello World!'
Hello World!

$ podman container run amazonlinux cat /etc/os-release
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Show All Locally Available Images

Again, Docker must be run as root for this operation, this represents an unnecessary security risk.

$ sudo docker images
REPOSITORY    TAG       IMAGE ID       CREATED        SIZE
amazonlinux   latest    53ef897d731f   21 hours ago   163MB

$ podman images
REPOSITORY                     TAG     IMAGE ID      CREATED       SIZE
localhost/hello                latest  40ef32b39cf4  5 hours ago   622 MB
docker.io/library/amazonlinux  latest  53ef897d731f  21 hours ago  170 MB

List OCI Containers

$ sudo docker container ls
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

$ podman container ls
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

View All OCI Containers (Running or Not)

$ sudo docker container ls -a
CONTAINER ID   IMAGE         COMMAND                 CREATED          STATUS                      PORTS     NAMES
250f56d9aced   amazonlinux   "echo 'Hello World!'"   14 minutes ago   Exited (0) 14 minutes ago             competent_einstein

$ podman container ls -a
CONTAINER ID  IMAGE                                 COMMAND            CREATED            STATUS                        PORTS   NAMES
  0f8203e9d3b8  docker.io/library/amazonlinux:latest  echo Hello world!  36 minutes ago     Exited (0) 36 minutes ago
    beautiful_mestorf
  14282ace8978  docker.io/library/amazonlinux:latest  echo Hello world!  36 minutes ago     Exited (0) 36 minutes ago
    beautiful_goldwasser
  1b9a8db52fb9  docker.io/library/alpine:latest       echo Hello World!  About an hour ago  Exited (0) About an hour ago          zealous_easley
  6444ee144488  docker.io/library/amazonlinux:latest  echo Hello World!  12 minutes ago     Exited (0) 12 minutes ago
    frosty_ritchie
  7444122cbc59  docker.io/library/alpine:latest       cat /etc/motd      About an hour ago  Exited (0) About an hour ago          elated_sammet
  aef84973d6ad  docker.io/library/amazonlinux:latest  echo Hello world!  About an hour ago  Exited (0) About an hour ago          lucid_sinoussi
  e210f74bc209  docker.io/library/amazonlinux:latest  cat /etc/motd      About an hour ago  Exited (0) About an hour ago          jovial_borg

List Running OCI containers

$ sudo docker container ps -a
CONTAINER ID   IMAGE         COMMAND                 CREATED         STATUS                     PORTS     NAMES
250f56d9aced   amazonlinux   "echo 'Hello World!'"   5 minutes ago   Exited (0) 5 minutes ago             competent_einstein

podman has a problem with the container ps sub-subcommand.

$ podman container ps -a
Error: unrecognized command `podman container ps`
Try 'podman container --help' for more information.

buildah push to Docker Daemon

Shell

$ buildah push hello:latest docker-daemon:hello:latest
Getting image source signatures
  Copying blob sha256:72fcdba8cff9f105a61370d930d7f184702eeea634ac986da0105d8422a17028
   247.02 MiB / 247.02 MiB [==================================================] 2s
  Copying blob sha256:e567905cf805891b514af250400cc75db3cb47d61219750e0db047c5308bd916
   144.75 MiB / 144.75 MiB [==================================================] 1s
  Copying config sha256:6d54bef73e638f2e2dd8b7bf1c4dfa26e7ed1188f1113ee787893e23151ff3ff
   1.59 KiB / 1.59 KiB [======================================================] 0s
  Writing manifest to image destination
  Storing signatures 

$ buildah images | head -n2
REPOSITORY              TAG             IMAGE ID        CREATED                 SIZE
docker.io/hello      latest       6d54bef73e63  2 minutes ago   398 MB 

$ buildah run -t hello:latest
Hello, world!

Delete an OCI Image

Delete an OCI image in Buildah's ~/.local/share/container directory with the rmi subcommand:

Shell

$ buildah rmi e12ea62c5582
e12ea62c5582f91a2228e3e284ea957f2df4f1cdb150fd2c189ef8f11d7633ce

Stack Overflow Culture: Zero-Sum, Authoritarian and Hormonally Imbalanced

2021-04-18T00:00:00-04:00

This blog post describes some problems that significantly impact Stack Overflow users, and offers suggestions for improvement. If you are unfamiliar with Stack Overflow, or would like to read a summary of what this blog post is based on, Kevin Workman wrote a terrific summary in March 2019 entitled “The Stack Overflow Culture Wars”. Very little has changed since then.

Introduction

Stack Overflow is the premier website world-wide for programmers to help each other by asking and answering questions. It has a defined protocol for this type of interaction, however new user on-boarding is often ineffective, so newcomers are not properly informed, and old-timers often do not exhibit appropriate people skills. The protocol is somewhat misguided and appropriate tools for better interaction are not provided.

As a result, this website has developed a well-documented reputation for being “a valuable resource, but a scary place to contribute due to potential hostility.”

Sometimes, loving something means caring enough to admit that it has a problem.

– Jay Hanlon, writing about Stack Overflow when he was EVP of Culture and Experience.

Stack Overflow suffers from militant moderators who close and delete reasonable submissions and answers due to Draconian rules.

– sleavey commenting on a Hacker News thread entitled Stack Overflow Culture.

Change Starts At the Top

Prashanth Chandrasekar, CEO

Prashanth Chandrasekar became CEO of Stack Overflow in September, 2019. Inside Intercom interviewed Mr. Chandrasekar in August 2020. This puff piece made no mention of any cultural problems. The focus was on the brilliance of Stack Overflow’s technology. If Mr. Chandrasekar has a vision for how to guide social change, he did not mention it. Instead, in the article and his publications since then he only speaks publicly about transitioning Stack Overflow to a product-led SaaS company.

Shortly after becoming CEO, Mr. Chandrasekar published “Scripting the Future of Stack Overflow, in which he wrote:

We learned that we needed much better channels to listen to our moderators and community members. We have not evolved the existing channels of engagement for power users in our community, like Meta, or articulated how we intended to make improvements going forward. This has caused friction as our user base and business have rapidly grown. We acknowledge these issues, apologize for our mistakes, and have plans for improving in the future.

Later in the article, he mentioned improvements to the code of conduct, a survey and establishing a moderator council. More than 2 years later, none of this has made the slightest difference in Stack Overflow's culture.

Mihir Pathak — EVP, Strategy & Transformation

Prior to his employment at Stack Overflow, Mr. Pathak was a derivatives strategist McKinsey & Company, a world-renouned change management company. That is to say, although Mr. Pathak worked at McKinsey, he was not there to assist other companies make structural changes; instead, he was responsible for pricing methodologies and hedging techniques underlying financial derivative products and options trading strategies – a heads-down money manager.

Mr. Pathak’s page at StackOverflow is no longer available. Google cached it. No announcement has been made about his departure or if there will be a replacement.

Mr. Pathak's job description is still visible at themuse.com:

Mihir is responsible for the long-term business strategy of Stack Overflow, which includes forming partnerships with like-minded organizations and understanding how to best serve the needs of future developers.

Mr. Pathak was employed from September 2016 at Stack Overflow as a business development executive, not a change management champion.

Teresa Dietrich, Chief Product and Technology Officer

In January 2020 Teresa Dietrich was made Chief Product and Technology Officer. She also came from McKinsey & Company, where she was Global Head of Product & Engineering. Two months after she took the job she wrote “Sharing our first quarter 2020 community roadmap”. That rather bland article did not seem to indicate any recognition of serious problems within the Stack Overflow culture, and I could not find any publicly available results of the studies that were mentioned.

16 months after Ms. Dietrich started her job, I do not see any cultural change. Has Ms. Dietrich been tasked with leading such a change? If so:

Does she have board-level support?
Does she have what she needs to make significant cultural changes?
Why has she not made any public acknowledgement of a cultural problem? One possible answer is that her boss, Mr. Chandrasekar, has not done so.
Is Ms. Dietrich the right person for the job? This is not a purely technical problem, it is a social problem. I believe that women in general possess more highly developed social skills, but the skills necessary to climb to the top are not the skills required to make this type of cultural shift.

Are Mr. Chandrasekar and Ms. Dietrich part of the cultural problem, or part of the solution?

Where Am I Coming From?

I have no agenda, no investment in the status quo, nothing to prove, no contacts at the company, I am not an undercover journalist, and I am not a competitor or investor. I am just Joe User... and I am not shy when I believe I have something that I think needs to be heard.

If I mispeak, please tell me. If I missed something, again please tell me. If there is a bigger picture I would like to learn about it.

I have used Stack Overflow and its sibling websites for more than 10 years. Until a few weeks ago, I contributed a few answers here and there, and otherwise did not spend much time on the sites. All contributors are volunteers, so the only reasons to contribute are for prestige, social bonding and altruism.

For almost 30 days, in my spare time, I helped people who asked questions on Stack Overflow. I am writing this blog post because although I enjoy helping others, the enjoyment I experienced while doing so within the Stack Overflow environment was overshadowed by the regressive behavior tolerated and even enforced by other contributors. I should also say that I did encounter certain other contributors with whom interaction was very pleasant. However, interactions with alpha contributors with the highest scores seemed more often than not to be quite unpleasant. Since I first published this blog post, I have mostly not interacted on the site. I remain open to contributing to improvements in the culture.

The next image is provided so readers know that I am able to effectively participate in the current Stack Overflow website.

The volume of accepted and upvoted answers put me in the top 0.14% of Stack Overflow answerers in under 30 days.

Over 100 of the answers I offered in a 30-day period were accepted as the preferred answer. In fact, most of my answers were selected as the preferred answer. That means many more alternative answers were not accepted.

After losing, sometimes a contributor will delete their post. I have done it myself, when the winning post was significantly better by all measures.

Some of the other contributors who had provided alternative answers that were not selected clearly felt they had lost a competition. This structure, and others that I describe below, define a system designed to generate hostility. Stack Overflow, as currently implemented, promotes dominance behavior, which for most primates (other than bonobos) is patriarchal.

Gamification

Stack Overflow's success has be in part due to its successful gamification of the interaction between questioners and answerers. Gamification is powerful and addictive. Unfortunately, the model chosen resembles FPS (first-person shooter) games, instead of co-operative games.

Jeff Atwood, one of the two original authors of Stack Overflow, wrote an article in 2011 entitled The Gamification, in which he writes:

Gaming elements are not tacked on to the Stack Exchange Q&A engine, but a natural and essential element of the design.

Just below that sentence, Mr. Atwood shows a screenshot from an FPS video game:

FPS Game screenshot from 'The Gamification'.

I haven't ever quite come out and said it this way, but … I played a lot of Counter-Strike from 1998 to 2001, and Stack Overflow is in many ways my personal Counter-Strike.

– Jeff Atwood, from “The Gamification”

FPS games are structured so the player only wins by killing others. This is an entirely different motivational structure than a scenario where a person only wins if others succeed.

Terminology

I use a few non-standard terms in this blog post:

Questioner: One who provides a question
Answerer: One who provides an answer
Downvoter: One who downvotes another person's contribution
Downvotee: One who has their contribution downvoted

Dialog Improves Most Questions

A small percentage of questions asked on Stack Overflow are unambiguous, contain all the necessary information, and are phrased well enough to be understood. For these questions, answers can be posted without any interaction between questioner and potential answerers.

However, most questions involve a dialog between potential answerers and the questioner. In the dialog, the question is refined, and the questioner's code and any other relevant data is elicited and provided. The tools provided for such a dialog are unfortunately problematic.

The only two mechanisms for interaction between questioner and potential answerers are comments and answers. Comments have severe limitations that greatly reduce their effectiveness for eliciting information from a questioner:

Comments must be very short
Comments cannot be formatted properly
Comments cannot be edited for more than a few minutes

This means that answerers who are trying to explain something to the questioner to elicit more information, or guide the questioner towards understanding their problem better, must resort to posting an incomplete answer. Posting an incomplete answer is risky; other potential answerers can attack the answer by downvoting it.

Downvotes typically last forever

Some Incentives Promote Hostility

Many long-time users have completely objectified other users, and act as if Stack Overflow is a video game. Points are accumulated, and at any given time there are a finite number of questions to answer. A person's reputation on Stack Overflow is represented by a single number, which is the number of points they have accumulated. This single number is the structural source of many problems. A more nuanced reputation score would be a giant step forward.

Many of these long-term answerers have come to view their participation on Stack Overflow as a zero-sum competition; they can only win (that is, have their answer accepted) if everyone else loses (that is, no-one else provides an answer that is accepted).

Some answerers employ intimidation order to suppress competition. Downvotes are often used in the same way against other answerers as a brush back pitch in baseball.

Downvoting has no negative consequences for the downvoter

Standard operating procedure for competitively-minded answerers is to downvote answers from others at every opportunity. There is no risk in downvoting:

Downvotes are untraceable; there is no public record of who downvoted or when a downvote was cast.
Downvotes are free to downvoters; this encourages liberal downvoting.

This scenario incentivizes competitively-minded answerers to strafe the competition with downvotes at every opportunity.

If downvotes cost the downvoter the same number of points as they penalize the downvotee, then downvotes would become much rarer.

Downvotes typically last forever. Yes, a downvoter could theoretically reverse a downvote, but it is awkward for them to find their old downvotes, and there is no incentive to do so.

Questions from newcomers are also frequently downvoted, without discussion, or along with hostile remarks. That leaves a permanent impression, and tends to select for new members who are comfortable with dominance-based hostility. This is a self-perpetuating, and highly toxic, social order.

Toxic Masculinity Harms Men and Society As A Whole, from Focus for Health

Suggestion: Gamified Onboarding

The only information for new users that is directly accessible from the Stack Overflow front page, is the Help Center, under the question mark icon. It is obvious from the often very polite and tentative inquiries made by new users when they ask their first question that they never noticed that information, or if they did it did not seem relevant. ... and then those new users are mercilessly slammed.

New users should be presented with a short instructional question and answer-style introduction, where information is provided on how to be a good Stack Overflow user. This should happen before they get the opportunity to post their first question. Different levels of users should be explained. Although Stack Overflow is all-English, the onboarding should be multilingual.

Suggestion: Multilingual Support

A high percentage of users do not speak English very well. They really struggle, and tolerance is low on Stack Overflow for bad English. Other sites, for example Facebook and LinkedIn, have a translation facility built right in. I think Facebook did a particularly good job. Why not do something similar on StackOverflow.com? English would be the official language, but everyone world-wide would be able to interact much more effectively.

This site is multilingual.
It is not that hard to do!

Machine translation is really quite good. I have it on this site. Want to view this site in one of over 100 languages? Just go to the top of this page and click on this pull-down menu labeled Select Language:

You will then see the list of languages that you can view this website in:

Morally speaking, not to provide a multilingual site discriminates against non-English-speaking people.

Suggestion: More Nuanced Reputation Score

Instead of a single metric, answerers should be rated along several dimensions. Economists and psychologists both use multidimensional diagrams. The following diagram represents data in 4 dimensions. More dimensions can easily be shown in this type of diagram.

Multi-dimensional data can easily be visualized by outlined shapes

Instead of "bigger is better" (a single number indicating status, with the high score indicating alpha status), more information would allow for more detail, so a fuzzy diagram would show little interaction, while a highly detailed and intricate design would indicate a lot of participation. Some answerers might be stronger regarding some metrics, while being weaker on other metrics.

Instead of displaying a person's score, the shape of their participation would be shown. Some people prefer to be seen as well-rounded, others prefer to be the best on selected aspects and ignore the others. One size does not fit all.

People would start to give pet names for various shapes. Jokes would be made about the shapes.

HR personnel would start to hire teams based on how well these shapes meshed together. Money will be made by timely entrepreneurs because these shapes will quickly be adopted industry-wide. Some will aspire to change their shape.

An entire industry will spring up servicing those who wish to modify their shape.

Suggestion: Encourage and Highlight Elicitation

Elicitation is a difficult skill to master. The current high-scorers have no incentives to employ elicitation, and they act as if on a campaign to eradicate it.

Introduce an Elicitation Phase

A button should be introduced that lets everyone who visits the question page that a potential answerer would like to elicit information. While at least one such button is enabled, no downvotes are possible relating to the question, and the question cannot be closed or moved to another forum by anyone, regardless of their privilege level.

The elicitation mode would time out, but not suddenly or unexpectedly. Instead, it would back off, rather like the Ethernet back-off algorithm for collision resolution used in random access MAC protocols.

Both the potential answerer and the questioner would be given cues that they have a question or a response waiting, and after a period of inactivity the special status ends. I leave the details of the timing undefined for others to discuss.

Suggestion: Restrict Downvoting

Downvoting needs to incorporate:

Accountability (no more anonymous downvoting)
Cost (no more drive-by shootings without consequences)
Time window (vote after the dust settles, not during the elicitation phase)
Public displays of user profiles should prominently display that user's downvotes and upvotes, with links
Aggregate statistics on user profiles of their percentage downvotes and upvotes, trends (absolute and relative), etc.

Serverless E-Commerce

2021-04-14T00:00:00-04:00

As readers of this blog know, I have been chronicling my adventure into Python-powered e-commerce for several months. I have been focusing on Django in general, and Django-Oscar in particular. Webapps made with this technology are almost exclusively run on dedicated real or virtual machines. Serverless computing is a method of providing backend services on an as-used basis. AWS Lambda is the best-known example of serverless computing, and it combines nicely with a CDN like AWS CloudFront.

This blog post discusses 3 goals for an e-commerce system. Two goals are provided by the technology behind serverless webapps:

Enormous and instantaneous scalability.
Pay-as-you-go without an up-front cost commitment.

I have one more goal: very low latency for online shoppers.

The Big Picture

AWS Lambda consists of two main parts: the Lambda service which manages the execution requests, and the Amazon Linux micro virtual machines provisioned using AWS Firecracker, which actually runs the code.

A Firecracker VM is started the first time a given Lambda function receives an execution request (the so-called “Cold Start”), and as soon as the VM starts, it begins to poll the Lambda service for messages. When the VM receives a message, it runs your function code handler, passing the received message JSON to the function as the event object.

Thus every time the Lambda service receives a Lambda execution request, it checks if there is a Firecracker microVM available to manage the execution request. If so, it delivers the message to the VM to be executed.

In contrast, if no available Firecracker VM is found, it starts a new VM to manage the message. Each VM executes one message at a time, so if a lot of concurrent requests are sent to the Lambda service, for example due to a traffic spike received by an API gateway, several new Firecracker VMs will be started to manage the requests and the average latency of the requests will be higher since each VM takes roughly a second to start.

– From How to run any programming language on AWS Lambda: Custom Runtimes by Matteo Moroni.

AWS Lambda Limits

AWS Lambda programs have access to considerable resources, enough for most e-commerce stores. The AWS Lambda runtime environment has the following limitations, some of which can be improved upon with some work:

The disk space (ephemeral) is limited to 512 MB.
The default deployment package size is 50 MB.
The memory range is from 128 to 3008 MB.
The maximum execution timeout for a function is 15 minutes.
Request and response (synchronous calls) body payload size can be up to 6 MB.
Event request (asynchronous calls) body can be up to 128 KB.

CloudFront

Brian Caffey wrote Building a Django application on AWS with Cloud Development Kit (CDK). The website Mr. Caffey's article discusses does not use Lambda, instead his website is always running. So, this option is quite informative and well-thought-out, but it is AWS-specific and does not discuss serverless architecture.

For me, the most interesting part about Mr. Caffey's article is it mentions using 3 origins with AWS CloudFront: (1) an origin for ALB (for hosting the Django API), (2) an origin for the S3 website (static Vue.js site), and (3) an S3 origin for Django assets. Mr. Caffey does not say why he used 3 origins, but feeding one CloudFront distribution from multiple origins would mean that all of their content would appear on the same Internet subdomain.

This means that the extra HTTP handshaking required for certain CORS (cross-origin HTTP requests) requests between subdomains would be avoided; specifically, there would be no need for pre-flight requests. This would make the website seem noticeably faster if users did lots of content editing and/or transactions with the website. My own pet project has users creating and modifying content, and purchasing product, so taking the requirement for the CORS handshakes away would be a win, plus the end user's web browser could reuse the origin HTTP connection, speeding up even non-cacheable requests.

Tamás Sallai wrote How to route to multiple origins with CloudFront – Set up path-based routing with Terraform. Mr. Sallai is a prolific writer!

Edge Computing

Performing computations and serving assets from a nearby point of presence minimizes latency for end users. E-commerce customers much prefer online stores that respond quickly. Edge computing can deliver that experience world-wide, and developers can deploy their work from wherever they are.

AWS Lambda@Edge (console) runs the Lambda computation in one of 13 regional AWS points of presence, one hop removed from the CloudFront edge locations, or at least in the same availability zone at the CloudFront point of presence. Distributed database issues would need to be addressed before significant benefits would accrue from this implementing this decentralized architecture. Unfortunately, Lambda@Edge has some significant restrictions that prevent it from running nontrivial Django apps.

Lambda@Edge Restrictions

From requirements and restrictions on using Lambda functions with CloudFront, it is apparent that it is not possible to run non-trivial Django apps securely at the edge with good performance.

You can add triggers only for functions in the US East (N. Virginia) Region.
You can’t configure your Lambda function to access resources inside your VPC.
AWS Lambda environment variables are not supported.
Lambda functions with AWS Lambda layers are not supported.
Using AWS X-Ray is not supported.
AWS Lambda reserved concurrency and provisioned concurrency are not supported.
Lambda functions defined as container images are not supported.

Until such time as Lambda@Edge removes the above restrictions, Django webapps will continue to be deployed as centralized webapps, which means that ultra-low latency is not possible world-wide.

CloudFront Functions

CloudFront Functions are closer to the user, but have even more restrictions than Lambda@Edge. Alas, CloudFront Functions do not seem likely to be able to support significant computation any time soon.

From “Introducing CloudFront Functions – Run Your Code at the Edge with Low Latency at Any Scale”

Infrastructure as Code (IaC)

For anything bigger than a toy cloud application, Infrastructure as Code (IaC) is table stakes. You’d be hard-pressed to find someone managing anything of scale who thinks letting folks point and click in the console is the optimal route.

– From CloudFormation, Terraform, or CDK? A guide to IaC on AWS by Jared Short, published by acloudguru.com.

Terraform, AWS CloudFormation, Packer, Pulumi, and GeoEngineer are the most popular tools in the category "Infrastructure Build Tools".
– from Stackshare.io

Infographic: Lambda Framework Comparison

Yan Cui at Lumigo.io made this terrific infographic, which compares 9 serverless application frameworks and infrastructure management tools according to opinionatedness and customizability. This article discusses some of those technologies.

From 'AWS Lambda Deployment Frameworks', by Yan Cui at lumigo.io

The trade-off between customizability and opinionatedness is that highly customizable frameworks require more code to do things that opinionated frameworks do more succinctly. On the other hand, very opinionated frameworks are more limited in their abilities. A classic example of an opinionated framework is Ruby on Rails, which is specifically designed for master/detail applications. Other types of applications should use a different framework, or no framework at all.

Two of the technologies on the above infographic are Zappa and Terraform, both of which I discuss in this blog post. Zappa is rather opinionated, while Terraform is very customizable.

AWS Cloud Development Kit (CDK)

AWS CDK provides a programmatic interface for modeling and provisioning cloud resources. Languages supported include Java, JavaScript, .NET, Node.js, Python and Typescript.

Even if AWS is not directly the service provider, awareness of the AWS CDK is important because some other options, for example the Cloud Development Kit for Terraform (cdktf), are based on AWS CDK.

Chalice – Serverless Django on AWS

Chalice is an AWS open-source project that has good traction. This Python serverless microframework for AWS allows applications that use Amazon API Gateway and AWS Lambda to be quickly created and deployed.

The name and logo of this project are suggestive of the Holy Grail. I found the thinly veiled references to Christianity to be off-putting. Religious references have no place in a professional environment. Programmers who work with this project have religious icons, words and phrases continuously presented to them, and they must write words that are strongly identified with Christian doctrine for them to write software. This is forced indoctrination.

Django w/ Zappa & AWS Lambda

Zappa is a popular library for serverless web hosting of Python webapps. Zappa allows Python WSGI webapps like Django to run on AWS Lambda instead of from within a container like AWS EC2. I am particularly interested in using Zappa to package and run django-oscar for AWS Lambda and CloudFront.

Zappa can perform two primary functions:

Packaging – Zappa can build a Django webapp into an AWS Lambda package. The package can be delivered via other mechanisms, for example mechanisms that are not even Python aware.
Deploying – Zappa can deploy and Django webapp to AWS Lambda, and configure several AWS services to feed events to the Django webapp.

Zappa does not provide a means to define additional resources as part of the overall infrastructure. It is also somewhat rigid in how it defines certain resources which can lead to friction when incorporating Zappa within organizations with more rigid requirements on cloud resource management. With Zappa, you are better off allowing it to manage all the pieces needed for your web application on its own and manage other resources with a separate tool such as stacker or Terraform.

… or use Zappa's package command to create an archive that is ready for upload to lambda and utilize the other helpful functions the project provides for use after code is deployed.

– from The Evolution of Maintainable Lambda Development Pt 2 by JBS Custom Software Solutions.

The Zappa documentation is excellent. The project has some rough edges, but the new regime coming on board seem competent and fired up. They have some work ahead to set things straight, but the technical path seems clear.

I think this project deserves special attention. Lots of moldy issues and PRs need to be processed, which a small team could get done fairly quickly. The project might also benefit from someone to hone the messaging. I opened an issue on the Zappa GitHub microsite to discuss this.

This seminal project has been around several years, and other well-known projects that have been developed since Zappa was first released have acknowledged that Zappa provided inspiration. Time to brush it up and set it straight again; its best days lie ahead!

Edgar Roman wrote this helpful document: Guide to using Django with Zappa.

I've messing around with Zappa, will report back.

Videos

Videos of Zappa exist.

This video has got all the right technologies mixed together for me: Serverless Deployment of a Django Project with AWS Lambda, Zappa, S3 and PostgreSQL.

Djambda / AWS Lambda / Terraform

Terraform does not impose a runtime dependency unless the realtime orchestration features are used.

Djambda is an example project setting up Django application in AWS Lambda managed by Terraform. I intend to play with it and write up my experience right here Real Soon Now.

This project uses GitHub Actions to create environments for the master branch and pull requests. I wonder if this project can be used without GitHub actions?

[Terraform] does not provide an abstraction layer for the AWS, Azure, or Google Cloud. It does that deliberately, as you should embrace all aspects when using cloud - not extract a common denominator from the services delivered by the cloud provider.

– From AWS CDK? Why not Terraform? by Wojciech Gawroński.

Serverless Framework with WSGI

The docs describe Serverless WSGI as:

Serverless plugin to deploy WSGI applications (Flask/Django/Pyramid etc.) and bundle Python packages.

I am concerned that the Serverless architecture requires an ongoing runtime dependency on the viability and good will of Serverless, Inc. Any hiccup on their part will immediately be felt by all their users. It would make me nervous to base daily operational infrastructure on this.

Bintray and JCenter Went Poof!

I do not want to rely upon online services from a software tool vendor to run my builds. The Scala community is still recovering from Bintray and JCenter shutting down. I had dozens of Scala libraries on Bintray. I do not plan to migrate them, they are gone from public access.

On February 3, 2021, JFrog announced that they will be shutting down Bintray and JCenter. A complete shutdown is planned for February 2022.

– JCenter shutdown impact on Gradle builds

Trading Autonomy for Minimal Convenience is a Poor Trade

Remember that free products are usually subject to change or termination without notice. Examples abound of many companies whose free (and non-free) products suddenly ceased. There is no need to assume this type of vulnerability, so I block my metaphoric ears to the siren sound that tempts trusting souls into assuming unnecessary dependencies, and I chose tooling that is completely under my control.

What happens if I exceed the fair use policy?

From the Serverless Pricing and Terms page.

We want to offer a lot of value for free so you can get your idea off the ground, before introducing any infrastructure cost. The intent of the fair use policy is to ensure that we can provide a high quality of service without incurring significant infrastructure costs. The great majority of users will fall well within range of the typical usage guidelines. While we reserve the right to throttle services if usage exceeds the fair use policy, we do not intend to do so as long as we can deliver a high quality of service without significant infrastructure costs.

If you anticipate your project will exceed these guidelines, please contact our support team. We’ll work with you on a plan which scales well.

AWS AppRunner

AWS just announced AppRunner. I wonder how suitable it is...

Merging a Remote File with a Local File

2021-04-12T00:00:00-04:00

Today I am once again re-installing WSL2 on one of my laptops. Seems that a Windows 10 installation’s half-life is measured in months, after which time a reset is required. The reset preserves data, but not installed programs and not the WSL setup.

When I set up an OS I often use a pre-existing system’s files as templates for the new system’s files.

Meld

Meld is a fantastic, F/OSS file and directory merge tool. 2-way and 3-way merges are supported. Meld uses X Windows for its user interface. GWSL makes it easy to run X apps on WSL and WSL2.

Shell

$ sudo apt install meld

Merging a remote file with a local file using Meld is easy once you know how. Unless the remote file system is mounted locally, Meld cannot be used to modify remote files and directories, just local files and directories.

Following is the incantation I used to display my local .profile and interactively merge it with my profile on an Ubuntu Linux machine called gojira.

Shell

$ meld ~/.profile <(ssh mslinn@gojira cat .profile)&

The above runs ssh in a subshell, logs in as mslinn to the machine called gojira and then displays the contents of .profile on gojira. Meld compares the output of cat with the local copy of ~/.profile, and displays the differences:

😁

Meld makes it easy to reconcile file versions.

Visual Studio Code Workspace Settings

2021-04-11T00:00:00-04:00

For me, the killer feature that Visual Studio Code is how it integrates the Windows user interface with working on WSL and WSL2. Programs residing on the active WSL OS image execute natively on that OS, while VSCode continues to run as a native Windows application. This is possible because VSCode installs a proxy on the target OS. The proxy does the bidding of the Windows executable.

Getting a project to execute on the target OS instead of the host OS can be tricky. I have found that using a workspace to hold a collection of VSCode projects is very helpful, because the definition of the collection also defines how they are handled.

WSL projects have different types of VSCode workspace entries than Windows entries do. They are easy to recognize and change once you know what to look for. The two possibile types of VSCode workspace project entries in a .workspace file are:

WSL Project – "uri": "vscode-remote://wsl+ubuntu/path/to/vscode/project"
Windows Project – "path": "C:\\path\\to\\vscode\\project"

The following VSCode workspace file has both types of entries. For me, this is an error; I only want WSL projects. My task is to change the yellow highlighted Windows project and make it look like the other WSL projects.

aw.workspace.code-workspace

{
  "folders": [
          {
                  "uri": "vscode-remote://wsl+ubuntu/var/sitesUbuntu/www.ancientwarmth.com"
          },
          {
                  "uri": "vscode-remote://wsl+ubuntu/var/work/django/django"
          },
          {
                  "uri": "vscode-remote://wsl+ubuntu/var/work/django/oscar"
          },
          {
                  "uri": "vscode-remote://wsl+ubuntu/var/work/ancientWarmth/ancientWarmth"
          },
          {
                  "path": "../../var/work/django/main"
          }
  ],
  "remoteAuthority": "wsl+Ubuntu",
  "settings": {
          "liveServer.settings.multiRootWorkspaceName": "www.mslinn.com",
          "python.pythonPath": "/var/work/django/oscar/bin/python",
          "git.ignoreLimitWarning": true,
          "sqltools.connections": [
                  {
                          "previewLimit": 50,
                          "server": "localhost",
                          "port": 5432,
                          "driver": "PostgreSQL",
                          "name": "Ancient Warmth on Camille",
                          "database": "ancient_warmth",
                          "username": "postgres",
                          "password": "hithere"
                  }
          ]
  }
}

All I need to do is change this entry:

"path": "../../var/work/django/main"

To:

"uri": "vscode-remote://wsl+ubuntu/var/var/work/django/main"

😁

The modified entry will cause VSCode to launch the project from WSL, instead of Windows.

A Python Virtual Environment For Every Project

2021-04-09T00:00:00-04:00

Python virtual environments are cheap to make and use – unless you are unfortunate enough to program in native Windows. I have adopted the habit of making a Python virtual environment (venv) for each significant Python project, plus a default venv for trivial Python work.

Why Virtualize Python?

It is better to use virtualized user- and project-specific Python instances, instead of working with a system-wide installation of Python. This allows you to install and upgrade Python packages without using supervisor privileges. Also, virtualized instances allows you to work on many different independent Python projects at the same time, without package version collisions.

Docker is over-sold. It adds unnecessary complexity to software projects. Instead of virtualizing the entire software environment, as docker attempts to do, virtualizing the Python programming environment as described in this blog post, Ruby rbenv, or node.js nvm are much easier and more productive approaches.

I think docker has been pushed hard in the media because it is a gateway technology to PaSS. This is a trend that PaSS vendors like AWS and Azure want to encourage, but customers are pushing back.

Python’s venv Virtualization Module

Venv is a tool to create isolated virtual Python environments. It has been included with Python since Python v3.3, which was released 11 years ago.

PEP 405 specifies Python’s venv virtualization module.

Deprecated Virtualization Modules

Python 3.6 was released 6 years ago. It deprecated the other virtualization modules, pyenv and virtualenv. Instead, use venv, as described in this blog post.

Extra Installation Steps for Ubuntu

Venv was included with Python on Ubuntu until Ubuntu 23.04 It virtualizes all the common Python executables.

Debian changed pip’s behavior as a result of PEP 668 – Marking Python base environments as “externally managed”. This affects Ubuntu because it is downstream. Starting with Ubuntu 23.04, you need to install venv by typing sudo apt install python3.xx-venv, where xx is the minor version number of Python that is installed. For Python 3.11, the following command is required:

Shell

$ yes | sudo apt install python3.11-venv

Some software projects use meta-packages to specify version-agnostic dependencies. Perhaps Python on Ubuntu will do so in the future.

In order to virtualize pip, venv needs to invoke ensurepip, which is not installed by default on Ubuntu. On Ubuntu systems, the ensurepip command is provided by a package called python3-pip.

Shell

$ yes | sudo apt install python3-pip

You could install both of the above packages together, of course:

Shell

$ yes | sudo apt install python3.11-venv python3-pip

Manually Creating a VEnv

The following demonstrates how to create a new virtual python environment in the ~/venv/default/ directory. Intermediate directories, such as venv in this example, will be created as required.

Shell

$ python3 -m venv ~/venv/default

Re-running this command at a later date will update the version of Python in an existing venv. Each time you run this command, all pip packages are removed.

At this point, the virtual environment just contained executable images for Python.

Shell

$ ls ~/venv/default/**
/home/mslinn/venv/default/lib64@  /home/mslinn/venv/default/pyvenv.cfg

/home/mslinn/venv/default/bin:
Activate.ps1  activate  activate.csh  activate.fish  pip*  pip3*  pip3.10*  python@  python3@  python3.10@

/home/mslinn/venv/default/include:

/home/mslinn/venv/default/lib:
python3.10/

VEnvs are Nearly Free

The cost of a venv is virtually free for all OSes except native Windows. This is because on all OSes except native Windows, the executable images are linked by default, so they do not require much storage space.

The ls command below shows that the python program in the default venv is linked to /usr/bin/python3.10.

Shell

$ ls -go ~/venv/default/bin/python
lrwxrwxrwx 1 18 Apr 9 06:01 /home/mslinn/venv/default/bin/python -> python3*

Create a VEnv for Every Python Project

My projects are stored under the directory pointed to by $work.

My standard procedure when making a Python project called $work/blah is to also create a venv for it at within the project, at $work/blah/.venv/. I add an entry to .gitignore that merely consists of a line that says .venv/.

A bash alias could be defined called blah that makes the project directory current and activates the venv:

~/.bash_aliases

alias blah="cd $work/blah; source ./venv/bin/activate"

😁

Now you could type blah at a shell prompt, and you would be working on that project. Boom!

Deactivate a VEnv

Stop using venvs with deactivate. Notice that the prompt changes.

Shell

(aw) $ deactivate

$

Directory-Locked Python Virtualization

After setting up a Python virtual environment, a quick examination of the pip script shows that it is hard-coded to the directory that it was made for:

Shell

$ head -n 1 ~/venv/aw/bin/pip
#!/home/mslinn/venv/aw/bin/python

For virtualized environments, such as Docker, this means that a Python virtual environment created without Docker can only be used within a Docker image if the path to it is the same from within the Docker image as when it was created.

For Further Reading

Python Best Practices for a New Project in 2021. This article is already becoming dated. YMMV.

Summary

Demonstrated how to make an alias for working with Python virtual environments (venvs) that are coupled with Python projects.
Deactivating the current venv was demonstrated using the deactivate command, provided with every venv.
Locked directories mean that Python virtual environments should normally only be created in the same environment they are intended to be used.