Mike Slinn's Blog

Working With JTabs

2024-07-22T00:00:00-04:00

I like the tabs provided by jsuites jtabs, because it does not use jQuery, so tabs are lightweight, and because they work well. However, the documentation is thin.

These are my notes.

URLs and URI Fragments

URL containing a hash with value tab2

https://domain.com/jtabs_demo.html#tab2

Defining a Few Terms

I expect readers of this website will generally know what a Uniform Resource Locator (URL) is. Wikipedia has a good page if you would like a quick refresher.

A URI fragment, is delimited from the web page by an octothorpe (#), also known as a pound sign, also known as a hash mark. For example, the URL http://domain.com/page1.html#tab2 has a URI fragment of tab2.

Popular documentation in 2024 often refers to URI fragments as hashes, so we would normally say that the hash value of the given URL is tab2.

Two Uses For a URL Hash

When serving a web page whose URL contains a hash, the user's web browser scans the web page, looking for an HTML element id that matches the hash value. If a match is found, the web browser tries to scroll the web page up or down, so the element with the matched id is as near to the top of the browser window as possible.
A few lines of JavaScript can cause a URI fragment to activate a specific HTML tab amongst a set of tabs. I will show how to do that using jsuites jtabs in this article.

Opening a Tab From a URL Hash

Jsuites needs the following HTML structure to define a jtabs instance for every element with class jtabs. The collection of jtabs is saved in the JavaScript variable, also called jtabs:

jsuites jtabs HTML

<div class="jtabs">
  <div id="jtabs-headers">
    <div id="tab1">Tab 1</div>
    <div id="tab2">Tab 2</div>
    <div id="tab3">Tab 3</div>
  </div>
  <div id="jtabs-content">
    <div>
      This is the content of tab 1.
    </div>
    <div>
      This is the content of tab 2.
    </div>
    <div>
      This is the content of tab 3.
    </div>
  </div>
</div>

JavaScript statement to create a jtabs instance

jtabs = jSuites.tabs(document.getElementsByClassName('jtabs')[0]);

The following 3 JavaScript incantations that all do the same thing in different ways. They all use the URL hash to activate the second tab of the jtabs instance, labeled "Tab 2", in the following jsuites jtab.

The following JavaScript statement invokes the open method provided by the jtabs instance and passes the value 1. Because the first tab has index 0, the second tab has index 1. The second tab is thus revealed and is made active.

Invoke open and pass an index

jtabs.open(1); // origin 0

The following JavaScript statement also also opens the tab with id tab2. This JavaScript statement obtains the HTML element with id jtabs2 and passes it to selectIndex. The second tab is then revealed and made active.

Invoke selectIndex and pass an HTML element

jtabs.selectIndex(document.getElementById("jtabs2"));

The third JavaScript incantation is the most interesting. If a URI fragment (aka a hash) was specified, obtain the HTML element for that id and pass it to selectIndex as before. The second tab is then revealed and made active.

Parse the URI fragment if specified and invoke selectIndex

const desiredTab = window.location.hash.split('#')[1];
if (typeof desiredTab !== 'undefined')
  jtabs.selectIndex(document.getElementById(desiredTab));

JTabs Demo

I wrote this example to show how to style tabs and programmatically open them. The following iframe was embedded with URL jtabs_demo.html#tab2, which is why Tab 2 was active. You can click on the tabs to verify the page is live.

Here is the complete HTML for the above JTabs demo:

jtabs_demo.html

<html>
  <head>
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/jsuites/dist/jsuites.min.css" type="text/css" />
    <style>
      body {
        background-color: rgba(242, 215, 158, 0.55);
        font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
        font-size: 16px;
        margin-left: 10%;
        margin-top: 2em;
        width: 80%;
      }

      .jtabs-content {
        background-color: #F7E8B4;
        border: 2px solid rgba(102, 102, 102, 0.226);
        border-radius: 0 8px 8px 8px;
        padding: 10px;
      }

      .jtabs .jtabs-headers {
        div {
          border-color: rgba(102, 102, 102, 0.226);
          border-style: solid;
          border-width: 1px 1px 0 1px;
          border-top-left-radius: 6px;
          border-top-right-radius: 6px;
          margin-left: 6px;
        }

        div:not(.jtabs-selected) {
          background-color: rgba(216, 162, 1, 0.184);
          color: rgba(102, 102, 102, .8);
        }

        div.jtabs-selected {
          background-color: rgba(216, 162, 1, 0.479);
          color: black;
        }
      }
    </style>
  </head>
  <body>
    <h1>JTabs Demo</h1>
    <div class="jtabs">
      <div id="jtabs-headers">
        <div id="tab1">Tab 1</div>
        <div id="tab2">Tab 2</div>
        <div id="tab3">Tab 3</div>
      </div>
      <div id="jtabs-content">
        <div>
          This is the content of tab 1.
        </div>
        <div>
          This is the content of tab 2.
        </div>
        <div>
          This is the content of tab 3.
        </div>
      </div>
    </div>
  </body>

  <script defer src="https://cdn.jsdelivr.net/npm/jsuites/dist/jsuites.min.js"></script>
  <script>
    document.addEventListener('DOMContentLoaded', function() {
      jtabs = jSuites.tabs(document.getElementsByClassName('jtabs')[0]);

      // The following two statements accomplish the same task: open tab 2
      // jtabs.open(1); // origin 0
      //jtabs.selectIndex(document.getElementById("tab2"));

      // The following opens tab2 if the URL ends with #tab2
      const desiredTab = window.location.hash.split('#')[1];
      if (typeof desiredTab !== 'undefined')
        jtabs.selectIndex(document.getElementById(desiredTab));
    });
  </script>
</html>

Thermal Control

2024-07-05T00:00:00-04:00

This article discusses some of the physics and a few of the electrical components for controlling temperature in a consumer electronics product.

Mini-Series

This article can be read standalone; however, is part of a mini-series that discusses how to increase the storage capacity and performance of the Ableton Push 3 Standalone (P3S). If you own a P3S, or are thinking of purchasing one, you might want to first read my review.

The articles in this mini-series are:

Thermal Pads

A thermal pad is made from thermal interface material; its purpose in the article I wrote entitled Ableton Push 3 NVMe Replacement is to thermally connect the chips on the NVMe drive to a metal heat sink. Thermal pads are often baby blue; however, as shown in the above photographs, the P3S uses a pink thermal pad between the CPU module and the bottom plate, and, as we saw at the beginning of this article, another pink thermal pad between the external heat sink and the bottom plate.

Various thicknesses of thermal pads

Depending on the quality and conditions of use, a thermal pad can last anywhere from 5 to 10 years.

The pink thermal pad cools the P3S CPU module

Folklore vs. Provable Facts

Some manufacturers state that thermal pads should not ever be reused, citing the introduction of air gaps and bubbles. Apparently reusing a thermal pad would cause the thermal pads to act more as an insulator than as a conductor. I do not have data that would indicate how well-founded the statement is. Someone should test this; the information source obviously has a profit motive.

Also from two manufacturers: When as thermal pad is removed, the surface should be cleaned with alcohol wipes and a microfiber cloth to ensure no traces are left behind. Again, I would like to see data on this.

Thermal Conductivity

Heat transfer only occurs 3 ways: via conduction, convection, and radiation. Of the three, conduction is by far the most effective.

NOAA: The Transfer of Heat Energy

Mode	Transfer Mechanism	Material
Conduction	Diffusion of electrons and phonon vibrations	Solids
Convection	Density differences cause currents	Gases and liquids
Radiation	Electromagnetic radiation	Non-opaque mediums

This article focuses on conduction, because it is the dominant cooling mode for the P3S.

The thermal conductivity of a material characterizes its ability to transfer heat. For solids, thermal conductivity is due to the flow of free electrons, rather like how electricity is conducted. Materials with high thermal conductivity transfer heat faster than materials with low thermal conductivity. The thermal conductivity tells you the rate at which thermal energy is transported across a unit length of a material that has a unit temperature difference applied across that length.

Thermal conductivity in an electrical device depends upon:

Density of material
Pressure
Temperature
Temperature differential
Material structure

In the SI unit system, thermal conductivity is measured in watts per meter per kelvin (W/mK). This defines the number of watts conducted per meter thickness of the material, per degree of temperature difference between the two surfaces.

The following table is ordered by increasing thermal conductivity, with thermal insulators at the top and the best thermal conductors at the bottom.

Material	Comment	Conductivity
Air	Effectively a thermal insulator.	0.026 W/mK
Powder coating	Used on computer chassis, and on the P3S bottom plate. Effectively a thermal insulator.	0.174 W/mK
Anodized aluminum coating	Used to coat the P3S heat sink underneath the P3S bottom plate. Anodizing means adding a layer of alumina to the aluminum surface. This thin layer of oxide is effectively a thermal insulator. Anodized aluminum has a higher surface area than unfinished aluminum and, therefore, radiates more infrared energy; however, radiation is orders of magnitude less effective than conduction for cooling.	0.53-1.62 W/mK
Marble	Used for kitchen table tops.	2.8 W/mK
Thermal paste	Commonly used between a CPU and its heatsink. Shelf life is 3 to 5 years, durability (working life) is 5 to 8 years.	5 — 8 W/mK
Thermal pads	Easily replaced	3 — 16 W/mK The blue thermal silica gel pads that I purchased were rated at 6 W/mK. The pink thermal pads of the P3S might be rated at 14.5 W/mK.
Low-carbon steel	Sheet metal, used for computer chassis, which might include the P3S bottom plate. Surface treatment, such as anodizing, generally reduces this value.	34-54 W/mK
Liquid metal paste (Conductonaut)	Can be highly dangerous for electronics and aluminum parts.	73 W/mK
Solid aluminum	Often used in heat sinks, for example underneath the P3S bottom plate. However, the P3S exterior heat sink is anodized, which greatly reduces its thermal conductivity.	150-205 W/mK
Solid copper	Used on motherboards	357-413 W/mK
Heat pipe	Used in a wide variety of electronics, including laptops and mobile devices	4,000 to 100,000 W/mK

Notice the huge disparity between the thermal conductivity of thermal pads, the metals and the coatings that the bottom of the P3S is constructed from. This tells us that although thermal pads are much better than simply relying on convection or radiation, thermal pads and the coatings greatly restrict the transfer of heat generated by the P3S electronics.

If a heat pipe was used effectively, the overheating problems of the P3S would completely go away. Furthermore, the heat pipe would allow more powerful and faster electrical components could be used.

Heat Pipes

Photo by Kristoferb at English Wikipedia, CC BY-SA 3.0' class="imgImg rounded shadow" src="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.png" style='width: 100%; ' title='Heat pipes in a laptop
Photo by Kristoferb at English Wikipedia, CC BY-SA 3.0' />

Heat pipes in a laptop
Photo by Kristoferb at English Wikipedia, CC BY-SA 3.0

Thermal pads, fans and heat pipes are commonly used in laptops for dissipating heat. Heat pipes do not require power, are very light, and provide extremely high heat flux — from 10x to 200x more than pure copper. Heat flux (also known as thermal flux, heat flux density, heat flow density and heat flow rate intensity) is the amount of heat energy passing through a surface, measured as the flow of energy per unit of area per unit of time. It is a vector quantity.

Heat pipes for thermally controlling a satellite

The following video by James Orgill of The Action Lab shows that heat pipes transfer heat energy dramatically faster than solid materials.

This video explains how heat pipesmic work in detail:

Perhaps Ableton might look at using a heat pipe in a future version of the P3S that features a more powerful CPU and larger NVMe drive.

Heat Sinks

Heat pipes can be combined with other components for heat transfer, such as extruded heat sinks, die-cast heat sinks, fin-stack heat sinks, and skived heat sinks. Heat transfer materials thermally bond the heat pipes to the heat sinks. This creates a thermal management system that can be individually tuned for each application.

Stacked fin heat sinks offer better performance than extrusion heat sinks, due to thinner fin profiles leading to higher surface area and lower pressure drop.

Skived heat sinks provide the best possible thermal conductivity between the fins and the base

Vapor Chambers

Vapor chambers, also known as planar heat pipes, are relatively new. Tubular heat pipes act as heat dissipaters, while planar heat pipes act as heat spreaders.

Vapor chamber assembly

Aftermarket Upgrade

I can imagine an aftermarket upgrade that would increase the height of a PS3 slightly, uses a heat pipe or vapor chamber for cooling, and has a variable-speed fan. This would support the biggest, fastest and hottest NVMe drive you can afford. I wonder if it might be possible to overclock the CPU if it was properly cooled?

Sources

The University of Cambridge has an online course which goes into the physics of conductivity: Introduction to thermal and electrical conductivity.

Upgrading Modular PSUs

2024-06-28T00:00:00-04:00

This article discusses modular PSUs conforming to the family of standards for power supplies designed for ATX form factor motherboards, specifically, the ATX 2.x and ATX 3.x PSU specifications. These two major versions of the ATX PSU specification are collectively called the ATX12V specifications. ATX12V PSUs are the most common types of PSU today, having evolved from the original ATX PSU specification.

Some of the information in this article applies generally to all modular PSUs, in particular the summary.

ATX12V Standard

The ATX12V standard is upwards compatible. This means you can install a PSU that conforms to a more recent version of the standard in an older motherboard that only supports an older version of the standard. For more information, please see the ATX / ATX12V Power Supply Design Guide.

ATX 2.x PSUs

The current version of the ATX12V standard, v2.53, was released in December 2021. The most recent version of the standard that I could find a PDF of is ATX Specificaiton Version 2.2, published in 2004.

This is the ATX 2.2 24-pin motherboard power connector pinout:

Pin numbering is clearly shown' class="imgImg rounded shadow" src="/blog/images/psu/atx_2_pinout.png" style='width: 100%; padding: 1em;' title='ATX 2.2 24-Pin Connector Pinout Diagram
Pin numbering is clearly shown' />

ATX 2.2 24-Pin Connector Pinout Diagram
Pin numbering is clearly shown

The above diagram shows that pins are numbered in columns from left to right. pin 1 is at the top left, and pin 24 (the highest-numbered pin) is at the bottom right.

See previous diagram for full pinout numbering' class="imgImg rounded shadow" src="/blog/images/psu/atx_2_2_pinout.png" style='width: 100%; ' title='All ATX 2.2 Cable Pinout Diagrams
See previous diagram for full pinout numbering' />

All ATX 2.2 Cable Pinout Diagrams
See previous diagram for full pinout numbering

ATX 3.x PSUs

ATX 3.x is upwards compatible with ATX 2.x, which means that an ATX 2.x tester can determine if an ATX 3.x PSU works, except for being able to test either of the new high-power PCIe 5.0 connectors for top-tier GPUs.

ATX 3.0 introduced the 12VHPWR connector, which was problematic and was replaced by the new 12V-2×6 connector for ATX 3.1. To learn more about ATX 3.0, see What is ATX 3.0 and why you need to know before your next PC upgrade.

Anandtech has an excellent description of ATX 3.1 that explains why the 12VHPWR connector introduced by ATX 3.0 was a failure and discusses the 12V-2×6 connector introduced by ATX 3.1 to replace the ATX 3.0 12VHPWR connector.

I would not install an ATX 3.x PSU unless the computer had an NVIDIA 4090 because this PSU standard is still bleeding edge.

Modular PSUs

I recently encountered a bad batch of computer power supplies (PSUs). This article specifically discusses replacing / upgrading modular PSUs. Most of the issues raised do not arise unless the PSU is at least partially modular.

Non-modular PSUs lack one significant source of failure: only the cables provided with the PSU can be used because they are permanently attached.

Cables

Modular power supplies have removable cables; you only connect the cables that you need. This makes building computers easier and improves airflow within a computer.

If you attach a cable from a modular PSU to a modular PSU of a different model, then some or all of the electronics in your computer may be fried. The actual results depend on many variables.

Replace old PSU cables with the cables that come with the replacement PSU

You might want to get in the habit of marking each end of every cable with nail polish, and also paint a dab on a small portion of the PSU label as well. Using a unique color of nail polish for every different model of PSU that you own will make it easy to recognize the cables that came with each model of PSU that you have. Nail polish will not stick to the plastic used in some connectors, so if that is the case, place the dabs of nail polish on the wire sleeves, if present, or on the wires themselves.

Four Special Wires

Four of the wires in 24-pin ATX v2 and v3 cables have special functions. It is important to test new PSUs before you install them, to verify that the signals in these wires are functioning properly. This section describes the signals to be tested. The next section describes how to test them.

PS_ON (power on) on pin 16 is a signal originating from the motherboard to the power supply. This pin’s voltage is internally pulled up to +5 V inside the power supply. The PSU will attempt turn on after the voltage on this pin is pulled low by being connected to ground.
PWR_OK (power good) on pin 8 is an signal from the power supply that indicates that the +5VDC and +3.3VDC voltages provided by the PSU’s 24-pin connector have stabilized. The motherboard will not attempt to power up unless this signal is provided. Normally the singal remains low for a brief time (100 to 500 milliseconds) after the PS_ON signal is pulled low; however, the signal will only go high after the PSU ascertains that its +5VDC and +3.3VDC outputs are stable and within normal operating parameters.
+5 V_SB (+5 V standby) on pin 9 supplies power even when the rest of the supply wire lines are off. This can be used to power the circuitry that controls the power-on signal.
+3.3 V sense on pin 13 allows the PSU to sense the voltage drop in the PSU wiring.

Use a PSU Tester

EVGA uses this tester to test their PSUs

Always use a PSU tester to test a new power supply before you install it.

Most PSU testers have no battery and are completely powered by the PSU's 20-pin or 24-pin interface. If no power is provided by the 20-pin or 24-pin interface, the tester will not function.

To use the PSU tester shown above, plug the 24-pin cable between the PSU and the right-hand-side connector on the tester. Now turn on the PSU. You should observe:

The tester should not start beeping.
PG stands for Power Good, also known as “Power OK”. As previously mentioned, the motherboard will not attempt to power up unless this signal is provided. The LCD indicates how many milliseconds it takes to receive the Power Good signal from the PSU. If it reads 0, your PSU is supposedly non-functional.

PG was inherited from the original IBM PC in the early 1980s. The PC’s PSU had a PG signal because of the RAM chips it used and the specific internal design of the PSU. The PSU’s design was such that the -5V rail wasn’t stable immediately, and this in turn meant that the RAM wasn’t 100% reliable at initial power on, so if the mobo tried to boot, it might crash due to memory corruption. PG was designed to make the mobo wait (in hardware) a few hundreds of milliseconds before booting so that the system would be stable.

0ms would indicate that the PS is telling the mainboard that power is good from the get-go when in fact it may not be.

Even today there is a small window of opportunity at the instant of switch on (around 50-100 ms) where supply voltages aren’t stable. If the system tries to begin POST during this time it may crash, so PG is used to generate a hardware delay so that booting doesn’t start until the voltages are stable. A 0ms PG means that the PG signal is probably faked.

Modern motherboards have largely done away with the need for hardware PG (they either simply don't require a delay, or implement the delay by other means), although it is still part of the ATX specs, page 21.

A faked PG signal says nothing about the actual functionality of the PSU, and simply cannot be used as a judgment of the functionality of the supply, although the use of a faked PG design can imply that the PSU is of an inferior design quality.
– @Merlin on Whirlpool 2007-Apr-12
The text under the +12V2 heading will flash LL when no PCI-E device is connected. Its value is only relevant when a PCI-E connector (4-pin, 6-pin or 8-pin) is used by an IDE (HDD) or a SATA drive, or a floppy disk.

Replace The Cabling When Replacing a Power Supply

When replacing a power supply with another model or brand, even if it seems almost identical, also replace the power cabling. Seemingly trivial differences between PSU models can cause the new power supply to fail if an old power supply's cabling is used, and the components it is electrically connected to might be damaged.

💥 💥

In other words, unless you are really, totally, absolutely 100% sure that the cables are in every way identical, replace them when you replace the PSU. Leaving the previous cabling in place and just swapping the power supply can fry the power supply, the motherboard, and more.

Corsair

I have been purchasing PSUs for the computers that I build since the early 1980s. Some of those PSUs were made by Corsair.

Following the lead of other vendors, Corsair introduced its first fully modular PSU in 2012 for computer enthusiasts to build their own computers.

Technical support is not provided, citing ‘legal liability issues’. I call bullshit.

‘Legal liability issues’

Technical information for specific products is not organized properly and contains inaccurate compatibility statements. Providing incomplete information seems to be a goal, perhaps again due to paranoia about legal liability. I attribute the inaccuracies to sloppiness.

Corsair's PSU products are described as mysterious black boxes, with stern warnings of fatal hazards if opened.

Connector Labels

Corsair has been inconsistent about how it has labeled the modular connectors over the years.

In the above photograph, we see:

Three connectors were clearly labeled with the PSUs that they were compatible with (“TXM/HX Series only”, “760/860AX ONLY”, “Type4”)
One connector had a label that only had meaning if you knew the PSU it was for (“CD08”)
One connector labels its purpose but does not provide any clue about the PSU that it was designed for (“PCI-E”). As we shall see, not all PCI-E cables are created equal.

Not shown are connectors without labels.

The inconsistent and unhelpful labeling continues today.

Inaccurate Compatibility Statements About Cabling Variants

The main Corsair product page for PSUs says:

The only difference between Type 3 and Type 4 cables is the pinout of the 24-pin ATX cable; all other cables (SATA, PCIe, etc) are the same.

– PSU Cable Compatibility Statement

However, that is incorrect. Corsair’s blog states that the 24-pin cables have the same pin-outs, and that the PCIe and EPS12V cables are not the same. This is also incorrect, because the 24-pin cables have different pin-outs.

The Type 4 cables have the same pin-out as Type 3 cables, but include small, solid capacitors on the +12V, +5V and +3.3V leads on the 24-pin, PCIe and EPS12V cables.

– Explanation of RMi's New Type 4 Cables

In summary, I understand all of the above to mean that, except for the 24-pin ATX cable, type 4 cables are compatible with type 3 cables, but they provide cleaner power.

Diagnostics

Little diagnostic information is provided. "Use a paperclip," they say, and then they also mention that might not be very helpful. In general, paperclips have limited value as a diagnostic tool for debugging PSUs. Next, Corsair discusses how to use a PSU tester for an ATX 24-pin cable but does not talk about how to test the other cables.

Corsair provides a table on the same page showing pin signal levels for an ATX 2 24-pin cable but neglects to indicate how pins are numbered.

Alternative Manufacturers

A brief search yielded the following PSU manufacturers, all of whom provide product support:

EVGA SuperNOVA 1000 GT

I moved away from the bad batch of Corsair PSUs and purchased an EVGA SuperNOVA 1000 GT.

NewEgg now has a new AI-generated customer recommendation summary for their products, and those comments helped me make a purchasing decision. I recommend the NewEgg purchasing experience.

NewEgg Customer Review Summary

Setup

I opened the EVGA SuperNOVA 1000 GT box and found a proper printed manual, in color. The manual showed a US phone number for support (888-881-3842) and an email address, support@evga.com.

The included 'tester' was just a jumper that could perform the same function as a paperclip. It was almost completely useless. I used my tester, shown above. The test failed with PG=0.

Communication Breakdown

I called the EVGA tech support phone number on a Thursday at 4 PM ET (1 PM PT - California time), but the message said EVGA was closed until Monday at 9 AM PT. That was odd. I also called their other customer support phone number, (714) 528-4500, but got the same message.

I emailed EVGA’s support address, but the email bounced back with the following error.

<support@evga.com>: host d290004a.ess.barracudanetworks.com[209.222.82.255]
  said: 550 permanent failure for one or more recipients
  (support@evga.com:blocked) (in reply to end of DATA command)

I emailed their European support address, supportEU@evga, with the same error.

I then attempted enrolled in the EVGA Member online service. However, I could not log in due to several bizarre errors.

The following Monday I was able to call EVGA. I left a message.

A tech from EVGA called back the same day, then escalated to Chris Bencivenga, EVGA Operations Manager, who called the day after (Tuesday). Both of them were on top of things; they had read this article and done their homework.

I had a half-hour conversation with the second tech on July 16, 2024. Chris said that EVGA PSUs should pass using the type of tester that I used without beeping or flashing LEDs. Chris also said that their PG values are typically 70-85 ms. Chris offered to exchange my unit via prepaid AIR shipping and customs duty. I thought that was pretty nice.

It turned out that something about my email signature tripped EVGA’s spam filters. Once I sent plain text emails, my transmissions to EVGA succeeded.

I received the replacement PSU, and immediately tested it. The PSU failed with PG=0, just like the others. I made a short video that demonstrated the failure and sent it to Chris. Now I am awaiting a response.

Summary

Replace all cables attached to a PSU when upgrading it.

I forgive Corsair's bad batch of product, it happens. I do not forgive the policy of not providing technical support.

I will continue to update this article.

Supersizing a Partition and Its File System

2024-06-12T00:00:00-04:00

This article discusses how to clone a Linux drive containing partitions with ext4 file systems. If the original drive has the same storage capacity as the clone, your task is complete, and you do not need to read this article.

Mini-Series

The articles in this mini-series are:

Because a Linux file system is being modified, Linux must be used to perform the modification. It does not matter if the Linux OS runs on a physical machine or in a virtualized machine, so long as the file system to be modified is accessible from Linux. I wrote this article using an Ubuntu instance running in VirtualBox, as described in VirtualBox Setup. It does not matter which OS you use to run VirtualBox; Windows, Mac and Linux all work equally well in this regard.

Two Scenarios

This article works through the following scenarios when working with a cloned Linux drive that has more storage capacity than the original drive:

Modestly enlarge a partition containing an ext4 file system.
Supersize a partition containing a ext4 file system by recreating it such that the resulting file system is much larger than the original.

But first, some important technical details.

Enlarging a File System

After testing to verify that a drive has been successfully cloned to a larger storage device, at least one of the partitions on the newly cloned NVMe drive must be enlarged, or the extra storage capacity of the larger drive will not be used. The file system within the partition also needs to be enlarged. However, there are limits to how much a file system should be enlarged instead of recreating it in the new and larger partition.

Tiny file systems like the default file system in a Push 3 Standalone, which uses 1K blocks, should not be enlarged much, if at all. Instead, tiny file systems should be recreated at the size they will be used. See the next section for an explanation of the technical details of the Linux file system internals behind this statement.

Linux File System Internals

Theodore Tso is a maintainer for the ext4 file system. He is also one of the two authors of e2fsprogs, a collection of 31 commands for manipulating Linux ext3 and ext4 file systems. Theodore is one of the best authorities that one might wish for regarding how to work with ext4 file systems.

For a small file system, we use a 1k block size, to reduce space wasted by internal fragmentation. We also reduce the size of the ext4 journal to reduce file system overhead. But both of these changes have significant performance impacts—which might not matter if the file system is meant for a small, slow USB thumb drive, but if you are trying to use this for a higher-performance system, you'll be sorry.

We also don't enable 64-bit block numbers, since this increases metadata overhead. But like the 1K block size, this limits the maximum size that is supported by the base file system, and it is not at all simple to change the fundamental aspects of the file system so that the file system is scalable and performant. So this idea of "drop a small file system on a device" and then blowing it up to a huge size is a Really, REALLY, REALLY bad idea.

There are some ways that this can be mitigated (you can give options to mke2fs to use a 4k block size, enable 64k blocksize, and force a larger journal size), but if the file system has some files placed in the small file system, the block placement of the small file system will not be optimal after the file system is blown up to a large size.

– e2fsprogs author Theodore Tso

Theodore also explains why a tiny file system like the one shipped with the Push 3 Standalone should not be resized past 1 TB:

The problem is that when you create a file system that is tiny (e.g., 200MiB), we use a blocksize of 1K. With a block size of 1K, the backup superblock is located at block 8193 (e.g., the first block of the second block group in the file system). But when using a block size of 1K, the number of block group descriptors is so large that it completely fills the first block group and spills into the second block group:

Group 0: (Blocks 1-8192) csum 0x0e73 Primary superblock at 1, Group descriptors at 2-8193

and so when the backup superblock is written, it trashes the block group descriptors:

Group 1: (Blocks 8193-16384) csum 0x98cc [INODE_UNINIT] Backup superblock at 8193, Group descriptors at 8194-16385

With the above in mind, let’s work through resizing the file system on a partition. We will consider both of the scenarios introduced at the beginning of this article.

Installing Gparted

The Linux gparted GUI program can change the size of a drive partition, and will attempt to enlarge or shrink the file system as required.

I was able to run gparted on a WSL instance; however, I did not know how to map USB drives to the WSL VM, so gparted could work its magic. Maybe one day I will fight through the issues involved, but there is no reason for me to work that hard for no benefit.

Instead, I installed Ubuntu Desktop 24.04 in a new virtual machine. Please see the article I wrote on VirtualBox for details. Once Ubuntu was running in a VM, I started a terminal session and installed gparted on the new Ubuntu VM with this incantation:

Shell

$ yes | sudo apt install gparted

Gparted Help

The help message for gparted is:

Shell

$ man gparted
GPARTED(8)                  GParted Manual                  GPARTED(8)

NAME
        gparted  -  GNOME Partition Editor for manipulating disk parti‐
        tions.

SYNOPSIS
        gparted [device]...

DESCRIPTION
        The gparted application is the GNOME partition editor for  cre‐
        ating, reorganizing, and deleting disk partitions.

        A  disk  device  can be subdivided into one or more partitions.
        The gparted application enables you to change the partition or‐
        ganization on a disk device while preserving  the  contents  of
        the partition.

        With gparted you can accomplish the following tasks:
        - Create a partition table on a disk device.
        - Enable and disable partition flags such as boot and hidden.
        -  Perform  actions with partitions such as create, delete, re‐
        size, move, check, label, copy, and paste.

        More documentation can be found in the application help manual,
        and online at:
        https://gparted.org

EXAMPLES
        You can run gparted from a command line and specify one or more
        disk devices.

        For example, to start gparted with  the  devices  /dev/sda  and
        /dev/sdc you would use the following command:

        gparted /dev/sda /dev/sdc

NOTES
        Editing partitions has the potential to cause LOSS of DATA.

        The  gparted application is designed to enable you to edit par‐
        titions while reducing the risk of data loss.  The  application
        is  carefully  tested  and is used by the GParted project team.
        However, loss of data might occur due to software  bugs,  hard‐
        ware problems, or power failure.

        You can help to reduce the risk of data loss by not mounting or
        unmounting  partitions outside of the gparted application while
        gparted is running.

        You are advised to BACKUP your DATA before  using  the  gparted
        application.

REPORTING BUGS
        Report bugs at:
        https://gparted.org/bugs.php

AUTHOR
        Manual page written by Curtis Gedak <gedakc@users.sf.net>

SEE ALSO
        parted(8), fdisk(8), mkfs(8), ntfsprogs(8)

gparted                     Jan 16th, 2011                  GPARTED(8)

To expand on what the gparted help information says, gparted does not just resize partitions, it also runs e2fsk before doing the resize to verify that the partition was successfully cloned, then if that succeeds, it tries to run resize2fs to resize the file system in the partition and thereby complete the job.

This means that if one of the steps in the above sequence fails, the remaining steps do not get executed. So the partition might resize, and the file system could successfully verify its integrity, but the file system resize could fail. If you clone a 256 GB NVMe drive to a 4 TB NVMe drive, then you are likely to encouter this problem. The Recreating a File System section later in this article demonstrates the solution to that problem.

Practicalities

Gparted has no problem enlarging a 256 GB partition with a 1K block size to 500 GB. However, resizing the partition/file system to a much larger partition, for example, 3 TB, will fail.

However, as Theodore Tso has already informed us, it would be unwise to expand a tiny file system at all, and also unwise to supersize a medium-sized file system to a much larger file system.

When enlarging a file system is not possible or advisable, the file system must be recreated and then the contents of the old file system must be copied file-by-file to the new file system.

Running Gparted

In Clonezilla, I showed how the 256 GB NVMe from an Ableton Push 3 Standalone could be cloned to a 4 TB NVMe. After cloning, the data partition should be enlarged to use all available space. This is a job for gparted! If the data partition is not enlarged, then the extra space on the drive would never be used.

Running gparted showed the VM drive used by Ubuntu as /dev/sda, and the 4 TB NVMe drive as /dev/sdb:

Selecting /dev/sdb showed 5 partitions:

The msdos partition has two flags: legacy_boot and msftdata. This partition’s file system is FAT16. UEFI boot partitions (Windows calls these EFI system partitions) are always formatted with FAT16 or FAT32 file systems. There should be no reason to ever modify this partition.
The swap1 partition has one flag: swap. This swap partition is equal to the amount of RAM in the P3S. There should no reason to ever modify this partition.
I do not know what the primary partition is for or why its file system type is unknown.
The secondary partition contains the Linux operating system for the P3S. This partition is 47.7% full; most users will probably not need to modify it.
The data partition is where all user data is stored. This partition should be expanded to utilize all available space.
3.41 TB of unused space follows. If you clone a P3S NVMe drive to an NVMe with different capacity, say 1 TB, the amount of unused space will differ.

Right-clicking on the data partition displayed the Resize/Move menu item, which I selected:

I dragged the right-most handle to the far right.

The data partition was now set to include all available free space on the drive:

I pressed the Resize/Move button, then I pressed the green checkbox icon to actually resize the data partition.

Finally, I pressed the Apply button:

The resize operation first verified the integrity of the contents of the data partition, which took about 5 minutes. Then it aborted with the following error before resizing the file system:

resize2fs: New size results in too many block group descriptors.

The data partition had been resized, but its file system had not. Gparted shows the unused portion of /dev/sdb5 in gray:

Not to worry, the partition is fine. The error message tells us that we must recreate the file system inside the partition, and that will necessitate copying all the files and directories into the file system once it has been recreated.

Modestly Enlarging the Filesystem

A file system can be resized to utilize all the space in the newly expanded partition, subject to the limitations that have already been discussed.

This section shows how I enlarged a partition and its file system after cloning a 256 GB NVMe drive to a 500 GB NVMe drive. As previously mentioned, I suggest that you not enlarge tiny file systems, so if you are reading this because you have a P3S and want to clone the 256 GB NVMe drive that it came with to a larger drive, you should skip this section and read the Recreating a File System section.

We can use the standard Linux command resize2fs to resize the filesystem. As the documentation says, resize2fs can expand the size of an ext4 mounted file system.

Since I discourage people from running with scissors, I suggest you first unmount any drives whose file systems will be resized.

Before running resize2fs it is a good idea to verify the integrity of the partition to be resized. The e2fsck is the command normally used to do that.

Shell

$ man e2fsck
E2FSCK(8)               System Manager's Manual              E2FSCK(8)

NAME
        e2fsck - check a Linux ext2/ext3/ext4 file system

SYNOPSIS
        e2fsck [ -pacnyrdfkvtDFV ] [ -b superblock ] [ -B blocksize ] [
        -l|-L  bad_blocks_file ] [ -C fd ] [ -j external-journal ] [ -E
        extended_options ] [ -z undo_file ] device

DESCRIPTION
        e2fsck is used to check the ext2/ext3/ext4 family of file  sys‐
        tems.   For  ext3  and ext4 file systems that use a journal, if
        the system has been shut down  uncleanly  without  any  errors,
        normally,  after  replaying  the committed transactions  in the
        journal, the file system should be marked  as  clean.    Hence,
        for  file systems that use journaling, e2fsck will normally re‐
        play the journal and exit, unless its superblock indicates that
        further checking is required.

        device is a block device (e.g., /dev/sdc1) or  file  containing
        the file system.

        Note  that  in  general it is not safe to run e2fsck on mounted
        file systems.  The only exception is if the -n option is speci‐
        fied, and -c, -l, or -L options are not  specified.    However,
        even  if it is safe to do so, the results printed by e2fsck are
        not valid if the file  system  is  mounted.    If  e2fsck  asks
        whether or not you should check a file system which is mounted,
        the  only  correct  answer  is ``no''.  Only experts who really
        know what they are doing should consider answering  this  ques‐
        tion in any other way.

        If  e2fsck is run in interactive mode (meaning that none of -y,
        -n, or -p are specified), the program will ask the user to  fix
        each  problem found in the file system.  A response of 'y' will
        fix the error; 'n' will leave the error unfixed; and  'a'  will
        fix  the  problem  and  all subsequent problems; pressing Enter
        will proceed with the default response, which is printed before
        the question mark.  Pressing Control-C terminates e2fsck  imme‐
        diately.

OPTIONS
        -a     This option does the same thing as the -p option.  It is
              provided  for  backwards  compatibility only; it is sug‐
              gested that people use -p option whenever possible.

        -b superblock
              Instead of using the normal superblock, use an  alterna‐
              tive superblock specified by superblock.  This option is
              normally  used when the primary superblock has been cor‐
              rupted.  The location of backup superblocks is dependent
              on the file system's blocksize, the number of blocks per
              group, and features such as sparse_super.

              Additional backup superblocks can be determined by using
              the mke2fs program using the  -n  option  to  print  out
              where  the  superblocks  exist, supposing mke2fs is sup‐
              plied with arguments that are consistent with  the  file
              system's  layout  (e.g.  blocksize,  blocks  per  group,
              sparse_super, etc.).

              If an alternative superblock is specified and  the  file
              system  is  not  opened read-only, e2fsck will make sure
              that the primary  superblock  is  updated  appropriately
              upon completion of the file system check.

        -B blocksize
              Normally, e2fsck will search for the superblock at vari‐
              ous  different block sizes in an attempt to find the ap‐
              propriate block size.  This search can be fooled in some
              cases.  This option forces e2fsck to only  try  locating
              the  superblock  at  a particular blocksize.  If the su‐
              perblock is not found, e2fsck will terminate with a  fa‐
              tal error.

        -c     This option causes e2fsck to use badblocks(8) program to
              do  a  read-only scan of the device in order to find any
              bad blocks.  If any bad blocks are found, they are added
              to the bad block inode to prevent them from being  allo‐
              cated  to a file or directory.  If this option is speci‐
              fied twice, then the bad block scan will be done using a
              non-destructive read-write test.

        -C fd  This option causes e2fsck to write  completion  informa‐
              tion  to  the  specified  file  descriptor  so  that the
              progress of the file  system  check  can  be  monitored.
              This option is typically used by programs which are run‐
              ning e2fsck.  If the file descriptor number is negative,
              then absolute value of the file descriptor will be used,
              and  the  progress  information  will be suppressed ini‐
              tially.  It can later be enabled by sending  the  e2fsck
              process a SIGUSR1 signal.  If the file descriptor speci‐
              fied is 0, e2fsck will print a completion bar as it goes
              about  its  business.  This requires that e2fsck is run‐
              ning on a video console or terminal.

        -d     Print debugging output (useless unless you are debugging
              e2fsck).

        -D     Optimize directories in file system.  This option causes
              e2fsck to try to optimize all directories, either by re-
              indexing them if the file system supports directory  in‐
              dexing,   or  by sorting and compressing directories for
              smaller directories, or for file  systems  using  tradi‐
              tional linear directories.

              Even  without  the -D option, e2fsck may sometimes opti‐
              mize a few directories --- for example, if directory in‐
              dexing is enabled and a directory  is  not  indexed  and
              would benefit from being indexed, or if the index struc‐
              tures  are corrupted and need to be rebuilt.  The -D op‐
              tion forces all directories in the file system to be op‐
              timized.  This can sometimes make them a little  smaller
              and  slightly  faster  to  search,  but in practice, you
              should rarely need to use this option.

              The -D option will detect directory entries with  dupli‐
              cate  names in a single directory, which e2fsck normally
              does not enforce for performance reasons.

        -E extended_options
              Set e2fsck extended options.  Extended options are comma
              separated, and may take an  argument  using  the  equals
              ('=') sign.  The following options are supported:

                    ea_ver=extended_attribute_version
                          Set  the  version  of the extended attribute
                          blocks  which  e2fsck  will  require   while
                          checking  the file system.  The version num‐
                          ber may be 1 or 2.  The default extended at‐
                          tribute version format is 2.

                    journal_only
                          Only replay the journal if required, but  do
                          not perform any further checks or repairs.

                    fragcheck
                          During  pass  1,  print a detailed report of
                          any discontiguous blocks for  files  in  the
                          file system.

                    discard
                          Attempt  to  discard  free blocks and unused
                          inode blocks  after  the  full  file  system
                          check  (discarding blocks is useful on solid
                          state devices and sparse /  thin-provisioned
                          storage).  Note that discard is done in pass
                          5 AFTER  the  file  system  has  been  fully
                          checked and only if it does not contain rec‐
                          ognizable  errors.  However  there  might be
                          cases where e2fsck does not fully  recognize
                          a problem and hence in this case this option
                          may prevent you from further manual data re‐
                          covery.

                    nodiscard
                          Do  not  attempt  to discard free blocks and
                          unused inode blocks. This option is  exactly
                          the  opposite of discard option. This is set
                          as default.

                    no_optimize_extents
                          Do not offer to optimize the extent tree  by
                          eliminating   unnecessary  width  or  depth.
                          This can also be enabled in the options sec‐
                          tion of /etc/e2fsck.conf.

                    optimize_extents
                          Offer to optimize the extent tree by  elimi‐
                          nating  unnecessary width or depth.  This is
                          the default unless  otherwise  specified  in
                          /etc/e2fsck.conf.

                    inode_count_fullmap
                          Trade off using memory for speed when check‐
                          ing  a  file  system  with a large number of
                          hard-linked files.  The amount of memory re‐
                          quired is proportional to the number of  in‐
                          odes  in  the  file  system.  For large file
                          systems, this can be  gigabytes  of  memory.
                          (For  example,  a  40TB file system with 2.8
                          billion inodes will  consume  an  additional
                          5.7  GB  memory  if this optimization is en‐
                          abled.)  This optimization can also  be  en‐
                          abled    in    the    options   section   of
                          /etc/e2fsck.conf.

                    no_inode_count_fullmap
                          Disable  the  inode_count_fullmap  optimiza‐
                          tion.   This is the default unless otherwise
                          specified in /etc/e2fsck.conf.

                    readahead_kb
                          Use this many KiB  of  memory  to  pre-fetch
                          metadata  in  the  hopes  of reducing e2fsck
                          runtime.  By default, this  is  set  to  the
                          size of two block groups' inode tables (typ‐
                          ically  4MiB on a regular ext4 file system);
                          if this amount is more than 1/50th of  total
                          physical memory, readahead is disabled.  Set
                          this to zero to disable readahead entirely.

                    bmap2extent
                          Convert  block-mapped files to extent-mapped
                          files.

                    fixes_only
                          Only fix damaged metadata; do  not  optimize
                          htree  directories or compress extent trees.
                          This option is incompatible with the -D  and
                          -E bmap2extent options.

                    check_encoding
                          Force  verification  of encoded filenames in
                          case-insensitive directories.  This  is  the
                          default  mode  if  the  file  system has the
                          strict flag enabled.

                    unshare_blocks
                          If the file system has shared  blocks,  with
                          the shared blocks read-only feature enabled,
                          then this will unshare all shared blocks and
                          unset the read-only feature bit. If there is
                          not  enough  free  space  then the operation
                          will fail.  If the file system does not have
                          the read-only feature bit,  but  has  shared
                          blocks anyway, then this option will have no
                          effect.  Note  when  using  this  option, if
                          there is no  free  space  to  clone  blocks,
                          there  is  no prompt to delete files and in‐
                          stead the operation will fail.

                          Note that unshare_blocks  implies  the  "-f"
                          option  to  ensure  that all passes are run.
                          Additionally, if  "-n"  is  also  specified,
                          e2fsck  will  simulate  trying  to  allocate
                          enough space to deduplicate. If this  fails,
                          the exit code will be non-zero.

        -f     Force checking even if the file system seems clean.

        -F     Flush  the file system device's buffer caches before be‐
              ginning.  Only really useful for doing e2fsck time  tri‐
              als.

        -j external-journal
              Set  the  pathname  where  the external-journal for this
              file system can be found.

        -k     When combined with  the  -c  option,  any  existing  bad
              blocks in the bad blocks list are preserved, and any new
              bad  blocks  found by running badblocks(8) will be added
              to the existing bad blocks list.

        -l filename
              Add the block numbers listed in the  file  specified  by
              filename  to the list of bad blocks.  The format of this
              file is the same  as  the  one  generated  by  the  bad‐
              blocks(8)  program.   Note  that  the  block numbers are
              based on the blocksize of the file system.  Hence,  bad‐
              blocks(8) must be given the blocksize of the file system
              in  order to obtain correct results.  As a result, it is
              much simpler and safer to use the -c option  to  e2fsck,
              since  it  will  assure  that the correct parameters are
              passed to the badblocks program.

        -L filename
              Set the bad blocks list to be the list of blocks  speci‐
              fied  by  filename.   (This option is the same as the -l
              option, except the bad blocks list is cleared before the
              blocks listed in the file are added to  the  bad  blocks
              list.)

        -n     Open  the file system read-only, and assume an answer of
              `no' to all questions.  Allows e2fsck to be used non-in‐
              teractively.  This option may not be  specified  at  the
              same time as the -p or -y options.

        -p     Automatically  repair  ("preen")  the file system.  This
              option will cause e2fsck to automatically fix  any  file
              system  problems  that can be safely fixed without human
              intervention.  If e2fsck discovers a problem  which  may
              require the system administrator to take additional cor‐
              rective  action,  e2fsck will print a description of the
              problem and then exit with the value 4  logically  or'ed
              into  the exit code.  (See the EXIT CODE section.)  This
              option is normally used by the  system's  boot  scripts.
              It may not be specified at the same time as the -n or -y
              options.

        -r     This option does nothing at all; it is provided only for
              backwards compatibility.

        -t     Print  timing  statistics for e2fsck.  If this option is
              used twice, additional timing statistics are printed  on
              a pass by pass basis.

        -v     Verbose mode.

        -V     Print version information and exit.

        -y     Assume  an  answer  of  `yes'  to  all questions; allows
              e2fsck to be used non-interactively.   This  option  may
              not  be  specified  at the same time as the -n or -p op‐
              tions.

        -z undo_file
              Before overwriting a file system block,  write  the  old
              contents  of  the block to an undo file.  This undo file
              can be used with e2undo(8) to restore the  old  contents
              of  the  file  system should something go wrong.  If the
              empty string is passed as the  undo_file  argument,  the
              undo  file  will  be  written to a file named e2fsck-de‐
              vice.e2undo  in  the   directory   specified   via   the
              E2FSPROGS_UNDO_DIR environment variable.

              WARNING:  The undo file cannot be used to recover from a
              power or system crash.

EXIT CODE
        The exit code returned by e2fsck is the sum  of  the  following
        conditions:
            0    - No errors
            1    - File system errors corrected
            2    - File system errors corrected, system should
                    be rebooted
            4    - File system errors left uncorrected
            8    - Operational error
            16   - Usage or syntax error
            32   - E2fsck canceled by user request
            128  - Shared library error

SIGNALS
        The  following  signals  have the following effect when sent to
        e2fsck.

        SIGUSR1
              This signal causes e2fsck to start displaying a  comple‐
              tion bar or emitting progress information.  (See discus‐
              sion of the -C option.)

        SIGUSR2
              This  signal  causes e2fsck to stop displaying a comple‐
              tion bar or emitting progress information.

REPORTING BUGS
        Almost any piece of software will have bugs.  If you manage  to
        find  a  file  system  which  causes  e2fsck to crash, or which
        e2fsck is unable to repair, please report it to the author.

        Please include as much information as possible in your bug  re‐
        port.   Ideally,  include  a  complete transcript of the e2fsck
        run, so I can see exactly what error  messages  are  displayed.
        (Make  sure  the  messages printed by e2fsck are in English; if
        your system has been configured so that e2fsck's messages  have
        been  translated  into  another  language,  please  set the the
        LC_ALL environment variable to C  so  that  the  transcript  of
        e2fsck's  output will be useful to me.)  If you have a writable
        file system where the transcript can be stored,  the  script(1)
        program is a handy way to save the output of e2fsck to a file.

        It is also useful to send the output of dumpe2fs(8).  If a spe‐
        cific  inode  or  inodes seems to be giving e2fsck trouble, try
        running the debugfs(8) command  and  send  the  output  of  the
        stat(1u) command run on the relevant inode(s).  If the inode is
        a directory, the debugfs dump command will allow you to extract
        the contents of the directory inode, which can sent to me after
        being  first run through uuencode(1).  The most useful data you
        can send to help reproduce the bug is a  compressed  raw  image
        dump  of  the file system, generated using e2image(8).  See the
        e2image(8) man page for more details.

        Always include the full version string  which  e2fsck  displays
        when it is run, so I know which version you are running.

ENVIRONMENT
        E2FSCK_CONFIG
              Determines  the  location of the configuration file (see
              e2fsck.conf(5)).

AUTHOR
        This  version  of  e2fsck  was   written   by   Theodore   Ts'o
        <tytso@mit.edu>.

SEE ALSO
        e2fsck.conf(5),  badblocks(8),  dumpe2fs(8),  debugfs(8), e2im‐
        age(8), mke2fs(8), tune2fs(8)

E2fsprogs version 1.47.0     February 2023                   E2FSCK(8)

Here is the documentation for resize2fs:

Shell

$ man resize2fs
RESIZE2FS(8)            System Manager's Manual           RESIZE2FS(8)

NAME
        resize2fs - ext2/ext3/ext4 file system resizer

SYNOPSIS
        resize2fs  [ -fFpPMbs ] [ -d debug-flags ] [ -S RAID-stride ] [
        -z undo_file ] device [ size ]

DESCRIPTION
        The resize2fs program will resize ext2, ext3, or ext4 file sys‐
        tems.  It can be used to enlarge or shrink  an  unmounted  file
        system  located  on  device.  If the file system is mounted, it
        can be used to expand the size of the mounted file system,  as‐
        suming  the  kernel and the file system supports on-line resiz‐
        ing.  (Modern Linux 2.6 kernels will support on-line resize for
        file systems mounted using ext3 and  ext4;  ext3  file  systems
        will require the use of file systems with the resize_inode fea‐
        ture enabled.)

        The size parameter specifies the requested new size of the file
        system.  If no units are specified, the units of the size para‐
        meter  shall  be  the file system blocksize of the file system.
        Optionally, the size parameter may be suffixed by  one  of  the
        following  units designators: 'K', 'M', 'G', 'T' (either upper-
        case  or  lower-case)  or  's'  for   power-of-two   kilobytes,
        megabytes,  gigabytes,  terabytes  or  512 byte sectors respec‐
        tively. The size of the file system may never  be  larger  than
        the size of the partition.  If size parameter is not specified,
        it will default to the size of the partition.

        The  resize2fs  program  does not manipulate the size of parti‐
        tions.  If you wish to enlarge a file  system,  you  must  make
        sure you can expand the size of the underlying partition first.
        This  can  be done using fdisk(8) by deleting the partition and
        recreating it with a  larger  size  or  using  lvextend(8),  if
        you're  using the logical volume manager lvm(8).  When recreat‐
        ing the partition, make sure you create it with the same start‐
        ing disk cylinder as before!  Otherwise, the  resize  operation
        will certainly not work, and you may lose your entire file sys‐
        tem.   After running fdisk(8), run resize2fs to resize the ext2
        file system to use all of the space in the newly enlarged  par‐
        tition.

        If you wish to shrink an ext2 partition, first use resize2fs to
        shrink  the  size of file system.  Then you may use fdisk(8) to
        shrink the size of the partition.  When shrinking the  size  of
        the  partition,  make  sure you do not make it smaller than the
        new size of the ext2 file system!

        The -b and -s options enable and disable the 64bit feature, re‐
        spectively.  The resize2fs program will, of course,  take  care
        of  resizing  the block group descriptors and moving other data
        blocks out of the way, as needed.  It is not possible to resize
        the file system concurrent with changing the 64bit status.

OPTIONS
        -b     Turns on the 64bit feature, resizes the  group  descrip‐
              tors  as  necessary, and moves other metadata out of the
              way.

        -d debug-flags
              Turns on various resize2fs debugging features,  if  they
              have  been compiled into the binary.  debug-flags should
              be computed by adding the numbers of  the  desired  fea‐
              tures from the following list:
                    2    - Debug block relocations
                    4    - Debug inode relocations
                    8    - Debug moving the inode table
                    16   - Print timing information
                    32   - Debug minimum file system size (-M) calcula‐
              tion

        -f     Forces  resize2fs to proceed with the file system resize
              operation, overriding some safety checks which resize2fs
              normally enforces.

        -F     Flush the file system device's buffer caches before  be‐
              ginning.   Only  really  useful for doing resize2fs time
              trials.

        -M     Shrink the file system to minimize its size as  much  as
              possible, given the files stored in the file system.

        -p     Print  out percentage completion bars for each resize2fs
              phase during an offline (non-trivial) resize  operation,
              so  that  the user can keep track of what the program is
              doing.  (For very fast resize  operations,  no  progress
              bars may be displayed.)

        -P     Print an estimate of the number of file system blocks in
              the file system if it is shrunk using resize2fs's -M op‐
              tion and then exit.

        -s     Turns off the 64bit feature and frees blocks that are no
              longer in use.

        -S RAID-stride
              The  resize2fs  program will heuristically determine the
              RAID stride that was specified when the file system  was
              created.   This  option  allows  the  user to explicitly
              specify a RAID stride setting to be  used  by  resize2fs
              instead.

        -z undo_file
              Before  overwriting  a  file system block, write the old
              contents of the block to an undo file.  This  undo  file
              can  be  used with e2undo(8) to restore the old contents
              of the file system should something go  wrong.   If  the
              empty  string  is  passed as the undo_file argument, the
              undo file will be written to a file named  resize2fs-de‐
              vice.e2undo   in   the   directory   specified  via  the
              E2FSPROGS_UNDO_DIR environment variable.

              WARNING: The undo file cannot be used to recover from  a
              power or system crash.

KNOWN BUGS
        The  minimum  size of the file system as estimated by resize2fs
        may be incorrect, especially for file systems with  1k  and  2k
        blocksizes.

AUTHOR
        resize2fs was written by Theodore Ts'o <tytso@mit.edu>.

COPYRIGHT
        Resize2fs  is  Copyright  1998 by Theodore Ts'o and PowerQuest,
        Inc.  All rights reserved.  As of April, 2000 Resize2fs may  be
        redistributed under the terms of the GPL.

SEE ALSO
        fdisk(8), e2fsck(8), mke2fs(8), lvm(8), lvextend(8)

E2fsprogs version 1.47.0     February 2023                RESIZE2FS(8)

Now let’s finally get the job done. First, we verify and optimize the filesystem on the data partition:

Shell

$ sudo e2fsck -Dfp /dev/sdb5
data: 48424/27583488 files (2.1% non-contiguous), 98269049/220661060 blocks

The file system is good, and very little of it is non-continuous, so let’s go ahead and resize it.

Shell

$ sudo resize2fs /dev/sdb5
resize2fs 1.47.0 (5-Feb-2023)
The filesystem is already 458988544 (1k) blocks long.  Nothing to do!

You are done, time to clean up.

Recreating a File System

When resizing an existing file system is either impossible or inadvisable, a new file system should be created using mke2fs, and the contents of the old file system copied to the new file system.

This is the help information for mke2fs:

man mke2fs

MKE2FS(8)               System Manager's Manual              MKE2FS(8)

NAME
       mke2fs - create an ext2/ext3/ext4 file system

SYNOPSIS
       mke2fs [ -c | -l filename ] [ -b block-size ] [ -C cluster-size
       ]  [  -d  root-directory  ] [ -D ] [ -g blocks-per-group ] [ -G
       number-of-groups ] [ -i bytes-per-inode ] [ -I inode-size  ]  [
       -j ] [ -J journal-options ] [ -N number-of-inodes ] [ -n ] [ -m
       reserved-blocks-percentage  ]  [  -o  creator-os ] [ -O [^]fea‐
       ture[,...]  ] [ -q ] [ -r fs-revision-level ] [ -E extended-op‐
       tions ] [ -v ] [ -F ] [ -L volume-label ] [ -M last-mounted-di‐
       rectory ] [ -S ] [ -t fs-type ] [ -T usage-type ] [ -U UUID ] [
       -V ] [ -e errors-behavior ] [ -z undo_file ] device [ fs-size ]

       mke2fs -O journal_dev [ -b block-size ] [ -L volume-label  ]  [
       -n ] [ -q ] [ -v ] external-journal [ fs-size ]

DESCRIPTION
       mke2fs  is  used  to create an ext2, ext3, or ext4 file system,
       usually in a disk partition (or file) named by device.

       The file system size is specified by fs-size.  If fs-size  does
       not have a suffix, it is interpreted as power-of-two kilobytes,
       unless  the -b blocksize option is specified, in which case fs-
       size is interpreted as the number of blocksize blocks.   If the
       fs-size is suffixed by 'k', 'm', 'g', 't' (either upper-case or
       lower-case), then it is interpreted in power-of-two  kilobytes,
       megabytes,  gigabytes,  terabytes, etc.  If fs-size is omitted,
       mke2fs will create the file system based on the device size.

       If mke2fs is run as mkfs.XXX (i.e.,  mkfs.ext2,  mkfs.ext3,  or
       mkfs.ext4) the option -t XXX is implied; so mkfs.ext3 will cre‐
       ate  a  file  system for use with ext3, mkfs.ext4 will create a
       file system for use with ext4, and so on.

       The defaults of the parameters for the newly created file  sys‐
       tem,  if  not  overridden by the options listed below, are con‐
       trolled by the /etc/mke2fs.conf configuration  file.   See  the
       mke2fs.conf(5) manual page for more details.

OPTIONS
       -b block-size
              Specify  the  size of blocks in bytes.  Valid block-size
              values are powers of two from 1024 up to 65536  (however
              note  that the kernel is able to mount only file systems
              with block-size smaller or equal to the system page size
              - 4k on x86 systems, up to 64k on ppc64 or  aarch64  de‐
              pending  on  kernel  configuration).  If omitted, block-
              size is heuristically determined by the file system size
              and the expected usage of the file system  (see  the  -T
              option).   In  most common cases, the default block size
              is 4k. If block-size is  preceded  by  a  negative  sign
              ('-'),  then mke2fs will use heuristics to determine the
              appropriate block size, with  the  constraint  that  the
              block  size  will be at least block-size bytes.  This is
              useful for certain hardware devices which  require  that
              the blocksize be a multiple of 2k.

       -c     Check the device for bad blocks before creating the file
              system.   If  this  option  is  specified  twice, then a
              slower read-write test is used instead of a  fast  read-
              only test.

       -C  cluster-size
              Specify  the  size  of cluster in bytes for file systems
              using the bigalloc feature.  Valid  cluster-size  values
              are  from 2048 to 256M bytes per cluster.  This can only
              be specified if the bigalloc feature is  enabled.   (See
              the  ext4 (5) man page for more details about bigalloc.)
              The default cluster size if bigalloc is  enabled  is  16
              times the block size.

       -d root-directory
              Copy  the  contents of the given directory into the root
              directory of the file system.

       -D     Use direct I/O when writing to the  disk.   This  avoids
              mke2fs  dirtying a lot of buffer cache memory, which may
              impact other applications  running  on  a  busy  server.
              This  option  will cause mke2fs to run much more slowly,
              however, so there is a tradeoff to using direct I/O.

       -e error-behavior
              Change the behavior of the kernel code when  errors  are
              detected.   In all cases, a file system error will cause
              e2fsck(8) to check the file system  on  the  next  boot.
              error-behavior can be one of the following:

                   continue    Continue normal execution.

                   remount-ro  Remount file system read-only.

                   panic       Cause a kernel panic.

       -E extended-options
              Set  extended options for the file system.  Extended op‐
              tions are comma separated, and may take an argument  us‐
              ing  the equals ('=') sign.  The -E option used to be -R
              in earlier versions of mke2fs.  The -R option  is  still
              accepted for backwards compatibility, but is deprecated.
              The following extended options are supported:

                   encoding=encoding-name
                          Enable  the  casefold  feature  in the super
                          block and set encoding-name as the  encoding
                          to  be used.  If encoding-name is not speci‐
                          fied, the encoding defined in mke2fs.conf(5)
                          is used.

                   encoding_flags=encoding-flags
                          Define parameters for  file  name  character
                          encoding  operations.   If  a  flag  is  not
                          changed using this  parameter,  its  default
                          value  is  used.  encoding-flags should be a
                          comma-separated lists of  flags  to  be  en‐
                          abled.   To  disable  a  flag, add it to the
                          list with the prefix "no".

                          The only flag that can be set right  now  is
                          strict  which  means  that  invalid  strings
                          should be rejected by the file  system.   In
                          the  default  configuration, the strict flag
                          is disabled.

                   mmp_update_interval=interval
                          Adjust the initial MMP  update  interval  to
                          interval seconds.  Specifying an interval of
                          0  means  to  use the default interval.  The
                          specified interval must  be  less  than  300
                          seconds.   Requires  that the mmp feature be
                          enabled.

                   stride=stride-size
                          Configure the file system for a  RAID  array
                          with stride-size file system blocks. This is
                          the number of blocks read or written to disk
                          before  moving  to  the  next disk, which is
                          sometimes referred to  as  the  chunk  size.
                          This mostly affects placement of file system
                          metadata  like  bitmaps  at  mke2fs  time to
                          avoid placing them on a single  disk,  which
                          can  hurt  performance.  It may also be used
                          by the block allocator.

                   stripe_width=stripe-width
                          Configure the file system for a  RAID  array
                          with  stripe-width  file  system  blocks per
                          stripe. This is typically stride-size  *  N,
                          where  N is the number of data-bearing disks
                          in the RAID (e.g. for RAID 5  there  is  one
                          parity  disk,  so  N  will  be the number of
                          disks in the array minus  1).   This  allows
                          the  block allocator to prevent read-modify-
                          write of the parity in a RAID stripe if pos‐
                          sible when the data is written.

                   offset=offset
                          Create the file system at an offset from the
                          beginning of the device or file.   This  can
                          be useful when creating disk images for vir‐
                          tual machines.

                   resize=max-online-resize
                          Reserve enough space so that the block group
                          descriptor  table can grow to support a file
                          system that has max-online-resize blocks.

                   lazy_itable_init[= <0 to disable, 1 to enable>]
                          If enabled and the uninit_bg feature is  en‐
                          abled,  the  inode  table  will not be fully
                          initialized by mke2fs.  This speeds up  file
                          system initialization noticeably, but it re‐
                          quires the kernel to finish initializing the
                          file  system in the background when the file
                          system is  first  mounted.   If  the  option
                          value is omitted, it defaults to 1 to enable
                          lazy inode table zeroing.

                   lazy_journal_init[= <0 to disable, 1 to enable>]
                          If  enabled,  the  journal inode will not be
                          fully zeroed out by mke2fs.  This speeds  up
                          file  system  initialization noticeably, but
                          carries  some  small  risk  if  the   system
                          crashes  before  the  journal has been over‐
                          written entirely one time.   If  the  option
                          value is omitted, it defaults to 1 to enable
                          lazy journal inode zeroing.

                   assume_storage_prezeroed[= <0 to disable, 1 to en‐
                   able>]
                          If  enabled, mke2fs assumes that the storage
                          device has been prezeroed, skips zeroing the
                          journal and inode tables, and annotates  the
                          block  group  flags to signal that the inode
                          table has been zeroed.

                   no_copy_xattrs
                          Normally mke2fs will copy the  extended  at‐
                          tributes of the files in the directory hier‐
                          archy  specified  via  the (optional) -d op‐
                          tion.  This will disable the copy and leaves
                          the files in the newly created  file  system
                          without any extended attributes.

                   num_backup_sb=<0|1|2>
                          If  the sparse_super2 file system feature is
                          enabled this option controls  whether  there
                          will  be  0, 1, or 2 backup superblocks cre‐
                          ated in the file system.

                   packed_meta_blocks[= <0 to disable, 1 to enable>]
                          Place the allocation bitmaps and  the  inode
                          table  at  the  beginning of the disk.  This
                          option requires that the flex_bg file system
                          feature to be enabled in  order  for  it  to
                          have  effect, and will also create the jour‐
                          nal at the beginning  of  the  file  system.
                          This option is useful for flash devices that
                          use  SLC flash at the beginning of the disk.
                          It also maximizes the  range  of  contiguous
                          data blocks, which can be useful for certain
                          specialized  use  cases,  such  as supported
                          Shingled Drives.

                   root_owner[=uid:gid]
                          Specify the numeric user and group ID of the
                          root directory.  If no UID:GID is specified,
                          use the user and group ID of the  user  run‐
                          ning mke2fs.  In mke2fs 1.42 and earlier the
                          UID  and  GID of the root directory were set
                          by default to the UID and GID  of  the  user
                          running the mke2fs command.  The root_owner=
                          option  allows  explicitly  specifying these
                          values, and  avoid  side-effects  for  users
                          that  do not expect the contents of the file
                          system to change based on the  user  running
                          mke2fs.

                   test_fs
                          Set a flag in the file system superblock in‐
                          dicating that it may be mounted using exper‐
                          imental  kernel  code,  such  as the ext4dev
                          file system.

                   orphan_file_size=size
                          Set size of the file for  tracking  unlinked
                          but  still open inodes and inodes with trun‐
                          cate in progress.  Larger  file  allows  for
                          better  scalability,  reserving a few blocks
                          per cpu is ideal.

                   discard
                          Attempt to discard blocks at mkfs time (dis‐
                          carding blocks initially is useful on  solid
                          state  devices and sparse / thin-provisioned
                          storage). When the  device  advertises  that
                          discard  also  zeroes  data  (any subsequent
                          read after the discard and before write  re‐
                          turns  zero),  then  mark all not-yet-zeroed
                          inode tables as zeroed.  This  significantly
                          speeds  up  file system initialization. This
                          is set as default.

                   nodiscard
                          Do not attempt to  discard  blocks  at  mkfs
                          time.

                   quotatype
                          Specify  the  which   quota types (usrquota,
                          grpquota, prjquota) which should be  enabled
                          in the created file system.  The argument of
                          this extended option should be a colon sepa‐
                          rated  list.  This option has effect only if
                          the quota  feature  is  set.    The  default
                          quota types to be initialized if this option
                          is not specified is both user and group quo‐
                          tas.  If the project feature is enabled that
                          project quotas will be initialized as well.

       -F     Force mke2fs to create a file system, even if the speci‐
              fied  device  is  not a partition on a block special de‐
              vice, or if other parameters do not make sense.  In  or‐
              der  to force mke2fs to create a file system even if the
              file system appears to be in use or is mounted (a  truly
              dangerous  thing  to  do), this option must be specified
              twice.

       -g blocks-per-group
              Specify the number of blocks in a block group.  There is
              generally no reason for the user to ever set this  para‐
              meter,  as  the  default is optimal for the file system.
              (For administrators who are  creating  file  systems  on
              RAID arrays, it is preferable to use the stride RAID pa‐
              rameter  as part of the -E option rather than manipulat‐
              ing the number of blocks per  group.)   This  option  is
              generally  used  by  developers  who are developing test
              cases.

              If the bigalloc feature is enabled, the -g  option  will
              specify the number of clusters in a block group.

       -G number-of-groups
              Specify  the  number of block groups that will be packed
              together to create a  larger  virtual  block  group  (or
              "flex_bg  group") in an ext4 file system.  This improves
              meta-data locality and performance  on  meta-data  heavy
              workloads.   The  number  of groups must be a power of 2
              and may only be specified if  the  flex_bg  file  system
              feature is enabled.

       -i bytes-per-inode
              Specify  the bytes/inode ratio.  mke2fs creates an inode
              for every bytes-per-inode bytes of space  on  the  disk.
              The  larger  the bytes-per-inode ratio, the fewer inodes
              will be created.   This  value  generally  shouldn't  be
              smaller  than the blocksize of the file system, since in
              that case more inodes would be made  than  can  ever  be
              used.   Be warned that it is not possible to change this
              ratio on a file system after it is created, so be  care‐
              ful deciding the correct value for this parameter.  Note
              that resizing a file system changes the number of inodes
              to maintain this ratio.

       -I inode-size
              Specify the size of each inode in bytes.  The inode-size
              value  must be a power of 2 larger or equal to 128.  The
              larger the inode-size the more  space  the  inode  table
              will  consume,  and this reduces the usable space in the
              file system and can also negatively impact  performance.
              It  is  not possible to change this value after the file
              system is created.

              File systems with an inode size of 128 bytes do not sup‐
              port timestamps beyond January 19, 2038.   Inodes  which
              are  256  bytes  or  larger  will support extended time‐
              stamps, project id's, and the ability to store some  ex‐
              tended  attributes  in the inode table for improved per‐
              formance.

              The  default   inode   size   is   controlled   by   the
              mke2fs.conf(5)  file.   In  the mke2fs.conf file shipped
              with e2fsprogs, the default inode size is 256 bytes  for
              most  file  systems, except for small file systems where
              the inode size will be 128 bytes.

       -j     Create the file system with an ext3 journal.  If the  -J
              option  is not specified, the default journal parameters
              will be used to create an  appropriately  sized  journal
              (given  the  size  of the file system) stored within the
              file system.  Note that you must be using a kernel which
              has ext3 support in order to actually make  use  of  the
              journal.

       -J journal-options
              Create  the  ext3 journal using options specified on the
              command-line.  Journal options are comma separated,  and
              may  take an argument using the equals ('=')  sign.  The
              following journal options are supported:

                   size=journal-size
                          Create an internal journal (i.e., stored in‐
                          side the file system) of  size  journal-size
                          megabytes.   The size of the journal must be
                          at least 1024 file system blocks (i.e.,  1MB
                          if  using 1k blocks, 4MB if using 4k blocks,
                          etc.)  and may be no  more  than  10,240,000
                          file  system  blocks  or half the total file
                          system size (whichever is smaller)

                   fast_commit_size=fast-commit-size
                          Create an  additional  fast  commit  journal
                          area  of  size  fast-commit-size  kilobytes.
                          This option is  only  valid  if  fast_commit
                          feature  is  enabled  on the file system. If
                          this  option  is  not   specified   and   if
                          fast_commit  feature is turned on, fast com‐
                          mit area size defaults to journal-size /  64
                          megabytes.  The  total  size  of the journal
                          with fast_commit feature set is journal-size
                          + ( fast-commit-size * 1024) megabytes.  The
                          total  journal  size  may  be  no  more than
                          10,240,000 file system blocks  or  half  the
                          total   file   system   size  (whichever  is
                          smaller).

                   location=journal-location
                          Specify the location of  the  journal.   The
                          argument   journal-location  can  either  be
                          specified as a block number, or if the  num‐
                          ber  has  a  units  suffix  (e.g., 'M', 'G',
                          etc.) interpret it as the  offset  from  the
                          beginning of the file system.

                   device=external-journal
                          Attach  the file system to the journal block
                          device located on external-journal.  The ex‐
                          ternal journal must already have  been  cre‐
                          ated using the command

                          mke2fs -O journal_dev external-journal

                          Note  that  external-journal  must have been
                          created with the same block size as the  new
                          file  system.   In  addition, while there is
                          support for attaching multiple file  systems
                          to a single external journal, the Linux ker‐
                          nel  and  e2fsck(8) do not currently support
                          shared external journals yet.

                          Instead of  specifying  a  device  name  di‐
                          rectly,  external-journal can also be speci‐
                          fied by either LABEL=label or  UUID=UUID  to
                          locate  the  external  journal by either the
                          volume label or UUID stored in the ext2  su‐
                          perblock  at  the start of the journal.  Use
                          dumpe2fs(8) to display  a  journal  device's
                          volume  label and UUID.  See also the -L op‐
                          tion of tune2fs(8).

              Only one of the size or device options can be given  for
              a file system.

       -l filename
              Read  the  bad blocks list from filename.  Note that the
              block numbers in the bad block list  must  be  generated
              using  the  same block size as used by mke2fs.  As a re‐
              sult, the -c option to mke2fs is a much simpler and less
              error-prone method of checking a disk for bad blocks be‐
              fore formatting it, as mke2fs  will  automatically  pass
              the correct parameters to the badblocks program.

       -L new-volume-label
              Set  the volume label for the file system to new-volume-
              label.  The maximum length of the  volume  label  is  16
              bytes.

       -m reserved-blocks-percentage
              Specify  the  percentage  of  the file system blocks re‐
              served for the super-user.  This  avoids  fragmentation,
              and  allows  root-owned  daemons, such as syslogd(8), to
              continue  to  function  correctly  after  non-privileged
              processes are prevented from writing to the file system.
              The default percentage is 5%.

       -M last-mounted-directory
              Set  the  last  mounted  directory  for the file system.
              This might be useful for the sake of utilities that  key
              off of the last mounted directory to determine where the
              file system should be mounted.

       -n     Causes  mke2fs to not actually create a file system, but
              display what it would do if it were  to  create  a  file
              system.   This  can be used to determine the location of
              the backup superblocks for a particular file system,  so
              long  as the mke2fs parameters that were passed when the
              file system  was  originally  created  are  used  again.
              (With the -n option added, of course!)

       -N number-of-inodes
              Overrides  the  default calculation of the number of in‐
              odes that should be reserved for the file system  (which
              is based on the number of blocks and the bytes-per-inode
              ratio).   This  allows the user to specify the number of
              desired inodes directly.

       -o creator-os
              Overrides the default value of  the  "creator  operating
              system"  field of the file system.  The creator field is
              set by default to the name of the  OS  the  mke2fs  exe‐
              cutable was compiled for.

       -O [^]feature[,...]
              Create  a file system with the given features (file sys‐
              tem options), overriding the  default  file  system  op‐
              tions.   The  features  that  are enabled by default are
              specified by the base_features relation, either  in  the
              [defaults] section in the /etc/mke2fs.conf configuration
              file,  or  in  the  [fs_types] subsections for the usage
              types as specified by the -T option, further modified by
              the features relation found in  the  [fs_types]  subsec‐
              tions  for  the  file  system  and usage types.  See the
              mke2fs.conf(5) manual page for more details.   The  file
              system  type-specific configuration setting found in the
              [fs_types] section  will  override  the  global  default
              found in [defaults].

              The file system feature set will be further edited using
              either  the  feature set specified by this option, or if
              this option is not given, by the default_features  rela‐
              tion  for  the file system type being created, or in the
              [defaults] section of the configuration file.

              The file system feature set is comprised of  a  list  of
              features,  separated  by commas, that are to be enabled.
              To disable a feature, simply  prefix  the  feature  name
              with  a  caret ('^') character.  Features with dependen‐
              cies will not be removed successfully.  The  pseudo-file
              system  feature  "none"  will clear all file system fea‐
              tures.

       For more information about the features which can be set,
       please see
              the manual page ext4(5).

       -q     Quiet execution.  Useful if mke2fs is run in a script.

       -r revision
              Set the file system revision for the  new  file  system.
              Note  that 1.2 kernels only support revision 0 file sys‐
              tems.  The default is to create revision 1 file systems.

       -S     Write superblock and group descriptors only.  This is an
              extreme measure to be taken only in  the  very  unlikely
              case  that  all of the superblock and backup superblocks
              are corrupted, and a last-ditch recovery method  is  de‐
              sired  by experienced users.  It causes mke2fs to reini‐
              tialize the superblock and group descriptors, while  not
              touching  the  inode  table  and  the  block  and  inode
              bitmaps.  The e2fsck program should be  run  immediately
              after  this  option  is  used, and there is no guarantee
              that any data will be salvageable.  Due to the wide  va‐
              riety  of possible options to mke2fs that affect the on-
              disk layout, it is critical to specify exactly the  same
              format  options,  such  as  blocksize,  fs-type, feature
              flags, and other tunables when using this option, or the
              file system will be further corrupted.  In  some  cases,
              such as file systems that have been resized, or have had
              features  enabled after format time, it is impossible to
              overwrite all of the superblocks correctly, and at least
              some file system corruption will occur.  It is  best  to
              run  this on a full copy of the file system so other op‐
              tions can be tried if this doesn't work.

       -t fs-type
              Specify the file system type (i.e.,  ext2,  ext3,  ext4,
              etc.)  that  is  to  be  created.  If this option is not
              specified, mke2fs will pick a default either via how the
              command was run (for example, using a name of  the  form
              mkfs.ext2,  mkfs.ext3, etc.) or via a default as defined
              by the /etc/mke2fs.conf  file.    This  option  controls
              which  file system options are used by default, based on
              the fstypes configuration stanza in /etc/mke2fs.conf.

              If the -O option is used to  explicitly  add  or  remove
              file system options that should be set in the newly cre‐
              ated  file  system, the resulting file system may not be
              supported by the requested fs-type.  (e.g.,  "mke2fs  -t
              ext3 -O extent /dev/sdXX" will create a file system that
              is  not supported by the ext3 implementation as found in
              the Linux kernel; and "mke2fs -t  ext3  -O  ^has_journal
              /dev/hdXX"  will create a file system that does not have
              a journal and hence will not be supported  by  the  ext3
              file system code in the Linux kernel.)

       -T usage-type[,...]
              Specify how the file system is going to be used, so that
              mke2fs  can  choose  optimal  file system parameters for
              that use.  The usage types that are  supported  are  de‐
              fined  in  the configuration file /etc/mke2fs.conf.  The
              user may specify one or more usage types using  a  comma
              separated list.

              If  this  option is is not specified, mke2fs will pick a
              single default usage type based on the size of the  file
              system  to  be created.  If the file system size is less
              than 3 megabytes, mke2fs will use the file  system  type
              floppy.   If  the  file  system  size is greater than or
              equal to 3 but less than 512 megabytes,  mke2fs(8)  will
              use the file system type small.  If the file system size
              is greater than or equal to 4 terabytes but less than 16
              terabytes,  mke2fs(8) will use the file system type big.
              If the file system size is greater than or equal  to  16
              terabytes, mke2fs(8) will use the file system type huge.
              Otherwise,  mke2fs(8)  will  use the default file system
              type default.

       -U UUID
              Set the universally unique identifier (UUID) of the file
              system to UUID.  The format of the UUID is a  series  of
              hex    digits   separated   by   hyphens,   like   this:
              "c1b9d5a2-f162-11cf-9ece-0020afc76f16".  The UUID  para‐
              meter may also be one of the following:

                   clear  clear the file system UUID

                   random generate a new randomly-generated UUID

                   time   generate a new time-based UUID

       -v     Verbose execution.

       -V     Print the version number of mke2fs and exit.

       -z undo_file
              Before  overwriting  a  file system block, write the old
              contents of the block to an undo file.  This  undo  file
              can  be  used with e2undo(8) to restore the old contents
              of the file system should something go  wrong.   If  the
              empty  string  is  passed as the undo_file argument, the
              undo file will be written to  a  file  named  mke2fs-de‐
              vice.e2undo   in   the   directory   specified  via  the
              E2FSPROGS_UNDO_DIR environment variable or the  undo_dir
              directive in the configuration file.

              WARNING:  The undo file cannot be used to recover from a
              power or system crash.

ENVIRONMENT
       MKE2FS_SYNC
              If set to non-zero integer value, its value is  used  to
              determine how often sync(2) is called during inode table
              initialization.

       MKE2FS_CONFIG
              Determines  the  location of the configuration file (see
              mke2fs.conf(5)).

       MKE2FS_FIRST_META_BG
              If set to non-zero integer value, its value is  used  to
              determine first meta block group. This is mostly for de‐
              bugging purposes.

       MKE2FS_DEVICE_SECTSIZE
              If  set  to non-zero integer value, its value is used to
              determine logical sector size of the device.

       MKE2FS_DEVICE_PHYS_SECTSIZE
              If set to non-zero integer value, its value is  used  to
              determine physical sector size of the device.

       MKE2FS_SKIP_CHECK_MSG
              If set, do not show the message of file system automatic
              check caused by mount count or check interval.

AUTHOR
       This  version  of  mke2fs  has  been  written  by Theodore Ts'o
       <tytso@mit.edu>.

AVAILABILITY
       mke2fs is part of the e2fsprogs package and is  available  from
       http://e2fsprogs.sourceforge.net.

SEE ALSO
       mke2fs.conf(5),     badblocks(8),    dumpe2fs(8),    e2fsck(8),
       tune2fs(8), ext4(5)

E2fsprogs version 1.47.0     February 2023                   MKE2FS(8)

Journaling

The mke2fs command will create an ext4 journal when creating an ext4 file system.

There is no mke2fs option to specify that an ext4 journal should be created; instead, an ext4 journal is created by default.

Unfortunately, lots of the ‘advice’ you can find on the interwebs advise people to issue options for the mke2fs command that create a journal … however, those commands create ext3 journals, not ext4 journals.

This is undesirable because ext4 journals are better than ext3 journals.

UUID

File systems are best mounted by specifying their UUID. In contrast, specifying the drive to be mounted by their device name is problematic when device assignments shift as hardware is modified.

When you clone a drive, you should also replicate the original UUIDs of the file systems that should be mounted. That allows you to replace the original drive with the clone without modifying the system.

You can obtain the original file system UUID for /dev/sdb5 whether or not it is mounted with the following incantation:

Shell

$ lsblk -no UUID /dev/sdb5
64f5365e-edfc-4781-a285-9a053e6937fa

The above shows that the UUID for /dev/sdb5 on my virtualized Ubuntu system was 64f5365e-edfc-4781-a285-9a053e6937fa.

Let’s store the file system UUID in an environment variable called UUID for convenience in the next step:

Shell

$ UUID="$( lsblk -no UUID /dev/sdb5 )"

$ echo $UUID
64f5365e-edfc-4781-a285-9a053e6937fa

Example

As an example, let’s consider the case of cloning the NVMe drive of an Ableton P3S to a larger NVMe drive. The data partition is the only partition that should be enlarged, and it should be enlarged to utilize all available storage capacity.

Before we throw away the file system on /dev/sdb5 we need to unmount it:

Shell

$ sudo umount /dev/sdb5

Let’s use mke2fs to create an optimal file system in the data partition, mapped as /dev/sdb5. This file system will automatically be optimized to use all available space, with the same UUID as the original file system.

The mke2fs command is destructive. It places a new, formatted filesystem onto the specified partition. Typing the wrong device name will wipe all existing data in that device!

We can use the -n option to do a dry run to see what would happen:

Shell

$ sudo mke2fs \
  -n \
  -c -L data -O filetype,extents -t ext4 -U "$UUID" \
  /dev/sdb5
mke2fs 1.47.0 (5-Feb-2023)
/dev/sdb5 contains a ext4 file system labelled 'data'
      last mounted on /media/mslinn/data1 on Thu Jun 20 06:18:21 2024
Proceed anyway? (y,N) y
Creating filesystem with 969404928 4k blocks and 242352128 inodes
Filesystem UUID: 64f5365e-edfc-4781-a285-9a053e6937fa
Superblock backups stored on blocks:
      32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
      4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
      102400000, 214990848, 512000000, 550731776, 644972544

This is the actual invocation:

Shell

$ sudo mke2fs \
  -c -L data -O filetype,extents -t ext4 -U "$UUID" \
  /dev/sdb5
mke2fs 1.47.0 (5-Feb-2023)
/dev/sdb5 contains a ext4 file system labelled 'data'
      last mounted on /media/mslinn/data1 on Thu Jun 20 06:18:21 2024
Proceed anyway? (y,N) y
Creating filesystem with 969404928 4k blocks and 242352128 inodes
Filesystem UUID: 64f5365e-edfc-4781-a285-9a053e6937fa
Superblock backups stored on blocks:
      32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
      4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
      102400000, 214990848, 512000000, 550731776, 644972544

Now the data partition contains an optimized file system that is completely empty. We need to copy the files and directories from the data partition of the original drive to the empty data partition on the clone.

Copying Files and Directories

Partitions need to be mounted before files and directories can be copied to them, or from them. I inserted the original P3S NVMe drive into my second Sabrent NVMe / USB enclosure, and connected the enclosure to the computer. The VirtualBox VM for the Ubuntu instance I was working with already knew to take control of both Sabrent USB devices.

Once plugged in, the virtualized Ubuntu instance running in VirtualBox presented me with all the file systems on both of the NVMe drives in Sabrent enclosures, including the file systems called data found on each of the NVMe drives.

Clicking on each of the data partitions in the Ubuntu file manager mounted them.

The data partition on the original drive (now mounted as /dev/sdc5) contained all the files and directories to be copied, and the data partition on the clone (still mounted as /dev/sdb5) was empty because we had just recreated its file system.

Cut and Paste

One way to copy all the files would be to just copy and paste. This is likely to require superuser privilege to copy some files or directories. Unfortunately, Ubuntu 20.04 removed the ability for a file manager such as nautilus or thunar to run with elevated permissions. Thus, you are likely to find that some or even most files copy fine, but the process terminates with an error.

If you were able to perform the copy without incident, skip to the All Done section.

Findmnt

If you prefer to use the command line instead of drag and drop, or if you used drag and drop, but an error occurred after copying some files, you should use the rsync command to complete the copy.

Before you can use rsync, however, you need to know the mount points for the file systems in both data partitions.

The findmnt command shows information about mounted file systems. Here is the man page:

man findmnt

FINDMNT(8)               System Administration              FINDMNT(8)

NAME
      findmnt - find a filesystem

SYNOPSIS
      findmnt [options]

      findmnt [options] device|mountpoint

      findmnt [options] [--source] device [--target path|--mountpoint
      mountpoint]

DESCRIPTION
      findmnt will list all mounted filesystems or search for a
      filesystem. The findmnt command is able to search in
      /etc/fstab, /etc/mtab or /proc/self/mountinfo. If device or
      mountpoint is not given, all filesystems are shown.

      The device may be specified by device name, major:minor
      numbers, filesystem label or UUID, or partition label or UUID.
      Note that findmnt follows mount(8) behavior where a device name
      may be interpreted as a mountpoint (and vice versa) if the
      --target, --mountpoint or --source options are not specified.

      The command-line option --target accepts any file or directory
      and then findmnt displays the filesystem for the given path.

      The command prints all mounted filesystems in the tree-like
      format by default. The default output, is subject to change. So
      whenever possible, you should avoid using default output in
      your scripts. Always explicitly define expected columns by
      using --output columns-list in environments where a stable
      output is required.

      The relationship between block devices and filesystems is not
      always one-to-one. The filesystem may use more block devices.
      This is why findmnt provides  SOURCE and SOURCES (pl.) columns.
      The column SOURCES displays all devices where it is possible to
      find the same filesystem  UUID (or another tag specified in
      fstab when executed with --fstab and --evaluate).

OPTIONS
      -A, --all
            Disable all built-in filters and print all filesystems.

      -a, --ascii
            Use ascii characters for tree formatting.

      -b, --bytes
            Print the sizes in bytes rather than in a human-readable
            format.

            By default, the unit, sizes are expressed in, is byte, and
            unit prefixes are in power of 2^10 (1024). Abbreviations of
            symbols are exhibited truncated in order to reach a better
            readability, by exhibiting alone the first letter of them;
            examples: "1 KiB" and "1 MiB" are respectively exhibited as
            "1 K" and "1 M", then omitting on purpose the mention "iB",
            which is part of these abbreviations.

      -C, --nocanonicalize
            Do not canonicalize paths at all. This option affects the
            comparing of paths and the evaluation of tags (LABEL, UUID,
            etc.).

      -c, --canonicalize
            Canonicalize all printed paths.

      --deleted
            Print filesystems where target (mountpoint) is marked as
            deleted by kernel.

      -D, --df
            Imitate the output of df(1). This option is equivalent to
            -o SOURCE,FSTYPE,SIZE,USED,AVAIL,USE%,TARGET but excludes
            all pseudo filesystems. Use --all to print all filesystems.

      -d, --direction word
            The search direction, either forward or backward.

      -e, --evaluate
            Convert all tags (LABEL, UUID, PARTUUID, or PARTLABEL) to
            the corresponding device names for the SOURCE column. It’s
            an unusual situation, but the same tag may be duplicated
            (used for more devices). For this purpose, there is SOURCES
            (pl.) column. This column displays by multi-line cell all
            devices where the tag is detected by libblkid. This option
            makes sense for fstab only.

      -F, --tab-file path
            Search in an alternative file. If used with --fstab, --mtab
            or --kernel, then it overrides the default paths. If
            specified more than once, then tree-like output is disabled
            (see the --list option).

      -f, --first-only
            Print the first matching filesystem only.

      -i, --invert
            Invert the sense of matching.

      -J, --json
            Use JSON output format.

      -k, --kernel
            Search in /proc/self/mountinfo. The output is in the
            tree-like format. This is the default. The output contains
            only mount options maintained by kernel (see also --mtab).

      -l, --list
            Use the list output format. This output format is
            automatically enabled if the output is restricted by the
            -t, -O, -S or -T option and the option --submounts is not
            used or if more that one source file (the option -F) is
            specified.

      -M, --mountpoint path
            Explicitly define the mountpoint file or directory. See
            also --target.

      -m, --mtab
            Search in /etc/mtab. The output is in the list format by
            default (see --tree). The output may include user space
            mount options.

      -N, --task tid
            Use alternative namespace /proc/<tid>/mountinfo rather than
            the default /proc/self/mountinfo. If the option is
            specified more than once, then tree-like output is disabled
            (see the --list option). See also the unshare(1) command.

      -n, --noheadings
            Do not print a header line.

      -O, --options list
            Limit the set of printed filesystems. More than one option
            may be specified in a comma-separated list. The -t and -O
            options are cumulative in effect. It is different from -t
            in that each option is matched exactly; a leading no at the
            beginning does not have global meaning. The "no" can used
            for individual items in the list. The "no" prefix
            interpretation can be disabled by "+" prefix.

      -o, --output list
            Define output columns. See the --help output to get a list
            of the currently supported columns. The TARGET column
            contains tree formatting if the --list or --raw options are
            not specified.

            The default list of columns may be extended if list is
            specified in the format +list (e.g., findmnt -o
            +PROPAGATION).

      --output-all
            Output almost all available columns. The columns that
            require --poll are not included.

      -P, --pairs
            Produce output in the form of key="value" pairs. All
            potentially unsafe value characters are hex-escaped
            (\x<code>). See also option --shell.

      -p, --poll[=list]
            Monitor changes in the /proc/self/mountinfo file. Supported
            actions are: mount, umount, remount and move. More than one
            action may be specified in a comma-separated list. All
            actions are monitored by default.

            The time for which --poll will block can be restricted with
            the --timeout or --first-only options.

            The standard columns always use the new version of the
            information from the mountinfo file, except the umount
            action which is based on the original information cached by
            findmnt. The poll mode allows using extra columns:

            ACTION
                  mount, umount, move or remount action name; this column
                  is enabled by default

            OLD-TARGET
                  available for umount and move actions

            OLD-OPTIONS
                  available for umount and remount actions

      --pseudo
            Print only pseudo filesystems.

      --shadow
            Print only filesystems over-mounted by another filesystem.

      -R, --submounts
            Print recursively all submounts for the selected
            filesystems. The restrictions defined by options -t, -O,
            -S, -T and --direction are not applied to submounts. All
            submounts are always printed in tree-like order. The option
            enables the tree-like output format by default. This option
            has no effect for --mtab or --fstab.

      -r, --raw
            Use raw output format. All potentially unsafe characters
            are hex-escaped (\x<code>).

      --real
            Print only real filesystems.

      -S, --source spec
            Explicitly define the mount source. Supported
            specifications are device, maj:min, LABEL=label, UUID=uuid,
            PARTLABEL=label and PARTUUID=uuid.

      -s, --fstab
            Search in /etc/fstab. The output is in the list format (see
            --list).

      -T, --target path
            Define the mount target. If path is not a mountpoint file
            or directory, then findmnt checks the path elements in
            reverse order to get the mountpoint (this feature is
            supported only when searching in kernel files and
            unsupported for --fstab). It’s recommended to use the
            option --mountpoint when checks of path elements are
            unwanted and path is a strictly specified mountpoint.

      -t, --types list
            Limit the set of printed filesystems. More than one type
            may be specified in a comma-separated list. The list of
            filesystem types can be prefixed with no to specify the
            filesystem types on which no action should be taken. For
            more details see mount(8).

      --tree
            Enable tree-like output if possible. The options is
            silently ignored for tables where is missing child-parent
            relation (e.g., fstab).

      --shadowed
            Print only filesystems over-mounted by another filesystem.

      -U, --uniq
            Ignore filesystems with duplicate mount targets, thus
            effectively skipping over-mounted mount points.

      -u, --notruncate
            Do not truncate text in columns. The default is to not
            truncate the TARGET, SOURCE, UUID, LABEL, PARTUUID,
            PARTLABEL columns. This option disables text truncation
            also in all other columns.

      -v, --nofsroot
            Do not print a [/dir] in the SOURCE column for bind mounts
            or btrfs subvolumes.

      -w, --timeout milliseconds
            Specify an upper limit on the time for which --poll will
            block, in milliseconds.

      -x, --verify
            Check mount table content. The default is to verify
            /etc/fstab parsability and usability. It’s possible to use
            this option also with --tab-file. It’s possible to specify
            source (device) or target (mountpoint) to filter mount
            table. The option --verbose forces findmnt to print more
            details.

      --verbose
            Force findmnt to print more information (--verify only for
            now).

      --vfs-all
            When used with VFS-OPTIONS column, print all VFS
            (fs-independent) flags. This option is designed for
            auditing purposes to list also default VFS kernel mount
            options which are normally not listed.

      -y, --shell
            The column name will be modified to contain only characters
            allowed for shell variable identifiers. This is usable, for
            example, with --pairs. Note that this feature has been
            automatically enabled for --pairs in version 2.37, but due
            to compatibility issues, now it’s necessary to request this
            behavior by --shell.

      -h, --help
            Display help text and exit.

      -V, --version
            Print version and exit.

EXIT STATUS
      The exit value is 0 if there is something to display, or 1 on
      any error (for example if no filesystem is found based on the
      user’s filter specification, or the device path or mountpoint
      does not exist).

ENVIRONMENT
      LIBMOUNT_FSTAB=<path>
            overrides the default location of the fstab file

      LIBMOUNT_MTAB=<path>
            overrides the default location of the mtab file

      LIBMOUNT_DEBUG=all
            enables libmount debug output

      LIBSMARTCOLS_DEBUG=all
            enables libsmartcols debug output

      LIBSMARTCOLS_DEBUG_PADDING=on
            use visible padding characters.

EXAMPLES
      findmnt --fstab -t nfs
            Prints all NFS filesystems defined in /etc/fstab.

      findmnt --fstab /mnt/foo
            Prints all /etc/fstab filesystems where the mountpoint
            directory is /mnt/foo. It also prints bind mounts where
            /mnt/foo is a source.

      findmnt --fstab --target /mnt/foo
            Prints all /etc/fstab filesystems where the mountpoint
            directory is /mnt/foo.

      findmnt --fstab --evaluate
            Prints all /etc/fstab filesystems and converts LABEL= and
            UUID= tags to the real device names.

      findmnt -n --raw --evaluate --output=target LABEL=/boot
            Prints only the mountpoint where the filesystem with label
            "/boot" is mounted.

      findmnt --poll --mountpoint /mnt/foo
            Monitors mount, unmount, remount and move on /mnt/foo.

      findmnt --poll=umount --first-only --mountpoint /mnt/foo
            Waits for /mnt/foo unmount.

      findmnt --poll=remount -t ext3 -O ro
            Monitors remounts to read-only mode on all ext3
            filesystems.

AUTHORS
      Karel Zak <kzak@redhat.com>

SEE ALSO
      fstab(5), mount(8)

REPORTING BUGS
      For bug reports, use the issue tracker at
      https://github.com/util-linux/util-linux/issues.

AVAILABILITY
      The findmnt command is part of the util-linux package which can
      be downloaded from Linux Kernel Archive
      <https://www.kernel.org/pub/linux/utils/util-linux/>.

util-linux 2.39.3             2023-12-01                    FINDMNT(8)

The following incantation shows the mount points for the data partitions:

Shell

$ findmnt -lo source,target | grep data
/dev/sdb5                      /media/mslinn/data
/dev/sdc5                      /media/mslinn/data1

We now know that for my Ubuntu instance, the files and directories mounted at /media/mslinn/data1/ need to be copied to /media/mslinn/data/.

Rsync

Rsync can resume a large copy that failed earlier, without recopying files that were successfully copied. You can rerun rsync as many times as you like without fear of duplicating information.

This is the general form of the rsync command that you should use. Instead of source_directory and destination_directory, specify the mount points for the two data directories.

Be sure to append a trailing slash to both directories when using rsync.

Shell

$ rsync -axHAXS --numeric-ids --info=progress2 \
  source_directory/ destination_directory/

The rsync options I used are:

`-a`	Copy all files, with permissions, etc.
`-x`	Stay on one file system
`-H`	Preserve hard links
`-A`	Preserve ACLs/permissions
`-X`	Preserve extended attributes
`-S`	Handle sparse files
`--numeric-ids`	Transfer numeric group and user IDs, rather than using user and group names and mapping them at both ends
`--info=progress2`	shows statistics based on the whole transfer, rather than individual files

We can now write the proper incantation to run rsync to copy from /media/mslinn/data1/ to /media/mslinn/data/.

Note that sudo is used to ensure that all files and directories are copied, without problems from insufficient permissions.

Shell

$ sudo rsync -axHAXS --numeric-ids --info=progress2 \
  /media/mslinn/data1/ /media/mslinn/data/

The copy took 25 seconds on my system.

All Done

Unmount both data partitions, unplug the Sabrent enclosures, and remove the NVMe drives. Store the original NVMe drive somewhere safe, and install the clone into the computer that it belongs to; in my case, this was the P3S.

Keep your NVMe drive safe

😁

We did what we set out to do: resized the partition and its file system.

Clonezilla

2024-06-12T00:00:00-04:00

Although other OSes are mentioned, this article focuses on using Clonezilla to clone drives used by Linux systems.

Mini-Series

The articles in this mini-series are:

Nomenclature

An original is the drive that needs to be cloned, and a clone is a drive containing an identical copy of the original drive’s contents.

A snapshot is a read-only copy of an entire file system and all the files contained in the file system. The contents of each snapshot reflect the state of the file system at the time the snapshot was created. It is easy to navigate through each snapshot as if it were still active. The directories, folders and files will appear as they were at the time that the snapshot was created.

By default, Linux systems are not configured to use snapshots; for example, Ableton Push 3 Standalone devices do not use snapshots. It is not difficult to set up snapshots using LVM; perhaps I will write an article about how to do that one day.

About Clonezilla

Clonezilla can clone just about any type of Linux hard drive.

Many file systems are supported:

ext2, ext3, ext4, reiserfs, reiser4, xfs, jfs, btrfs, f2fs and nilfs2 of GNU/Linux
FAT12, FAT16, FAT32, exFAT and NTFS of MS Windows
HFS+ and APFS of macOS
UFS of FreeBSD, NetBSD, and OpenBSD
Minix
VMFS3 and VMFS5 of VMWare ESX

Therefore, you can clone GNU/Linux, Microsoft Windows, Intel-based macOS, FreeBSD, NetBSD, OpenBSD, Minix, VMWare ESXi and Chrome OS/Chromium OS, no matter if it's 32-bit (x86) or 64-bit (x86-64) OS. For these file systems, only used blocks in partitions are saved and restored by Partclone. For unsupported file systems, sector-to-sector copy is done by dd in Clonezilla.

– From What is Clonezilla?

Clonezilla works with MBR- and GPT-formatted hard drives. However, I am only going to discuss GPT-formatted drives. There is no point in discussing obsolete technology unless a legal firm is paying me for my time.

GPT Drives

The GUID Partition Table (GPT) is a standard for the layout of the partition tables on a physical hard disk. It forms a part of the Unified Extensible Firmware Interface (UEFI) standard, which Microsoft calls EFI. GPT allows for a maximum disk and partition size of 9.4 ZB (a zetabyte is a billion terabytes).

The bootloader for GPT drives is stored in the EFI system partition, formatted as FAT16 or FAT32, and resides at the beginning of the disk — Partition 1 in the following diagram.

From 'How to Remove, Delete or Format GPT Disk Partition'

As illustrated by the above diagram, drives contain a primary partition table, at least one partition, and a backup partition table. You could think of the two partition tables as bookends and the partitions as books between them, with the introductory book (the partition dedicated to containing the bootloader) in first place.

Clonezilla Overview

Clonezilla can clone entire drives, including the two GPT partition tables that bookend all the partitions in between, or it can just clone specific partitions.

The Clonezilla procedure is the same, no matter what size file system you start with. The process after using Clonezilla changes, depending on the scenario. To summarize:

Once Clonezilla finishes cloning a drive, all of the partitions in the clone are identical to the partitions in the original drive; any extra storage capacity is not used.
If the cloned drive is the same size as the original, you are done.
As I discuss in Supersizing a Partition and Its File System, tiny Linux file systems (those with 1K blocks) should be recreated larger, not enlarged, and the same is true for normal-sized file systems that are being migrated to much larger partitions. Normally, only one of the partitions on a drive should be enlarged, so this article makes that assumption.

To recreate a file system in a partition:
1. The file system in the partition being enlarged should be deleted.
2. The partition should be enlarged.
3. A new file system should be created in the enlarged partition. The creation process heavily optimizes the file system for the partition it is placed in.
4. The files from the partition on the original drive should be copied to the new file system on the cloned drive.

Usage

The Clonezilla installation process depends on how you want to use it. I describe the installation processes separately for each scenario discussed in this article.

I would like to encourage you to get in the habit of running Clonezilla in a VirtualBox virtual machine, as I did when I wrote this article. Unless you want to clone the system drive in the machine that you prefer to run VirtualBox on, this is generally your best option for running Clonezilla.

You have many other options as well. You can run Clonezilla from a normal Ubuntu system if you just want to clone a drive that is not mounted or if your file system supports snapshots. To clone a system disk that does not support snapshots, the drives on the disk will first have to be attached to the OS as a /dev device but not mounted, then the virtualized system should be booted, or an ISO system image on a CD / DVD / SSD should be booted. Once Ubuntu is running, Clonezilla can be run as a console text program in a terminal window.

I was unable to use Clonezilla from WSL because I did not know how to map USB drives to the WSL VM so Clonezilla could work its magic. Maybe one day I will fight through the issues involved, but there is no reason for me to work that hard for no benefit.

Cloning an Unmounted Drive

Installation

If you want to clone unmounted drives, install Clonezilla on an Ubuntu system like this:

Shell

$ sudo apt install clonezilla

Once installed, read the clonezilla help message:

Shell

$ sudo clonezilla -h
/usr/sbin/clonezilla: -h: invalid option
Usage:
Run clonezilla:
/usr/sbin/clonezilla [OPTION]
  -l, --language INDEX Set the language to be shown by index number:
        [0|en_US.UTF-8]: English,
        [2|zh_TW.UTF-8]: Traditional Chinese (UTF-8, Unicode) - Taiwan
        [a|ask]: Prompt to ask the language index
  This option is for backward compatibility. It's recommended to use
  locales to assign that.
  -d0, --dialog         Use dialog
  -d1, --Xdialog        Use Xdialog
  -d2, --whiptail       Use whiptail
  -d3, --gdialog        Use gdialog
  -d4, --kdialog        Use kdialog
  -k, --skip-ocs-prep-repo  Skip preparing the clonezilla image home
     directory (assume it's ready), this is specially for device
     <-> image clone.
  -p, --postaction [choose|poweroff|reboot|command|CMD]
    When save/restoration finishs, choose action in the client,
    poweroff, reboot (default), in command prompt or run CMD
  -s, --skip-lite-menu  Do not show live-server and lite-client in the dialog menu.
Ex. /usr/sbin/clonezilla -l en

Launch Clonezilla

Now launch clonezilla:

Shell

$ sudo clonezilla

If you are cloning an unmounted drive, you can skip the next section and continue to Running Clonezilla.

Cloning a System Drive

This section actually describes how to clone any Linux-compatible drive, not just system drives.

Generally speaking, a system cannot clone itself unless it has the ability to snapshot the drives that you want to back up. Without a snapshot, cloning a system drive requires Clonezilla to be booted, such that:

The drive to be cloned is connected as a /dev/ Linux device
None of the file systems in any of the drive's partitions may be mounted

Installation in a VirtualBox VM

Clonezilla is available as a package that includes a minimal Linux operating system. After the dedicated Linux OS boots, it runs Clonezilla.

You can install Clonezilla on a CD, DVD, or SSD drive. It was more convenient for me to run Clonezilla in a VirtualBox virtual machine (VM) instead of running on bare metal using a CD or SSD drive.

I downloaded the ISO image for standalone Clonezilla for the amd64 architecture. This will run on physical Windows desktops, Mac and Linux desktop computers that can boot from ISO images, CDs, DVDs and SSD drives. It will also run on a virtual 64-bit x86 computer.

Then I started VirtualBox as described in my VirtualBox Setup article and created a new virtual machine using the Clonezilla ISO image.

Notice that Clonezilla does not require a virtual hard drive; it strictly runs from the downloaded ISO image.

I launched the Clonezilla VM, and it instantly booted the Linux setup program.

The Linux setup program asked me to select the keyboard layout I was using, and I pressed Enter to accept the default en_US layout.

Yes, I am sure.

Once Linux is configured, it is time to run the Clonezilla program.

Running Clonezilla

Clonezilla uses different background colors when running from a command line, and when running from an ISO image. Otherwise, the program is exactly the same.

The next two screenshots show the difference between Clonezilla running from a command-line, and when running from an ISO image. Both of the screens accomplish the same thing: defining the mode of operation.

I wanted to copy from one physical disk to another, so I selected the second choice, device-device, using the up- and down-arrow keys, then pressing Enter.

In order for the VM to 'see' the NVMe drives, I right-clicked on the little rocket icon at the bottom of the VirtualBox window. This displayed all the USB devices attached to the Windows computer. I then enabled the two Sabrent [2001] USB devices, which made them exclusively available to the VM running Clonezilla.

Now I selected disk_to_local_disk and pressed the Enter key.

After several long minutes, I was presented with the two USB drives and was asked to select the source drive to clone, which was of course the 256 GB NVMe.

After pressing the Enter key, I had to wait a while once again while Clonezilla laboriously enumerated all available drives. Eventually, it displayed the 4 TB drive as the only possible candidate for the destination, which was correct. I pressed the Enter key once more.

Another long pause ensued, and then I was presented with this screen:

I pressed Enter and soon saw:

I just want to resize the data partition, so I selected the first option, -k0. I will resize the partition later using gparted.

I will decide what to do later, so I picked the first option, -p choose.

A helpful hit was displayed in case I need to run this command again. The information was presented in green text over a black background at the bottom of the window.

The small blue LEDs on both Sabrent enclosures flashed for several minutes. Finally, I was asked twice if I wanted the contents of sda (the new 4 TB drive) to be overwritten. Yes!

/dev/sda3 had a file with errors. I will run fsck on it shortly.

I left Clonezilla running overnight. When I awoke, it had completed:

I turned off the Clonezilla VM.

Store the Original NVMe

I unplugged the original 256 GB NVMe drive and placed it into an empty prescription bottle, then stored the bottle with my extra drives. Small spice bottles also work well for storing NVMe drives.

Clone Verification

Now we can verify that the cloned drive works by putting it into the system that the original drive came from and verifying that the system still works.

😁

If the drive containing the clone has the same storage capacity as the original drive, you are done.

Next: Resizing

If the drive containing the clone is larger or smaller than the original drive, at least one of the partitions on the clone will need to be resized. Remove the cloned drive from the system, place it back into the Sabrent NVMe / USB enclosure, and reattach the enclosure to the computer for further work. Please continue on to read Supersizing a Partition and Its File System for more information.

VirtualBox

2024-06-11T00:00:00-04:00

This article describes how I set up Ubuntu Desktop 24.04 in a VirtualBox VM.

Mini-Series

The articles in this mini-series are:

VirtualBox

VirtualBox allows you to run another operating system within your computer’s operating system. VirtualBox is available for Windows, Macintosh, Linux and other OSes. It can virtualize Windows, Linux, macOS, Solaris, and the BSD UNIX work-alikes.

I often use an Ubuntu Desktop instance in a VirtualBox VM. It is my default Ubuntu instance for general-purpose work.

Terminology

Host: This is the OS of the physical computer on which Oracle VM VirtualBox was installed.
Guest: This is the OS of the physical computer on which Oracle VM VirtualBox was installed.
Virtual machine (VM): This is the special environment that VirtualBox creates for your guest OS while it is running. In other words, you run your guest OS in a VM.

Installation

I installed VirtualBox and the VirtualBox Extension Pack according to these instructions.

I then installed Ubuntu Desktop 24.04 as a guest in a new VirtualBox VM. The steps are:

Download the Ubuntu Desktop 24.04 ISO file.
Open VirtualBox and click on the New button to create a new virtual machine.
Name your virtual machine and select Linux as the type and Ubuntu (64-bit) as the version.
Allocate at least 2 GB of RAM to the virtual machine; 4 GB is better.
Choose Create a virtual hard disk now and click Create.
Select VDI (VirtualBox Disk Image) as the hard disk file type.

7 . Choose " Dynamically allocated " for the storage type . 8 . Allocate at least 20 GB of storage space for the virtual machine . 9 . Click on the " Start " button to launch the virtual machine . 10 . In the " Select start - up disk " window , click on the folder icon and browse for the Ubuntu 24.04 ISO file you downloaded . 11 . Click " Start " to begin the installation process . 12 . Follow the on - screen instructions to complete the installation .

VirtualBox Guest Additions

A third item needs to be installed in the guest OS, the VirtualBox Guest Additions. It is included in the VirtualBox download and provides:

Seamless mouse support
Shared folders
Better video support
Seamless windows
Generic host/guest communication channels
Time synchronization
Shared clipboard
Automated logins

For Windows, the VirtualBox Guest Additions are installed at %ProgramFiles%\Oracle\VirtualBox\VBoxGuestAdditions.iso. For Mac OS X hosts, this file is provided in the VirtualBox application bundle.

Installation

Before installing the VirtualBox Guest Additions in a guest Linux instance, you should install gcc, make and perl. Launch the virtual machine containing the guest Ubuntu image and enter the following at a shell prompt:

Shell

$ sudo apt install build-essential gcc make perl dkms

Now reboot the guest Ubuntu instance.

Each time a VirtualBox Ubuntu instance boots, the VirtualBox Guest Additions CD-ROM image automounts at /media/$USER/VBox_GAs_7.0.18/.

Shell

$ ls /media/$USER/VBox_GAs_7.0.18/
AUTORUN.INF  runasroot.sh                       VBoxSolarisAdditions.pkg
autorun.sh   TRANS.TBL                          VBoxWindowsAdditions-amd64.exe
cert         VBoxDarwinAdditions.pkg            VBoxWindowsAdditions.exe
NT3x         VBoxDarwinAdditionsUninstall.tool  VBoxWindowsAdditions-x86.exe
OS2          VBoxLinuxAdditions.run             windows11-bypass.reg

To install the VirtualBox Guest Additions in the guest Ubuntu instance, run VBoxLinuxAdditions.run, and then reboot, as follows:

Shell

$ sudo /media/$USER/VBox_GAs_7.0.18/VBoxLinuxAdditions.run
Verifying archive integrity...  100%   MD5 checksums are OK. All good.
Uncompressing VirtualBox 7.0.18 Guest Additions for Linux  100%
VirtualBox Guest Additions installer
Removing installed version 7.0.18 of VirtualBox Guest Additions...
update-initramfs: Generating /boot/initrd.img-6.8.0-35-generic
Copying additional installer modules ...
Installing additional modules ...
VirtualBox Guest Additions: Starting.
VirtualBox Guest Additions: Setting up modules
VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel
modules.  This may take a while.
VirtualBox Guest Additions: To build modules for other installed kernels, run
VirtualBox Guest Additions:   /sbin/rcvboxadd quicksetup 
VirtualBox Guest Additions: or
VirtualBox Guest Additions:   /sbin/rcvboxadd quicksetup all
VirtualBox Guest Additions: Building the modules for kernel 6.8.0-35-generic.
update-initramfs: Generating /boot/initrd.img-6.8.0-35-generic 

$ sudo reboot

Rust

2024-05-30T00:00:00-04:00

Rust is a low-level programming language done right, using modern tools with good documentation and extensive support.

I have a lot of experience with the Scala and Ruby languages, as well as dozens of other computer languages. I believe that Rust has adopted the best parts of the Scala and Ruby philosophies and melded the syntaxes together nicely. For example, all of these languages provide expressions, closures, structs (value classes) and generics.

The package management is a bit like Ruby’s gems. Cargo crates and Ruby gems are provided as source code and compiled on demand.

Rust has many of the Scala features I enjoy the most:

Object-oriented and functional programming styles are both supported.
Pattern matching.
Traits.
Algebraic type system.

What sets Rust apart, however, is:

Error handling.
Data ownership (borrow checker) and smart pointers.
Data has lifetimes.
Stable, yet advanced compile-time metaprogramming (macros).
No runtime types.
Unsafe system.
Memory safety without a garbage collector; eal-time behavior is deterministic. Real-time audio/video applications are therefore possible.

rust-lang.org is the home of the Rust language.

Background

Rust was initially released in May 2015 by Graydon Hoare (no relation to Tony Hoare). Graydon had created the Swift language used by Apple the previous year. Chris Krycho has written a detailed comparison of the Swift and Rust languages.

Microsoft vice president David Weston said on Apr 29, 2023 that Microsoft had rewritten 36,000 lines of code in the Windows kernel in Rust, in addition to another 152,000 lines of code it had written for a proof of concept involving the DirectWrite Core library. The performance is purportedly excellent, with no regressions compared to the old C++ code.

Linus Torvalds accepts contributions for Linux written in Rust, and the Rust infrastructure now present in most Linux installations is mature. Linux 6.8, released on March 10, 2024, was the first version of Linux to include a device driver written in Rust, replacing an older network device driver written in C.

The XKCD Rust API looks like a fun way to blow off an afternoon.

If you need help, see the #beginners Rust Discord channel and the Rust Users Forum.

Bindings To Other Languages

Ruby / Rust bindings — Magnus allows you to write Ruby gems in Rust, and call Ruby code from Rust code.
Python / Rust bindings. — Py03 allows you to mix Rust with Python. Enjoy Python’s convenience with Rust’s speed.
Calling C from Rust.
How to create C binding for a Rust library: Rust can generate C dynamic libraries (.so files) as well as static libraries (.a files), which can be easily wrapped in Go bindings and Python bindings and used in code written in those languages.
bindgen automatically generates Rust FFI bindings to C (and some C++) libraries.
Robusta — easy interoperability between Rust and Java. This library provides a procedural macro to make it easier to write JNI-compatible code in Rust and automatically convert Rust input and output types.
Execute a shell command from Rust.

rustup

Rustup is an installer for the Rust systems programming language.

WSL / Ubuntu

For WSL / Ubuntu you can easily install rustup, then install the Rust toolchain with the following commands:

Shell

$ yes | sudo apt install rustup

$ rustup default stable
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2024-05-02, rust version 1.78.0 (9b00956e5 2024-04-29)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
 63.7 MiB /  63.7 MiB (100 %)  38.2 MiB/s in  1s ETA:  0s
info: downloading component 'rustfmt'
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'rust-docs'
info: installing component 'rust-std'
 24.3 MiB /  24.3 MiB (100 %)  23.8 MiB/s in  1s ETA:  0s
info: installing component 'rustc'
 63.7 MiB /  63.7 MiB (100 %)  20.5 MiB/s in  2s ETA:  0s
info: installing component 'rustfmt'
info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'

  stable-x86_64-unknown-linux-gnu installed - rustc 1.78.0 (9b00956e5 2024-04-29)

Other OSes

Installation instructions for your computer's OS appear when you visit rustup.rs.

If you want to customize your setup, you need to know the host triple for your system. A ‘host triple’ identifies the computer architecture and OS to install to. In truth, the value is no longer a triple but a quintic value.

For WSL / Ubuntu, the host triple is x86_64-unknown-linux-gnu. For Mac, the host triple is x86_64-apple-darwin. The toolchain string consists of the channel prefix, followed by the host triple; the channels are stable, beta and nightly. Thus, the complete toolchain strings for the stable channel are:

WSL / Ubuntu: stable-x86_64-unknown-linux-gnu
Mac: stable-x86_64-apple-darwin

Product Development Collaboration

2024-02-18T00:00:00-05:00

For product development, the goal of effective collaboration is to fully engage all parties. Good product-oriented collaborators maintain a flow of information, ideas and commensurate action. Hesitation is minimized. Feedback is rapid. Reactions are prompt, yet measured through constant practice.

Product Collaborator Checklist

The following criteria are a guide to effective collaboration candidates for product development.

Desirable Traits

Strong interest in optimizing the architecture and engineering processes.
A focus on the contributing to the desired results for the team, which are: reliably providing an easily understood and valuable benefit to a well-defined customer base.
Manage the difference between desired outcomes and actual outcomes.
Strive for process and product consistency.
Willing to expend the overhead of time and energy required for effective communication:
- Driving their activities from a Kanban-style status board
- Planning interactions with others, minimizing impulsive chatter
- Preparing and distributing notes before meetings
- Describing alternatives for decisions
- Documenting the rationale for design decisions
- Summarizing discussions in project documentation
- Contributing to an up-to-date product roadmap

Undesirable Traits

People with the following traits disqualify themselves from being a member of the core team because they are unable to consistently contribute effectively to the ever-evolving vision for the project.

"Guns for hire" just want to get paid for time spent; they have no vested interest in the long-term outcome. They are tools to be exploited, not valuable resources to be cultivated. The best use of a mercenary is to give them orders, supervise them, learn from them, and then dispose of them as soon as possible.
"Putting in time" demonstrates a lack of personal engagement. Those people require constant supervision, and the cost of the supervision must be subtracted from any value they might provide when computing their net contribution.

Software Architecture and Engineering Processes

ChatGPT gave me the following response when I asked, “Compare and contrast the processes of architecture and engineering for software”. I like the quality of this response, and provide it verbatim because it exactly matches my understanding of those terms.

When it comes to software development, there are parallels between the processes of architecture and engineering, but there are also significant differences due to the intangible and rapidly evolving nature of software. Let's compare and contrast the processes of software architecture and engineering:

Purpose and Focus

Software Architecture: Similar to architectural design in physical structures, software architecture focuses on designing the overall structure and organization of a software system. It involves making high-level decisions about the system's components, their interactions, and the principles that guide their design.

Software Engineering: Software engineering encompasses the systematic approach to developing software solutions. It involves various disciplines such as requirements analysis, design, coding, testing, deployment, and maintenance. The focus is on creating reliable, efficient, and maintainable software systems.

Design Process

Software Architecture: The architectural design process involves defining the system's key components, interfaces, and their interactions. Architectural patterns, such as layered architecture, microservices, or event-driven architecture, are often used to guide the design. Considerations include scalability, performance, security, and maintainability.

Software Engineering: Software engineering encompasses the entire software development lifecycle (SDLC). It begins with requirements gathering and analysis, followed by design, implementation (coding), testing, deployment, and maintenance. Various methodologies like Waterfall, Agile, and DevOps guide this process.

Creativity vs. Technicality

Software Architecture: Software architects must balance creative design decisions with technical constraints. They need to envision the overall structure of the system and choose appropriate technologies and patterns to achieve the desired functionality and quality attributes.

Software Engineering: Software engineers focus on implementing the design created by architects. While creativity is still important, engineers primarily deal with the technical aspects of coding, testing, and debugging. They translate the architectural design into working software components.

Collaboration

Software Architecture: Collaboration between architects, stakeholders, and development teams is crucial to ensure that architectural decisions align with business goals and technical requirements. Architects often communicate their designs through diagrams, documentation, and discussions.

Software Engineering: Software engineers collaborate closely with other team members, including architects, designers, testers, and project managers. They work together to implement features, resolve technical challenges, and deliver software increments according to project timelines.

Regulatory and Environmental Considerations

Software Architecture: Architectural decisions may need to comply with regulatory standards, industry best practices, and security guidelines. Architects must consider factors like data privacy, compliance requirements (such as GDPR or HIPAA), and interoperability with other systems.

Software Engineering: Engineers implement features and functionalities while adhering to coding standards, security protocols, and testing practices. They ensure that the software meets quality standards, performance targets, and regulatory requirements specified by the architecture and project stakeholders.

In summary, while both software architecture and engineering involve designing and building software systems, they have distinct focuses and processes. Architecture concentrates on high-level design decisions and system structure, while engineering encompasses the entire software development lifecycle, from requirements to deployment. Collaboration and communication are essential in both disciplines to deliver successful software solutions.

Collaboration

ChatGPT defined collaboration as the following. Again, I very much like the response, so I provide it verbatim.

Collaboration refers to the process of individuals or groups working together to achieve a common goal or objective. It involves the sharing of ideas, resources, responsibilities, and tasks in a coordinated manner to accomplish tasks more effectively and efficiently than if done individually.

Collaboration often emphasizes communication, cooperation, and mutual respect among participants. It can occur within various contexts, including businesses, organizations, teams, communities, and even across different disciplines or sectors.

Successful collaboration typically results in synergy, where the combined efforts of the participants produce outcomes that exceed what could be achieved by any individual or subgroup working alone.

Following is the response when I asked "what is required for effective collaboration in a new software project?" I highlighted a passage that is important, and ~~struck out~~ portions that I disagree with.

Effective collaboration in a new software project requires several key elements:

Clear Goals and Objectives: Establishing clear, well-defined goals and objectives for the project ensures that all collaborators understand what they are working towards and why.
Communication Channels: Implementing reliable communication channels is essential for keeping all team members informed and facilitating the exchange of ideas, updates, and feedback. This can include tools like instant messaging, email, project management software, or regular team meetings.
Defined Roles and Responsibilities: Clearly defining roles and responsibilities for each team member helps avoid confusion and ensures that tasks are completed efficiently. This includes designating a project manager, developers, testers, designers, and other relevant roles.
Collaboration Tools: Utilizing collaboration tools such as version control systems (e.g., Git), issue tracking systems ~~(e.g., Jira)~~, document sharing platforms (e.g., Google Drive), and collaborative coding environments (e.g., GitHub) can streamline the development process and enhance teamwork.
Regular Feedback and Reviews: Encouraging open and constructive feedback throughout the project lifecycle allows team members to address issues promptly, make necessary adjustments, and continuously improve the software.
Flexibility and Adaptability: Remaining flexible and adaptable to changes in requirements, timelines, and priorities is crucial for navigating unforeseen challenges and ensuring the project stays on track.
Respect and Trust: Cultivating an environment of respect and trust among team members fosters collaboration and encourages open communication, creativity, and innovation.
Documentation: Maintaining thorough documentation of project requirements, design decisions, codebase, and processes helps ensure continuity and facilitates knowledge transfer among team members.
Regular Progress Tracking: Implementing mechanisms to track progress, ~~such as sprint planning, daily stand-up meetings, and~~ regular status reports, enables the team to monitor project milestones, identify potential bottlenecks, and make necessary adjustments to stay on schedule.
Quality Assurance and Testing: Integrating quality assurance processes and testing procedures into the development workflow ensures that the software meets the specified requirements, functions as intended, and delivers value to end-users.

By incorporating these elements into the collaborative process, teams can increase the likelihood of success in their new software projects.

Design Decisions

I asked ChatGPT "how would someone document design decisions in a software project?" The highlighted portion is covered in more detail later in this article.

Documenting design decisions in a software project is essential for ensuring clarity, facilitating collaboration, and providing a reference for future development. Here's a structured approach to documenting design decisions:

Establish a Design Document: Create a designated document or section within project documentation specifically for recording design decisions. This could be a separate document, a section in the project wiki, or part of the version control repository.
Provide Context: Start by providing context for the decision. Explain the problem or requirement that the design decision aims to address. Include any relevant background information, constraints, or considerations that influenced the decision-making process.
Describe Alternatives: Outline the various alternatives considered during the decision-making process. This could include different approaches, technologies, architectures, or design patterns. Briefly discuss the pros and cons of each alternative.
Justify the Chosen Solution: Explain why the chosen solution was selected over the alternatives. Highlight the reasons, benefits, and trade-offs associated with the chosen approach. Consider factors such as performance, scalability, maintainability, ease of implementation, and alignment with project goals.
Detail the Design: Provide a detailed description of the chosen design solution. Break down the design into its constituent components, modules, or layers. Use diagrams, flowcharts, UML diagrams, or other visual aids to illustrate the architecture, structure, and relationships between components.
Document Dependencies and Interactions: Document dependencies between different components or modules within the design. Describe how these components interact with each other and with external systems or services. Identify any interfaces, APIs, or protocols used for communication.
Address Risks and Mitigations: Identify potential risks or challenges associated with the chosen design. Discuss strategies for mitigating these risks and ensuring the robustness and resilience of the design solution.
Include Implementation Details: If applicable, include implementation details or guidelines for developers to follow when implementing the design. This could include coding standards, best practices, design patterns, or frameworks to be used.
Update and Maintain: Keep the design document up to date as the project evolves. Update it to reflect any changes, refinements, or new decisions made during the development process. Ensure that the document remains a reliable reference for developers and stakeholders.
Review and Iterate: Encourage team members to review the design document periodically and provide feedback. Iterate on the design as necessary based on feedback, changes in requirements, or lessons learned from implementation.

By following these steps, you can effectively document design decisions in a software project, providing a comprehensive and structured reference for developers, stakeholders, and future contributors.

Comparative Product Matrix

Comparative product matrices are useful for guiding and documenting design decisions. Search for them when you need to identify pre-existing solutions to a problem, and generate them when considering the implementation of various approaches.

Comparative product matrix

Comparative product matrices take time to read and to prepare, but they help guide the team to the best technical decisions. Without them, you are just hoping to get lucky.

Expanding the WSL virtual hard drive

2024-02-12T00:00:00-05:00

I ran out of room on my WSL2 virtual hard drive. Microsoft’s instructions worked perfectly to increase the virtual drive size.

Shut Down WSL

The following command work from WSL bash, and cmd, and PowerShell:

Shell

$ wsl.exe --shutdown

Expand VHD With Diskpart

The virtual disk was in %LocalAppData%\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState\ext4.vhdx

Open a cmd.exe window with admininstrator privileges and run diskpart.

diskpart

DISKPART> Select vdisk file="%LocalAppData%\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState\ext4.vhdx"

DISKPART> detail vdisk
Device type ID: 0 (Unknown)
Vendor ID: {00000000-0000-0000-0000-000000000000} (Unknown)
State: Added
Virtual size:  256 GB
Physical size:  247 GB
Filename: C:\Users\Mike Slinn\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState\ext4.vhdx
Is Child: No
Parent Filename:
Associated disk#: Not found.

The virtual disk was 256 GB (256,000 MB). I want it to be twice as big: 512,000 MB.

diskpart (continued)

DISKPART> expand vdisk maximum=512000

100 percent completed

DiskPart successfully expanded the virtual disk file. 
DISKPART> exit

Leaving DiskPart...

Restart WSL and Run resize2fs

Now I restarted WSL from a cmd window. The following command work from WSL bash, and cmd, and PowerShell:

cmd.exe

$ wsl.exe

Now tell Linux to use all available space:

Shell

$ sudo resize2fs /dev/sdc 512000M
resize2fs 1.47.0 (5-Feb-2023)
Filesystem at /dev/sdc is mounted on /; on-line resizing required
old_desc_blocks = 32, new_desc_blocks = 63
The filesystem on /dev/sdc is now 131072000 (4k) blocks long.

😁

Done!

Installing JDK 17 on Ubuntu

2023-09-28T00:00:00-04:00

The Java Development Kit (JDK) version 17 is compatible with v11 and v8. Installing the JDK also installs the JRE, as you can see from the highlighted dependency shown below:

Shell

$ apt show -a openjdk-17-jdk
Package: openjdk-17-jdk
  Version: 17.0.8.1+1~us1-0ubuntu1~23.04
  Priority: optional
  Section: java
  Source: openjdk-17
  Origin: Ubuntu
  Maintainer: Ubuntu Developers 
  Original-Maintainer: OpenJDK Team 
  Bugs: https://bugs.launchpad.net/ubuntu/+filebug
  Installed-Size: 1544 kB
  Provides: java-compiler, java-sdk (= 17), java10-sdk, java11-sdk, java12-sdk, java13-sdk, java14-sdk, java15-sdk, java16-sdk, java17-sdk, java2-sdk, java5-sdk, java6-sdk, java7-sdk, java8-sdk, java9-sdk
  Depends: openjdk-17-jre (= 17.0.8.1+1~us1-0ubuntu1~23.04), openjdk-17-jdk-headless (= 17.0.8.1+1~us1-0ubuntu1~23.04), libc6 (>= 2.34), zlib1g (>= 1:1.1.4)
  Recommends: libxt-dev
  Suggests: openjdk-17-demo, openjdk-17-source, visualvm
  Homepage: https://openjdk.java.net/
  Download-Size: 1486 kB
  APT-Sources: http://archive.ubuntu.com/ubuntu lunar-updates/main amd64 Packages
  Description: OpenJDK Development Kit (JDK)
    OpenJDK is a development environment for building applications,
    applets, and components using the Java programming language.

  Package: openjdk-17-jdk
  Version: 17.0.6+10-1ubuntu2
  Priority: optional
  Section: java
  Source: openjdk-17
  Origin: Ubuntu
  Maintainer: Ubuntu Developers 
  Original-Maintainer: OpenJDK Team 
  Bugs: https://bugs.launchpad.net/ubuntu/+filebug
  Installed-Size: 4732 kB
  Provides: java-compiler, java-sdk, java10-sdk, java11-sdk, java12-sdk, java13-sdk, java14-sdk, java15-sdk, java16-sdk, java17-sdk, java2-sdk, java5-sdk, java6-sdk, java7-sdk, java8-sdk, java9-sdk
  Depends: openjdk-17-jre (= 17.0.6+10-1ubuntu2), openjdk-17-jdk-headless (= 17.0.6+10-1ubuntu2), libc6 (>= 2.34)
  Recommends: libxt-dev
  Suggests: openjdk-17-demo, openjdk-17-source, visualvm
  Homepage: https://openjdk.java.net/
  Download-Size: 4585 kB
  APT-Sources: http://archive.ubuntu.com/ubuntu lunar/main amd64 Packages
  Description: OpenJDK Development Kit (JDK)
    OpenJDK is a development environment for building applications,
    applets, and components using the Java programming language.

Install OpenJDK 17 and its dependencies:

Shell

$ yes | sudo apt install openjdk-17-jdk

Setting the Default Java Version

The man page for update-java-alternatives is:

Shell

$ man update-java-alternatives
UPDATE-JAVA-ALTERNATIVESystem Manager's MUPDATE-JAVA-ALTERNATIVES(8)

NAME
        update-java-alternatives  -  update  alternatives for jre/sdk
        installations

SYNOPSIS
        update-java-alternatives [--jre] [--plugin] [-v|--verbose]
              -l|--list [<jname>]
              -s|--set <jname>
              -a|--auto
              -h|-?|--help

DESCRIPTION
        update-java-alternatives updates all  alternatives  belonging
        to  one  runtime or development kit for the Java language.  A
        package does provide these information of  it's  alternatives
        in /usr/lib/jvm/.<jname>.jinfo.

OPTIONS
        -l|--list [<jname>]
              List  all installed packages (or just <jname>) provid‐
              ing information to set a bunch of  java  alternatives.
              Verbose  output shows each alternative provided by the
              packages.

        -a|--auto
              Switch all alternatives of registered jre/sdk  instal‐
              lations to automatic mode.

        -s|--set <jname>
              Set all alternatives of the registered jre/sdk instal‐
              lation to the program path provided by the <jname> in‐
              stallation.

        --jre  Limit  the actions to alternatives belong to a runtime
              environment, not a development kit.

        --jre-headless
              Limit the actions to alternatives belong to the  head‐
              less part of a runtime environment.

        --plugin
              Limit  the  actions  to alternatives providing browser
              plugins.

        -h|--help
              Display a help message.

        -v|--verbose
              Verbose output.

FILES
        /usr/lib/jvm/.*.jinfo
              A text file describing a  jre/sdk  installation.  Con‐
              sists  of some variables of the form <var>=<value> and
              a list of alternatives  of  the  form  jre|jdk  <name>
              <path>.

AUTHOR
        update-java-alternatives  and this manual page was written by
        Matthias Klose <doko@ubuntu.com>.

                              May 2006   UPDATE-JAVA-ALTERNATIVES(8)

The above man page neglects to say that a package version’s priority is set from its version number. Thus a newer version would normally have a higher priority. To get a list of the installed Java versions, type:

Shell

$ sudo update-java-alternatives --list
java-1.11.0-openjdk-amd64      1111       /usr/lib/jvm/java-1.11.0-openjdk-amd64
java-1.17.0-openjdk-amd64      1711       /usr/lib/jvm/java-1.17.0-openjdk-amd64

As you can see, my machine had Java 11 and 17 installed. To set the newest version to be the default, type:

Shell

$ sudo update-java-alternatives -a

$ java -version
openjdk version "17.0.8.1" 2023-08-24
OpenJDK Runtime Environment (build 17.0.8.1+1-Ubuntu-0ubuntu123.04)
OpenJDK 64-Bit Server VM (build 17.0.8.1+1-Ubuntu-0ubuntu123.04, mixed mode, sharing)

C++ Boost library

2023-09-14T00:00:00-04:00

About Boost

Boost is a general-purpose open source library of utility functions for C++. The list of categories of fuctionality is formidable.

Boost is used in many commercial products, in-house projects, and open-source projects.

Ostensibly a C++ library for wide usage, it has become an techical underpinning for Python; that is why Boost documentation also contains information about Python.

Getting Started

I started by reading the online Getting Started guide.

For most Boost functionality, only headers are required, and libraries need not be built. The exceptions are described here.

The Boost C++ Libraries is a good free online book, and you can order non-free printed copies.

Ubuntu Installation

Discover the Latest Boost Version Number

The latest version of Boost is shown on https://www.boost.org/users/history/?ref=learnubuntu.com:

Let’s scrape the lastest version number from the above web page:

Shell

$ yes | sudo apt install libxml2-utils

$ VER=$(
  wget -q -O - https://www.boost.org/users/history/?ref=learnubuntu.com | \
  xmllint --html --xpath '//*[@id="intro"]/div/h2[1]/a[2]' - 2>/dev/null | \
  grep -oEi 'Version ([0-9].)*' | \
  cut -d' ' -f 2
)

$ echo $VER
1.83

Download Source & Dependencies

Download and unpack the library source to a new directory within your home directory as follows:

Shell

$ cd

$ V_R=`echo $VER | tr . _`
$ echo $V_R
1_83 

$ NAME=boost_${V_R}_0
$ echo $NAME
boost_1_83_0 

$ wget -O $NAME.tar.gz \
  https://sourceforge.net/projects/boost/files/boost/$VER.0/$NAME.tar.gz/download

$ tar xzvf $NAME.tar.gz

$ cd $NAME/

You should now have the following files and directories:

Shell

$ ls -w72
INSTALL          boost/           boostcpp.jam   index.htm   rst.css
Jamroot          boost-build.jam  bootstrap.bat  index.html  status/
LICENSE_1_0.txt  boost.css        bootstrap.sh*  libs/       tools/
README.md        boost.png        doc/           more/

Install the required dependencies for building Boost.

Shell

$ sudo apt-get update

$ yes | sudo apt install autotools-dev build-essential g++ \
  libbz2-dev libicu-dev python3-dev

Build Installer Using Bootstrap

Two scripts for building the installation program Boost are provided: bootstrap.sh (for Bash) and bootstrap.bat (for Windows/DOS). By default the scripts build all possible static/shared debug/release Boost libraries. Another default is that the script uses all the CPU cores your computer has for the build process. Even using all the cores on a fast laptop, the build might take 10 minutes.

Here is the bootstrap.sh help message:

Shell

$ ./bootstrap.sh -h
`./bootstrap.sh\' builds the Boost build system B2 and prepares Boost for
building. This includes setting defaults in the project-config.jam which you
can adjust prior to invoking B2.

Usage: ./bootstrap.sh [OPTION]...

Defaults for the options are specified in brackets.

Configuration:
  -h, --help                display this help and exit
  --with-bjam=BJAM          use existing Boost.Jam executable (bjam)
                            [automatically built]
  --with-toolset=TOOLSET    use specific TOOLSET to build B2 and as default
                            for building Boost
                            [automatically detected]
  --show-libraries          show the set of libraries that require build
                            and installation steps (i.e., those libraries
                            that can be used with --with-libraries or
                            --without-libraries), then exit
  --with-libraries=list     build only a particular set of libraries,
                            describing using either a comma-separated list of
                            library names or "all"
                            [all]
  --without-libraries=list  build all libraries except the ones listed []
  --with-icu                enable Unicode/ICU support in Regex
                            [automatically detected]
  --without-icu             disable Unicode/ICU support in Regex
  --with-icu=DIR            specify the root of the ICU library installation
                            and enable Unicode/ICU support in Regex
                            [automatically detected]
  --with-python=PYTHON      specify the Python executable [python]
  --with-python-root=DIR    specify the root of the Python installation
                            [automatically detected]
  --with-python-version=X.Y specify the Python version as X.Y
                            [automatically detected]

Installation directories:
  --prefix=PREFIX           install Boost into the given PREFIX
                            [/usr/local]
  --exec-prefix=EPREFIX     install Boost binaries into the given EPREFIX
                            [PREFIX]

More precise control over installation directories:
  --libdir=DIR              install libraries here [EPREFIX/lib]
  --includedir=DIR          install headers here [PREFIX/include]

Bootstrap.sh compiles and links a build program called b2, which by default installs all of Boost into /usr/local/. To be more specific, the default location for Boost include files to be installed into is /usr/local/include/boost/, and Boost libraries are installed by default into /usr/local/lib/.

Create (and recreate) the b2 installation program for Boost for each Python virtual environment. By default, bootstrap.sh uses the currently active Python virtual environment, like this:

Shell

$ ./bootstrap.sh

Following is how to build b2 for the virtual environment at ~/venv/blah/.

Shell

$ ./bootstrap.sh --with-python-root=~/venv/blah
Building B2 engine..

###
###
### Using 'gcc' toolset.
###
###

g++ (Ubuntu 12.3.0-1ubuntu1~23.04) 12.3.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


###
###

> g++ -x c++ -std=c++11 -O2 -s -DNDEBUG builtins.cpp class.cpp command.cpp compile.cpp constants.cpp cwd.cpp debug.cpp debugger.cpp execcmd.cpp execnt.cpp execunix.cpp filesys.cpp filent.cpp fileunix.cpp frames.cpp function.cpp glob.cpp hash.cpp hcache.cpp hdrmacro.cpp headers.cpp jam_strings.cpp jam.cpp jamgram.cpp lists.cpp make.cpp make1.cpp md5.cpp mem.cpp modules.cpp native.cpp object.cpp option.cpp output.cpp parse.cpp pathnt.cpp pathsys.cpp pathunix.cpp regexp.cpp rules.cpp scan.cpp search.cpp startup.cpp subst.cpp sysinfo.cpp timestamp.cpp variable.cpp w32_getreg.cpp modules/order.cpp modules/path.cpp modules/property-set.cpp modules/regex.cpp modules/sequence.cpp modules/set.cpp -o b2
tools/build/src/engine/b2
Detecting Python version... 3.11
Unicode/ICU support for Boost.Regex?... /usr
Backing up existing B2 configuration in project-config.jam.2
Generating B2 configuration in project-config.jam for gcc...

Bootstrapping is done. To build, run:

    ./b2

To generate header files, run:

    ./b2 headers

The configuration generated uses gcc to build by default. If that is
unintended either use the --with-toolset option or adjust configuration, by
editing 'project-config.jam'.

Further information:

   - Command line help:
     ./b2 --help

   - Getting started guide:
     http://www.boost.org/more/getting_started/unix-variants.html

   - B2 documentation:
     http://www.boost.org/build/

Running B2, the Boost Installer

Following is the help message for the newly created b2 Boost installation program. As you can see, the defaults specified when creating the b2 script can be overidden. Note that not all of the available options are described in this help message; for example, the -j option is only described in the online help for b2.

Shell

$ ./b2 --help
B2 4.10-git

Project-specific help:

  Project has jamfile at Jamroot

Usage:

  b2 [options] [properties] [install|stage]

  Builds and installs Boost.

Targets and Related Options:

  install                 Install headers and compiled library files to the
  =======                 configured locations (below).

  --prefix=<PREFIX>       Install architecture independent files here.
                          Default: C:\Boost on Windows
                          Default: /usr/local on Unix, Linux, etc.

  --exec-prefix=<EPREFIX> Install architecture dependent files here.
                          Default: <PREFIX>

  --libdir=<LIBDIR>       Install library files here.
                          Default: <EPREFIX>/lib

  --includedir=<HDRDIR>   Install header files here.
                          Default: <PREFIX>/include

  --cmakedir=<CMAKEDIR>   Install CMake configuration files here.
                          Default: <LIBDIR>/cmake

  --no-cmake-config       Do not install CMake configuration files.

  stage                   Build and install only compiled library files to the
  =====                   stage directory.

  --stagedir=<STAGEDIR>   Install library files here
                          Default: ./stage

Other Options:

  --build-type=<type>     Build the specified pre-defined set of variations of
                          the libraries. Note, that which variants get built
                          depends on what each library supports.

                              -- minimal -- (default) Builds a minimal set of
                              variants. On Windows, these are static
                              multithreaded libraries in debug and release
                              modes, using shared runtime. On Linux, these are
                              static and shared multithreaded libraries in
                              release mode.

                              -- complete -- Build all possible variations.

  --build-dir=DIR         Build in this location instead of building within
                          the distribution tree. Recommended!

  --show-libraries        Display the list of Boost libraries that require
                          build and installation steps, and then exit.

  --layout=<layout>       Determine whether to choose library names and header
                          locations such that multiple versions of Boost or
                          multiple compilers can be used on the same system.

                              -- versioned -- Names of boost binaries include
                              the Boost version number, name and version of
                              the compiler and encoded build properties. Boost
                              headers are installed in a subdirectory of
                              <HDRDIR> whose name contains the Boost version
                              number.

                              -- tagged -- Names of boost binaries include the
                              encoded build properties such as variant and
                              threading, but do not including compiler name
                              and version, or Boost version. This option is
                              useful if you build several variants of Boost,
                              using the same compiler.

                              -- system -- Binaries names do not include the
                              Boost version number or the name and version
                              number of the compiler. Boost headers are
                              installed directly into <HDRDIR>. This option is
                              intended for system integrators building
                              distribution packages.

                          The default value is 'versioned' on Windows, and
                          'system' on Unix.

  --buildid=ID            Add the specified ID to the name of built libraries.
                          The default is to not add anything.

  --python-buildid=ID     Add the specified ID to the name of built libraries
                          that depend on Python. The default is to not add
                          anything. This ID is added in addition to --buildid.

  --help                  This message.

  --with-<library>        Build and install the specified <library>. If this
                          option is used, only libraries specified using this
                          option will be built.

  --without-<library>     Do not build, stage, or install the specified
                          <library>. By default, all libraries are built.

Properties:

  toolset=toolset         Indicate the toolset to build with.

  variant=debug|release   Select the build variant

  link=static|shared      Whether to build static or shared libraries

  threading=single|multi  Whether to build single or multithreaded binaries

  runtime-link=static|shared
                          Whether to link to static or shared C and C++
                          runtime.


General command line usage:

    b2 [options] [properties] [targets]

  Options, properties and targets can be specified in any order.

Important Options:

  * --clean Remove targets instead of building
  * -a Rebuild everything
  * -n Don't execute the commands, only print them
  * -d+2 Show commands as they are executed
  * -d0 Suppress all informational messages
  * -q Stop at first error
  * --reconfigure Rerun all configuration checks
  * --durations[=N] Report top N targets by execution time
  * --debug-configuration Diagnose configuration
  * --debug-building Report which targets are built with what properties
  * --debug-generator Diagnose generator search/execution

Further Help:

  The following options can be used to obtain additional documentation.

  * --help-options Print more obscure command line options.
  * --help-internal B2 implementation details.
  * --help-doc-options Implementation details doc formatting.

...found 1 target...

The following runs the newly created b2 program, which builds and installs Boost using all CPU cores.

The Python version is stored into an environment variable called PVER.
The CPLUS_INCLUDE_PATH environment variable is pointed at the location of the Python 3 include files. PVER was used to set the proper value for CPLUS_INCLUDE_PATH.
B2 generates screen upon screen of compiler warnings. The -d 1 option suppresses them.

Shell

$ PVER="$(python --version | cut -d' ' -f 2 | grep -oEi '(3.[0-9]*)')"

$ echo $PVER
3.11 

$ sudo CPLUS_INCLUDE_PATH=/usr/include/python$PVER ./b2 -d 1 install
Performing configuration checks

  - default address-model    : 64-bit (cached) [1]
  - default architecture     : x86 (cached) [1]
  - compiler supports SSE2   : yes (cached) [2]
  - compiler supports SSE4.1 : yes (cached) [2]
  - has std::atomic_ref      : no  (cached) [2]
  - has -Wl,--no-undefined   : yes (cached) [2]
  - has statx                : yes (cached) [2]
  - has init_priority attribute : yes (cached) [2]
  - has stat::st_blksize     : yes (cached) [2]
  - has stat::st_mtim        : yes (cached) [2]
  - has stat::st_mtimensec   : no  (cached) [2]
  - has stat::st_mtimespec   : no  (cached) [2]
  - has stat::st_birthtim    : no  (cached) [2]
  - has stat::st_birthtimensec : no  (cached) [2]
  - has stat::st_birthtimespec : no  (cached) [2]
  - has fdopendir(O_NOFOLLOW) : yes (cached) [2]
  - has dirent::d_type       : yes (cached) [2]
  - has POSIX *at APIs       : yes (cached) [2]
  - cxx11_auto_declarations  : yes (cached) [2]
  - cxx11_constexpr          : yes (cached) [2]
  - cxx11_defaulted_functions : yes (cached) [2]
  - cxx11_final              : yes (cached) [2]
  - cxx11_hdr_mutex          : yes (cached) [2]
  - cxx11_hdr_tuple          : yes (cached) [2]
  - cxx11_lambdas            : yes (cached) [2]
  - cxx11_noexcept           : yes (cached) [2]
  - cxx11_nullptr            : yes (cached) [2]
  - cxx11_rvalue_references  : yes (cached) [2]
  - cxx11_template_aliases   : yes (cached) [2]
  - cxx11_thread_local       : yes (cached) [2]
  - cxx11_variadic_templates : yes (cached) [2]
  - has_icu builds           : yes (cached) [2]
warning: Graph library does not contain MPI-based parallel components.
note: to enable them, add "using mpi ;" to your user-config.jam.
note: to suppress this message, pass "--without-graph_parallel" to bjam.
  - zlib                     : yes (cached)
  - bzip2                    : yes (cached)
  - lzma                     : yes (cached)
  - zstd                     : yes (cached)
  - has_lzma_cputhreads builds : yes (cached) [2]
  - cxx11_decltype           : yes (cached) [2]
  - cxx11_basic_alignas      : yes (cached) [2]
  - iconv (libc)             : yes (cached) [2]
  - icu                      : yes (cached) [2]
  - cxx11_defaulted_moves    : yes (cached) [2]
  - cxx11_hdr_functional     : yes (cached) [2]
  - cxx11_hdr_type_traits    : yes (cached) [2]
  - cxx11_override           : yes (cached) [2]
  - cxx11_range_based_for    : yes (cached) [2]
  - cxx11_scoped_enums       : yes (cached) [2]
  - cxx11_smart_ptr          : yes (cached) [2]
  - cxx11_static_assert      : yes (cached) [2]
  - lockfree boost::atomic_flag : yes (cached) [2]
  - native atomic int32 supported : yes (cached) [2]
  - native syslog supported  : yes (cached) [2]
  - pthread supports robust mutexes : yes (cached) [2]
  - compiler supports SSSE3  : yes (cached) [2]
  - compiler supports AVX2   : yes (cached) [2]
  - gcc visibility           : yes (cached) [2]
  - sfinae_expr              : yes (cached) [2]
  - cxx11_unified_initialization_syntax : yes (cached) [2]
  - cxx11_hdr_initializer_list : yes (cached) [2]
  - cxx11_hdr_chrono         : yes (cached) [2]
  - cxx11_numeric_limits     : yes (cached) [2]
  - cxx11_hdr_array          : yes (cached) [2]
  - cxx11_hdr_atomic         : yes (cached) [2]
  - cxx11_allocator          : yes (cached) [2]
  - cxx11_explicit_conversion_operators : yes (cached) [2]
  - long double support      : yes (cached) [2]
warning: skipping optional Message Passing Interface (MPI) library.
note: to enable MPI support, add "using mpi ;" to user-config.jam.
note: to suppress this message, pass "--without-mpi" to bjam.
note: otherwise, you can safely ignore this message.
  - cxx11_char16_t           : yes (cached) [2]
  - cxx11_char32_t           : yes (cached) [2]
  - Has Large File Support   : yes (cached) [2]
  - Has attribute init_priority : yes (cached) [2]
  - libbacktrace builds      : yes (cached) [2]
  - addr2line builds         : yes (cached) [2]
  - WinDbg builds            : no  (cached) [2]
  - WinDbg builds            : no  (cached) [3]
  - WinDbgCached builds      : no  (cached) [2]
  - WinDbgCached builds      : no  (cached) [3]
  - BOOST_COMP_GNUC >= 4.3.0 : yes (cached) [2]
  - BOOST_COMP_GNUC >= 4.3.0 : yes (cached) [4]
  - cxx11_hdr_thread         : yes (cached) [2]
  - cxx11_hdr_regex          : yes (cached) [2]
  - compiler supports SSE2   : yes (cached) [4]
  - compiler supports SSE4.1 : yes (cached) [4]
  - has std::atomic_ref      : no  (cached) [4]
  - has statx                : yes (cached) [4]
  - has init_priority attribute : yes (cached) [4]
  - has stat::st_blksize     : yes (cached) [4]
  - has stat::st_mtim        : yes (cached) [4]
  - has stat::st_mtimensec   : no  (cached) [4]
  - has stat::st_mtimespec   : no  (cached) [4]
  - has stat::st_birthtim    : no  (cached) [4]
  - has stat::st_birthtimensec : no  (cached) [4]
  - has stat::st_birthtimespec : no  (cached) [4]
  - has fdopendir(O_NOFOLLOW) : yes (cached) [4]
  - has dirent::d_type       : yes (cached) [4]
  - has POSIX *at APIs       : yes (cached) [4]
  - cxx11_auto_declarations  : yes (cached) [4]
  - cxx11_constexpr          : yes (cached) [4]
  - cxx11_defaulted_functions : yes (cached) [4]
  - cxx11_final              : yes (cached) [4]
  - cxx11_hdr_mutex          : yes (cached) [4]
  - cxx11_hdr_tuple          : yes (cached) [4]
  - cxx11_lambdas            : yes (cached) [4]
  - cxx11_noexcept           : yes (cached) [4]
  - cxx11_nullptr            : yes (cached) [4]
  - cxx11_rvalue_references  : yes (cached) [4]
  - cxx11_template_aliases   : yes (cached) [4]
  - cxx11_thread_local       : yes (cached) [4]
  - cxx11_variadic_templates : yes (cached) [4]
  - has_icu builds           : yes (cached) [4]
  - zlib                     : yes (cached) [5]
  - bzip2                    : yes (cached) [5]
  - lzma                     : yes (cached) [5]
  - zstd                     : yes (cached) [5]
  - has_lzma_cputhreads builds : yes (cached) [4]
  - cxx11_decltype           : yes (cached) [4]
  - cxx11_basic_alignas      : yes (cached) [4]
  - iconv (libc)             : yes (cached) [4]
  - icu                      : yes (cached) [4]
  - cxx11_defaulted_moves    : yes (cached) [4]
  - cxx11_hdr_functional     : yes (cached) [4]
  - cxx11_hdr_type_traits    : yes (cached) [4]
  - cxx11_override           : yes (cached) [4]
  - cxx11_range_based_for    : yes (cached) [4]
  - cxx11_scoped_enums       : yes (cached) [4]
  - cxx11_smart_ptr          : yes (cached) [4]
  - cxx11_static_assert      : yes (cached) [4]
  - lockfree boost::atomic_flag : yes (cached) [4]
  - native atomic int32 supported : yes (cached) [4]
  - native syslog supported  : yes (cached) [4]
  - pthread supports robust mutexes : yes (cached) [4]
  - compiler supports SSSE3  : yes (cached) [4]
  - compiler supports AVX2   : yes (cached) [4]
  - gcc visibility           : yes (cached) [4]
  - sfinae_expr              : yes (cached) [4]
  - cxx11_unified_initialization_syntax : yes (cached) [4]
  - cxx11_hdr_initializer_list : yes (cached) [4]
  - cxx11_hdr_chrono         : yes (cached) [4]
  - cxx11_numeric_limits     : yes (cached) [4]
  - cxx11_hdr_array          : yes (cached) [4]
  - cxx11_hdr_atomic         : yes (cached) [4]
  - cxx11_allocator          : yes (cached) [4]
  - cxx11_explicit_conversion_operators : yes (cached) [4]
  - long double support      : yes (cached) [4]
  - cxx11_char16_t           : yes (cached) [4]
  - cxx11_char32_t           : yes (cached) [4]
  - Has Large File Support   : yes (cached) [4]
  - Has attribute init_priority : yes (cached) [4]
  - libbacktrace builds      : yes (cached) [4]
  - addr2line builds         : yes (cached) [4]
  - WinDbg builds            : no  (cached) [4]
  - WinDbg builds            : no  (cached) [6]
  - WinDbgCached builds      : no  (cached) [4]
  - WinDbgCached builds      : no  (cached) [6]
  - cxx11_hdr_thread         : yes (cached) [4]
  - cxx11_hdr_regex          : yes (cached) [4]

[1] gcc-12
[2] gcc-12/release/python-3.11/threading-multi/visibility-hidden
[3] gcc-12/release/build-no/python-3.11/threading-multi/visibility-hidden
[4] gcc-12/release/link-static/python-3.11/threading-multi/visibility-hidden
[5] link-static
[6] gcc-12/release/build-no/link-static/python-3.11/threading-multi/visibility-hidden

Component configuration:

  - atomic                   : building
  - chrono                   : building
  - container                : building
  - context                  : building
  - contract                 : building
  - coroutine                : building
  - date_time                : building
  - exception                : building
  - fiber                    : building
  - filesystem               : building
  - graph                    : building
  - graph_parallel           : building
  - headers                  : building
  - iostreams                : building
  - json                     : building
  - locale                   : building
  - log                      : building
  - math                     : building
  - mpi                      : building
  - nowide                   : building
  - program_options          : building
  - python                   : building
  - random                   : building
  - regex                    : building
  - serialization            : building
  - stacktrace               : building
  - system                   : building
  - test                     : building
  - thread                   : building
  - timer                    : building
  - type_erasure             : building
  - url                      : building
  - wave                     : building

...patience...
...patience...
...patience...
...patience...
...patience...
...patience...
...patience...
...found 52031 targets...

The b2 installation program stored the following include files in /usr/local/include/boost/

Shell

$ ls -w72 /usr/local/include/boost/
accumulators/                 make_unique.hpp
algorithm/                    math/
align/                        math_fwd.hpp
align.hpp                     mem_fn.hpp
aligned_storage.hpp           memory_order.hpp
any/                          metaparse/
any.hpp                       metaparse.hpp
archive/                      move/
array.hpp                     mp11/
asio/                         mp11.hpp
asio.hpp                      mpi/
assert/                       mpi.hpp
assert.hpp                    mpl/
assign/                       msm/
assign.hpp                    multi_array/
atomic/                       multi_array.hpp
atomic.hpp                    multi_index/
beast/                        multi_index_container.hpp
beast.hpp                     multi_index_container_fwd.hpp
bimap/                        multiprecision/
bimap.hpp                     mysql/
bind/                         mysql.hpp
bind.hpp                      next_prior.hpp
blank.hpp                     non_type.hpp
blank_fwd.hpp                 noncopyable.hpp
call_traits.hpp               nondet_random.hpp
callable_traits/              none.hpp
callable_traits.hpp           none_t.hpp
cast.hpp                      nowide/
cerrno.hpp                    numeric/
checked_delete.hpp            operators.hpp
chrono/                       operators_v1.hpp
chrono.hpp                    optional/
circular_buffer/              optional.hpp
circular_buffer.hpp           outcome/
circular_buffer_fwd.hpp       outcome.hpp
compat/                       parameter/
compatibility/                parameter.hpp
compressed_pair.hpp           pending/
compute/                      pfr/
compute.hpp                   pfr.hpp
concept/                      phoenix/
concept_archetype.hpp         phoenix.hpp
concept_check/                pointee.hpp
concept_check.hpp             pointer_cast.hpp
config/                       pointer_to_other.hpp
config.hpp                    poly_collection/
container/                    polygon/
container_hash/               polymorphic_cast.hpp
context/                      polymorphic_pointer_cast.hpp
contract/                     pool/
contract.hpp                  predef/
contract_macro.hpp            predef.h
convert/                      preprocessor/
convert.hpp                   preprocessor.hpp
core/                         process/
coroutine/                    process.hpp
coroutine2/                   program_options/
crc.hpp                       program_options.hpp
cregex.hpp                    progress.hpp
cstdfloat.hpp                 property_map/
cstdint.hpp                   property_tree/
cstdlib.hpp                   proto/
current_function.hpp          ptr_container/
cxx11_char_types.hpp          python/
date_time/                    python.hpp
date_time.hpp                 qvm/
describe/                     qvm.hpp
describe.hpp                  qvm_lite.hpp
detail/                       random/
dll/                          random.hpp
dll.hpp                       range/
dynamic_bitset/               range.hpp
dynamic_bitset.hpp            ratio/
dynamic_bitset_fwd.hpp        ratio.hpp
enable_shared_from_this.hpp   rational.hpp
endian/                       ref.hpp
endian.hpp                    regex/
exception/                    regex.h
exception_ptr.hpp             regex.hpp
fiber/                        regex_fwd.hpp
filesystem/                   safe_numerics/
filesystem.hpp                scope_exit.hpp
flyweight/                    scoped_array.hpp
flyweight.hpp                 scoped_ptr.hpp
foreach.hpp                   serialization/
foreach_fwd.hpp               shared_array.hpp
format/                       shared_container_iterator.hpp
format.hpp                    shared_ptr.hpp
function/                     signals2/
function.hpp                  signals2.hpp
function_equal.hpp            smart_ptr/
function_output_iterator.hpp  smart_ptr.hpp
function_types/               sort/
functional/                   spirit/
functional.hpp                spirit.hpp
fusion/                       stacktrace/
generator_iterator.hpp        stacktrace.hpp
geometry/                     statechart/
geometry.hpp                  static_assert.hpp
get_pointer.hpp               static_string/
gil/                          static_string.hpp
gil.hpp                       stl_interfaces/
graph/                        swap.hpp
hana/                         system/
hana.hpp                      system.hpp
heap/                         test/
histogram/                    thread/
histogram.hpp                 thread.hpp
hof/                          throw_exception.hpp
hof.hpp                       timer/
icl/                          timer.hpp
implicit_cast.hpp             token_functions.hpp
indirect_reference.hpp        token_iterator.hpp
integer/                      tokenizer.hpp
integer.hpp                   tti/
integer_fwd.hpp               tuple/
integer_traits.hpp            type.hpp
interprocess/                 type_erasure/
intrusive/                    type_index/
intrusive_ptr.hpp             type_index.hpp
io/                           type_traits/
io_fwd.hpp                    type_traits.hpp
iostreams/                    typeof/
is_placeholder.hpp            units/
iterator/                     unordered/
iterator.hpp                  unordered_map.hpp
iterator_adaptors.hpp         unordered_set.hpp
json/                         url/
json.hpp                      url.hpp
lambda/                       utility/
lambda2/                      utility.hpp
lambda2.hpp                   uuid/
leaf/                         variant/
leaf.hpp                      variant.hpp
lexical_cast/                 variant2/
lexical_cast.hpp              variant2.hpp
limits.hpp                    version.hpp
local_function/               visit_each.hpp
local_function.hpp            vmd/
locale/                       wave/
locale.hpp                    wave.hpp
lockfree/                     weak_ptr.hpp
log/                          winapi/
logic/                        xpressive/
make_default.hpp              yap/
make_shared.hpp

The b2 installation program stored the following libraries in /usr/local/lib/:

Shell

$ ls -w72 /usr/local/lib
cmake/
libboost_atomic.a
libboost_atomic.so@
libboost_atomic.so.1.83.0*
libboost_chrono.a
libboost_chrono.so@
libboost_chrono.so.1.83.0*
libboost_container.a
libboost_container.so@
libboost_container.so.1.83.0*
libboost_context.a
libboost_context.so@
libboost_context.so.1.83.0*
libboost_contract.a
libboost_contract.so@
libboost_contract.so.1.83.0*
libboost_coroutine.a
libboost_coroutine.so@
libboost_coroutine.so.1.83.0*
libboost_date_time.a
libboost_date_time.so@
libboost_date_time.so.1.83.0*
libboost_exception.a
libboost_fiber.a
libboost_fiber.so@
libboost_fiber.so.1.83.0*
libboost_filesystem.a
libboost_filesystem.so@
libboost_filesystem.so.1.83.0*
libboost_graph.a
libboost_graph.so@
libboost_graph.so.1.83.0*
libboost_iostreams.a
libboost_iostreams.so@
libboost_iostreams.so.1.83.0*
libboost_json.a
libboost_json.so@
libboost_json.so.1.83.0*
libboost_locale.a
libboost_locale.so@
libboost_locale.so.1.83.0*
libboost_log.a
libboost_log.so@
libboost_log.so.1.83.0*
libboost_log_setup.a
libboost_log_setup.so@
libboost_log_setup.so.1.83.0*
libboost_math_c99.a
libboost_math_c99.so@
libboost_math_c99.so.1.83.0*
libboost_math_c99f.a
libboost_math_c99f.so@
libboost_math_c99f.so.1.83.0*
libboost_math_c99l.a
libboost_math_c99l.so@
libboost_math_c99l.so.1.83.0*
libboost_math_tr1.a
libboost_math_tr1.so@
libboost_math_tr1.so.1.83.0*
libboost_math_tr1f.a
libboost_math_tr1f.so@
libboost_math_tr1f.so.1.83.0*
libboost_math_tr1l.a
libboost_math_tr1l.so@
libboost_math_tr1l.so.1.83.0*
libboost_nowide.a
libboost_nowide.so@
libboost_nowide.so.1.83.0*
libboost_prg_exec_monitor.a
libboost_prg_exec_monitor.so@
libboost_prg_exec_monitor.so.1.83.0*
libboost_program_options.a
libboost_program_options.so@
libboost_program_options.so.1.83.0*
libboost_random.a
libboost_random.so@
libboost_random.so.1.83.0*
libboost_regex.a
libboost_regex.so@
libboost_regex.so.1.83.0*
libboost_serialization.a
libboost_serialization.so@
libboost_serialization.so.1.83.0*
libboost_stacktrace_addr2line.a
libboost_stacktrace_addr2line.so@
libboost_stacktrace_addr2line.so.1.83.0*
libboost_stacktrace_backtrace.a
libboost_stacktrace_backtrace.so@
libboost_stacktrace_backtrace.so.1.83.0*
libboost_stacktrace_basic.a
libboost_stacktrace_basic.so@
libboost_stacktrace_basic.so.1.83.0*
libboost_stacktrace_noop.a
libboost_stacktrace_noop.so@
libboost_stacktrace_noop.so.1.83.0*
libboost_system.a
libboost_system.so@
libboost_system.so.1.83.0*
libboost_test_exec_monitor.a
libboost_thread.a
libboost_thread.so@
libboost_thread.so.1.83.0*
libboost_timer.a
libboost_timer.so@
libboost_timer.so.1.83.0*
libboost_type_erasure.a
libboost_type_erasure.so@
libboost_type_erasure.so.1.83.0*
libboost_unit_test_framework.a
libboost_unit_test_framework.so@
libboost_unit_test_framework.so.1.83.0*
libboost_url.a
libboost_url.so@
libboost_url.so.1.83.0*
libboost_wave.a
libboost_wave.so@
libboost_wave.so.1.83.0*
libboost_wserialization.a
libboost_wserialization.so@
libboost_wserialization.so.1.83.0*
node_modules/
python3.11/

Verify Boost System Libraries

The ldconfig command manages Linux system libraries. Here is the help message:

Shell

$ man ldconfig
ldconfig(8)             System Manager's Manual            ldconfig(8)

NAME
        ldconfig - configure dynamic linker run-time bindings

SYNOPSIS
        /sbin/ldconfig [-nNvVX] [-C cache] [-f conf] [-r root]
                      directory ...

        /sbin/ldconfig -l [-v] library ...

        /sbin/ldconfig -p

DESCRIPTION
        ldconfig creates the necessary links and cache to the most  re‐
        cent shared libraries found in the directories specified on the
        command line, in the file /etc/ld.so.conf, and in  the  trusted
        directories,  /lib  and /usr/lib.  On some 64-bit architectures
        such as x86-64, /lib and /usr/lib are the  trusted  directories
        for  32-bit libraries, while /lib64 and /usr/lib64 are used for
        64-bit libraries.

        The cache is used by the run-time linker, ld.so or ld-linux.so.
        ldconfig  checks  the  header and filenames of the libraries it
        encounters when determining which versions  should  have  their
        links  updated.   ldconfig  should normally be run by the supe‐
        ruser as it may require write permission on some root owned di‐
        rectories and files.

        ldconfig  will  look only at files that are named lib*.so* (for
        regular shared objects) or ld-*.so* (for the dynamic loader it‐
        self).   Other files will be ignored.  Also, ldconfig expects a
        certain pattern to how the symbolic links are set up, like this
        example, where the middle file (libfoo.so.1 here) is the SONAME
        for the library:

            libfoo.so -> libfoo.so.1 -> libfoo.so.1.12

        Failure to follow this pattern may result in compatibility  is‐
        sues after an upgrade.

OPTIONS
        -c fmt
        --format=fmt
              (Since  glibc 2.2) Use cache format fmt, which is one of
              old, new, or compat.  Since glibc 2.32, the  default  is
              new.  Before that, it was compat.

        -C cache
              Use cache instead of /etc/ld.so.cache.

        -f conf
              Use conf instead of /etc/ld.so.conf.

        -i
        --ignore-aux-cache
              (Since glibc 2.7) Ignore auxiliary cache file.

        -l     (Since  glibc  2.2)  Interpret  each operand as a libary
              name and configure its links.  Intended for use only  by
              experts.

        -n     Process  only  the  directories specified on the command
              line; don't process the trusted directories,  nor  those
              specified in /etc/ld.so.conf.  Implies -N.

        -N     Don't  rebuild  the cache.  Unless -X is also specified,
              links are still updated.

        -p
        --print-cache
              Print the lists of directories and  candidate  libraries
              stored in the current cache.

        -r root
              Change to and use root as the root directory.

        -v
        --verbose
              Verbose mode.  Print current version number, the name of
              each directory as it is scanned, and any links that  are
              created.  Overrides quiet mode.

        -V
        --version
              Print program version.

        -X     Don't  update  links.   Unless -N is also specified, the
              cache is still rebuilt.

FILES
        /lib/ld.so
              is the run-time linker/loader.
        /etc/ld.so.conf
              contains a list of directories, one per line,  in  which
              to search for libraries.
        /etc/ld.so.cache
              contains  an  ordered list of libraries found in the di‐
              rectories specified in /etc/ld.so.conf, as well as those
              found in the trusted directories.

SEE ALSO
        ldd(1), ld.so(8)

Linux man-pages 6.03          2023-01-07                   ldconfig(8)

The Boost installation procedure automatically causes the newly built Boost libraries to be added to the configured system libraries when bootstrap.sh is not invoked with a target path, for example by specifying the --with-python-root option.

To manually add the newly built Boost libraries to the system load path, use ldconfig. The following adds the newly compiled and installed Boost libraries in /usr/local/lib/ (and any other libraries that might happen to be in that directory) to the ld.so cache.

Shell

$ sudo ldconfig -n /usr/local/lib

We can verify that the library cache now contains the newly built Boost libraries by using ldconfig:

Shell

$ ldconfig -p | grep boost
libboost_wserialization.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_wserialization.so.1.83.0
libboost_wserialization.so (libc6,x86-64) => /usr/local/lib/libboost_wserialization.so
libboost_wave.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_wave.so.1.83.0
libboost_wave.so (libc6,x86-64) => /usr/local/lib/libboost_wave.so
libboost_url.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_url.so.1.83.0
libboost_url.so (libc6,x86-64) => /usr/local/lib/libboost_url.so
libboost_unit_test_framework.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_unit_test_framework.so.1.83.0
libboost_unit_test_framework.so (libc6,x86-64) => /usr/local/lib/libboost_unit_test_framework.so
libboost_type_erasure.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_type_erasure.so.1.83.0
libboost_type_erasure.so (libc6,x86-64) => /usr/local/lib/libboost_type_erasure.so
libboost_timer.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_timer.so.1.83.0
libboost_timer.so (libc6,x86-64) => /usr/local/lib/libboost_timer.so
libboost_thread.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_thread.so.1.83.0
libboost_thread.so (libc6,x86-64) => /usr/local/lib/libboost_thread.so
libboost_system.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_system.so.1.83.0
libboost_system.so (libc6,x86-64) => /usr/local/lib/libboost_system.so
libboost_stacktrace_noop.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_stacktrace_noop.so.1.83.0
libboost_stacktrace_noop.so (libc6,x86-64) => /usr/local/lib/libboost_stacktrace_noop.so
libboost_stacktrace_basic.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_stacktrace_basic.so.1.83.0
libboost_stacktrace_basic.so (libc6,x86-64) => /usr/local/lib/libboost_stacktrace_basic.so
libboost_stacktrace_backtrace.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_stacktrace_backtrace.so.1.83.0
libboost_stacktrace_backtrace.so (libc6,x86-64) => /usr/local/lib/libboost_stacktrace_backtrace.so
libboost_stacktrace_addr2line.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_stacktrace_addr2line.so.1.83.0
libboost_stacktrace_addr2line.so (libc6,x86-64) => /usr/local/lib/libboost_stacktrace_addr2line.so
libboost_serialization.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_serialization.so.1.83.0
libboost_serialization.so (libc6,x86-64) => /usr/local/lib/libboost_serialization.so
libboost_regex.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_regex.so.1.83.0
libboost_regex.so.1.74.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/libboost_regex.so.1.74.0
libboost_regex.so (libc6,x86-64) => /usr/local/lib/libboost_regex.so
libboost_random.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_random.so.1.83.0
libboost_random.so (libc6,x86-64) => /usr/local/lib/libboost_random.so
libboost_python311.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_python311.so.1.83.0
libboost_python311.so (libc6,x86-64) => /usr/local/lib/libboost_python311.so
libboost_program_options.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_program_options.so.1.83.0
libboost_program_options.so (libc6,x86-64) => /usr/local/lib/libboost_program_options.so
libboost_prg_exec_monitor.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_prg_exec_monitor.so.1.83.0
libboost_prg_exec_monitor.so (libc6,x86-64) => /usr/local/lib/libboost_prg_exec_monitor.so
libboost_nowide.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_nowide.so.1.83.0
libboost_nowide.so (libc6,x86-64) => /usr/local/lib/libboost_nowide.so
libboost_math_tr1l.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_math_tr1l.so.1.83.0
libboost_math_tr1l.so (libc6,x86-64) => /usr/local/lib/libboost_math_tr1l.so
libboost_math_tr1f.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_math_tr1f.so.1.83.0
libboost_math_tr1f.so (libc6,x86-64) => /usr/local/lib/libboost_math_tr1f.so
libboost_math_tr1.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_math_tr1.so.1.83.0
libboost_math_tr1.so (libc6,x86-64) => /usr/local/lib/libboost_math_tr1.so
libboost_math_c99l.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_math_c99l.so.1.83.0
libboost_math_c99l.so (libc6,x86-64) => /usr/local/lib/libboost_math_c99l.so
libboost_math_c99f.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_math_c99f.so.1.83.0
libboost_math_c99f.so (libc6,x86-64) => /usr/local/lib/libboost_math_c99f.so
libboost_math_c99.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_math_c99.so.1.83.0
libboost_math_c99.so (libc6,x86-64) => /usr/local/lib/libboost_math_c99.so
libboost_log_setup.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_log_setup.so.1.83.0
libboost_log_setup.so (libc6,x86-64) => /usr/local/lib/libboost_log_setup.so
libboost_log.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_log.so.1.83.0
libboost_log.so (libc6,x86-64) => /usr/local/lib/libboost_log.so
libboost_locale.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_locale.so.1.83.0
libboost_locale.so (libc6,x86-64) => /usr/local/lib/libboost_locale.so
libboost_json.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_json.so.1.83.0
libboost_json.so (libc6,x86-64) => /usr/local/lib/libboost_json.so
libboost_iostreams.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_iostreams.so.1.83.0
libboost_iostreams.so (libc6,x86-64) => /usr/local/lib/libboost_iostreams.so
libboost_graph.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_graph.so.1.83.0
libboost_graph.so (libc6,x86-64) => /usr/local/lib/libboost_graph.so
libboost_filesystem.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_filesystem.so.1.83.0
libboost_filesystem.so (libc6,x86-64) => /usr/local/lib/libboost_filesystem.so
libboost_fiber.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_fiber.so.1.83.0
libboost_fiber.so (libc6,x86-64) => /usr/local/lib/libboost_fiber.so
libboost_date_time.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_date_time.so.1.83.0
libboost_date_time.so (libc6,x86-64) => /usr/local/lib/libboost_date_time.so
libboost_coroutine.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_coroutine.so.1.83.0
libboost_coroutine.so (libc6,x86-64) => /usr/local/lib/libboost_coroutine.so
libboost_contract.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_contract.so.1.83.0
libboost_contract.so (libc6,x86-64) => /usr/local/lib/libboost_contract.so
libboost_context.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_context.so.1.83.0
libboost_context.so (libc6,x86-64) => /usr/local/lib/libboost_context.so
libboost_container.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_container.so.1.83.0
libboost_container.so (libc6,x86-64) => /usr/local/lib/libboost_container.so
libboost_chrono.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_chrono.so.1.83.0
libboost_chrono.so (libc6,x86-64) => /usr/local/lib/libboost_chrono.so
libboost_atomic.so.1.83.0 (libc6,x86-64) => /usr/local/lib/libboost_atomic.so.1.83.0
libboost_atomic.so (libc6,x86-64) => /usr/local/lib/libboost_atomic.so

As a further check, examine the full path of an arbitrary include file (src.hpp) and list the Boost libraries:

Shell

$ locate src.hpp
/usr/include/boost/asio/impl/src.hpp
/usr/include/boost/asio/ssl/impl/src.hpp
/usr/include/boost/beast/src.hpp 

$ find /usr/local/lib/ -iname *boost*
/usr/local/lib/libboost_math_tr1f.a
/usr/local/lib/libboost_log_setup.so
/usr/local/lib/libboost_system.so.1.83.0
/usr/local/lib/libboost_system.so
/usr/local/lib/libboost_serialization.a
/usr/local/lib/libboost_context.a
/usr/local/lib/libboost_nowide.so
/usr/local/lib/libboost_contract.so.1.83.0
/usr/local/lib/libboost_prg_exec_monitor.a
/usr/local/lib/libboost_stacktrace_backtrace.so
/usr/local/lib/libboost_iostreams.so
/usr/local/lib/libboost_math_c99.so.1.83.0
/usr/local/lib/libboost_stacktrace_addr2line.so
/usr/local/lib/libboost_locale.so.1.83.0
/usr/local/lib/libboost_math_c99l.so.1.83.0
/usr/local/lib/libboost_timer.a
/usr/local/lib/libboost_atomic.so.1.83.0
/usr/local/lib/libboost_url.so.1.83.0
/usr/local/lib/libboost_math_c99.a
/usr/local/lib/libboost_date_time.so
/usr/local/lib/libboost_fiber.a
/usr/local/lib/libboost_filesystem.so.1.83.0
/usr/local/lib/libboost_date_time.a
/usr/local/lib/libboost_log_setup.a
/usr/local/lib/libboost_type_erasure.so
/usr/local/lib/libboost_json.so.1.83.0
/usr/local/lib/libboost_wave.so.1.83.0
/usr/local/lib/libboost_unit_test_framework.so
/usr/local/lib/libboost_type_erasure.so.1.83.0
/usr/local/lib/libboost_random.so.1.83.0
/usr/local/lib/libboost_filesystem.so
/usr/local/lib/libboost_stacktrace_basic.a
/usr/local/lib/libboost_math_c99.so
/usr/local/lib/libboost_regex.a
/usr/local/lib/libboost_coroutine.so.1.83.0
/usr/local/lib/libboost_filesystem.a
/usr/local/lib/libboost_date_time.so.1.83.0
/usr/local/lib/libboost_wave.a
/usr/local/lib/libboost_regex.so
/usr/local/lib/libboost_container.a
/usr/local/lib/libboost_locale.a
/usr/local/lib/libboost_graph.so
/usr/local/lib/libboost_graph.so.1.83.0
/usr/local/lib/libboost_math_tr1f.so.1.83.0
/usr/local/lib/libboost_math_tr1.so.1.83.0
/usr/local/lib/libboost_program_options.so.1.83.0
/usr/local/lib/libboost_iostreams.so.1.83.0
/usr/local/lib/libboost_math_c99f.so.1.83.0
/usr/local/lib/libboost_nowide.so.1.83.0
/usr/local/lib/libboost_thread.so.1.83.0
/usr/local/lib/libboost_unit_test_framework.a
/usr/local/lib/cmake/boost_math_c99f-1.83.0
/usr/local/lib/cmake/boost_math_c99f-1.83.0/libboost_math_c99f-variant-shared.cmake
/usr/local/lib/cmake/boost_math_c99f-1.83.0/boost_math_c99f-config.cmake
/usr/local/lib/cmake/boost_math_c99f-1.83.0/libboost_math_c99f-variant-static.cmake
/usr/local/lib/cmake/boost_math_c99f-1.83.0/boost_math_c99f-config-version.cmake
/usr/local/lib/cmake/boost_test_exec_monitor-1.83.0
/usr/local/lib/cmake/boost_test_exec_monitor-1.83.0/libboost_test_exec_monitor-variant-static.cmake
/usr/local/lib/cmake/boost_test_exec_monitor-1.83.0/libboost_test_exec_monitor-variant-shared.cmake
/usr/local/lib/cmake/boost_test_exec_monitor-1.83.0/boost_test_exec_monitor-config.cmake
/usr/local/lib/cmake/boost_test_exec_monitor-1.83.0/boost_test_exec_monitor-config-version.cmake
/usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0
/usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0/libboost_stacktrace_backtrace-variant-shared.cmake
/usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0/boost_stacktrace_backtrace-config-version.cmake
/usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0/libboost_stacktrace_backtrace-variant-static.cmake
/usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0/boost_stacktrace_backtrace-config.cmake
/usr/local/lib/cmake/boost_wave-1.83.0
/usr/local/lib/cmake/boost_wave-1.83.0/libboost_wave-variant-shared.cmake
/usr/local/lib/cmake/boost_wave-1.83.0/boost_wave-config-version.cmake
/usr/local/lib/cmake/boost_wave-1.83.0/libboost_wave-variant-static.cmake
/usr/local/lib/cmake/boost_wave-1.83.0/boost_wave-config.cmake
/usr/local/lib/cmake/boost_log-1.83.0
/usr/local/lib/cmake/boost_log-1.83.0/libboost_log-variant-shared.cmake
/usr/local/lib/cmake/boost_log-1.83.0/boost_log-config.cmake
/usr/local/lib/cmake/boost_log-1.83.0/libboost_log-variant-static.cmake
/usr/local/lib/cmake/boost_log-1.83.0/boost_log-config-version.cmake
/usr/local/lib/cmake/boost_stacktrace_basic-1.83.0
/usr/local/lib/cmake/boost_stacktrace_basic-1.83.0/boost_stacktrace_basic-config.cmake
/usr/local/lib/cmake/boost_stacktrace_basic-1.83.0/boost_stacktrace_basic-config-version.cmake
/usr/local/lib/cmake/boost_stacktrace_basic-1.83.0/libboost_stacktrace_basic-variant-static.cmake
/usr/local/lib/cmake/boost_stacktrace_basic-1.83.0/libboost_stacktrace_basic-variant-shared.cmake
/usr/local/lib/cmake/boost_coroutine-1.83.0
/usr/local/lib/cmake/boost_coroutine-1.83.0/boost_coroutine-config-version.cmake
/usr/local/lib/cmake/boost_coroutine-1.83.0/boost_coroutine-config.cmake
/usr/local/lib/cmake/boost_coroutine-1.83.0/libboost_coroutine-variant-shared.cmake
/usr/local/lib/cmake/boost_coroutine-1.83.0/libboost_coroutine-variant-static.cmake
/usr/local/lib/cmake/boost_timer-1.83.0
/usr/local/lib/cmake/boost_timer-1.83.0/boost_timer-config-version.cmake
/usr/local/lib/cmake/boost_timer-1.83.0/boost_timer-config.cmake
/usr/local/lib/cmake/boost_timer-1.83.0/libboost_timer-variant-shared.cmake
/usr/local/lib/cmake/boost_timer-1.83.0/libboost_timer-variant-static.cmake
/usr/local/lib/cmake/boost_program_options-1.83.0
/usr/local/lib/cmake/boost_program_options-1.83.0/boost_program_options-config.cmake
/usr/local/lib/cmake/boost_program_options-1.83.0/libboost_program_options-variant-static.cmake
/usr/local/lib/cmake/boost_program_options-1.83.0/libboost_program_options-variant-shared.cmake
/usr/local/lib/cmake/boost_program_options-1.83.0/boost_program_options-config-version.cmake
/usr/local/lib/cmake/boost_math_tr1f-1.83.0
/usr/local/lib/cmake/boost_math_tr1f-1.83.0/libboost_math_tr1f-variant-static.cmake
/usr/local/lib/cmake/boost_math_tr1f-1.83.0/boost_math_tr1f-config.cmake
/usr/local/lib/cmake/boost_math_tr1f-1.83.0/boost_math_tr1f-config-version.cmake
/usr/local/lib/cmake/boost_math_tr1f-1.83.0/libboost_math_tr1f-variant-shared.cmake
/usr/local/lib/cmake/boost_date_time-1.83.0
/usr/local/lib/cmake/boost_date_time-1.83.0/libboost_date_time-variant-static.cmake
/usr/local/lib/cmake/boost_date_time-1.83.0/libboost_date_time-variant-shared.cmake
/usr/local/lib/cmake/boost_date_time-1.83.0/boost_date_time-config.cmake
/usr/local/lib/cmake/boost_date_time-1.83.0/boost_date_time-config-version.cmake
/usr/local/lib/cmake/boost_math_c99l-1.83.0
/usr/local/lib/cmake/boost_math_c99l-1.83.0/libboost_math_c99l-variant-static.cmake
/usr/local/lib/cmake/boost_math_c99l-1.83.0/libboost_math_c99l-variant-shared.cmake
/usr/local/lib/cmake/boost_math_c99l-1.83.0/boost_math_c99l-config.cmake
/usr/local/lib/cmake/boost_math_c99l-1.83.0/boost_math_c99l-config-version.cmake
/usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0
/usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0/libboost_prg_exec_monitor-variant-shared.cmake
/usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0/libboost_prg_exec_monitor-variant-static.cmake
/usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0/boost_prg_exec_monitor-config.cmake
/usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0/boost_prg_exec_monitor-config-version.cmake
/usr/local/lib/cmake/boost_wserialization-1.83.0
/usr/local/lib/cmake/boost_wserialization-1.83.0/libboost_wserialization-variant-shared.cmake
/usr/local/lib/cmake/boost_wserialization-1.83.0/boost_wserialization-config-version.cmake
/usr/local/lib/cmake/boost_wserialization-1.83.0/libboost_wserialization-variant-static.cmake
/usr/local/lib/cmake/boost_wserialization-1.83.0/boost_wserialization-config.cmake
/usr/local/lib/cmake/boost_serialization-1.83.0
/usr/local/lib/cmake/boost_serialization-1.83.0/boost_serialization-config.cmake
/usr/local/lib/cmake/boost_serialization-1.83.0/libboost_serialization-variant-static.cmake
/usr/local/lib/cmake/boost_serialization-1.83.0/boost_serialization-config-version.cmake
/usr/local/lib/cmake/boost_serialization-1.83.0/libboost_serialization-variant-shared.cmake
/usr/local/lib/cmake/boost_random-1.83.0
/usr/local/lib/cmake/boost_random-1.83.0/boost_random-config.cmake
/usr/local/lib/cmake/boost_random-1.83.0/libboost_random-variant-static.cmake
/usr/local/lib/cmake/boost_random-1.83.0/libboost_random-variant-shared.cmake
/usr/local/lib/cmake/boost_random-1.83.0/boost_random-config-version.cmake
/usr/local/lib/cmake/boost_log_setup-1.83.0
/usr/local/lib/cmake/boost_log_setup-1.83.0/boost_log_setup-config-version.cmake
/usr/local/lib/cmake/boost_log_setup-1.83.0/libboost_log_setup-variant-shared.cmake
/usr/local/lib/cmake/boost_log_setup-1.83.0/libboost_log_setup-variant-static.cmake
/usr/local/lib/cmake/boost_log_setup-1.83.0/boost_log_setup-config.cmake
/usr/local/lib/cmake/boost_exception-1.83.0
/usr/local/lib/cmake/boost_exception-1.83.0/boost_exception-config.cmake
/usr/local/lib/cmake/boost_exception-1.83.0/boost_exception-config-version.cmake
/usr/local/lib/cmake/boost_system-1.83.0
/usr/local/lib/cmake/boost_system-1.83.0/libboost_system-variant-shared.cmake
/usr/local/lib/cmake/boost_system-1.83.0/boost_system-config-version.cmake
/usr/local/lib/cmake/boost_system-1.83.0/libboost_system-variant-static.cmake
/usr/local/lib/cmake/boost_system-1.83.0/boost_system-config.cmake
/usr/local/lib/cmake/boost_python-1.83.0
/usr/local/lib/cmake/boost_python-1.83.0/boost_python-config.cmake
/usr/local/lib/cmake/boost_python-1.83.0/boost_python-config-version.cmake
/usr/local/lib/cmake/boost_thread-1.83.0
/usr/local/lib/cmake/boost_thread-1.83.0/libboost_thread-variant-shared.cmake
/usr/local/lib/cmake/boost_thread-1.83.0/boost_thread-config.cmake
/usr/local/lib/cmake/boost_thread-1.83.0/boost_thread-config-version.cmake
/usr/local/lib/cmake/boost_thread-1.83.0/libboost_thread-variant-static.cmake
/usr/local/lib/cmake/boost_json-1.83.0
/usr/local/lib/cmake/boost_json-1.83.0/libboost_json-variant-shared.cmake
/usr/local/lib/cmake/boost_json-1.83.0/libboost_json-variant-static.cmake
/usr/local/lib/cmake/boost_json-1.83.0/boost_json-config.cmake
/usr/local/lib/cmake/boost_json-1.83.0/boost_json-config-version.cmake
/usr/local/lib/cmake/boost_iostreams-1.83.0
/usr/local/lib/cmake/boost_iostreams-1.83.0/libboost_iostreams-variant-shared.cmake
/usr/local/lib/cmake/boost_iostreams-1.83.0/boost_iostreams-config.cmake
/usr/local/lib/cmake/boost_iostreams-1.83.0/libboost_iostreams-variant-static.cmake
/usr/local/lib/cmake/boost_iostreams-1.83.0/boost_iostreams-config-version.cmake
/usr/local/lib/cmake/boost_math_c99-1.83.0
/usr/local/lib/cmake/boost_math_c99-1.83.0/libboost_math_c99-variant-static.cmake
/usr/local/lib/cmake/boost_math_c99-1.83.0/boost_math_c99-config-version.cmake
/usr/local/lib/cmake/boost_math_c99-1.83.0/libboost_math_c99-variant-shared.cmake
/usr/local/lib/cmake/boost_math_c99-1.83.0/boost_math_c99-config.cmake
/usr/local/lib/cmake/boost_unit_test_framework-1.83.0
/usr/local/lib/cmake/boost_unit_test_framework-1.83.0/libboost_unit_test_framework-variant-static.cmake
/usr/local/lib/cmake/boost_unit_test_framework-1.83.0/boost_unit_test_framework-config-version.cmake
/usr/local/lib/cmake/boost_unit_test_framework-1.83.0/libboost_unit_test_framework-variant-shared.cmake
/usr/local/lib/cmake/boost_unit_test_framework-1.83.0/boost_unit_test_framework-config.cmake
/usr/local/lib/cmake/boost_filesystem-1.83.0
/usr/local/lib/cmake/boost_filesystem-1.83.0/libboost_filesystem-variant-static.cmake
/usr/local/lib/cmake/boost_filesystem-1.83.0/boost_filesystem-config.cmake
/usr/local/lib/cmake/boost_filesystem-1.83.0/boost_filesystem-config-version.cmake
/usr/local/lib/cmake/boost_filesystem-1.83.0/libboost_filesystem-variant-shared.cmake
/usr/local/lib/cmake/Boost-1.83.0
/usr/local/lib/cmake/Boost-1.83.0/BoostConfigVersion.cmake
/usr/local/lib/cmake/Boost-1.83.0/BoostConfig.cmake
/usr/local/lib/cmake/boost_headers-1.83.0
/usr/local/lib/cmake/boost_headers-1.83.0/boost_headers-config.cmake
/usr/local/lib/cmake/boost_headers-1.83.0/boost_headers-config-version.cmake
/usr/local/lib/cmake/boost_nowide-1.83.0
/usr/local/lib/cmake/boost_nowide-1.83.0/libboost_nowide-variant-static.cmake
/usr/local/lib/cmake/boost_nowide-1.83.0/libboost_nowide-variant-shared.cmake
/usr/local/lib/cmake/boost_nowide-1.83.0/boost_nowide-config.cmake
/usr/local/lib/cmake/boost_nowide-1.83.0/boost_nowide-config-version.cmake
/usr/local/lib/cmake/boost_atomic-1.83.0
/usr/local/lib/cmake/boost_atomic-1.83.0/boost_atomic-config-version.cmake
/usr/local/lib/cmake/boost_atomic-1.83.0/boost_atomic-config.cmake
/usr/local/lib/cmake/boost_atomic-1.83.0/libboost_atomic-variant-shared.cmake
/usr/local/lib/cmake/boost_atomic-1.83.0/libboost_atomic-variant-static.cmake
/usr/local/lib/cmake/boost_graph-1.83.0
/usr/local/lib/cmake/boost_graph-1.83.0/boost_graph-config-version.cmake
/usr/local/lib/cmake/boost_graph-1.83.0/libboost_graph-variant-static.cmake
/usr/local/lib/cmake/boost_graph-1.83.0/libboost_graph-variant-shared.cmake
/usr/local/lib/cmake/boost_graph-1.83.0/boost_graph-config.cmake
/usr/local/lib/cmake/boost_math_tr1-1.83.0
/usr/local/lib/cmake/boost_math_tr1-1.83.0/boost_math_tr1-config-version.cmake
/usr/local/lib/cmake/boost_math_tr1-1.83.0/libboost_math_tr1-variant-static.cmake
/usr/local/lib/cmake/boost_math_tr1-1.83.0/libboost_math_tr1-variant-shared.cmake
/usr/local/lib/cmake/boost_math_tr1-1.83.0/boost_math_tr1-config.cmake
/usr/local/lib/cmake/boost_graph_parallel-1.83.0
/usr/local/lib/cmake/boost_graph_parallel-1.83.0/boost_graph_parallel-config-version.cmake
/usr/local/lib/cmake/boost_graph_parallel-1.83.0/boost_graph_parallel-config.cmake
/usr/local/lib/cmake/boost_stacktrace_noop-1.83.0
/usr/local/lib/cmake/boost_stacktrace_noop-1.83.0/libboost_stacktrace_noop-variant-static.cmake
/usr/local/lib/cmake/boost_stacktrace_noop-1.83.0/boost_stacktrace_noop-config-version.cmake
/usr/local/lib/cmake/boost_stacktrace_noop-1.83.0/boost_stacktrace_noop-config.cmake
/usr/local/lib/cmake/boost_stacktrace_noop-1.83.0/libboost_stacktrace_noop-variant-shared.cmake
/usr/local/lib/cmake/boost_mpi-1.83.0
/usr/local/lib/cmake/boost_mpi-1.83.0/boost_mpi-config-version.cmake
/usr/local/lib/cmake/boost_mpi-1.83.0/boost_mpi-config.cmake
/usr/local/lib/cmake/boost_type_erasure-1.83.0
/usr/local/lib/cmake/boost_type_erasure-1.83.0/libboost_type_erasure-variant-shared.cmake
/usr/local/lib/cmake/boost_type_erasure-1.83.0/libboost_type_erasure-variant-static.cmake
/usr/local/lib/cmake/boost_type_erasure-1.83.0/boost_type_erasure-config-version.cmake
/usr/local/lib/cmake/boost_type_erasure-1.83.0/boost_type_erasure-config.cmake
/usr/local/lib/cmake/boost_chrono-1.83.0
/usr/local/lib/cmake/boost_chrono-1.83.0/boost_chrono-config-version.cmake
/usr/local/lib/cmake/boost_chrono-1.83.0/libboost_chrono-variant-static.cmake
/usr/local/lib/cmake/boost_chrono-1.83.0/libboost_chrono-variant-shared.cmake
/usr/local/lib/cmake/boost_chrono-1.83.0/boost_chrono-config.cmake
/usr/local/lib/cmake/boost_math_tr1l-1.83.0
/usr/local/lib/cmake/boost_math_tr1l-1.83.0/libboost_math_tr1l-variant-shared.cmake
/usr/local/lib/cmake/boost_math_tr1l-1.83.0/boost_math_tr1l-config.cmake
/usr/local/lib/cmake/boost_math_tr1l-1.83.0/boost_math_tr1l-config-version.cmake
/usr/local/lib/cmake/boost_math_tr1l-1.83.0/libboost_math_tr1l-variant-static.cmake
/usr/local/lib/cmake/boost_contract-1.83.0
/usr/local/lib/cmake/boost_contract-1.83.0/libboost_contract-variant-shared.cmake
/usr/local/lib/cmake/boost_contract-1.83.0/boost_contract-config-version.cmake
/usr/local/lib/cmake/boost_contract-1.83.0/boost_contract-config.cmake
/usr/local/lib/cmake/boost_contract-1.83.0/libboost_contract-variant-static.cmake
/usr/local/lib/cmake/boost_fiber-1.83.0
/usr/local/lib/cmake/boost_fiber-1.83.0/libboost_fiber-variant-static.cmake
/usr/local/lib/cmake/boost_fiber-1.83.0/boost_fiber-config-version.cmake
/usr/local/lib/cmake/boost_fiber-1.83.0/libboost_fiber-variant-shared.cmake
/usr/local/lib/cmake/boost_fiber-1.83.0/boost_fiber-config.cmake
/usr/local/lib/cmake/BoostDetectToolset-1.83.0.cmake
/usr/local/lib/cmake/boost_context-1.83.0
/usr/local/lib/cmake/boost_context-1.83.0/boost_context-config.cmake
/usr/local/lib/cmake/boost_context-1.83.0/libboost_context-variant-shared.cmake
/usr/local/lib/cmake/boost_context-1.83.0/libboost_context-variant-static.cmake
/usr/local/lib/cmake/boost_context-1.83.0/boost_context-config-version.cmake
/usr/local/lib/cmake/boost_container-1.83.0
/usr/local/lib/cmake/boost_container-1.83.0/libboost_container-variant-shared.cmake
/usr/local/lib/cmake/boost_container-1.83.0/boost_container-config-version.cmake
/usr/local/lib/cmake/boost_container-1.83.0/boost_container-config.cmake
/usr/local/lib/cmake/boost_container-1.83.0/libboost_container-variant-static.cmake
/usr/local/lib/cmake/boost_regex-1.83.0
/usr/local/lib/cmake/boost_regex-1.83.0/libboost_regex-variant-shared.cmake
/usr/local/lib/cmake/boost_regex-1.83.0/boost_regex-config.cmake
/usr/local/lib/cmake/boost_regex-1.83.0/libboost_regex-variant-static.cmake
/usr/local/lib/cmake/boost_regex-1.83.0/boost_regex-config-version.cmake
/usr/local/lib/cmake/boost_url-1.83.0
/usr/local/lib/cmake/boost_url-1.83.0/boost_url-config-version.cmake
/usr/local/lib/cmake/boost_url-1.83.0/libboost_url-variant-static.cmake
/usr/local/lib/cmake/boost_url-1.83.0/libboost_url-variant-shared.cmake
/usr/local/lib/cmake/boost_url-1.83.0/boost_url-config.cmake
/usr/local/lib/cmake/boost_locale-1.83.0
/usr/local/lib/cmake/boost_locale-1.83.0/libboost_locale-variant-static.cmake
/usr/local/lib/cmake/boost_locale-1.83.0/boost_locale-config.cmake
/usr/local/lib/cmake/boost_locale-1.83.0/libboost_locale-variant-shared.cmake
/usr/local/lib/cmake/boost_locale-1.83.0/boost_locale-config-version.cmake
/usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0
/usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0/libboost_stacktrace_addr2line-variant-static.cmake
/usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0/libboost_stacktrace_addr2line-variant-shared.cmake
/usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0/boost_stacktrace_addr2line-config-version.cmake
/usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0/boost_stacktrace_addr2line-config.cmake
/usr/local/lib/libboost_fiber.so.1.83.0
/usr/local/lib/libboost_chrono.a
/usr/local/lib/libboost_serialization.so
/usr/local/lib/libboost_fiber.so
/usr/local/lib/libboost_wserialization.a
/usr/local/lib/libboost_unit_test_framework.so.1.83.0
/usr/local/lib/libboost_stacktrace_noop.a
/usr/local/lib/libboost_iostreams.a
/usr/local/lib/libboost_math_c99f.a
/usr/local/lib/libboost_exception.a
/usr/local/lib/libboost_type_erasure.a
/usr/local/lib/libboost_thread.a
/usr/local/lib/libboost_chrono.so.1.83.0
/usr/local/lib/libboost_math_c99l.so
/usr/local/lib/libboost_log_setup.so.1.83.0
/usr/local/lib/libboost_stacktrace_addr2line.a
/usr/local/lib/libboost_container.so.1.83.0
/usr/local/lib/libboost_context.so
/usr/local/lib/libboost_program_options.so
/usr/local/lib/libboost_stacktrace_basic.so
/usr/local/lib/libboost_graph.a
/usr/local/lib/libboost_prg_exec_monitor.so.1.83.0
/usr/local/lib/libboost_atomic.a
/usr/local/lib/libboost_math_tr1l.a
/usr/local/lib/libboost_log.so.1.83.0
/usr/local/lib/libboost_random.a
/usr/local/lib/libboost_math_tr1f.so
/usr/local/lib/libboost_contract.a
/usr/local/lib/libboost_atomic.so
/usr/local/lib/libboost_serialization.so.1.83.0
/usr/local/lib/libboost_thread.so
/usr/local/lib/libboost_wave.so
/usr/local/lib/libboost_program_options.a
/usr/local/lib/libboost_context.so.1.83.0
/usr/local/lib/libboost_coroutine.a
/usr/local/lib/libboost_json.a
/usr/local/lib/libboost_stacktrace_noop.so.1.83.0
/usr/local/lib/libboost_wserialization.so.1.83.0
/usr/local/lib/libboost_math_tr1.a
/usr/local/lib/libboost_chrono.so
/usr/local/lib/libboost_stacktrace_backtrace.a
/usr/local/lib/libboost_stacktrace_addr2line.so.1.83.0
/usr/local/lib/libboost_stacktrace_basic.so.1.83.0
/usr/local/lib/libboost_math_c99l.a
/usr/local/lib/libboost_log.so
/usr/local/lib/libboost_log.a
/usr/local/lib/libboost_math_tr1l.so
/usr/local/lib/libboost_container.so
/usr/local/lib/libboost_coroutine.so
/usr/local/lib/libboost_prg_exec_monitor.so
/usr/local/lib/libboost_nowide.a
/usr/local/lib/libboost_url.so
/usr/local/lib/libboost_url.a
/usr/local/lib/libboost_math_c99f.so
/usr/local/lib/libboost_stacktrace_backtrace.so.1.83.0
/usr/local/lib/libboost_test_exec_monitor.a
/usr/local/lib/libboost_json.so
/usr/local/lib/libboost_wserialization.so
/usr/local/lib/libboost_system.a
/usr/local/lib/libboost_regex.so.1.83.0
/usr/local/lib/libboost_contract.so
/usr/local/lib/libboost_stacktrace_noop.so
/usr/local/lib/libboost_locale.so
/usr/local/lib/libboost_math_tr1.so
/usr/local/lib/libboost_timer.so.1.83.0
/usr/local/lib/libboost_random.so
/usr/local/lib/libboost_math_tr1l.so.1.83.0
/usr/local/lib/libboost_timer.so

Clean Up

Many Ubuntu/Debian packages are left behind by the Boost installation program. You can view them as follows:

Shell

$ apt --dry-run autoremove | grep -Po '^Remv \K[^ ]+'
libboost-mpi-python-dev
libboost-mpi-python1.74-dev
libboost-mpi-python1.74.0
mpi-default-bin
libboost-mpi-dev
libboost-mpi1.74-dev
mpi-default-dev
libopenmpi-dev
openmpi-bin
libcoarrays-openmpi-dev
libopenmpi3
libucx0
libfabric1
ibverbs-providers
libboost-atomic-dev
libboost-coroutine-dev
libboost-coroutine1.74-dev
libboost-fiber-dev
libboost-fiber1.74-dev
libboost-context1.74-dev
libboost-type-erasure-dev
libboost-type-erasure1.74-dev
libboost-thread1.74-dev
libboost-log-dev
libboost-log1.74-dev
libboost-atomic1.74-dev
libboost-atomic1.74.0
libboost-chrono-dev
libboost-timer-dev
libboost-timer1.74-dev
libboost-chrono1.74-dev
libboost-timer1.74.0
libboost-chrono1.74.0
libboost-container-dev
libboost-container1.74-dev
libboost-container1.74.0
libboost-context-dev
libboost-fiber1.74.0
libboost-coroutine1.74.0
libboost-context1.74.0
libboost-date-time-dev
libboost-date-time1.74-dev
libboost-date-time1.74.0
libboost-dev
libboost-exception-dev
libboost-exception1.74-dev
libboost-filesystem-dev
libboost-wave-dev
libboost-wave1.74-dev
libboost-filesystem1.74-dev
libboost-graph-dev
libboost-graph-parallel-dev
libboost-graph-parallel1.74-dev
libboost-graph-parallel1.74.0
libboost-graph1.74-dev
libboost-graph1.74.0
libboost-iostreams-dev
libboost-iostreams1.74-dev
libboost-locale-dev
libboost-locale1.74-dev
libboost-log1.74.0
libboost-math-dev
libboost-math1.74-dev
libboost-math1.74.0
libboost-mpi1.74.0
libboost-nowide-dev
libboost-nowide1.74-dev
libboost-nowide1.74.0
libboost-numpy-dev
libboost-numpy1.74-dev
libboost-numpy1.74.0
libboost-program-options-dev
libboost-program-options1.74-dev
libboost-program-options1.74.0
libboost-python-dev
libboost-python1.74-dev
libboost-python1.74.0
libboost-random-dev
libboost-random1.74-dev
libboost-random1.74.0
libboost-regex-dev
libboost-regex1.74-dev
libboost-serialization-dev
libboost-serialization1.74-dev
libboost-serialization1.74.0
libboost-stacktrace-dev
libboost-stacktrace1.74-dev
libboost-stacktrace1.74.0
libboost-system-dev
libboost-system1.74-dev
libboost-system1.74.0
libboost-test-dev
libboost-test1.74-dev
libboost-test1.74.0
libboost-thread-dev
libboost-tools-dev
libboost-type-erasure1.74.0
libboost-wave1.74.0
libboost1.74-dev
libboost1.74-tools-dev
libcaf-openmpi-3
libpmix-dev
libevent-dev
libevent-extra-2.1-7
libevent-openssl-2.1-7
libpmix2
libevent-pthreads-2.1-7
libhwloc-dev
libhwloc-plugins
libhwloc15
libibverbs-dev
librdmacm1
libibverbs1
libjs-jquery-ui
libmunge2
libnl-route-3-dev
libnl-3-dev
libnuma-dev
libpsm-infinipath1
libpsm2-2
openmpi-common

Remove the packages as follows:

Shell

$ yes | sudo apt autoremove
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  ibverbs-providers libboost-atomic-dev libboost-atomic1.74-dev libboost-atomic1.74.0 libboost-chrono-dev libboost-chrono1.74-dev libboost-chrono1.74.0
  libboost-container-dev libboost-container1.74-dev libboost-container1.74.0 libboost-context-dev libboost-context1.74-dev libboost-context1.74.0
  libboost-coroutine-dev libboost-coroutine1.74-dev libboost-coroutine1.74.0 libboost-date-time-dev libboost-date-time1.74-dev libboost-date-time1.74.0
  libboost-dev libboost-exception-dev libboost-exception1.74-dev libboost-fiber-dev libboost-fiber1.74-dev libboost-fiber1.74.0 libboost-filesystem-dev
  libboost-filesystem1.74-dev libboost-graph-dev libboost-graph-parallel-dev libboost-graph-parallel1.74-dev libboost-graph-parallel1.74.0
  libboost-graph1.74-dev libboost-graph1.74.0 libboost-iostreams-dev libboost-iostreams1.74-dev libboost-locale-dev libboost-locale1.74-dev
  libboost-log-dev libboost-log1.74-dev libboost-log1.74.0 libboost-math-dev libboost-math1.74-dev libboost-math1.74.0 libboost-mpi-dev
  libboost-mpi-python-dev libboost-mpi-python1.74-dev libboost-mpi-python1.74.0 libboost-mpi1.74-dev libboost-mpi1.74.0 libboost-nowide-dev
  libboost-nowide1.74-dev libboost-nowide1.74.0 libboost-numpy-dev libboost-numpy1.74-dev libboost-numpy1.74.0 libboost-program-options-dev
  libboost-program-options1.74-dev libboost-program-options1.74.0 libboost-python-dev libboost-python1.74-dev libboost-python1.74.0 libboost-random-dev
  libboost-random1.74-dev libboost-random1.74.0 libboost-regex-dev libboost-regex1.74-dev libboost-serialization-dev libboost-serialization1.74-dev
  libboost-serialization1.74.0 libboost-stacktrace-dev libboost-stacktrace1.74-dev libboost-stacktrace1.74.0 libboost-system-dev libboost-system1.74-dev
  libboost-system1.74.0 libboost-test-dev libboost-test1.74-dev libboost-test1.74.0 libboost-thread-dev libboost-thread1.74-dev libboost-timer-dev
  libboost-timer1.74-dev libboost-timer1.74.0 libboost-tools-dev libboost-type-erasure-dev libboost-type-erasure1.74-dev libboost-type-erasure1.74.0
  libboost-wave-dev libboost-wave1.74-dev libboost-wave1.74.0 libboost1.74-dev libboost1.74-tools-dev libcaf-openmpi-3 libcoarrays-openmpi-dev libevent-dev
  libevent-extra-2.1-7 libevent-openssl-2.1-7 libevent-pthreads-2.1-7 libfabric1 libhwloc-dev libhwloc-plugins libhwloc15 libibverbs-dev libibverbs1
  libjs-jquery-ui libmunge2 libnl-3-dev libnl-route-3-dev libnuma-dev libopenmpi-dev libopenmpi3 libpmix-dev libpmix2 libpsm-infinipath1 libpsm2-2
  librdmacm1 libucx0 mpi-default-bin mpi-default-dev openmpi-bin openmpi-common
0 upgraded, 0 newly installed, 121 to remove and 4 not upgraded.
After this operation, 413 MB disk space will be freed.
(Reading database ... 399799 files and directories currently installed.)
Removing libboost-mpi-python-dev (1.74.0.3ubuntu7) ...
Removing libboost-mpi-python1.74-dev (1.74.0-18.1ubuntu3) ...
Removing libboost-mpi-python1.74.0 (1.74.0-18.1ubuntu3) ...
Removing mpi-default-bin (1.14) ...
Removing libboost-mpi-dev (1.74.0.3ubuntu7) ...
Removing libboost-mpi1.74-dev (1.74.0-18.1ubuntu3) ...
Removing mpi-default-dev (1.14) ...
Removing libopenmpi-dev:amd64 (4.1.4-3ubuntu2) ...
Removing libcoarrays-openmpi-dev:amd64 (2.10.1-1) ...
update-alternatives: warning: alternative /usr/bin/caf.openmpi (part of link group caf) doesn't exist; removing from list of alternatives
update-alternatives: warning: /etc/alternatives/caf is dangling; it will be updated with best choice
Removing libboost-atomic-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-coroutine-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-coroutine1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-fiber-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-fiber1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-type-erasure-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-type-erasure1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-log-dev (1.74.0.3ubuntu7) ...
Removing libboost-log1.74-dev (1.74.0-18.1ubuntu3) ...
Removing libboost-chrono-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-timer-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-timer1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-timer1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-container-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-container1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-container1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-context-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-fiber1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-coroutine1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-date-time-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-exception-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-exception1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-filesystem-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-wave-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-wave1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-filesystem1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
Removing libboost-graph-dev:amd64 (1.74.0.3ubuntu7) ...
Removing libboost-graph-parallel-dev (1.74.0.3ubuntu7) ...
  Removing libboost-graph-parallel1.74-dev (1.74.0-18.1ubuntu3) ...
  Removing libboost-graph-parallel1.74.0 (1.74.0-18.1ubuntu3) ...
  Removing libboost-graph1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-graph1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-iostreams-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-iostreams1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-locale-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-locale1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-log1.74.0 (1.74.0-18.1ubuntu3) ...
  Removing libboost-math-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-math1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-math1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-mpi1.74.0 (1.74.0-18.1ubuntu3) ...
  Removing libboost-nowide-dev (1.74.0.3ubuntu7) ...
  Removing libboost-nowide1.74-dev (1.74.0-18.1ubuntu3) ...
  Removing libboost-nowide1.74.0 (1.74.0-18.1ubuntu3) ...
  Removing libboost-numpy-dev (1.74.0.3ubuntu7) ...
  Removing libboost-numpy1.74-dev (1.74.0-18.1ubuntu3) ...
  Removing libboost-numpy1.74.0 (1.74.0-18.1ubuntu3) ...
  Removing libboost-program-options-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-program-options1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-program-options1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-python-dev (1.74.0.3ubuntu7) ...
  Removing libboost-python1.74-dev (1.74.0-18.1ubuntu3) ...
  Removing libboost-python1.74.0 (1.74.0-18.1ubuntu3) ...
  Removing libboost-random-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-random1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-random1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-regex-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-regex1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-serialization-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-stacktrace-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-stacktrace1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-stacktrace1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-system-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-test-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-test1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-test1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-thread-dev:amd64 (1.74.0.3ubuntu7) ...
  Removing libboost-tools-dev (1.74.0.3ubuntu7) ...
  Removing libboost-type-erasure1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-wave1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost1.74-tools-dev (1.74.0-18.1ubuntu3) ...
  Removing libcaf-openmpi-3:amd64 (2.10.1-1) ...
  Removing libpmix-dev:amd64 (4.2.2-1) ...
  Removing libevent-dev (2.1.12-stable-8ubuntu3) ...
  Removing libevent-extra-2.1-7:amd64 (2.1.12-stable-8ubuntu3) ...
  Removing libevent-openssl-2.1-7:amd64 (2.1.12-stable-8ubuntu3) ...
  Removing libhwloc-dev:amd64 (2.9.0-1) ...
  Removing libibverbs-dev:amd64 (44.0-2) ...
  Removing libjs-jquery-ui (1.13.2+dfsg-1) ...
  Removing libnl-route-3-dev:amd64 (3.7.0-0.2) ...
  Removing libnl-3-dev:amd64 (3.7.0-0.2) ...
  Removing libnuma-dev:amd64 (2.0.16-1) ...
  Removing openmpi-bin (4.1.4-3ubuntu2) ...
  Removing libopenmpi3:amd64 (4.1.4-3ubuntu2) ...
  Removing libucx0:amd64 (1.13.1-1) ...
  Removing libfabric1:amd64 (1.17.0-3) ...
  Removing ibverbs-providers:amd64 (44.0-2) ...
  Removing libboost-context1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-thread1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-atomic1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-atomic1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-chrono1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-chrono1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-context1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-date-time1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-date-time1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-serialization1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-serialization1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-system1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost-system1.74.0:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libboost1.74-dev:amd64 (1.74.0-18.1ubuntu3) ...
  Removing libpmix2:amd64 (4.2.2-1) ...
  Removing libevent-pthreads-2.1-7:amd64 (2.1.12-stable-8ubuntu3) ...
  Removing libhwloc-plugins:amd64 (2.9.0-1) ...
  Removing libhwloc15:amd64 (2.9.0-1) ...
  Removing librdmacm1:amd64 (44.0-2) ...
  Removing libibverbs1:amd64 (44.0-2) ...
  Removing libmunge2 (0.5.15-2) ...
  Removing libpsm-infinipath1 (3.3+20.604758e7-6.2) ...
  update-alternatives: warning: alternative /usr/lib/libpsm1/libpsm_infinipath.so.1.16 (part of link group libpsm_infinipath.so.1) doesn't exist; removing from list of alternatives
  update-alternatives: warning: /etc/alternatives/libpsm_infinipath.so.1 is dangling; it will be updated with best choice
  Removing libpsm2-2 (11.2.185-2) ...
  Removing openmpi-common (4.1.4-3ubuntu2) ...
  Processing triggers for man-db (2.11.2-1) ...
  Processing triggers for libc-bin (2.37-0ubuntu2) ...
  /sbin/ldconfig.real: /usr/lib/wsl/lib/libcuda.so.1 is not a symbolic link

Go Forth and Get Boosted

😁

Boost usage as a C++ general-pupose library is widespread throughout a diverse range of industries and requirements.

PDF Manipulation

2023-09-08T00:00:00-04:00

Installation

The pdftk website discusses pdktk (a F/OSS command-line program), a Windows GUI, and mentions the book PDF Hacks.

An alternative free GUI is PDFTK Builder.

Other command-line utilties for manipulating PDF documents are:

Poppler (GPL)
QPDF

Install the command-line pdftk program on Ubuntu:

Shell

$ yes | sudo apt install pdftk

This is the pdftk help text:

Shell

$ man pdftk
pdftk.pdftk-java(1)     General Commands Manual    pdftk.pdftk-java(1)

NAME
        pdftk - A handy tool for manipulating PDF

SYNOPSIS
        pdftk <input PDF files | - | PROMPT>
            [ input_pw <input PDF owner passwords | PROMPT> ]
            [ <operation> <operation arguments> ]
            [ output <output filename | - | PROMPT> ]
            [ encrypt_40bit | encrypt_128bit | encrypt_aes128 ]
            [ allow <permissions> ]
            [ owner_pw <owner password | PROMPT> ]
            [ user_pw <user password | PROMPT> ]
            [ flatten ] [ need_appearances ]
            [ compress | uncompress ]
            [ keep_first_id | keep_final_id ] [ drop_xfa ] [ drop_xmp
        ]
            [ replacement_font <font name> ]
            [ verbose ] [ dont_ask | do_ask ]
        Where:
            <operation> may be empty, or:
            [ cat | shuffle | burst | rotate |
              generate_fdf | fill_form |
              background | multibackground |
              stamp | multistamp |
              dump_data | dump_data_utf8 |
              dump_data_fields | dump_data_fields_utf8 |
              dump_data_annots |
              update_info | update_info_utf8 |
              attach_files | unpack_files ]

        For Complete Help: pdftk --help

DESCRIPTION
        If PDF is electronic paper, then pdftk is an electronic staple-
        remover, hole-punch, binder, secret-decoder-ring, and X-Ray-
        glasses.  Pdftk is a simple tool for doing everyday things with
        PDF documents.  Use it to:

        * Merge PDF Documents or Collate PDF Page Scans
        * Split PDF Pages into a New Document
        * Rotate PDF Documents or Pages
        * Decrypt Input as Necessary (Password Required)
        * Encrypt Output as Desired
        * Fill PDF Forms with X/FDF Data and/or Flatten Forms
        * Generate FDF Data Stencils from PDF Forms
        * Apply a Background Watermark or a Foreground Stamp
        * Report PDF Metrics, Bookmarks and Metadata
        * Add/Update PDF Metrics, Bookmarks or Metadata
        * Attach Files to PDF Pages or the PDF Document
        * Unpack PDF Attachments
        * Burst a PDF Document into Single Pages
        * Uncompress and Re-Compress Page Streams
        * Repair Corrupted PDF (Where Possible)

OPTIONS
        A summary of options is included below.

        --help, -h
              Show this summary of options.

        <input PDF files | - | PROMPT>
              A list of the input PDF files. If you plan to combine
              these PDFs (without using handles) then list files in
              the order you want them combined.  Use - to pass a sin‐
              gle PDF into pdftk via stdin.  Input files can be asso‐
              ciated with handles, where a handle is one or more up‐
              per-case letters:

              <input PDF handle>=<input PDF filename>

              Handles are often omitted.  They are useful when speci‐
              fying PDF passwords or page ranges, later.

              For example: A=input1.pdf QT=input2.pdf M=input3.pdf

        [input_pw <input PDF owner passwords | PROMPT>]
              Input PDF owner passwords, if necessary, are associated
              with files by using their handles:

              <input PDF handle>=<input PDF file owner password>

              If handles are not given, then passwords are associated
              with input files by order.

              Most pdftk features require that encrypted input PDF are
              accompanied by the ~owner~ password. If the input PDF
              has no owner password, then the user password must be
              given, instead.  If the input PDF has no passwords, then
              no password should be given.

              When running in do_ask mode, pdftk will prompt you for a
              password if the supplied password is incorrect or none
              was given.

        [<operation> <operation arguments>]
              Available operations are: cat, shuffle, burst, rotate,
              generate_fdf, fill_form, background, multibackground,
              stamp, multistamp, dump_data, dump_data_utf8,
              dump_data_fields, dump_data_fields_utf8, dump_data_an‐
              nots, update_info, update_info_utf8, attach_files, un‐
              pack_files. Some operations takes additional arguments,
              described below.

              If this optional argument is omitted, then pdftk runs in
              'filter' mode.  Filter mode takes only one PDF input and
              creates a new PDF after applying all of the output op‐
              tions, like encryption and compression.

          cat [<page ranges>]
                  Assembles (catenates) pages from input PDFs to create
                  a new PDF. Use cat to merge PDF pages or to split PDF
                  pages from documents. You can also use it to rotate
                  PDF pages. Page order in the new PDF is specified by
                  the order of the given page ranges. Page ranges are
                  described like this:

                  <input PDF handle>[<begin page number>[-<end page
                  number>[<qualifier>]]][<page rotation>]

                  Where the handle identifies one of the input PDF
                  files, and the beginning and ending page numbers are
                  one-based references to pages in the PDF file.  The
                  qualifier can be even, odd, or ~, and the page rota‐
                  tion can be north, south, east, west, left, right, or
                  down.

                  If a PDF handle is given but no pages are specified,
                  then the entire PDF is used. If no pages are speci‐
                  fied for any of the input PDFs, then the input PDFs'
                  bookmarks are also merged and included in the output.

                  If the handle is omitted from the page range, then
                  the pages are taken from the first input PDF.

                  The even qualifier causes pdftk to use only the even-
                  numbered PDF pages, so 1-6even yields pages 2, 4 and
                  6 in that order.  6-1even yields pages 6, 4 and 2 in
                  that order.

                  The odd qualifier works similarly to the even.

                  Pages can be subtracted from a page range using the ~
                  qualifier followed by a page range. For instance,
                  1-20~5-6 and 1-20~5~6 are equivalent to 1-4 7-20, and
                  ~5 yields all pages except page 5. Depending on your
                  shell, you may need to quote this argument because of
                  the ~ at the beginning.

                  The page rotation setting can cause pdftk to rotate
                  pages and documents.  Each option sets the page rota‐
                  tion as follows (in degrees): north: 0, east: 90,
                  south: 180, west: 270, left: -90, right: +90, down:
                  +180. left, right, and down make relative adjustments
                  to a page's rotation.

                  If no arguments are passed to cat, then pdftk com‐
                  bines all input PDFs in the order they were given to
                  create the output.

                  NOTES:
                  * <end page number> may be less than <begin page num‐
                  ber>.
                  * The keyword end may be used to reference the final
                  page of a document instead of a page number.
                  * Reference a single page by omitting the ending page
                  number.
                  * The handle may be used alone to represent the en‐
                  tire PDF document, e.g., B1-end is the same as B.
                  * You can reference page numbers in reverse order by
                  prefixing them with the letter r. For example, page
                  r1 is the last page of the document, r2 is the next-
                  to-last page of the document, and rend is the first
                  page of the document. You can use this prefix in
                  ranges, too, for example r3-r1 is the last three
                  pages of a PDF.

                  Page Range Examples without Handles:
                  1\-endeast – rotate entire document 90 degrees
                  5 11 20 – take single pages from input PDF
                  5-25oddwest – take odd pages in range, rotate 90 de‐
                  grees
                  6-1 – reverse pages in range from input PDF

                  Page Range Examples Using Handles:
                  Say A=in1.pdf B=in2.pdf, then:
                  A1-21 – take range from in1.pdf
                  Bend-1odd – take all odd pages from in2.pdf in re‐
                  verse order
                  A72 – take a single page from in1.pdf
                  A1-21 Beven A72 – assemble pages from both in1.pdf
                  and in2.pdf
                  Awest – rotate entire in1.pdf document 90 degrees
                  B – use all of in2.pdf
                  A2-30evenleft – take the even pages from the range,
                  remove 90 degrees from each page's rotation
                  A A – catenate in1.pdf with in1.pdf
                  Aevenwest Aoddeast – apply rotations to even pages,
                  odd pages from in1.pdf
                  Awest Bwest Bdown – catenate rotated documents

          shuffle [<page ranges>]
                  Collates pages from input PDFs to create a new PDF.
                  Works like the cat operation except that it takes one
                  page at a time from each page range to assemble the
                  output PDF.  If one range runs out of pages, it con‐
                  tinues with the remaining ranges.  Ranges can use all
                  of the features described above for cat, like reverse
                  page ranges, multiple ranges from a single PDF, and
                  page rotation.  This feature was designed to help
                  collate PDF pages after scanning paper documents.

          burst  Splits a single input PDF document into individual
                  pages. Also creates a report named doc_data.txt which
                  is the same as the output from dump_data.  The output
                  section can contain a printf-styled format string to
                  name these pages.  For example, if you want pages
                  named page_01.pdf, page_02.pdf, etc., pass output
                  page_%02d.pdf to pdftk. If the pattern is omitted,
                  then a default pattern g_%04d.pdf is appended and
                  produces pages named pg_0001.pdf, pg_0002.pdf, etc.
                  Encryption can be applied to the output by appending
                  output options such as owner_pw, e.g.:

                  pdftk in.pdf burst owner_pw foopass

          rotate [<page ranges>]
                  Takes a single input PDF and rotates just the speci‐
                  fied pages.  All other pages remain unchanged.  The
                  page order remains unchanged.  Specify the pages to
                  rotate using the same notation as you would with cat,
                  except you omit the pages that you aren't rotating:

                  [<begin page number>[-<end page number>[<quali‐
                  fier>]]][<page rotation>]

                  The qualifier can be even or odd, and the page rota‐
                  tion can be north, south, east, west, left, right, or
                  down.

                  Each option sets the page rotation as follows (in de‐
                  grees): north: 0, east: 90, south: 180, west: 270,
                  left: -90, right: +90, down: +180. left, right, and
                  down make relative adjustments to a page's rotation.

                  The given order of the pages doesn't change the page
                  order in the output.

          generate_fdf
                  Reads a single input PDF file and generates an FDF
                  file suitable for fill_form out of it to the given
                  output filename or (if no output is given) to stdout.
                  Does not create a new PDF.

          fill_form <FDF data filename | XFDF data filename | - |
          PROMPT>
                  Fills the single input PDF's form fields with the
                  data from an FDF file, XFDF file or stdin. Enter the
                  data filename after fill_form, or use - to pass the
                  data via stdin, like so:

                  pdftk form.pdf fill_form data.fdf output
                  form.filled.pdf

                  If the input FDF file includes Rich Text formatted
                  data in addition to plain text, then the Rich Text
                  data is packed into the form fields as well as the
                  plain text.  Pdftk also sets a flag that cues
                  Reader/Acrobat to generate new field appearances
                  based on the Rich Text data.  So when the user opens
                  the PDF, the viewer will create the Rich Text appear‐
                  ance on the spot.  If the user's PDF viewer does not
                  support Rich Text, then the user will see the plain
                  text data instead.  If you flatten this form before
                  Acrobat has a chance to create (and save) new field
                  appearances, then the plain text field data is what
                  you'll see.

                  Also see the flatten, need_appearances, and replace‐
                  ment_font options.

          background <background PDF filename | - | PROMPT>
                  Applies a PDF watermark to the background of a single
                  input PDF.  Pass the background PDF's filename after
                  background like so:

                  pdftk in.pdf background back.pdf output out.pdf

                  Pdftk uses only the first page from the background
                  PDF and applies it to every page of the input PDF.
                  This page is scaled and rotated as needed to fit the
                  input page.  You can use - to pass a background PDF
                  into pdftk via stdin.

                  If the input PDF does not have a transparent back‐
                  ground (such as a PDF created from page scans) then
                  the resulting background won't be visible – use the
                  stamp operation instead.

          multibackground <background PDF filename | - | PROMPT>
                  Same as the background operation, but applies each
                  page of the background PDF to the corresponding page
                  of the input PDF.  If the input PDF has more pages
                  than the stamp PDF, then the final stamp page is re‐
                  peated across these remaining pages in the input PDF.

          stamp <stamp PDF filename | - | PROMPT>
                  This behaves just like the background operation ex‐
                  cept it overlays the stamp PDF page on top of the in‐
                  put PDF document's pages.  This works best if the
                  stamp PDF page has a transparent background.

          multistamp <stamp PDF filename | - | PROMPT>
                  Same as the stamp operation, but applies each page of
                  the background PDF to the corresponding page of the
                  input PDF.  If the input PDF has more pages than the
                  stamp PDF, then the final stamp page is repeated
                  across these remaining pages in the input PDF.

          dump_data
                  Reads a single input PDF file and reports its meta‐
                  data, bookmarks (a/k/a outlines), page metrics (me‐
                  dia, rotation and labels), data embedded by STAMPtk
                  (see STAMPtk's embed option) and other data to the
                  given output filename or (if no output is given) to
                  stdout.  Non-ASCII characters are encoded as XML nu‐
                  merical entities.  Does not create a new PDF.

          dump_data_utf8
                  Same as dump_data except that the output is encoded
                  as UTF-8.

          dump_data_fields
                  Reads a single input PDF file and reports form field
                  statistics to the given output filename or (if no
                  output is given) to stdout. Non-ASCII characters are
                  encoded as XML numerical entities. Does not create a
                  new PDF.

          dump_data_fields_utf8
                  Same as dump_data_fields except that the output is
                  encoded as UTF-8.

          dump_data_annots
                  This operation currently reports only link annota‐
                  tions.  Reads a single input PDF file and reports an‐
                  notation information to the given output filename or
                  (if no output is given) to stdout. Non-ASCII charac‐
                  ters are encoded as XML numerical entities. Does not
                  create a new PDF.

          update_info <info data filename | - | PROMPT>
                  Changes the bookmarks, page labels, page sizes, page
                  rotations, and metadata in a single PDF's Info dic‐
                  tionary to match the input data file. The input data
                  file uses the same syntax as the output from
                  dump_data. Non-ASCII characters should be encoded as
                  XML numerical entities.

                  This operation does not change the metadata stored in
                  the PDF's XMP stream, if it has one. (For this reason
                  you should include a ModDate entry in your updated
                  info with a current date/timestamp, format: D:YYYYM‐
                  MDDHHmmSS, e.g. D:201307241346 – omitted data after
                  YYYY revert to default values.)

                  For example:

                  pdftk in.pdf update_info in.info output out.pdf

          update_info_utf8 <info data filename | - | PROMPT>
                  Same as update_info except that the input is encoded
                  as UTF-8.

          attach_files <attachment filenames | PROMPT> [to_page <page
          number | PROMPT> | relation <relationship>]
                  Packs arbitrary files into a PDF using PDF's file at‐
                  tachment features. More than one attachment may be
                  listed after attach_files. Attachments are added at
                  the document level unless the optional to_page option
                  is given, in which case the files are attached to the
                  given page number (the first page is 1, the final
                  page is end). Attachments at the document level may
                  be tagged with a relationship among Source, Data, Al‐
                  ternative, Supplement, and Unspecified (default).

                  For example:

                  pdftk in.pdf attach_files table1.html table2.html
                  to_page 6 output out.pdf

                  pdftk in.pdf attach_files in.tex relation Source out‐
                  put out.pdf

          unpack_files
                  Copies all of the attachments from the input PDF into
                  the current folder or to an output directory given
                  after output. For example:

                  pdftk report.pdf unpack_files output ~/atts/

                  or, interactively:

                  pdftk report.pdf unpack_files output PROMPT

        [output <output filename | - | PROMPT>]
              The output PDF filename may not be set to the name of an
              input filename. Use - to output to stdout.  When using
              the dump_data operation, use output to set the name of
              the output data file. When using the unpack_files opera‐
              tion, use output to set the name of an output directory.
              When using the burst operation, you can use output to
              control the resulting PDF page filenames (described
              above).

        [encrypt_40bit | encrypt_128bit | encrypt_aes128]
              If an output PDF user or owner password is given, the
              output PDF encryption algorithm defaults to AES-128. The
              weaker RC4 40-bit and RC4 128-bit algorithms can be cho‐
              sen by specifying encrypt_40bit or encrypt_128bit (dis‐
              couraged).

        [allow <permissions>]
              Permissions are applied to the output PDF only if an en‐
              cryption strength is specified or an owner or user pass‐
              word is given.  If permissions are not specified, they
              default to 'none,' which means all of the following fea‐
              tures are disabled.

              The permissions section may include one or more of the
              following features:

              Printing
                      Top Quality Printing

              DegradedPrinting
                      Lower Quality Printing

              ModifyContents
                      Also allows Assembly

              Assembly

              CopyContents
                      Also allows ScreenReaders

              ScreenReaders

              ModifyAnnotations
                      Also allows FillIn

              FillIn

              AllFeatures
                      Allows the user to perform all of the above, and
                      top quality printing.

        [owner_pw <owner password | PROMPT>]

        [user_pw <user password | PROMPT>]
              If an encryption strength is given but no passwords are
              supplied, then the owner and user passwords remain
              empty, which means that the resulting PDF may be opened
              and its security parameters altered by anybody.

        [compress | uncompress]
              These are only useful when you want to edit PDF code in
              a text editor like vim or emacs.  Remove PDF page stream
              compression by applying the uncompress filter. Use the
              compress filter to restore compression.

        [flatten]
              Use this option to merge an input PDF's interactive form
              fields (and their data) with the PDF's pages. Only one
              input PDF may be given. Sometimes used with the
              fill_form operation.

        [need_appearances]
              Sets a flag that cues Reader/Acrobat to generate new
              field appearances based on the form field values.  Use
              this when filling a form with non-ASCII text to ensure
              the best presentation in Adobe Reader or Acrobat.  It
              won't work when combined with the flatten option.

        [replacement_font <font name>]
              Use the specified font to display text in form fields.
              This option is useful when filling a form with non-ASCII
              text that is not supported by the fonts included in the
              input PDF. font name may be either the file name or the
              family name of a font, but using a file name is more re‐
              liable. Currently only TrueType fonts with Unicode text
              are supported.

        [keep_first_id | keep_final_id]
              When combining pages from multiple PDFs, use one of
              these options to copy the document ID from either the
              first or final input document into the new output PDF.
              Otherwise pdftk creates a new document ID for the output
              PDF. When no operation is given, pdftk always uses the
              ID from the (single) input PDF.

        [drop_xfa]
              If your input PDF is a form created using Acrobat 7 or
              Adobe Designer, then it probably has XFA data.  Filling
              such a form using pdftk yields a PDF with data that
              fails to display in Acrobat 7 (and 6?).  The workaround
              solution is to remove the form's XFA data, either before
              you fill the form using pdftk or at the time you fill
              the form. Using this option causes pdftk to omit the XFA
              data from the output PDF form.

              This option is only useful when running pdftk on a sin‐
              gle input PDF.  When assembling a PDF from multiple in‐
              puts using pdftk, any XFA data in the input is automati‐
              cally omitted.

        [drop_xmp]
              Many PDFs store document metadata using both an Info
              dictionary (old school) and an XMP stream (new school).
              Pdftk's update_info operation can update the Info dic‐
              tionary, but not the XMP stream.  The proper remedy for
              this is to include a ModDate entry in your updated info
              with a current date/timestamp. The date/timestamp format
              is: D:YYYYMMDDHHmmSS, e.g. D:201307241346 – omitted data
              after YYYY revert to default values. This newer ModDate
              should cue PDF viewers that the Info metadata is more
              current than the XMP data.

              Alternatively, you might prefer to remove the XMP stream
              from the PDF altogether – that's what this option does.
              Note that objects inside the PDF might have their own,
              separate XMP metadata streams, and that drop_xmp does
              not remove those.  It only removes the PDF's document-
              level XMP stream.

        [verbose]
              By default, pdftk runs quietly. Append verbose to the
              end and it will speak up.

        [dont_ask | do_ask]
              Depending on the compile-time settings (see
              ASK_ABOUT_WARNINGS), pdftk might prompt you for further
              input when it encounters a problem, such as a bad pass‐
              word. Override this default behavior by adding dont_ask
              (so pdftk won't ask you what to do) or do_ask (so pdftk
              will ask you what to do).

              When running in dont_ask mode, pdftk will over-write
              files with its output without notice.

EXAMPLES
        Collate scanned pages
          pdftk A=even.pdf B=odd.pdf shuffle A B output collated.pdf
          or if odd.pdf is in reverse order:
          pdftk A=even.pdf B=odd.pdf shuffle A Bend-1 output col‐
          lated.pdf

        The following examples use actual passwords as command line pa‐
        rameters, which is discouraged (see the SECURITY CONSIDERATIONS
        section).

        Decrypt a PDF
          pdftk secured.pdf input_pw foopass output unsecured.pdf

        Encrypt a PDF using AES-128 (the default), withhold all permis‐
        sions (the default)
          pdftk 1.pdf output 1.128.pdf owner_pw foopass

        Same as above, except password 'baz' must also be used to open
        output PDF
          pdftk 1.pdf output 1.128.pdf owner_pw foo user_pw baz

        Same as above, except printing is allowed (once the PDF is
        open)
          pdftk 1.pdf output 1.128.pdf owner_pw foo user_pw baz allow
          printing

        Apply RCA 40-bit encryption to output, revoking all permissions
        (the default). Set the owner PW to 'foopass'.
          pdftk 1.pdf 2.pdf cat output 3.pdf encrypt_40bit owner_pw
          foopass

        Join two files, one of which requires the password 'foopass'.
        The output is not encrypted.
          pdftk A=secured.pdf 2.pdf input_pw A=foopass cat output 3.pdf

        Join in1.pdf and in2.pdf into a new PDF, out1.pdf
          pdftk in1.pdf in2.pdf cat output out1.pdf
          or (using handles):
          pdftk A=in1.pdf B=in2.pdf cat A B output out1.pdf
          or (using wildcards):
          pdftk *.pdf cat output combined.pdf

        Remove page 13 from in1.pdf to create out1.pdf
          pdftk in.pdf cat 1-12 14-end output out1.pdf
          or:
          pdftk A=in1.pdf cat A1-12 A14-end output out1.pdf

        Uncompress PDF page streams for editing the PDF in a text edi‐
        tor (e.g., vim, emacs)
          pdftk doc.pdf output doc.unc.pdf uncompress

        Repair a PDF's corrupted XREF table and stream lengths, if pos‐
        sible
          pdftk broken.pdf output fixed.pdf

        Burst a single PDF document into pages and dump its data to
        doc_data.txt
          pdftk in.pdf burst

        Burst a single PDF document into encrypted pages. Allow low-
        quality printing
          pdftk in.pdf burst owner_pw foopass allow DegradedPrinting

        Write a report on PDF document metadata and bookmarks to re‐
        port.txt
          pdftk in.pdf dump_data output report.txt

        Rotate the first PDF page to 90 degrees clockwise
          pdftk in.pdf cat 1east 2-end output out.pdf

        Rotate an entire PDF document to 180 degrees
          pdftk in.pdf cat 1-endsouth output out.pdf

NOTES
        This is a port of pdftk to java. See
        https://gitlab.com/pdftk-java/pdftk
        The original program can be found at www.pdftk.com

AUTHOR
        Original author of pdftk is Sid Steward (sid.steward at pdflabs
        dot com).

SECURITY CONSIDERATIONS
        Passing a password as a command line parameter is insecure be‐
        cause it can get saved into the shell's history and be accessi‐
        ble by other users via /proc. Use the keyword PROMPT and input
        any passwords via standard input instead.

                            December 7, 2020        pdftk.pdftk-java(1)

OCR

https://tools.pdf24.org/en/ocr-pdf provides fast, free OCR, without ads.

This website works well, and offers a total of 30 PDF operations. The other operations of https://tools.pdf24.org/ include merging, splitting, compressing, editing, signing, redacting, watermarking, locking, unlocking, etc.

PdfTk Operations

Rotate page 2 only

Given my_file.pdf, a 2-page document:

Do not transform page 1 (keep it as-is)
Rotate page 2 only
Save as my_file_fixed.pdf

Shell

$ pdftk my_file.pdf cat 1 2south output my_file_fixed.pdf

Merge PDFs

Concatenate file_a.pdf, file_b.pdf and file_c.pdf, then save as big_file.pdf:

Shell

$ pdftk file_a.pdf file_b.pdf file_c.pdf cat output big_file.pdf

Assemble Book Excerpt

If you need to create multiple excerpts of a book, one way to proceed is to:

Create a PDF containing 3 pages: the front cover, the page containing the edition notice, and the back cover. Let's call this PDF file book_wrapper.pdf.
Scan each of the excerpts into separate PDFs. Let's call these PDF files book_excerpt_raw_1.pdf, book_excerpt_raw_2.pdf, etc.
Wrap the each excerpt within the front cover and edition notice, and the back cover. To do that, concatenate the first 2 pages of book_wrapper.pdf, book_excerpt_raw_N.pdf and the last page of book_wrapper.pdf, then save as book_excerpt_N.pdf. Following is an example of how to do that for 3 excerpts:

Shell

$ pdftk book_wrapper.pdf 1-2 \
  book_excerpt_raw_1.pdf \
  book_wrapper.pdf 3 \
  cat output book_excerpt_1.pdf

$ pdftk book_wrapper.pdf 1-2 \
  book_excerpt_raw_2.pdf \
  book_wrapper.pdf 3 \
  cat output book_excerpt_2.pdf

$ pdftk book_wrapper.pdf 1-2 \
  book_excerpt_raw_3.pdf \
  book_wrapper.pdf 3 \
  cat output book_excerpt_3.pdf

Cut a large PDF into 2 Parts

Cut bigfile.pdf into two parts:

bigfile_part1.pdf (containing pages 1-150)
bigfile_part2.pdf (containing pages 151-350)

Shell

$ pdftk A=bigfile.pdf cat 1-150   output bigfile_part1.pdf

$ pdftk A=bigfile.pdf cat 151-end output bigfile_part2.pdf

Interleaving Double-Sided Originals

If you scan a double-sided document, you might end up with the odd-numbered pages in one PDF document, and the even-numbered pages in another.

A problem that you will encouter is that when you turn over the document to scan the even-numbered pages, they will be in reverse order because the pages will be scanned from the back to the front of the document. The following examples show two ways of combining the odd- and even-numbered pages in the proper order.

Manually Reversing Even-Numbered Pages

This example shows how to interleave the pages in both documents to create complete.pdf.

Scan the paper document from front to back, so only the odd pages are saved as odd.pdf.
Manually reverse the order of the pages in the paper document.
Turn the paper document over.
Scan the even pages into even.pdf.
Run the following incantation:

Shell

$ pdftk A=odd.pdf B=even.pdf shuffle A B output complete.pdf

$ rm even.pdf odd.pdf

Automatically Reversing Even-Numbered Pages

This example does not require you to manually re-order the pages, because pdftk can perform that task.

Scan the document from front to back, so only the odd pages are saved as odd.pdf.
Turn over the paper document.
Scan the even pages from back to front into even_reversed.pdf.
Run the following incantation:

Shell

$ pdftk even_reversed.pdf cat end-1 output even.pdf

$ pdftk A=odd.pdf B=even.pdf shuffle A B output complete.pdf

$ rm even.pdf even_reversed.pdf odd.pdf

Delete a Page

Delete page 69 from bigfile.pdf and save as smaller.pdf:

Shell

$ pdftk A=bigfile.pdf cat ~69 output smaller.pdf

Delete Several Pages

Delete pages 69-96 from bigfile.pdf and save as smaller.pdf:

Shell

$ pdftk A=bigfile.pdf cat ~69-96 output smaller.pdf

Remove Password

Assuming protected.pdf is encrypted with password myPwd, make an unencrypted copy and save as output.pdf:

Shell

$ pdftk protected.pdf input_pw myPwd output output.pdf

You might need to enclose the password within single quotes.

Edit Contents

Use LibreOffice Writer to edit the PDF, then export as PDF.

Shell

$ libreoffice --writer my_file.pdf

Pytest and Visual Studio Code

2023-08-20T00:00:00-04:00

Python has several well-known testing frameworks. Popular choices for functional and unit testing include pytest, the Robot framework, and unittest. Lettuce and Behave are popular for behavior-driven testing.

I decided to use pytest for a recent Python project.

I found that pytest’s test discovery mechanism was finicky, and demanded that Python packages and modules be set up before tests could be run. This is in sharp contrast to writing unit tests for Ruby, for example, using rspec, because Ruby modules are so much more flexible, forgiving and malleable.

However, getting pytest to work in Visual Studio Code was nearly impossible. Microsoft's documentation leaves a lot unsaid. I share what I learned, and how I made things work in this article.

Pytest

You can run pytest from the command line 3 ways.

Pytest Command

The most common way to run pytest is to type the pytest command, like the following, which runs all tests:

Shell

$ pytest

Pytest Module

The second way to run pytest is to invoke the pytest Python module, as follows:

Shell

$ python -m pytest

Running pytest as a Python module is nearly the same as running the pytest command, except that running it as a module adds the current directory to sys.path, which is standard python behavior, and can be helpful.

I had best results running pytest tests from Visual Studio Code when pytest was invoked as a module.

Calling Pytest from Python

Your code can call pytest. Lots of interesting and advanced development setups could be crafted using this approach. Baby steps.

Help Message

The help message for pytest is:

Shell

$ pytest -h
usage: pytest [options] [file_or_dir] [file_or_dir] [...]

positional arguments:
  file_or_dir

general:
  -k EXPRESSION         Only run tests which match the given substring
                        expression. An expression is a Python evaluatable
                        expression where all names are substring-matched against
                        test names and their parent classes. Example: -k
                        'test_method or test_other' matches all test functions
                        and classes whose name contains 'test_method' or
                        'test_other', while -k 'not test_method' matches those
                        that don't contain 'test_method' in their names. -k 'not
                        test_method and not test_other' will eliminate the
                        matches. Additionally keywords are matched to classes
                        and functions containing extra names in their
                        'extra_keyword_matches' set, as well as functions which
                        have names assigned directly to them. The matching is
                        case-insensitive.
  -m MARKEXPR           Only run tests matching given mark expression. For
                        example: -m 'mark1 and not mark2'.
  --markers             show markers (builtin, plugin and per-project ones).
  -x, --exitfirst       Exit instantly on first error or failed test
  --fixtures, --funcargs
                        Show available fixtures, sorted by plugin appearance
                        (fixtures with leading '_' are only shown with '-v')
  --fixtures-per-test   Show fixtures per test
  --pdb                 Start the interactive Python debugger on errors or
                        KeyboardInterrupt
  --pdbcls=modulename:classname
                        Specify a custom interactive Python debugger for use
                        with --pdb.For example:
                        --pdbcls=IPython.terminal.debugger:TerminalPdb
  --trace               Immediately break when running each test
  --capture=method      Per-test capturing method: one of fd|sys|no|tee-sys
  -s                    Shortcut for --capture=no
  --runxfail            Report the results of xfail tests as if they were not
                        marked
  --lf, --last-failed   Rerun only the tests that failed at the last run (or all
                        if none failed)
  --ff, --failed-first  Run all tests, but run the last failures first. This may
                        re-order tests and thus lead to repeated fixture
                        setup/teardown.
  --nf, --new-first     Run tests from new files first, then the rest of the
                        tests sorted by file mtime
  --cache-show=[CACHESHOW]
                        Show cache contents, don't perform collection or tests.
                        Optional argument: glob (default: '*').
  --cache-clear         Remove all cache contents at start of test run
  --lfnf={all,none}, --last-failed-no-failures={all,none}
                        Which tests to run with no previously (known) failures
  --sw, --stepwise      Exit on test failure and continue from last failing test
                        next time
  --sw-skip, --stepwise-skip
                        Ignore the first failing test but stop on the next
                        failing test. Implicitly enables --stepwise.

Reporting:
  --durations=N         Show N slowest setup/test durations (N=0 for all)
  --durations-min=N     Minimal duration in seconds for inclusion in slowest
                        list. Default: 0.005.
  -v, --verbose         Increase verbosity
  --no-header           Disable header
  --no-summary          Disable summary
  -q, --quiet           Decrease verbosity
  --verbosity=VERBOSE   Set verbosity. Default: 0.
  -r chars              Show extra test summary info as specified by chars:
                        (f)ailed, (E)rror, (s)kipped, (x)failed, (X)passed,
                        (p)assed, (P)assed with output, (a)ll except passed
                        (p/P), or (A)ll. (w)arnings are enabled by default (see
                        --disable-warnings), 'N' can be used to reset the list.
                        (default: 'fE').
  --disable-warnings, --disable-pytest-warnings
                        Disable warnings summary
  -l, --showlocals      Show locals in tracebacks (disabled by default)
  --no-showlocals       Hide locals in tracebacks (negate --showlocals passed
                        through addopts)
  --tb=style            Traceback print mode (auto/long/short/line/native/no)
  --show-capture={no,stdout,stderr,log,all}
                        Controls how captured stdout/stderr/log is shown on
                        failed tests. Default: all.
  --full-trace          Don't cut any tracebacks (default is to cut)
  --color=color         Color terminal output (yes/no/auto)
  --code-highlight={yes,no}
                        Whether code should be highlighted (only if --color is
                        also enabled). Default: yes.
  --pastebin=mode       Send failed|all info to bpaste.net pastebin service
  --junit-xml=path      Create junit-xml style report file at given path
  --junit-prefix=str    Prepend prefix to classnames in junit-xml output

pytest-warnings:
  -W PYTHONWARNINGS, --pythonwarnings=PYTHONWARNINGS
                        Set which warnings to report, see -W option of Python
                        itself
  --maxfail=num         Exit after first num failures or errors
  --strict-config       Any warnings encountered while parsing the `pytest`
                        section of the configuration file raise errors
  --strict-markers      Markers not registered in the `markers` section of the
                        configuration file raise errors
  --strict              (Deprecated) alias to --strict-markers
  -c FILE, --config-file=FILE
                        Load configuration from `FILE` instead of trying to
                        locate one of the implicit configuration files.
  --continue-on-collection-errors
                        Force test execution even if collection errors occur
  --rootdir=ROOTDIR     Define root directory for tests. Can be relative path:
                        'root_dir', './root_dir', 'root_dir/another_dir/';
                        absolute path: '/home/user/root_dir'; path with
                        variables: '$HOME/root_dir'.

collection:
  --collect-only, --co  Only collect tests, don't execute them
  --pyargs              Try to interpret all arguments as Python packages
  --ignore=path         Ignore path during collection (multi-allowed)
  --ignore-glob=path    Ignore path pattern during collection (multi-allowed)
  --deselect=nodeid_prefix
                        Deselect item (via node id prefix) during collection
                        (multi-allowed)
  --confcutdir=dir      Only load conftest.py's relative to specified dir
  --noconftest          Don't load any conftest.py files
  --keep-duplicates     Keep duplicate tests
  --collect-in-virtualenv
                        Don't ignore tests in a local virtualenv directory
  --import-mode={prepend,append,importlib}
                        Prepend/append to sys.path when importing test modules
                        and conftest files. Default: prepend.
  --doctest-modules     Run doctests in all .py modules
  --doctest-report={none,cdiff,ndiff,udiff,only_first_failure}
                        Choose another output format for diffs on doctest
                        failure
  --doctest-glob=pat    Doctests file matching pattern, default: test*.txt
  --doctest-ignore-import-errors
                        Ignore doctest ImportErrors
  --doctest-continue-on-failure
                        For a given doctest, continue to run after the first
                        failure

test session debugging and configuration:
  --basetemp=dir        Base temporary directory for this test run. (Warning:
                        this directory is removed if it exists.)
  -V, --version         Display pytest version and information about plugins.
                        When given twice, also display information about
                        plugins.
  -h, --help            Show help message and configuration info
  -p name               Early-load given plugin module name or entry point
                        (multi-allowed). To avoid loading of plugins, use the
                        `no:` prefix, e.g. `no:doctest`.
  --trace-config        Trace considerations of conftest.py files
  --debug=[DEBUG_FILE_NAME]
                        Store internal tracing debug information in this log
                        file. This file is opened with 'w' and truncated as a
                        result, care advised. Default: pytestdebug.log.
  -o OVERRIDE_INI, --override-ini=OVERRIDE_INI
                        Override ini option with "option=value" style, e.g. `-o
                        xfail_strict=True -o cache_dir=cache`.
  --assert=MODE         Control assertion debugging tools.
                        'plain' performs no assertion debugging.
                        'rewrite' (the default) rewrites assert statements in
                        test modules on import to provide assert expression
                        information.
  --setup-only          Only setup fixtures, do not execute tests
  --setup-show          Show setup of fixtures while executing tests
  --setup-plan          Show what fixtures and tests would be executed but don't
                        execute anything

logging:
  --log-level=LEVEL     Level of messages to catch/display. Not set by default,
                        so it depends on the root/parent log handler's effective
                        level, where it is "WARNING" by default.
  --log-format=LOG_FORMAT
                        Log format used by the logging module
  --log-date-format=LOG_DATE_FORMAT
                        Log date format used by the logging module
  --log-cli-level=LOG_CLI_LEVEL
                        CLI logging level
  --log-cli-format=LOG_CLI_FORMAT
                        Log format used by the logging module
  --log-cli-date-format=LOG_CLI_DATE_FORMAT
                        Log date format used by the logging module
  --log-file=LOG_FILE   Path to a file when logging will be written to
  --log-file-level=LOG_FILE_LEVEL
                        Log file logging level
  --log-file-format=LOG_FILE_FORMAT
                        Log format used by the logging module
  --log-file-date-format=LOG_FILE_DATE_FORMAT
                        Log date format used by the logging module
  --log-auto-indent=LOG_AUTO_INDENT
                        Auto-indent multiline messages passed to the logging
                        module. Accepts true|on, false|off or an integer.
  --log-disable=LOGGER_DISABLE
                        Disable a logger by name. Can be passed multiple times.

[pytest] ini-options in the first pytest.ini|tox.ini|setup.cfg|pyproject.toml file found:

  markers (linelist):   Markers for test functions
  empty_parameter_set_mark (string):
                        Default marker for empty parametersets
  norecursedirs (args): Directory patterns to avoid for recursion
  testpaths (args):     Directories to search for tests when no files or
                        directories are given on the command line
  filterwarnings (linelist):
                        Each line specifies a pattern for
                        warnings.filterwarnings. Processed after
                        -W/--pythonwarnings.
  usefixtures (args):   List of default fixtures to be used with this project
  python_files (args):  Glob-style file patterns for Python test module
                        discovery
  python_classes (args):
                        Prefixes or glob names for Python test class discovery
  python_functions (args):
                        Prefixes or glob names for Python test function and
                        method discovery
  disable_test_id_escaping_and_forfeit_all_rights_to_community_support (bool):
                        Disable string escape non-ASCII characters, might cause
                        unwanted side effects(use at your own risk)
  console_output_style (string):
                        Console output: "classic", or with additional progress
                        information ("progress" (percentage) | "count" |
                        "progress-even-when-capture-no" (forces progress even
                        when capture=no)
  xfail_strict (bool):  Default for the strict parameter of xfail markers when
                        not given explicitly (default: False)
  tmp_path_retention_count (string):
                        How many sessions should we keep the `tmp_path`
                        directories, according to `tmp_path_retention_policy`.
  tmp_path_retention_policy (string):
                        Controls which directories created by the `tmp_path`
                        fixture are kept around, based on test outcome.
                        (all/failed/none)
  enable_assertion_pass_hook (bool):
                        Enables the pytest_assertion_pass hook. Make sure to
                        delete any previously generated pyc cache files.
  junit_suite_name (string):
                        Test suite name for JUnit report
  junit_logging (string):
                        Write captured log messages to JUnit report: one of
                        no|log|system-out|system-err|out-err|all
  junit_log_passing_tests (bool):
                        Capture log information for passing tests to JUnit
                        report:
  junit_duration_report (string):
                        Duration time to report: one of total|call
  junit_family (string):
                        Emit XML for schema: one of legacy|xunit1|xunit2
  doctest_optionflags (args):
                        Option flags for doctests
  doctest_encoding (string):
                        Encoding used for doctest files
  cache_dir (string):   Cache directory path
  log_level (string):   Default value for --log-level
  log_format (string):  Default value for --log-format
  log_date_format (string):
                        Default value for --log-date-format
  log_cli (bool):       Enable log display during test run (also known as "live
                        logging")
  log_cli_level (string):
                        Default value for --log-cli-level
  log_cli_format (string):
                        Default value for --log-cli-format
  log_cli_date_format (string):
                        Default value for --log-cli-date-format
  log_file (string):    Default value for --log-file
  log_file_level (string):
                        Default value for --log-file-level
  log_file_format (string):
                        Default value for --log-file-format
  log_file_date_format (string):
                        Default value for --log-file-date-format
  log_auto_indent (string):
                        Default value for --log-auto-indent
  pythonpath (paths):   Add paths to sys.path
  faulthandler_timeout (string):
                        Dump the traceback of all threads if a test takes more
                        than TIMEOUT seconds to finish
  addopts (args):       Extra command line options
  minversion (string):  Minimally required pytest version
  required_plugins (args):
                        Plugins that must be present for pytest to run

Environment variables:
  PYTEST_ADDOPTS           Extra command line options
  PYTEST_PLUGINS           Comma-separated plugins to load during startup
  PYTEST_DISABLE_PLUGIN_AUTOLOAD Set to disable plugin auto-loading
  PYTEST_DEBUG             Set to enable debug tracing of pytest's internals


to see available markers type: pytest --markers
to see available fixtures type: pytest --fixtures
(shown according to specified file_or_dir or current dir if not specified; fixtures with leading '_' are only shown with the '-v' option

Helpful Options

It is handy to run new and failing tests, but to skip previously passing tests. The ‑‑nf and ‑‑lf options, respectively, invoke that mode of behavior:

Shell

$ pytest --nf --lf

Setup

Configuration

Pytest needed to know that the source code for my Python project is stored in the src directory. I created pyproject.toml as follows, which configured pytest:

pyproject.toml

[tool.pytest.ini_options]
addopts = [
    "--import-mode=importlib",
]
pythonpath = "src"

The way in which pytest discovers tests is complex. TL;DR: specify the newest and best mechanism for test discovery by using the ‑‑import-mode=importlib option as shown above.

To always run pytest with the ‑‑nf and ‑‑lf options, add those options to pyproject.toml:

pyproject.toml

[tool.pytest.ini_options]
addopts = [
    "--import-mode=importlib",
    "--lf",
    "--nf",
]
pythonpath = "src"

Test Directories

Flexibility generally adds complexity. Pytest allows for a lot of flexibility regarding the location of where tests can reside. Unfortunately, Python’s package and module mechanism in combination with the pytest discovery mechanism for tests can lead to a frustrating experience when setting up a new project.

You can read about good practices for pytest. Those recommendations are intimidating for first-time pytest programmers. I did not follow all of them.

I found the simplest approach was to create a subdirectory for the bulk of the Python project code. If a file called __init__.py is created in that directory (use touch for this) then the subdirectory becomes a submodule of your Python program. For this type of pytest setup, each submodule needs a tests directory.

Here is how you could use touch to define an imaginary subdirectory/submodule called my_submodule:

Shell

$ mkdir -p src/my_submodule/

$ touch src/my_submodule/__init__.py

My project structure was as follows. Tree docs are here.

pytest project structure

$ tree -I .git -I __pycache__
.
├── README.md
├── dl
├── dl.config
├── pyproject.toml
├── requirements.txt
└── src
    ├── __main__.py
    └── dl
        ├── __init__.py
        ├── argument_parse.py
        ├── dl_config.py
        ├── media_file.py
        ├── remote.py
        ├── tests
        │   ├── __init__.py
        │   ├── test_dl_config.py
        │   └── test_uti.py
        └── util.py

3 directories, 15 files

src/dl/tests/__init__.py was an empty marker file, but the ___init__.py file in the parent module defined imports for the submodule:

src/dl/__init__.py

from dl.argument_parse import ArgParse
from dl.dl_config import DLConfig
from dl.media_file import MediaFile
from dl.remote import Remote
from dl.util import *

Sample Test

Here is a sample pytest file, containing 2 unit tests. It is simple and clear.

src/dl/tests/test_util.py

import pytest
from pathlib import Path
from dl import samba_mount, samba_parse

class TestUtil:
    def test_samba_parse(self):
        remote_drive, local_path = samba_parse('e:/media/renders')
        assert remote_drive=='e'
        assert local_path=='/media/renders'

    def test_samba_mount(self):
        samba_root = samba_mount('bear', 'e', False)
        assert samba_root==Path('/mnt/bear/e')
        assert samba_root.is_mount()

Secret Sauce

If you have been paying attention, you should realize that one option for running new and failing tests is to type the following from the command line. We will use this method of invocation in the next section to integrate with Visual Studio Code.

Shell

$ python -m pytest --nf --lf

Visual Studio Code Integration

Visual Studio Code has a Testing explorer and a Run and Debug explorer. I find the existance of two very similar explorers annoying, because you often want to debug a test. Yes, you can do that by pushing the right button in the Testing Explorer ... if it works. This did not work for me.

Shopify Ruby, the latest and greatest Ruby extension for Visual Studio Code, uses the Run and Debug explorer to run and debug tests as well as entire programs. This makes sense to me.

Microsoft Pylance (for Python) follows a different convention, and uses the Run and Debug explorer to run and debug Python programs. You must use the Testing explorer for running or debugging Python unit tests. Annoying.

In the Testing explorer, I was able to see my dl module, but no tests were discovered. This was really annoying, and this is where more documentation from Microsoft would have been helpful. After wasting lots of time trying to solve this problem, I discovered a solution.

Invoke Pytest As a Module from VSCode

Run and Debug configurations that invoke the pytest module work perfectly. Here is an example .vscode/launch.json that demonstrates this:

.vscode/launch.json

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "PyTest All",
      "type": "python",
      "request": "launch",
      "module": "pytest",
      "justMyCode": true
    },
    {
      "args": ["--nf", "--lf"],
      "name": "PyTest New and Failing",
      "type": "python",
      "request": "launch",
      "module": "pytest",
      "justMyCode": true
    },
  ]
}

😁

You can set breakpoints in your code and debug your tests this way.

C and C++ Online and On Ubuntu

2023-05-12T00:00:00-04:00

Lattice C

K&R Book

Borland Turbo C

Recently I decided to brush up on my C programming.

I first discovered C during a visit in 1978 to the Computer Science department of University of California, Berkeley, where I also encounted UNIX for the first time.

From about 1983 to 1985 I was the unofficial Canadian distributor of the Lattice C compiler. I used that compiler for most of the programming work I was doing on PCs at the time. FedEx Express Canada did not come into being until 1987, UPS was even worse with the Canadian border than it is now, and Canada Post took forever to bring packages across the border. I bought in bulk at a discount, and sold at list price. Distribution is all about time and place.

I also sold the first edition of the K&R book (“The C Programming Language”) via mail order, and I provided technical support, which brought in interesting consulting work.

The Borland Turbo C compiler, released in 1990, was the original IDE. It was eventually replaced by Borland C++ compiler, which also compiled C. In 2005 I became the product marketing manager for most of Borland’s compilers. One of my responsibilities was to give free copies to book authors and members of the trade press. Borland C++ shipped in a massive box that was responsible for a lot of dead trees.

The C language has evolved since K&R days. The current and previous iterations are well summarized on cppreference.com, which also provides an online C/C++ IDE for sample code.

Revision History

1978: K&R book first published.

1989: C89, also known as ANSI X3. 159-1989, or ANSI C.

1995: NA1 (aka C94 and C95) added support for wide-characters, wide-strings, and international keyboard layouts.

1999: C99 added restricted pointers, variable-length arrays, flexible array members, complex numbers support, type-generic math, long long int, extended identifiers, hexadecimal floating-point constants, compound literals, designated initializers, single line comments, extended integer type, the ability to mix declarations and code, variadic macros, new math functions, inline functions, boolean types, _Pragma and standard pragmas, and VA_COPY. C99 removed implicit function declaration.

2011: C11 added concurrency support, type-generic expressions, alignment-related facilities, static assertion, Unicode support, floating-point characteristic macros, no-return functions, anonymous structures and unions, bound-checking, and reentrancy functions.

2018: C18 addressed defects in C11 without introducing new language features.

2023: The most recent publicly available draft of the next release, C23, was released on April 1, 2023. Unfortunately, that document was written for insiders, and is very difficult to read. Happily, JeanHeyd Meneide, the Project Editor for ISO/IEC JTC1 SC22 WG14 - Programming Languages, C, wrote an easy-to-read summary of C23.

Linux Is Written in C

It has been reported that Linux was originally written in C89, but Linux moved to C11 in 2022. I am curious as to why C18 is not used, since C18 just contains bug fixes for C11. Most of the Linux kernel code is written using the GNU extensions of GCC.

Online IDEs

I pioneered online programming with SMX Explorer in 1994, JSP Explorer in 2000, and supported most available languages in 2002 with Zamples.com.

Now it seems that every day another online programming option becomes available. Many are free:

Coliru is used by cppreference.com; GitHub project
GitHub Codespaces is tightly integrated with Microsoft Azure.
JetBrains Space
OnlineGDB claims to be to first Online IDE, but the WayBack Machine shows that this site was created in 2016.
Programiz
TutorialsPoint CodingGround
Replit
OneCompiler
... and many more ... even more!

Installing C/C++ on WSL/Ubuntu

Online programming is fine for learning, but I prefer to be independent and self-reliant whenever practical for doing actual work.

Command-Line Toolchain

To program in C or C++ on Ubuntu using the traditional Linux command-line toolchain, install the build-essential Debian meta-package on Ubuntu. It includes the GNU Compiler Collection (GCC), which is a collection of compilers and libraries for C, C++, Objective-C, Fortran, Ada, Go, and D programming languages. It also includes make and libc6.1-dev, the GNU C development libraries and header files.

You probably should also install gdb, the GNU Project Debugger, and the manual pages about using GNU/Linux for development.

Shell

$ yes | sudo apt install build-essential gdb manpages-dev

Now you are able to compile and run C programs at the command line:

Shell

$ mkdir test && cd test

$ cat >test.c <<EOF
#include <stdio.h>
int main(void) {
  printf("Hello, World!\n");
}
EOF

$ gcc test.c -o test && ./test
Hello, World!

Visual Studio Code

Visual Studio Code is a high-quality (and free) IDE. To work with C and C++ in Visual Studio Code, install the Microsoft C/C++ extension.

Don't Poke the Bear

2023-03-03T00:00:00-05:00

To the developers of the excellent ‘’ GitHub project

The name of this project is problematic, and you publicly dismissed the issue when it was brought to your attention. While ignorance of the law is no defense; demonstrably flaunting another party’s rights never has positive consequences.

The LEGO Group has not taken legal action yet, but the laws of most countries (including the US) are that if a trademark is not defended, then the trademark is at risk of being lost.

You should expect to receive a legal notice at some point. Not because the trademark owners are evil, but because they have a duty to their company, and the well-being of their employees, and a duty to their shareholders.

Your disregard for the LEGO Group's rights constituted a real and present danger to them, their employees and shareholders. They are compelled to address threats against their well-being.

CC BY-NC-SA 2.0' class="imgImg rounded shadow" src="/blog/images/bearpokelego.png" style='width: 100%; ' title='‘lego bear’ by angie
CC BY-NC-SA 2.0' />

‘lego bear’ by angie
CC BY-NC-SA 2.0

Think

Not only must a trademark holder defend their trademark, they must be seen to defend it. That means taking action that is publicly visible. For example, they might include GitHub and maybe even Microsoft in the legal action against you.

No-one would have reason to defend you

The LEGO Group would likely include GitHub/Microsoft in the case, even though they know that Section 230 would probably be found to apply (in due course), and GitHub/Microsoft would not be found to be liable – after they spent several million dollars in a successful legal defense.

It is easy to understand the motivation for the LEGO Group deliberately ‘losing’ at the attempt to ensnare GitHub/Microsoft in this court case. You would have caused GitHub/Microsoft an unnecessary and pointless waste of time and resources, and it would be shown that you knew better. Remember, you have already been publicly been warned that there would likely be consequences – yet you chose to ignore the warning.

This would likely motivate GitHub/Microsoft to protect themselves against similar misguided behavior of other GitHub users. Restrictive policies and procedures would likely get implemented, and the entire world would know that the resulting degraded GitHub experience was your fault. You would be vilified by every other GitHub user. Not a great thing for your resume.

... And GitHub/Microsoft might ask you to pay their legal expenses.

Do You Have Families?

Please understand that because you are acting as an informal collection of individuals, everyone who ever contributed to this project is exposed to signficant financial risk. The LEGO Group would probably not attempt to narrow down the guilty individuals – identifying the guilty and assigning punishment would be the outcomes of the court case. Instead, they would likely include every contributor in the litigation; that means every committer.

You are risking the financial well-being of your families ... just because you like a cute play on words?

Rational people would not expose themselves to significant risk without any prospect of signficant benefit, and when not faced with dire necessity.

If you presently balk at paying a lawyer to advise you on this matter, that cost will be as nothing compared to the world of hurt you are likely to find yourselves in if you do not change the name of your project.

From a technical aspect, you have done excellent work on your project, and you are freely providing the world a valuable benefit. Unfortunately, you trampled the rights of another in doing so... and for no reason. Please act on this before the inevitable negative consequences arise. Do not poke the bear because you are curious to see what it will do.

The wheels of corporate law grind slowly, and huge amounts of money are spent. According to the path you are now on, the committers of this project may be subjected to immense financial and mental burdens. Broken families and suicides may result.

Please heed this warning. If nothing bad has ever happened to you yet in life, rest assured that Lachesis, Atropos and Clotho will inevitably make themselves known. Do not let arrogance and pride ruin you.

I am trying to help you people.
Because you have done good work,
and it would be a shame to see you suffer.

Disclaimer

I have no relationship with the LEGO Group, or any individuals associated with them. Please consider this as a notice freely offered for public service.

Although I have experience working as an expert witness in the US Federal legal system as a software expert, I am not a lawyer and I have no legal training. However, I have seen corporate legal machinery at work many times, up close. I have spent thousands of hours working with corporate attorneys in the US and Europe. My clients have included Adobe, Amazon/AWS, Apple, and SAP.

Letsencrypt/ACME Wildcard SSL Certificates by Lego

2023-03-02T00:00:00-05:00

Letsencrypt certificates are only valid for 90 days. It is common practice to renew them every 60 days, a task that one should automate when the website is first published.

I wrote about setting up wildcard SSL certificates with Nginx 9 months ago. That article included a demonstration of how to create an SSL certificate manually, but did not discuss how to maintain it.

This article discusses how to automatically generate and maintain the SSL certificate using lego, a Letsencrypt/ACME client and library written in Go that has become popular since I last wrote about this topic.

Lego handles many moving parts transparently. It is much simpler than using DNS delegates with acme-dns.

We’ll start by briefly discussing some background information.

Letsencrypt’s Certbot and Wildcard SSL Certificates

You must prove to Letsencrypt that you control the DNS for a domain before it issues a wildcard SSL certificate for that domain. Letsencrypt’s certbot currently uses the DNS-01 challenge for this purpose.

DNS-01 Challenge

The DNS-01 challenge requires that DNS TXT records for the domain be created with specific values as part of the authentication mechanism. Several of these TXT records can co-exist among the DNS records for a domain. The name of the TXT records is always _acme-challenge.

When certbot requests that a new wildcard SSL certificate be created by Letsencrypt, it sends DNS-01 TXT queries to the primary DNS. The Letsencrypt service verifies that the required _acme-challenge TXT records are available with the correct values.

A Technical Deep Dive: Securing the Automation of ACME DNS Challenge Validation, published by the Electronic Frontier Foundation, explains how this works in detail.

Certbot plugins have been created for various DNS providers to make this process easier. Lego is a higher-level program that makes the entire process even easier.

Agenda

The high-level outline of the remainder of this blog is:

Install go language support.
Install lego.
Generate the wildcard SSL certificate.
Provide the certificate to your production web server.
Restart the webserver and verify the new certificate is served.
Add a new entry to crontab to automate the SSL certificate generation.

Install Go Language Support

Lego is written in Go. Ensure that Go language support is installed. For Ubuntu Linux (and the default WSL distro), type:

Shell

$ yes | sudo apt install golang-go
The following NEW packages will be installed:
  golang-1.18-go golang-1.18-src golang-go golang-src
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 82.2 MB of archives.
After this operation, 436 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 golang-1.18-src all 1.18.1-1ubuntu1 [16.2 MB]
Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 golang-1.18-go amd64 1.18.1-1ubuntu1 [66.0 MB]
Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 golang-src all 2:1.18~0ubuntu2 [4438 B]
Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 golang-go amd64 2:1.18~0ubuntu2 [41.8 kB]
Fetched 82.2 MB in 2s (33.6 MB/s)
Selecting previously unselected package golang-1.18-src.
(Reading database ... 173763 files and directories currently installed.)
Preparing to unpack .../golang-1.18-src_1.18.1-1ubuntu1_all.deb ...
Unpacking golang-1.18-src (1.18.1-1ubuntu1) ...
Selecting previously unselected package golang-1.18-go.
Preparing to unpack .../golang-1.18-go_1.18.1-1ubuntu1_amd64.deb ...
Unpacking golang-1.18-go (1.18.1-1ubuntu1) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../golang-src_2%3a1.18~0ubuntu2_all.deb ...
Unpacking golang-src (2:1.18~0ubuntu2) ...
Selecting previously unselected package golang-go:amd64.
Preparing to unpack .../golang-go_2%3a1.18~0ubuntu2_amd64.deb ...
Unpacking golang-go:amd64 (2:1.18~0ubuntu2) ...
Setting up golang-1.18-src (1.18.1-1ubuntu1) ...
Setting up golang-src (2:1.18~0ubuntu2) ...
Setting up golang-1.18-go (1.18.1-1ubuntu1) ...
Setting up golang-go:amd64 (2:1.18~0ubuntu2) ...
Processing triggers for man-db (2.10.2-1) ...
Scanning processes...
Scanning processor microcode...
Scanning linux images...

Failed to retrieve available kernel versions.

Failed to check for processor microcode upgrades.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

View the version of go that was installed:

Shell

$ go version
go version go1.19.2 linux/amd64

Installing Lego

The official lego installation instructions are here. They do not mention Ubuntu instructions, but the lego package for Ubuntu can be installed in the usual way:

Shell

$ yes | sudo apt install lego
Reading package lists... Done
  Building dependency tree... Done
  Reading state information... Done
  The following NEW packages will be installed:
    lego
  0 upgraded, 1 newly installed, 0 to remove and 7 not upgraded.
  Need to get 5322 kB of archives.
  After this operation, 19.7 MB of additional disk space will be used.
  Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 lego amd64 4.1.3-3ubuntu1.22.04.1 [5322 kB]
  Fetched 5322 kB in 1s (10.2 MB/s)
  Selecting previously unselected package lego.
  (Reading database ... 86044 files and directories currently installed.)
  Preparing to unpack .../lego_4.1.3-3ubuntu1.22.04.1_amd64.deb ...
  Unpacking lego (4.1.3-3ubuntu1.22.04.1) ...
  Setting up lego (4.1.3-3ubuntu1.22.04.1) ...

Here is the lego help message:

Shell

$ lego -h
NAME:
lego - Let's Encrypt client written in Go

USAGE:
lego [global options] command [command options] [arguments...]

VERSION:
dev

COMMANDS:
run      Register an account, then create and install a certificate
revoke   Revoke a certificate
renew    Renew a certificate
dnshelp  Shows additional help for the '--dns' global option
list     Display certificates and accounts information.
help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
--domains value, -d value    Add a domain to the process. Can be specified multiple times.
--server value, -s value     CA hostname (and optionally :port). The server certificate must be trusted in order to avoid further modifications to the client. (default: "https://acme-v02.api.letsencrypt.org/directory")
--accept-tos, -a             By setting this flag to true you indicate that you accept the current Let's Encrypt terms of service.
--email value, -m value      Email used for registration and recovery contact.
--csr value, -c value        Certificate signing request filename, if an external CSR is to be used.
--eab                        Use External Account Binding for account registration. Requires --kid and --hmac.
--kid value                  Key identifier from External CA. Used for External Account Binding.
--hmac value                 MAC key from External CA. Should be in Base64 URL Encoding without padding format. Used for External Account Binding.
--key-type value, -k value   Key type to use for private keys. Supported: rsa2048, rsa4096, rsa8192, ec256, ec384. (default: "ec256")
--filename value             (deprecated) Filename of the generated certificate.
--path value                 Directory to use for storing the data. (default: "/mnt/_/work/lego/.lego") [$LEGO_PATH]
--http                       Use the HTTP challenge to solve challenges. Can be mixed with other types of challenges.
--http.port value            Set the port and interface to use for HTTP based challenges to listen on.Supported: interface:port or :port. (default: ":80")
--http.proxy-header value    Validate against this HTTP header when solving HTTP based challenges behind a reverse proxy. (default: "Host")
--http.webroot value         Set the webroot folder to use for HTTP based challenges to write directly in a file in .well-known/acme-challenge. This disables the built-in server and expects the given directory to be publicly served with access to .well-known/acme-challenge
--http.memcached-host value  Set the memcached host(s) to use for HTTP based challenges. Challenges will be written to all specified hosts.
--tls                        Use the TLS challenge to solve challenges. Can be mixed with other types of challenges.
--tls.port value             Set the port and interface to use for TLS based challenges to listen on. Supported: interface:port or :port. (default: ":443")
--dns value                  Solve a DNS challenge using the specified provider. Can be mixed with other types of challenges. Run 'lego dnshelp' for help on usage.
--dns.disable-cp             By setting this flag to true, disables the need to wait the propagation of the TXT record to all authoritative name servers.
--dns.resolvers value        Set the resolvers to use for performing recursive DNS queries. Supported: host:port. The default is to use the system resolvers, or Google's DNS resolvers if the system's cannot be determined.
--http-timeout value         Set the HTTP timeout value to a specific value in seconds. (default: 0)
--dns-timeout value          Set the DNS timeout value to a specific value in seconds. Used only when performing authoritative name servers queries. (default: 10)
--pem                        Generate a .pem file by concatenating the .key and .crt files together.
--cert.timeout value         Set the certificate timeout value to a specific value in seconds. Only used when obtaining certificates. (default: 30)
--help, -h                   show help
--version, -v                print the version

Generating the Wildcard SSL Certificate

/etc/environment

My projects are defined in a directory tree, which is pointed to by an environment variable called work. This is helpful when working across many machines, each of which has a different directory layout.

I define the work environment variable in the system-wide /etc/environment file, where it affects all users, all scripts, and crontab entries:

/etc/environment

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
work=/mnt/_/work
sites=/mnt/_/www

$work is used in the script shown below, and that script will be launched via crontab, which is why /etc/environment is used instead of $HOME/.bashrc.

WSL does not run systemd services, and WSL2 only runs them if they are enabled. You can easily convert WSL instances into WSL2 instances.

Unless systemd starts automatically at boot time, /etc/environment will not be processed.

Read Microsoft’s announcement to learn more: Systemd support is now available in WSL!

Generation

When working with the lego CLI with DNS authentication, a directory is needed to hold all the files that are required. I created a directory for this purpose at $work/lego:

Shell

$ mkdir $work/lego

$ cd $work/lego

The lego --dns namecheap option connects to Namecheap, which is my DNS provider, and uses the Namecheap DNS for the DNS-01 challenge. Lego supports many other DNS providers.

Namecheap Users

The Namecheap API documentation is here. For some reason, that page reloads to a different page, which is annoying. Press the Esc key right after the page opens, so you can read it.

Please also read the FAQ.
Enable the Namecheap API here.
You could enable the Namecheap sandbox API here.

Here is a short script that displays your current public IP address:

whatismyip

#!/bin/bash

dig +short myip.opendns.com @resolver1.opendns.com

If your modem goes offline, Namecheap’s DNS will assign a new IP address the next time the modem reconnects. That will cause the scripts shown below to stop working until you manually provide the new address to the Namecheap API. Annoying! Be sure to plug your modem into a UPS to maintain the IP address even during short outages, power brownouts and voltage dropouts. A sine-wave UPS is strongly preferred, since modems are sensitive to low-quality power.

The Namecheap API needs 2 environment variables for authentication (NAMECHEAP_API_USER and NAMECHEAP_API_KEY). Here is how I defined those environment variables so they were available for the lego process launched by sudo:

Shell

$ export NAMECHEAP_API_USER=asdf

$ export NAMECHEAP_API_KEY=asdfasdf

To generate a wildcard certificate, you must specify the --domains option twice: once with the domain name, and once for all subdomains, like this:

Shell

$ lego \
  --accept-tos \
  --dns namecheap \
  --domains="scalacourses.com" \
  --domains="*.scalacourses.com" \
  --email="mslinn@scalacourses.com" \
  run
2023/03/01 13:40:03 [INFO] [*.scalacourses.com] acme: Obtaining bundled SAN certificate
2023/03/01 13:40:03 [INFO] [*.scalacourses.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/207409481036
2023/03/01 13:40:03 [INFO] [*.scalacourses.com] acme: use dns-01 solver
2023/03/01 13:40:03 [INFO] [*.scalacourses.com] acme: Preparing to solve DNS-01
2023/03/01 13:40:04 [INFO] [*.scalacourses.com] acme: Trying to solve DNS-01
2023/03/01 13:40:04 [INFO] [*.scalacourses.com] acme: Checking DNS record propagation using [172.19.176.1:53]
2023/03/01 13:40:19 [INFO] Wait for propagation [timeout: 1h0m0s, interval: 15s]
2023/03/01 13:40:20 [INFO] [*.scalacourses.com] acme: Waiting for DNS record propagation.
2023/03/01 13:40:35 [INFO] [*.scalacourses.com] acme: Waiting for DNS record propagation.
2023/03/01 13:40:56 [INFO] [*.scalacourses.com] The server validated our request
2023/03/01 13:40:56 [INFO] [*.scalacourses.com] acme: Cleaning DNS-01 challenge
2023/03/01 13:40:56 [INFO] [*.scalacourses.com] acme: Validations succeeded; requesting certificates
2023/03/01 13:40:57 [INFO] [*.scalacourses.com] Server responded with a certificate.

The wildcard SSL certificate and associated files are generated into the .lego/certificates directory.

Shell

$ ls -alF .lego/certificates
total 12
drwx------ 1 mslinn mslinn 4096 Mar  1 13:40 ./
drwx------ 1 mslinn mslinn 4096 Mar  1 13:15 ../
-rw------- 1 mslinn mslinn 5325 Mar  1 13:40 scalacourses.com.crt
-rw------- 1 mslinn mslinn 3751 Mar  1 13:40 scalacourses.com.issuer.crt
-rw------- 1 mslinn mslinn  239 Mar  1 13:40 scalacourses.com.json
-rw------- 1 mslinn mslinn  227 Mar  1 13:40 scalacourses.com.key

scalacourses.com.crt is the server certificate that nginx needs, already combined with the CA certificate.
scalacourses.com.key is the private key for the server certificate.
scalacourses.com.issuer.crt is the issuing Certificate Authority's certificate.

scalacourses.com.json contains JSON encoded meta information. It looked like this:

$work/lego/.lego/certificates/scalacourses.com.json

{
  "domain": "*.scalacourses.com",
  "certUrl": "https://acme-v02.api.letsencrypt.org/acme/cert/3939493849340275024024",
  "certStableUrl": "https://acme-v02.api.letsencrypt.org/acme/cert/738378373923492384932874932"
}

Using the value for certStableUrl in the above JSON file, we can view the generated (and combined) certificate that was saved on letsencrypt.org.

Shell

$ wget -O aw.crt \
https://acme-v02.api.letsencrypt.org/acme/cert/03b099f52b4841db17e035c5c2b390f0219b

certGenScalaCourses Script

60 days from now, when the certificate should be renewed, the last parameter passed to lego in the above command line should be changed from run to renew. Here is a bash script that you can edit for this purpose; at the end of this article this script is incorporated into a new crontab entry:

$work/lego/certGenScalaCourses

#!/bin/bash

export NAMECHEAP_API_USER=asdf
export NAMECHEAP_API_KEY=asdfasdf

ACTION=run
if [ -f .lego/certificates/scalacourses.com.crt ]; then
  ACTION=renew
fi

cd $work/lego

lego \
  --accept-tos \
  --dns namecheap \
  --domains="scalacourses.com" \
  --domains="*.scalacourses.com" \
  --email="mslinn@scalacourses.com" \
  "$ACTION"

sudo systemctl reload nginx

Viewing a Certificate

For Ubuntu 22.10, the filetype associations are defined in a hierarchy of files called mimeapps.list:

Shell

$ locate mimeapps.list
~/.config/mimeapps.list
~/.local/share/applications/mimeapps.list
/snap/core/14447/usr/share/applications/mimeapps.list
/snap/core/14784/usr/share/applications/mimeapps.list
/snap/core18/2679/usr/share/applications/mimeapps.list
/snap/core18/2697/usr/share/applications/mimeapps.list
/snap/core20/1778/usr/share/applications/mimeapps.list
/snap/core20/1822/usr/share/applications/mimeapps.list
/snap/core22/509/usr/share/applications/mimeapps.list
/snap/core22/522/usr/share/applications/mimeapps.list
/usr/share/gdm/greeter/applications/mimeapps.list

/snap/core22/509/usr/share/applications/mimeapps.list contains the default association for *.crt files:

/snap/core22/509/usr/share/applications/mimeapps.list

[Default Applications]
x-scheme-handler/http=xdg-open.desktop
x-scheme-handler/https=xdg-open.desktop
x-scheme-handler/mailto=xdg-open.desktop
x-scheme-handler/help=xdg-open.desktop

The above associates a program called xdg-open with CRT files.

Double-clicking on a crt file displayed by an Ubuntu file manager such as Nautilus causes xdg-open to open the file. You can also use the command line to open the file in xdg-open. For example, typing the following causes aw.crt to open:

Shell

$ xdg-open aw.crt

This is what xdg-open displays:

As you can see, this is a combined certificate, containing not just one certificate, but an entire certificate chain. Clicking on any of the red > Details buttons causes more information to be displayed about the corresponding certificate in the chain.

Warning

If you forget to include the domain, and just specify the wildcard for subdomains, the certificate will not be valid for the domain. For example, if above I had only specified one --domains option, like this:

Shell

--domains="scalacourses.com"

... instead of

Shell

--domains="scalacourses.com" \
--domains="*.scalacourses.com"

... then that would be an error. You can notice your error three ways:

The name of the generated certificate will start with an underscore (_), for example _.scalacourses.com.crt, instead of being named scalacourses.com.crt.
When you examine the certificate, the domain listed will show the subdomain wildcard, as shown below, with a leading underscore asterisk (*).
When you open the certificate, the section entitled Subject Alternative Names will just contain the subdomain wildcard, like this:

Subject Alternative Names
DNS: *.scalacourses.com

Or just the domain, without a wildcard:

Subject Alternative Names
DNS: scalacourses.com

Instead of both being present, like this:

Subject Alternative Names
DNS: *.scalacourses.com DNS: scalacourses.com

Provide the certificate to Nginx

Previously, before working with lego, I used raw certbot to generate the SSL wildcard certificates. That process generated files in ~/.certbot/scalacourses.com/config/live/scalacourses.com:

fullchain.pem (the ssl_certificate)
privkey.pem (the ssl_certificate_key)

From /etc/nginx/sites-enabled/scalacourses.com

ssl_certificate     /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/fullchain.pem;
ssl_certificate_key /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/privkey.pem;

Now I need to modify /etc/nginx/sites-enabled/scalacourses.com to reference the files generated by lego:

scalacourses.com.crt (the ssl_certificate)
scalacourses.com.key (the ssl_certificate_key)

If the webserver had access to the directory containing the new wildcard certificate, it would be simplest to provide the full path to the certificate and its key, like this:

Modified portion of /etc/nginx/sites-enabled/scalacourses.com

ssl_certificate     /mnt/_/work/lego/.lego/certificates/scalacourses.com.crt;
ssl_certificate_key /mnt/_/work/lego/.lego/certificates/scalacourses.com.key;

Restart Nginx

I tested the nginx configuration:

Shell

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Then I reloaded nginx:

Shell

$ sudo systemctl reload nginx

The new SSL certificate needs to be checked to make sure it was installed properly. I just visually examine the start and expire dates:

Shell

$ curl -Lvs https://www.scalacourses.com \
  2>&1 1>/dev/null | \
  grep '\(start\|expire\) date:'
*  start date: Mar  1 17:40:56 2023 GMT
*  expire date: May 30 17:40:55 2023 GMT

😁

Add a New Entry to Crontab

Automating the regeneration of the SSL wildcard certificate is easy. Edit your crontab:

Shell

$ crontab -e

Add the following reference to the certGenScalaCourses script that I gave you earlier to crontab:

crontab Entry

# Runs every 2 months / 60 days (more or less)
30 0 1 */2 * $work/lego/certGenScalaCourses

The above only works because we defined the work environment variable in /etc/environment.

Checking the Certificate With A Web Browser

Microsoft Windows caches SSL certificates, which can prevent your web browser from detecting newly updated SSL certificates. To clear the Windows SSL cache:

Press the Windows key, type Internet Options, and press Enter.
Select the Content tab.
Click the Clear SSL state button.
Click the OK button.

HTML Hyphens

2023-01-23T00:00:00-05:00

Web browsers now have built-in automatic hyphenation support, but it is turned off by default for compatibity with older browsers. You can enable hyphenation with two words of CSS, like this:

your_stylesheet.css

body {
  hyphens: auto;
}

W3 CSS3 Hyphenation Standard

Hyphenation occurs when the line breaks at a valid hyphenation opportunity, which is a type of soft wrap opportunity that exists within a word where hyphenation is allowed.

In CSS, hyphenation opportunities are controlled with the hyphens property. CSS Text Level 3 does not define the exact rules for hyphenation; however, UAs are strongly encouraged to optimize their choice of break points and to chose language-appropriate hyphenation points.

Possible hyphens values: none | manual | auto
Initial value: manual

– W3 CSS3 Standard

However, the W3 CSS3 hyphenation standard also says:

Correct automatic hyphenation requires a hyphenation resource appropriate to the language of the text being broken. The UA must therefore only automatically hyphenate text for which the content language is known and for which it has an appropriate hyphenation resource.

Authors should correctly tag their content’s language (e.g. using the HTML lang attribute or XML xml:lang attribute) in order to obtain correct automatic hyphenation.

– W3 CSS3 Standard

Let’s put this into practice now.

Your Website Stylesheet

You may not want certain passages to be hyphenated. Defining a CSS style that disables automatic hyphenation for portions of a document is easy; you can still use  in those portions of the document to manually define optional hyphenation points.

Here is a suggestion for your stylesheet, so it has the hyphenation support you need. Note that hypenation for kbd tags is completely disabled.

your_stylesheet.css

body, .hyphen {
  hyphens: auto;
}

.nohyphen {
  hyphens: manual;
}

kbd {
  hyphens: none;
}

Complete Example

The following example enables automatic hyphenation of the entire HTML body, according to US English rules. The HTML document contains nested tags, some of which disable or enable automatic hyphenation. One line demonstrates manual hyphenation, through the use of  entities:

index.html

<html lang="en-US">
  <head>
    <style>
      body, .hyphen {
        hyphens: auto;
      }

      .nohyphen {
        hyphens: manual;
      }
    </style>
  </head>
  <body>
    <p>This element is automatically hyphenated.</p>
    <p class="nohyphen">This element is not hyphenated.</p>
    <ol class="nohyphen">
      <li>This elem&shy;ent is manual&shy;ly hyphen&shy;ated.</li>
      <li class="hyphen">This element is automatically hyphenated.</li>
    </ol>
  </body>
</html>

😁

Easy!

Alternative Manual Hyphenation

Invisible hypenation is sometimes desirable. For example, if I have a long directory path, like $HOME/.rbenv/versions/$RUBY_VERSION/lib/ruby/gems/$RUBY_VERSION/$GEM_NAME-$GEM_VERSION I might want to tell the web browser where the string could be broken.

You have two options — they are actually different syntaxes for the same zero-width space.

Zero-Width Space HTML Entity

 is the HTML entity for a unicode character called the zero-width space (ZWSP).

For example, $HOME/.rbenv/versions/$RUBY_VERSION/lib/ruby/gems/$RUBY_VERSION/$GEM_NAME-$GEM_VERSION

renders as:

$HOME/.rbenv/versions/$RUBY_VERSION/lib/ruby/gems/$RUBY_VERSION/$GEM_NAME-$GEM_VERSION

Line Break Opportunity Tag

You could also use the line break opportunity tag,  .

For example, $HOME/.rbenv/versions/$RUBY_VERSION/lib/ruby/gems/$RUBY_VERSION/$GEM_NAME-$GEM_VERSION

renders as:

$HOME/.rbenv/versions/$RUBY_VERSION/lib/ruby/gems/$RUBY_VERSION/$GEM_NAME-$GEM_VERSION

Both Options Yield the Same Rendered Text

I prefer to use . YMMV.

Non-Breaking Hyphen

The non-breaking hyphen HTML entity (‑) displays a hyphen character that does not break.

For CSS, use content: "\2011" to produce a non-breaking hyphen.

A Curmudgeon’s Social Networking

2023-01-07T00:00:00-05:00

I am an unapologetic curmudgeon. I am not here to please anyone else, I am only here to earnestly be myself to the best of my ability.

If we do not continuously demonstrate that we endeavor to treat each other right, then we should be dismissed and forgotten.

Bullshit walks; money drives away, unsatisfied.

Fairness and equity are essential in all things without reservation. Otherwise, go away, life is short and you are just noise.

Twitter is now a circus. I don't need or want another circus in my communication. Do not look for me on Twitter any longer.

If someone connects with me on Facebook or LinkedIn, I expect them to interact with me in person, on the phone, and to want to engage with me. Failing that, I disconnect from them without hesitation.

Yes, please connect with me. Tell me your truth. Listen to me. I will listen to you.

Action Over Empty Words

Even better, we might endeavor to do things together. Imagining, conceptualizing, and building something real with others gives me joy. I want to share that joy.

People who talk about what ‘others’ should do, but are unwilling to act themselves, are a waste of time. They irritate me. Only approach me if you are an action-oriented individual.

The most difficult thing is the decision to act, the rest is merely tenacity. – Amelia Earhart

Peace and love. σ

Working With Volumes and Directories Under Ubuntu

2022-12-01T00:00:00-05:00

I've been reorganizing directories across several volumes on an Ubuntu server. This article documents how I used the commands I found to be most useful: rsync, ncdu, and fdupes. The latter two programs are not installed by default. You can install them this way:

Shell

$ yes | sudo apt install fdupes ncdu

Display Directory Sizes

Shell

$ ncdu /directory

Find Duplicate Files

This displays duplicate files and interactively asks which duplicate files to delete.

Shell

$ fdupes -r1Sd /directory

Compare Two Folders For Missing Files

This tip was inspired by an answer on unix.stackexchange.com.

Dry Run

The -n option displays the names of missing files in the destination directory, but makes no changes to the destination file system. If that option is not specified, then the missing files are copied.

Shell

$ sync -ri --ignore-existing -n /srcDir/ /destDir

There are a few things to note:

The first directory is the source, and it must end with a slash (/).
The second directory is the target, and it must NOT end with a slash.

Copy Missing Files

Simply do not provide the -n (dry run) option:

Shell

$ rsync -ri --ignore-existing /srcDir/ /destDir

JiraCLI, a Feature-rich Interactive Jira Command Line

2022-08-12T00:00:00-04:00

When first released in 2002, Jira was merely a bug tracker. Since then, it has gained features and is now used as a project management tool. Jira only provides a web user interface.

JiraCLI, an independent F/OSS project, provides command-line explorers for Jira issues, epics, and sprints.

Installation

JiraCLI is a Go program, so it is quick and easy to install on Ubuntu once Go is installed.

Shell

$ yes | sudo apt install golang-go

$ go install github.com/ankitpokhrel/jira-cli/cmd/jira@latest
go: downloading github.com/ankitpokhrel/jira-cli v1.0.0
go: downloading github.com/kr/text v0.2.0
go: downloading github.com/spf13/cobra v1.5.0
go: downloading github.com/spf13/viper v1.12.0
go: downloading github.com/zalando/go-keyring v0.2.1
go: downloading github.com/briandowns/spinner v1.18.1
go: downloading github.com/fatih/color v1.13.0
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/AlecAivazis/survey/v2 v2.3.5
go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go: downloading github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8
go: downloading github.com/cpuguy83/go-md2man/v2 v2.0.2
go: downloading github.com/spf13/pflag v1.0.5
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/atotto/clipboard v0.1.4
go: downloading github.com/charmbracelet/glamour v0.5.0
go: downloading github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d
go: downloading github.com/cli/safeexec v1.0.0
go: downloading github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51
go: downloading github.com/kentaro-m/blackfriday-confluence v0.0.0-20220126124413-8e85477b49b3
go: downloading github.com/russross/blackfriday/v2 v2.1.0
go: downloading github.com/mattn/go-colorable v0.1.12
go: downloading github.com/mattn/go-isatty v0.0.14
go: downloading github.com/gdamore/tcell/v2 v2.5.1
go: downloading github.com/rivo/tview v0.0.0-20220610163003-691f46d6f500
go: downloading github.com/fsnotify/fsnotify v1.5.4
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading github.com/spf13/afero v1.8.2
go: downloading github.com/spf13/cast v1.5.0
go: downloading github.com/spf13/jwalterweatherman v1.1.0
go: downloading github.com/godbus/dbus/v5 v5.1.0
go: downloading golang.org/x/text v0.3.7
go: downloading golang.org/x/term v0.0.0-20220526004731-065cf7ba2467
go: downloading golang.org/x/sys v0.0.0-20220615213510-4f61da869c0c
go: downloading github.com/subosito/gotenv v1.4.0
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading gopkg.in/ini.v1 v1.66.6
go: downloading github.com/magiconair/properties v1.8.6
go: downloading github.com/pelletier/go-toml/v2 v2.0.2
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/gdamore/encoding v1.0.0
go: downloading github.com/pelletier/go-toml v1.9.5
go: downloading github.com/lucasb-eyer/go-colorful v1.2.0
go: downloading github.com/mattn/go-runewidth v0.0.13
go: downloading github.com/muesli/termenv v0.12.0
go: downloading github.com/yuin/goldmark v1.4.12
go: downloading github.com/yuin/goldmark-emoji v1.0.1
go: downloading github.com/rivo/uniseg v0.2.0
go: downloading github.com/alecthomas/chroma v0.10.0
go: downloading github.com/microcosm-cc/bluemonday v1.0.18
go: downloading github.com/muesli/reflow v0.3.0
go: downloading github.com/olekukonko/tablewriter v0.0.5
go: downloading github.com/aymerick/douceur v0.2.0
go: downloading golang.org/x/net v0.0.0-20220621193019-9d032be2e588
go: downloading github.com/dlclark/regexp2 v1.4.0
go: downloading github.com/gorilla/css v1.0.0

Add $HOME/go/bin to the PATH so jira can be found:

Shell

$ echo 'PATH=$HOME/go/bin:$PATH' >> ~/.bashrc

$ source ~/.bashrc

$ which jira
/home/mslinn/go/bin/jira

Usage

This is the jira help message:

Shell

$ jira
Interactive Jira CLI.

USAGE
jira [flags]

MAIN COMMANDS
board       Board manages Jira boards in a project
epic        Epic manage epics in a project
issue       Issue manage issues in a project
open        Open issue in a browser
project     Project manages Jira projects
sprint      Sprint manage sprints in a project board

OTHER COMMANDS
completion  Output shell completion code for the specified shell (bash or zsh)
help        Help about any command
init        Init initializes jira config
man         Help generate man(7) pages for Jira CLI.
me          Displays configured jira user
version     Print the app version information

FLAGS
-c, --config string    Config file (default is /home/mslinn/.config/.jira/.config.yml)
--debug            Turn on debug output
-h, --help             help for jira
-p, --project string   Jira project to look into (defaults to /home/mslinn/.config/.jira/.config.yml)

LEARN MORE
Use 'jira <command> <subcommand> --help' for more information about a command.

You need an Jira API token before you can use JiraCLI. Get it from here.

You need to configure the program before you use it. Set the JIRA_API_TOKEN environment variable before running jira init. If you do not, then you will get an error like Received unexpected response '401 Unauthorized' from jira. in the next step.

Shell

$ export JIRA_API_TOKEN=asdfasdfasdf

$ jira init
? Installation type: Cloud
? Link to Jira server: https://xxx.atlassian.net/
? Login email: mslinn@mslinn.com
⠼ Verifying login details...
? Default project: My
? Default board:  [Use arrows to move, type to filter, ? for more help]
> [Search...]
  ----------
  My Scrum Board
  None

  ✓ Configuration generated: /home/mslinn/.config/.jira/.config.yml

This is the configuration file that was generated:

/home/mslinn/.config/.jira/.config.yml/

board:
    id: 350
    name: My Scrum Board
    type: scrum
epic:
    name: customfield_10009
    link: customfield_10008
installation: Cloud
issue:
    fields:
        custom:
            - name: Epic Link
              key: customfield_10008
              schema:
                datatype: any
            - name: Epic Name
              key: customfield_10009
              schema:
                datatype: string
    types:
        - id: "3"
          name: Task
          handle: Task
          subtask: false
        - id: "5"
          name: Sub-task
          handle: Sub-task
          subtask: true
        - id: "7"
          name: Story
          handle: Story
          subtask: false
        - id: "1"
          name: Bug
          handle: Bug
          subtask: false
        - id: "6"
          name: Epic
          handle: Epic
          subtask: false
login: mslinn@mslinn.com
project:
    key: My
    type: classic
server: https://xxx.atlassian.net

Example Commands

Help for the jira issue subcommand:

Shell

$ jira issue -h
Issue manage issues in a given project. See available commands below.

  USAGE
  jira issue [flags]

  MAIN COMMANDS
  assign      Assign issue to a user
  clone       Clone duplicates an issue
  comment     Manage issue comments
  create      Create an issue in a project
  delete      Delete an issue
  edit        Edit an issue in a project
  link        Link connects two issues
  list        List lists issues in a project
  move        Transition an issue to a given state
  unlink      Unlink disconnects two issues from each other
  view        View displays contents of an issue
  worklog     Manage issue worklog

  FLAGS
  -h, --help   help for issue

  INHERITED FLAGS
  -c, --config string    Config file (default is /home/mslinn/.config/.jira/.config.yml)
  --debug            Turn on debug output
  -p, --project string   Jira project to look into (defaults to /home/mslinn/.config/.jira/.config.yml)

  ALIASES
  issues

  LEARN MORE
  Use 'jira <command> <subcommand> --help' for more information about a command.

Help for the jira issue list sub-subcommand:

Shell

$ jira issue list -h
List lists issues in a given project.

You can combine different flags to create a unique query. For instance,

# Issues that are of high priority, is in progress, was created this month, and has given labels
jira issue list -yHigh -s"In Progress" --created month -lbackend -l"high prio"

Issues are displayed in an interactive list view by default. You can use a --plain flag
to display output in a plain text mode. A --no-headers flag will hide the table headers
in plain view. A --no-truncate flag will display all available fields in plain mode.

USAGE
jira issue list [flags]

FLAGS
-t, --type string             Filter issues by type
-R, --resolution string       Filter issues by resolution type
-s, --status string           Filter issues by status
-y, --priority string         Filter issues by priority
-r, --reporter string         Filter issues by reporter (email or display name)
-a, --assignee string         Filter issues by assignee (email or display name)
-C, --component string        Filter issues by component
-l, --label stringArray       Filter issues by label
-P, --parent string           Filter issues by parent
--history                 Issues you accessed recently
-w, --watching                Issues you are watching
--created string          Filter issues by created date
Accepts: today, week, month, year, or a date in yyyy-mm-dd and yyyy/mm/dd format,
or a period format using w = weeks, d = days, h = hours, m = minutes. eg: -10d
Created filter will have precedence over created-after and created-before filter
--updated string          Filter issues by updated date
Accepts: today, week, month, year, or a date in yyyy-mm-dd and yyyy/mm/dd format,
or a period format using w = weeks, d = days, h = hours, m = minutes. eg: -10d
Updated filter will have precedence over updated-after and updated-before filter
--created-after string    Filter by issues created after certain date
--updated-after string    Filter by issues updated after certain date
--created-before string   Filter by issues created before certain date
--updated-before string   Filter by issues updated before certain date
-q, --jql string              Run a raw JQL query in a given project context
--order-by string         Field to order the list with (default "created")
--reverse                 Reverse the display order (default "DESC")
--paginate string         Paginate the result. Max 100 at a time, format: <from>:<limit> where <from> is optional (default "0:100")
--plain                   Display output in plain mode
--no-headers              Don't display table headers in plain mode. Works only with --plain
--no-truncate             Show all available columns in plain mode. Works only with --plain
--columns string          Comma separated list of columns to display in the plain mode.
Accepts: TYPE, KEY, SUMMARY, STATUS, ASSIGNEE, REPORTER, PRIORITY, RESOLUTION, CREATED, UPDATED
-h, --help                    help for list

INHERITED FLAGS
-c, --config string    Config file (default is /home/mslinn/.config/.jira/.config.yml)
--debug            Turn on debug output
-p, --project string   Jira project to look into (defaults to /home/mslinn/.config/.jira/.config.yml)

EXAMPLES
$ jira issue list

# Limit list to 20 items
$ jira issue list --paginate 20

# Get 50 items starting from 10
$ jira issue list --paginate 10:50

# List issues in a plain table view without headers
$ jira issue list --plain --no-headers

# List some columns of the issue in a plain table view
$ jira issue list --plain --columns key,assignee,status

# List issues in a plain table view and show all fields
$ jira issue list --plain --no-truncate

# List issues of type "Epic" in status "Done"
$ jira issue list -tEpic -sDone

# List issues in status other than "Open" and is assigned to no one
$ jira issue list -s~Open -ax

# List issues from all projects
$ jira issue list -q"project IS NOT EMPTY"

ALIASES
lists
ls

LEARN MORE
Use 'jira <command> <subcommand> --help' for more information about a command.

Issues Updated on a Certain Date

Shell

$ jira issue list --updated 2021-11-17

Issues Selected by Complex Criteria

The following command will yield a list of high-priority issues, created this month, with status To Do, that are assigned to you, and have the label backend.

Shell

$ jira issue list -yHigh -s"To Do" --created month -lbackend -a$(jira me)

Output might look something like the following, which was redacted:

The --plain and --no-headers options are useful for driving scripts.

Shell

$ jira issue list --updated 2021-11-17 --plain --no-headers --columns KEY

View Issue

The following returns the available information about an issue, in a plain-text format.

Shell

$ jira issue view ISSUE-1 --comments 9999 --plain

Verdict

JiraCLI is useful project! 😁

ImageMagick Slicing on Ubuntu/WSL

2022-07-28T00:00:00-04:00

Lawyers like the Microsoft Office software suite; so when I am working on a court case as an expert, I endeavor to provide my clients with Word documents that contain necessary information. I like working in WSL/WSL2 because I can use Windows programs and Ubuntu programs together effectively.

Grab Image, Then Slice

Recently, I used SnagIt, a Windows program, to capture large web pages as single images. Some of these images were quite tall.

The CSS for the web pages made some content invisible on the printed page. Yes, I could have injected CSS using a Chrome plugin like My Style to ensure that all content will be printed, like this:

Injected Style

@media print {
  * { display: initial; visibility: visible; }
}

I decided to use screen grabs, which would guarantee that the contents of my report would exactly match what had been displayed on the screen, without injecting anything into the web pages.

ImageMagick is preinstalled on Ubuntu Desktop. I used ImageMagick to slice the image captures into smaller page-sized images, so they could be inserted into a Word document.

The Computer Worked Hard

Grabbing such large web pages was a lot of work for my desktop computer. The only programs active during the screen grab process were the Google Chrome browser and SnagIt. I found that 10GB RAM and 30% of the GPU capability (an NVidia GTX 1660 Super) was used.

The screen grab failed if I did not start scrolling from the top of the web page; while it is possible to scrub up and down smaller web pages in order to grab portions of interest, this fails for large pages.

I also found that scrolling too fast caused the screen grabbing process to fail. Clicking and holding the bottom scroll arrowhead at the bottom right of the screen seemed to result in a smooth and optimal scrolling speed. This meant that grabbing large web pages took a few minutes as the page slowly scrolled downward.

Setting Up the Conversion

The Word documents I usually work with are formatted for North American standards. This means one-inch margins on letter-sized paper (8.5" x 11"), which gives a working area of 6.5" x 9", yielding an aspect ratio of 0.72.

The tall captured images needed to be sliced into rectangles that fit efficiently into Word documents. The computations are as follows.

Determine the width of a screen grab and save it into W. The ImageMagick identify command does not provide a newline after its output, however I have inserted one for readability:
Shell
```
$ identify -ping -format '%w' ../IMG2005.png
1536 

$ export W="$( identify -ping -format '%w' ../IMG2005.png )"
```

Determine the height of a screen grab and save it into H:

Shell

$ identify -ping -format '%h' ../IMG2005.png

$ export H="$( identify -ping -format '%h' ../IMG2005.png )"

The width can be divided by the aspect ratio to obtain the desired height of each slice so they can be inserted optimally into the Word documents. I used the bc calculator provided with Bash to divide W / ASPECT_RATIO. The H2 integer variable contains the computed height for the images.
Shell
```
$ export ASPECT_RATIO=0.72

$ export H2="$( echo "scale=0 ; $W / $ASPECT_RATIO" | bc )"
```
Now the image called IMG2005.png can be sliced using ImageMagick’s convert command. The slices are stored into a subdirectory called slices, with file names like IMG2005-1.jpg, IMG2005-2.jpg, etc.
Shell
```
$ convert IMG2005.png -crop ${W}x${H2} \
  -quality 100% -scene 0 slices/IMG2005-%d.jpg
```

Automating the Conversion

I wrote the following bash script, which incorporates the above computations. It slices all the images in a directory and saves the results to a second directory.

sliceImages

#!/bin/bash

function help {
  if [ "$1" ]; then echo "Error: $1"; fi
  echo "
$(basename $0): slice all images in the given directory and place them into a specified directory,
which will be created if required.
"
  exit 1
}

function setup {
  export ASPECT_RATIO=0.72
  export W="$( identify -ping -format '%w' "$1" )"
  export H="$( identify -ping -format '%h' "$1" )"
  export H2="$( echo "scale=0 ; $W / $ASPECT_RATIO" | bc )"
}

function convert1 {
  FULLNAME=$(basename -- "$1")
  FILENAME="${FULLNAME%.*}"
  FILETYPE="${FULLNAME##*.}"

  convert "$1" \
    -crop "${W}x${H2}" \
    -quality 100% \
    -scene 0 \
    "$DIR_OUTPUT/$FILENAME-%d.png"
}


if [ -z "$1" ]; then help "No directory path for images to be converted was provided."; fi
export DIR_INPUT="$( realpath $1 )"

if [ -z "$2" ]; then help "No directory path for the image slices to be saved into was provided."; fi
export DIR_OUTPUT="$( realpath $2 )"

mkdir -p "$DIR_OUTPUT"

find $DIR_INPUT -type f -exec file --mime-type {} \+ | awk -F: '{if ($2 ~/image\//) print $1}' |
  while read FILE; do
    setup "$FILE"
    echo "Slicing $FILE into ${W}x${H2} pixels"
    convert1 "$FILE"
  done

Overcoming ImageMagick Processing Limits

Some of the web pages that I needed to grab were quite long, which resulted in those images requiring more computational resources than the default Imagemagick configuration allows. This caused errors such as the following to appear:

Shell

convert-im6.q16: no images defined `/mnt/c/images/slices/IMG1466-%d.png' @ error/convert.c/ConvertImageCommand/3229.
convert-im6.q16: cache resources exhausted `/mnt/c/images/IMG1091.png' @ error/cache.c/OpenPixelCache/4095.

Imagemagick defines computational resources limits in /etc/ImageMagick-6/policy.xml. The default maximum memory is 256 KB, the default maximum allowable height is 16,000 pixels (16KP), and the default maximum area is 128M pixels. These values are defined by the following entries:

/etc/ImageMagick-6/policy.xml

<policy domain="resource" name="memory" value="256MiB"/>
<policy domain="resource" name="height" value="16KP"/>
<policy domain="resource" name="area" value="128MP"/>

I changed the maximum memory limit to 2 GB RAM, the maximum height limit to 10,000,000 pixels (10MP), and the maximum area limit to 2G pixels with these entries:

/etc/ImageMagick-6/policy.xml

<policy domain="resource" name="memory" value="2GiB"/>
<policy domain="resource" name="height" value="10MP"/>
<policy domain="resource" name="area" value="2GP"/>

Alternatively, I could have simply commented out the limits, as shown in highlighted text below.

/etc/ImageMagick-6/policy.xml

<!--
<policy domain="resource" name="memory" value="256MiB"/>
<policy domain="resource" name="height" value="16KP"/>
<policy domain="resource" name="area" value="128MP"/>
-->

The largest web page to be sliced was converted to a very tall image, which was 83,703 pixels high. It was sliced into 40 images.

Word Macro

A Word macro is also needed to insert the images into the currently open Word document in alphabetical order. I modified this one.

Microsoft Word Macro

Sub insertImages()
    Dim intResult As Integer
    Dim strPath As String
    Dim strFolderPath As String
    Dim objFSO As Object
    Dim objFolder As Object
    Dim objFile As Object
    Dim i As Integer

    intResult = Application.FileDialog(msoFileDialogFolderPicker).Show
    'Check if user canceled the dialog
    If intResult <> 0 Then
        'dispaly message box
        strFolderPath = Application.FileDialog(msoFileDialogFolderPicker).SelectedItems(1)
        'Create an instance of the FileSystemObject
        Set objFSO = CreateObject("Scripting.FileSystemObject")
        'Get the folder object
        Set objFolder = objFSO.GetFolder(strFolderPath)
        i = 1
        'loops through each file in the directory and prints their names and path
        For Each objFile In objFolder.Files
            'get file path
            strPath = objFile.Path
            'insert the image
            Selection.InlineShapes.AddPicture FileName:= _
               strPath, LinkToFile:=False, _
               SaveWithDocument:=True
        Next objFile
    End If
End Sub

Done!

😁

Thanks to the above automation, I was able to deliver the Word documents containing the sliced web pages to my client soon after they were requested.

Optimizing Plex Media Server on Ubuntu

2022-07-23T00:00:00-04:00

Plex Media Server is a free, full-featured media manager that is available for Linux, Windows, Mac, and is built into some NAS devices. I run it along with other processes on an Ubuntu server in my apartment.

From Plex.tv documentation

We can learn a lot about the Plex Media Server processes using the information provided by systemctl status:

Shell

$ sudo systemctl status plexmediaserver
● plexmediaserver.service - Plex Media Server
     Loaded: loaded (/lib/systemd/system/plexmediaserver.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-07-22 12:13:32 EDT; 23h ago
    Process: 2307 ExecStartPre=/bin/sh -c /usr/bin/test -d "${PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR}" || /bin/mkdir -p "${PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR}" (code=exited, status=0/SUCCESS)
   Main PID: 2329 (Plex Media Serv)
      Tasks: 45 (limit: 38369)
     Memory: 25.2G
        CPU: 42min 54.673s
     CGroup: /system.slice/plexmediaserver.service
             ├─2329 "/usr/lib/plexmediaserver/Plex Media Server"
             ├─3404 "Plex Plug-in [com.plexapp.system]" /usr/lib/plexmediaserver/Resources/Plug-ins-6b0e31a64/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.27.1.5916-6b0e31a64 /usr/lib/plexmediaserver/Resources/Plug-ins-6b0e31a64/System.bundle
             └─3521 "/usr/lib/plexmediaserver/Plex Tuner Service" /usr/lib/plexmediaserver/Resources/Tuner/Private /usr/lib/plexmediaserver/Resources/Tuner/Shared 1.27.1.5916-6b0e31a64 32600

The above tells us that Plex Media Server runs under a cgroup, which is “a Linux kernel feature which allow processes to be organized into hierarchical groups whose usage of various types of resources can then be limited and monitored”. The cgroup is defined in /lib/systemd/system/plexmediaserver.service. Here is mine:

/lib/systemd/system/plexmediaserver.service

# DO NOT EDIT THIS FILE DIRECTLY!
#
# Plex Media Server's variables can be customized by creating an 'overide.conf'
# file using 'systemctl edit plexmediaserver' which will create the following;
# /etc/systemd/system/plexmediaserver.service.d/override.conf
#
# An example of the override.conf would be as follows if you wished to edit
# your user, group, temp directory, or app support directory (without the leading #)
#
# [Service]
# Environment="TMPDIR=/path/to/new/tmp"
# Environment="PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR=/home/myusername/Library/Application Support"
# User=myusername
# Group=mygroup
#

[Unit]
Description=Plex Media Server
After=network.target network-online.target

[Service]
Environment="PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR=/var/lib/plexmediaserver/Library/Application Support"
Environment=PLEX_MEDIA_SERVER_HOME=/usr/lib/plexmediaserver
Environment=PLEX_MEDIA_SERVER_MAX_PLUGIN_PROCS=6
ExecStartPre=/bin/sh -c '/usr/bin/test -d "${PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR}" || /bin/mkdir -p "${PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR}"'
ExecStart=/bin/sh -c '\
export PLEX_MEDIA_SERVER_INFO_VENDOR="$(grep ^NAME= /etc/os-release | awk -F= "{print \\$2}" | tr -d \\" )"; \
export PLEX_MEDIA_SERVER_INFO_DEVICE="PC"; \
export PLEX_MEDIA_SERVER_INFO_MODEL="$(uname -m)"; \
export PLEX_MEDIA_SERVER_INFO_PLATFORM_VERSION="$(grep ^VERSION= /etc/os-release | awk -F= "{print \\$2}" | tr -d \\" )"; \
exec "/usr/lib/plexmediaserver/Plex Media Server"'
Type=simple
User=plex
Group=plex
Restart=on-failure
RestartSec=5
StartLimitInterval=60s
StartLimitBurst=3
SyslogIdentifier=Plex Media Server
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Directories and Files

On Linux, Plex Media Server uses the following directories and files by default. These are defined by the cgroup information shown above.

Programs	`/usr/lib/plexmediaserver/`
Service Configuration	`/usr/lib/plexmediaserver/lib/plexmediaserver.service`
Content library metadata	`/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/`

Adjusting CPU Priority

By default, the Plex processes run at nice factor 35 (priority 20), which is standard. When transcoding this means that most other processes, such as web servers, must wait. Transcoding can take a while, which means that the websites served from the same server time out.

There are several ways to control how much CPU resource is allocated to a Linux process. I ran my Plex processes under nice so other processes can continue unimpeded. Nice only throttles the process it manages when other processes request a time slice. If the system load is low then nice does not throttle the process that is supervises.

All the Plex processes run under Linux as user plex, group plex. It is easy to change the priority of these processes using nice. The following changes all currently running Plex processes to priority 22:

Shell

$ sudo renice 22 -u plex

I noticed that the video stream would stall at this priority, so I raised it by 10% (10% "less nice" equates to renicing by -2). Aaron Kili wrote a nice explanation of renice.

Shell

$ sudo renice -2 -u plex

One way to ensure that Plex Media Server always runs at normal priority is to add this to the root crontab:

sudo crontab -e

@reboot sleep 60 && renice 18 -u plex

Rygel

While playing with Plex on Ubuntu I noticed that rygel became active when Plex started transcoding. Coincidence? I did not know about rygel before. The GitHub repo.

Uncomplicated Firewall on Ubuntu

2022-07-16T00:00:00-04:00

All websites, especially ecommerce sites, need to be secure. A properly set up firewall is an essential component for a secure server.

Ubuntu 22.04 uses the Uncomplicated Firewall ufw firewall frontend by default. Ufw has been provided for Ubuntu since v8.04 (Hardy Heron).

Quick Setup

Enable ufw as follows:

Shell

$ sudo ufw enable
Firewall is active and enabled on system startup

Enable the ufw application profiles for ssh and nginx (HTTP/HTTPS) like this:

Shell

$ sudo ufw allow OpenSSH
Output
  Rule added
  Rule added (v6) 

$ sudo ufw allow 'Nginx Full'
Output
  Rule added
  Rule added (v6)

Diving Deeper

The following is mostly true:

The default firewall on Ubuntu 22.04 Jammy Jellyfish is ufw, which is short for “uncomplicated firewall.” Ufw is a frontend for the typical Linux iptables commands, but it is developed in such a way that basic firewall tasks can be performed without the knowledge of iptables.

Additionally, ufw can be managed from a graphical interface. In this tutorial, you will learn how to enable and disable the ufw firewall on Ubuntu 22.04 Jammy Jellyfish from both command line and GUI.

– From How to enable/disable firewall on Ubuntu 22.04 LTS Jammy Jellyfish Linux

The above makes no mention of how Ubuntu 22.04 replaced iptables with nftables, as described below.

nftables as the default firewall backend

Firewalling on Linux consists of two components – the firewall mechanism within the Linux kernel, and the tools used to configure this from userspace. The Linux kernel has traditionally supported two different subsystems for firewall policies – iptables / xtables and the newer nftables.

Nftables brings significant benefits both in terms of performance and flexibility when creating and deploying firewall rules, particularly for dual stack IPv4/IPv6 systems.

The traditional iptables userspace management tool now configures the nftables kernel backend, whilst the new nft userspace tool is also present to allow the creation of more flexible rules not supported by the traditional iptables paradigm.

– From What’s new in Security for Ubuntu 22.04 LTS?

Digital Ocean has a good ufw tutorial.

Using Nginx As a Reverse Proxy With SSL

2022-07-08T00:00:00-04:00

ScalaCourses.com

Recently I migrated ScalaCourses.com from AWS EC2/S3/CloudFront to a server in my apartment, which has fiber optic internet service. The server’s motherboard is an ASUS Sabertooth x79 with an Intel i7 4820, 32 GB DDR3 RAM, and a 4TB SATA SSD. This motherboard is unable to boot from NVMe drives, so SATA is necessary. At the time of this writing, the server runs Ubuntu 22.04.

The backup server is currently being set up. I am repurposing an old Hackintosh as a fallback to the Ubuntu server. That motherboard is another ASUS Sabertooth x79, with 32 GB DDR3 RAM and two 2 TB SATA drives.

Why Am I Doing This?

Once again, I control my hardware, my software, my network, and all ancillary services. After several years of using PaaS vendor servers, I am now reverting to running ScalaCourses.com on my own hardware and software, using my own network.

No longer will I subject myself to the unlimited financial liability that current PaaS vendors expose their customers to.

Definitions

A proxy is a person or process serving as an authorized agent or substitute for another. In computer science, a more specific term is forward proxy.

A proxy server is a server process that acts as an intermediary between a client requesting a resource, and the process that provides the resource.

A reverse proxy is a process that sits in front of other processes, and forwards client requests to them. The term forwarding process is similar to reverse proxy.

The definitions for proxy, forward proxy and reverse proxy all sound identical. The key difference between a reverse proxy and a forward proxy is that a forward proxy enables computers isolated on a private network to connect to the public internet, while a reverse proxy enables computers on the internet to access a private subnet.

Dealing With Details

The old Pound v2.8-2 reverse proxy that was the front end for the old Play Framework app that runs ScalaCourses.com is no longer viable, and the new version 3 of Pound is incomplete. Depending on configuration, reverse proxies can provide extra security from external attacks on a website, decrypt https requests to http, and act as stream editors for the content.

This site still uses AWS CloudFront, for the moment. I am researching alternatives, with the goal of closing my AWS account... unless they introduce a way to cap financial liability before I fully migrate off AWS.

Apache httpd vs. Nginx

Two popular reverse proxies are Apache mod_proxy_http2 and nginx. Anything Pound can do, both nginx and Apache httpd/2 can do. While they both work well, Apache httpd has an older code base, in fact, many of my websites ran on Apache httpd at the turn of the century.

Nginx is relatively newer than Apache httpd. It is performant, well supported, well documented, and widely available. I decided to use nginx as the reverse proxy because I had found that nginx worked well on previous projects.

Installing nginx and the Letsencrypt software is easy on Debian distros such as Ubuntu:

Shell

$ yes | sudo apt install nginx certbot python3-certbot-nginx

Verifying the Nginx Build

When acting as a proxy server, nginx requires the http_sub_module to translate HTTP content. This allows the website content to be stream edited, so various links and paths that are not translated properly can be fixed up.

The nginx package provided by Ubuntu includes the ‑‑with‑http_sub_module option. You can verify that your instance of nginx was built with the ‑‑with‑http_sub_module option as follows:

Shell

$ nginx -V
nginx version: nginx/1.18.0 (Ubuntu)
built with OpenSSL 3.0.2 15 Mar 2022
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -ffile-prefix-map=/build/nginx-9P0wNJ/nginx-1.18.0=.
-flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat
-Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions
-flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx
--conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log
--error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid
--modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy
--http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat
--with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module
--with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module
--with-threads --add-dynamic-module=/build/nginx-9P0wNJ/nginx-1.18.0/debian/modules/http-geoip2
--with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module
--with-http_sub_module

Nginx SSL Configuration

Two files are needed for Letsencrypt to be able to create SSL certificates for nginx:

The contents of /etc/letsencrypt/options-ssl-nginx.conf need to be stored into /etc/letsencrypt/options-ssl-nginx.conf. I later discovered this file was also available locally at /usr/lib/python3/dist-packages/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf.
The contents of /etc/letsencrypt/ssl-dhparams.pem need to be stored into /etc/letsencrypt/ssl-dhparams.pem. I later discovered that this file was also available locally at /usr/lib/python3/dist-packages/certbot/ssl-dhparams.pem.

Making a Wildcard SSL Certificate

I knew an SSL wildcard certificate would be needed, so I made one using Letsencrypt. Please read Creating and Renewing Letsencrypt Wildcard SSL Certificates for details.

Update – 8 months after writing this article, I wrote about a better way to generate wildcard SSL certificates: Wildcard SSL Certificates for Let's Encrypt with Lego.

Defining the Nginx Website Reverse Proxy

Please see my article entitled Cross-Origin Resource Sharing (CORS) for a discussion of how to configure servers whose content needs to be proxied.

I saved the following configuration in a new file called /etc/nginx/sites-available/scalacourses.com. Note that a single server block answers on ports 80 and 443, using IPv4 and IPv6, for SSL and non-SSL requests, for the apex domain (scalacourses.com) and the www.scalacourses.com subdomain. No redirects are used.

Shell

server {
  listen 80;
  listen [::]:80;
  listen 443 ssl;
  listen [::]:443 ssl;

  server_name scalacourses.com www.scalacourses.com;

  ssl_certificate         /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/fullchain.pem;
  ssl_certificate_key     /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/privkey.pem;
  ssl_trusted_certificate /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/chain.pem;
  include /etc/letsencrypt/options-ssl-nginx.conf;
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

  root /var/www/html;
  index index.html;  # This gets served if the proxied website is down

  location / {
    proxy_pass http://localhost:9000/;
    proxy_set_header Accept-Encoding "";  # sub_filter requires this

    sub_filter 'https://localhost:9000/authenticate/' 'https://www.scalacourses.com/authenticate/';
    sub_filter_once off;
  }
}

I disabled the default site:

Shell

$ sudo rm /etc/nginx/sites-enabled/default

I enabled the new scalacourses.com site:

Shell

$ sudo ln /etc/nginx/sites-{available,enabled}/scalacourses.com

The nginx configuration was tested for syntax:

Shell

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

The nginx configuration was reloaded:

Shell

$ sudo systemctl reload nginx

Flush DNS Cache

Ensure that DNS requests receive up-to-date values by flushing the DNS cache. The following works on Ubuntu, but not when running on WSL/WSL2.

Shell

$ sudo resolvectl flush-caches

The following works on Windows 10:

Command and PowerShell consoles

C:\Users\Mike Slinn> ipconfig /flushdns

Verifying nginx Works

I verified that nginx was listening on ports 80 and 443. I highlighted the nginx process number – you might need to scroll the output below to the right to see it.

Shell

$ sudo netstat -tulpn | grep ':\(443\|80\)'
tcp    0    0 0.0.0.0:80       0.0.0.0:*       LISTEN   -
tcp6   0    0 :::80            :::*            LISTEN   -
tcp    0    0 0.0.0.0:443      0.0.0.0:*       LISTEN   87487/nginx: master

The executable for process 87487 can be found by:

Shell

$ ls -l /proc/87487/exe
lrwxrwxrwx 1 mslinn mslinn 0 Jul  7 09:13 /proc/5166/exe -> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java*

If more detail is desired, get it from the process list. Enclosing a character of the process id within square brackets is an old trick for only showing the desired process.

Shell

$ ps aux | grep [8]7487
mslinn      87487  2.4  1.3 6596476 439456 ?      Sl   09:13   0:20 java -Xms1024m -Xmx1024m -Dhttp.port=9000 /path/to/jar

If there is any problem getting things to work, it is often helpful to monitor the logs as you click on the web pages:

Shell

$ sudo tail -f /var/log/nginx/*.log

Finishing Up

Make nginx start each time the system starts as follows.

Shell

$ sudo systemctl enable nginx
Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable nginx 

$ sudo update-rc.d nginx defaults

Performance Test

KeyCDN offfers free website performance statistics. The result column labeled TTFB means “time to first byte”, which is the length of time required for the website content to begin being received by a user's web browser.

I am in Montreal, Canada. Response time for TTFB reported by KeyCDN varies, from a minimum of 68ms in New York, to a maximum of 815ms in Bangalore. This range of response times is typical for websites that have a centralized process. The speed of light is finite, after all.

Free Availability Monitoring

HetrixTools offers free availability monitoring for up to 10 websites. It was quick and easy to set up.

We Are Live!

ScalaCourses.com now serves Scala students from its newly refurbished server! 😁

Trialing mslinn.com on Linode Storage

2022-07-01T00:00:00-04:00

This is another article in my ongoing saga of moving off AWS, which does not integrate security with real-time billing. I recently learned this the hard way: when my AWS account was hijacked, in less than 15 minutes, a huge bill was incurred.

“Pay-as-you-go” is shorthand for “there is nothing you can do to limit your financial liability”

The world of pain that I experienced after the breach was inflicted by broken and wasteful AWS remedial processes, and an ineffective AWS management structure. This type of issue only is enabled because of deficiencies in the AWS architecture. Those AWS architectural deficiencies feel like the result of an exploitive mindset:

Unlimited financial liability is our customer’s problem, not ours – as a result, exploits are quite profitable for us.

– From a mythical retrospective discussion at an AWS offsite.

At this moment that feature of setting limits does not exist, Azure is not able to safeguard customers from unlimited financial liability.

– From an email sent to me from Microsoft Azure support staff on 2022-06-22.

Demand Limits to Financial Liability From PaaS Vendors

PaaS vendors currently provide accounts with all services ready to go, without limit. That is good for the vendor's bottom line, but highly dangerous for their customers.

All PaaS customers should demand the ability for themselves to be able to set firm limits on budgeted expenses, along with the ability to deny all services not explicitly authorized.

You can buy insurance against losses resulting from various calamities, but you cannot limit your financial liability with PaaS vendors.

Yet.

Website Hosting Market

Recently, I have spent a lot of time looking at options for hosting websites. I found the following types of products:

	WordPress	General Web Server	VM	S3 Compatible
CDN	No	No	No	Yes
Speed	Slow to Medium	Slow to Medium	Slow to Fast	Fast
Reliability	Fair	Fair	Depends on you	Good
Financial Liability	Fixed monthly cost	Fixed monthly cost	Depends	Unlimited
Storage	Small	Small	Depends	Unlimited
CLI & API	No	No	Depends	Yes

This website requires about 220 GB of storage. It has numerous images, in 2 versions: webp and png. So for this website, ‘small’ means less than 250 GB storage.

I decided to give the S3-compatible product Linode Storage a try.

Why Linode?

First, the positive: Linode is a pioneer in virtual computing. It has been on my short list of places for hosting my pet projects for many years. Prices have always been quite competitive, products solid, with good support ... and smart people have answered the phone when I have called. Being able to easily speak with a capable human provides huge value.

On the other hand, as I shall demonstrate in this article, Linode’s documentation presents as a significant barrier to customers considering adopting their services. I had to work a lot harder than I should have to get my evaluation website up and running. Hopefully, this document is complete enough, so others can follow along and host their static websites on Linode Storage.

Acquired by Akamai

Linode was acquired by Akamai 3 months ago. Akamai is the original CDN, and their network is gigantic.

Linode does not yet offer a CDN product, but the person I spoke to at Linode when I started writing this article suggested that a CDN product from Linode based on Akamai’s network might be available soon. He also said that they were planning to working on a mechanism for limiting financial liability to their customers in 2022.

This was music to my financially risk-averse ears!

The remainder of this article will take you through all the steps necessary to:

Install and configure software tools on a WSL / WSL2 / Ubuntu computer
Make an S3-compatible bucket and set it up to hold a website
Upload the website, and easily handle mimetype issues
Generate and install a free 4096-bit SSL wildcard certificate
Get a security rating from Qualys / SSL Labs

Trialing Linode Storage

Installing s3cmd

I installed the recommended S3-compatible command-line program s3cmd on WSL2 / Ubuntu like this:

Shell

$ yes | sudo apt install s3cmd
Reading package lists... Done
  Building dependency tree... Done
  Reading state information... Done
  The following additional packages will be installed:
    python3-magic
  The following NEW packages will be installed:
    python3-magic s3cmd
  0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
  Need to get 133 kB of archives.
  After this operation, 584 kB of additional disk space will be used.
  Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-magic all 2:0.4.24-2 [12.6 kB]
  Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 s3cmd all 2.2.0-1 [120 kB]
  Fetched 133 kB in 0s (278 kB/s)
  Selecting previously unselected package python3-magic.
  (Reading database ... 169821 files and directories currently installed.)
  Preparing to unpack .../python3-magic_2%3a0.4.24-2_all.deb ...
  Unpacking python3-magic (2:0.4.24-2) ...
  Selecting previously unselected package s3cmd.
  Preparing to unpack .../archives/s3cmd_2.2.0-1_all.deb ...
  Unpacking s3cmd (2.2.0-1) ...
  Setting up python3-magic (2:0.4.24-2) ...
  Setting up s3cmd (2.2.0-1) ...
  Processing triggers for man-db (2.10.2-1) ...
  Scanning processes...
  Scanning processor microcode...
  Scanning linux images...

  Failed to retrieve available kernel versions.

  Failed to check for processor microcode upgrades.

  No services need to be restarted.

  No containers need to be restarted.

  No user sessions are running outdated binaries.

  No VM guests are running outdated hypervisor (qemu) binaries on this host.

Here is the s3cmd help message:

Shell

$ s3cmd
Usage: s3cmd [options] COMMAND [parameters]

S3cmd is a tool for managing objects in Amazon S3 storage. It allows for
making and removing "buckets" and uploading, downloading and removing
"objects" from these buckets.

Options:
-h, --help            show this help message and exit
--configure           Invoke interactive (re)configuration tool. Optionally
use as '--configure s3://some-bucket' to test access
to a specific bucket instead of attempting to list
them all.
-c FILE, --config=FILE
Config file name. Defaults to $HOME/.s3cfg
--dump-config         Dump current configuration after parsing config files
and command line options and exit.
--access_key=ACCESS_KEY
AWS Access Key
--secret_key=SECRET_KEY
AWS Secret Key
--access_token=ACCESS_TOKEN
AWS Access Token
-n, --dry-run         Only show what should be uploaded or downloaded but
don't actually do it. May still perform S3 requests to
get bucket listings and other information though (only
for file transfer commands)
-s, --ssl             Use HTTPS connection when communicating with S3.
(default)
--no-ssl              Don't use HTTPS.
-e, --encrypt         Encrypt files before uploading to S3.
--no-encrypt          Don't encrypt files.
-f, --force           Force overwrite and other dangerous operations.
--continue            Continue getting a partially downloaded file (only for
[get] command).
--continue-put        Continue uploading partially uploaded files or
multipart upload parts.  Restarts parts/files that
don't have matching size and md5.  Skips files/parts
that do.  Note: md5sum checks are not always
sufficient to check (part) file equality.  Enable this
at your own risk.
--upload-id=UPLOAD_ID
UploadId for Multipart Upload, in case you want
continue an existing upload (equivalent to --continue-
put) and there are multiple partial uploads.  Use
s3cmd multipart [URI] to see what UploadIds are
associated with the given URI.
--skip-existing       Skip over files that exist at the destination (only
for [get] and [sync] commands).
-r, --recursive       Recursive upload, download or removal.
--check-md5           Check MD5 sums when comparing files for [sync].
(default)
--no-check-md5        Do not check MD5 sums when comparing files for [sync].
Only size will be compared. May significantly speed up
transfer but may also miss some changed files.
-P, --acl-public      Store objects with ACL allowing read for anyone.
--acl-private         Store objects with default ACL allowing access for you
only.
--acl-grant=PERMISSION:EMAIL or USER_CANONICAL_ID
Grant stated permission to a given amazon user.
Permission is one of: read, write, read_acp,
write_acp, full_control, all
--acl-revoke=PERMISSION:USER_CANONICAL_ID
Revoke stated permission for a given amazon user.
Permission is one of: read, write, read_acp,
write_acp, full_control, all
-D NUM, --restore-days=NUM
Number of days to keep restored file available (only
for 'restore' command). Default is 1 day.
--restore-priority=RESTORE_PRIORITY
Priority for restoring files from S3 Glacier (only for
'restore' command). Choices available: bulk, standard,
expedited
--delete-removed      Delete destination objects with no corresponding
source file [sync]
--no-delete-removed   Don't delete destination objects [sync]
--delete-after        Perform deletes AFTER new uploads when delete-removed
is enabled [sync]
--delay-updates       *OBSOLETE* Put all updated files into place at end
[sync]
--max-delete=NUM      Do not delete more than NUM files. [del] and [sync]
--limit=NUM           Limit number of objects returned in the response body
(only for [ls] and [la] commands)
--add-destination=ADDITIONAL_DESTINATIONS
Additional destination for parallel uploads, in
addition to last arg.  May be repeated.
--delete-after-fetch  Delete remote objects after fetching to local file
(only for [get] and [sync] commands).
-p, --preserve        Preserve filesystem attributes (mode, ownership,
timestamps). Default for [sync] command.
--no-preserve         Don't store FS attributes
--exclude=GLOB        Filenames and paths matching GLOB will be excluded
from sync
--exclude-from=FILE   Read --exclude GLOBs from FILE
--rexclude=REGEXP     Filenames and paths matching REGEXP (regular
expression) will be excluded from sync
--rexclude-from=FILE  Read --rexclude REGEXPs from FILE
--include=GLOB        Filenames and paths matching GLOB will be included
even if previously excluded by one of
--(r)exclude(-from) patterns
--include-from=FILE   Read --include GLOBs from FILE
--rinclude=REGEXP     Same as --include but uses REGEXP (regular expression)
instead of GLOB
--rinclude-from=FILE  Read --rinclude REGEXPs from FILE
--files-from=FILE     Read list of source-file names from FILE. Use - to
read from stdin.
--region=REGION, --bucket-location=REGION
Region to create bucket in. As of now the regions are:
us-east-1, us-west-1, us-west-2, eu-west-1, eu-
central-1, ap-northeast-1, ap-southeast-1, ap-
southeast-2, sa-east-1
--host=HOSTNAME       HOSTNAME:PORT for S3 endpoint (default:
s3.amazonaws.com, alternatives such as s3-eu-
west-1.amazonaws.com). You should also set --host-
bucket.
--host-bucket=HOST_BUCKET
DNS-style bucket+hostname:port template for accessing
a bucket (default: %(bucket)s.s3.amazonaws.com)
--reduced-redundancy, --rr
Store object with 'Reduced redundancy'. Lower per-GB
price. [put, cp, mv]
--no-reduced-redundancy, --no-rr
Store object without 'Reduced redundancy'. Higher per-
GB price. [put, cp, mv]
--storage-class=CLASS
Store object with specified CLASS (STANDARD,
STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER
or DEEP_ARCHIVE). [put, cp, mv]
--access-logging-target-prefix=LOG_TARGET_PREFIX
Target prefix for access logs (S3 URI) (for [cfmodify]
and [accesslog] commands)
--no-access-logging   Disable access logging (for [cfmodify] and [accesslog]
commands)
--default-mime-type=DEFAULT_MIME_TYPE
Default MIME-type for stored objects. Application
default is binary/octet-stream.
-M, --guess-mime-type
Guess MIME-type of files by their extension or mime
magic. Fall back to default MIME-Type as specified by
--default-mime-type option
--no-guess-mime-type  Don't guess MIME-type and use the default type
instead.
--no-mime-magic       Don't use mime magic when guessing MIME-type.
-m MIME/TYPE, --mime-type=MIME/TYPE
Force MIME-type. Override both --default-mime-type and
--guess-mime-type.
--add-header=NAME:VALUE
Add a given HTTP header to the upload request. Can be
used multiple times. For instance set 'Expires' or
'Cache-Control' headers (or both) using this option.
--remove-header=NAME  Remove a given HTTP header.  Can be used multiple
times.  For instance, remove 'Expires' or 'Cache-
Control' headers (or both) using this option. [modify]
--server-side-encryption
Specifies that server-side encryption will be used
when putting objects. [put, sync, cp, modify]
--server-side-encryption-kms-id=KMS_KEY
Specifies the key id used for server-side encryption
with AWS KMS-Managed Keys (SSE-KMS) when putting
objects. [put, sync, cp, modify]
--encoding=ENCODING   Override autodetected terminal and filesystem encoding
(character set). Autodetected: UTF-8
--add-encoding-exts=EXTENSIONs
Add encoding to these comma delimited extensions i.e.
(css,js,html) when uploading to S3 )
--verbatim            Use the S3 name as given on the command line. No pre-
processing, encoding, etc. Use with caution!
--disable-multipart   Disable multipart upload on files bigger than
--multipart-chunk-size-mb
--multipart-chunk-size-mb=SIZE
Size of each chunk of a multipart upload. Files bigger
than SIZE are automatically uploaded as multithreaded-
multipart, smaller files are uploaded using the
traditional method. SIZE is in Mega-Bytes, default
chunk size is 15MB, minimum allowed chunk size is 5MB,
maximum is 5GB.
--list-md5            Include MD5 sums in bucket listings (only for 'ls'
command).
-H, --human-readable-sizes
Print sizes in human readable form (eg 1kB instead of
1234).
--ws-index=WEBSITE_INDEX
Name of index-document (only for [ws-create] command)
--ws-error=WEBSITE_ERROR
Name of error-document (only for [ws-create] command)
--expiry-date=EXPIRY_DATE
Indicates when the expiration rule takes effect. (only
for [expire] command)
--expiry-days=EXPIRY_DAYS
Indicates the number of days after object creation the
expiration rule takes effect. (only for [expire]
command)
--expiry-prefix=EXPIRY_PREFIX
Identifying one or more objects with the prefix to
which the expiration rule applies. (only for [expire]
command)
--progress            Display progress meter (default on TTY).
--no-progress         Don't display progress meter (default on non-TTY).
--stats               Give some file-transfer stats.
--enable              Enable given CloudFront distribution (only for
[cfmodify] command)
--disable             Disable given CloudFront distribution (only for
[cfmodify] command)
--cf-invalidate       Invalidate the uploaded filed in CloudFront. Also see
[cfinval] command.
--cf-invalidate-default-index
When using Custom Origin and S3 static website,
invalidate the default index file.
--cf-no-invalidate-default-index-root
When using Custom Origin and S3 static website, don't
invalidate the path to the default index file.
--cf-add-cname=CNAME  Add given CNAME to a CloudFront distribution (only for
[cfcreate] and [cfmodify] commands)
--cf-remove-cname=CNAME
Remove given CNAME from a CloudFront distribution
(only for [cfmodify] command)
--cf-comment=COMMENT  Set COMMENT for a given CloudFront distribution (only
for [cfcreate] and [cfmodify] commands)
--cf-default-root-object=DEFAULT_ROOT_OBJECT
Set the default root object to return when no object
is specified in the URL. Use a relative path, i.e.
default/index.html instead of /default/index.html or
s3://bucket/default/index.html (only for [cfcreate]
and [cfmodify] commands)
-v, --verbose         Enable verbose output.
-d, --debug           Enable debug output.
--version             Show s3cmd version (2.2.0) and exit.
-F, --follow-symlinks
Follow symbolic links as if they are regular files
--cache-file=FILE     Cache FILE containing local source MD5 values
-q, --quiet           Silence output on stdout
--ca-certs=CA_CERTS_FILE
Path to SSL CA certificate FILE (instead of system
default)
--ssl-cert=SSL_CLIENT_CERT_FILE
Path to client own SSL certificate CRT_FILE
--ssl-key=SSL_CLIENT_KEY_FILE
Path to client own SSL certificate private key
KEY_FILE
--check-certificate   Check SSL certificate validity
--no-check-certificate
Do not check SSL certificate validity
--check-hostname      Check SSL certificate hostname validity
--no-check-hostname   Do not check SSL certificate hostname validity
--signature-v2        Use AWS Signature version 2 instead of newer signature
methods. Helpful for S3-like systems that don't have
AWS Signature v4 yet.
--limit-rate=LIMITRATE
Limit the upload or download speed to amount bytes per
second.  Amount may be expressed in bytes, kilobytes
with the k suffix, or megabytes with the m suffix
--no-connection-pooling
Disable connection re-use
--requester-pays      Set the REQUESTER PAYS flag for operations
-l, --long-listing    Produce long listing [ls]
--stop-on-error       stop if error in transfer
--content-disposition=CONTENT_DISPOSITION
Provide a Content-Disposition for signed URLs, e.g.,
"inline; filename=myvideo.mp4"
--content-type=CONTENT_TYPE
Provide a Content-Type for signed URLs, e.g.,
"video/mp4"

Commands:
Make bucket
s3cmd mb s3://BUCKET
Remove bucket
s3cmd rb s3://BUCKET
List objects or buckets
s3cmd ls [s3://BUCKET[/PREFIX]]
List all object in all buckets
s3cmd la
Put file into bucket
s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX]
Get file from bucket
s3cmd get s3://BUCKET/OBJECT LOCAL_FILE
Delete file from bucket
s3cmd del s3://BUCKET/OBJECT
Delete file from bucket (alias for del)
s3cmd rm s3://BUCKET/OBJECT
Restore file from Glacier storage
s3cmd restore s3://BUCKET/OBJECT
Synchronize a directory tree to S3 (checks files freshness using size and md5 checksum, unless overridden by options, see below)
s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR or s3://BUCKET[/PREFIX] s3://BUCKET[/PREFIX]
Disk usage by buckets
s3cmd du [s3://BUCKET[/PREFIX]]
Get various information about Buckets or Files
s3cmd info s3://BUCKET[/OBJECT]
Copy object
s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
Modify object metadata
s3cmd modify s3://BUCKET1/OBJECT
Move object
s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
Modify Access control list for Bucket or Files
s3cmd setacl s3://BUCKET[/OBJECT]
Modify Bucket Policy
s3cmd setpolicy FILE s3://BUCKET
Delete Bucket Policy
s3cmd delpolicy s3://BUCKET
Modify Bucket CORS
s3cmd setcors FILE s3://BUCKET
Delete Bucket CORS
s3cmd delcors s3://BUCKET
Modify Bucket Requester Pays policy
s3cmd payer s3://BUCKET
Show multipart uploads
s3cmd multipart s3://BUCKET [Id]
Abort a multipart upload
s3cmd abortmp s3://BUCKET/OBJECT Id
List parts of a multipart upload
s3cmd listmp s3://BUCKET/OBJECT Id
Enable/disable bucket access logging
s3cmd accesslog s3://BUCKET
Sign arbitrary string using the secret key
s3cmd sign STRING-TO-SIGN
Sign an S3 URL to provide limited public access with expiry
s3cmd signurl s3://BUCKET/OBJECT <expiry_epoch|+expiry_offset>
Fix invalid file names in a bucket
s3cmd fixbucket s3://BUCKET[/PREFIX]
Create Website from bucket
s3cmd ws-create s3://BUCKET
Delete Website
s3cmd ws-delete s3://BUCKET
Info about Website
s3cmd ws-info s3://BUCKET
Set or delete expiration rule for the bucket
s3cmd expire s3://BUCKET
Upload a lifecycle policy for the bucket
s3cmd setlifecycle FILE s3://BUCKET
Get a lifecycle policy for the bucket
s3cmd getlifecycle s3://BUCKET
Remove a lifecycle policy for the bucket
s3cmd dellifecycle s3://BUCKET
List CloudFront distribution points
s3cmd cflist
Display CloudFront distribution point parameters
s3cmd cfinfo [cf://DIST_ID]
Create CloudFront distribution point
s3cmd cfcreate s3://BUCKET
Delete CloudFront distribution point
s3cmd cfdelete cf://DIST_ID
Change CloudFront distribution point parameters
s3cmd cfmodify cf://DIST_ID
Display CloudFront invalidation request(s) status
s3cmd cfinvalinfo cf://DIST_ID[/INVAL_ID]

For more information, updates and news, visit the s3cmd website:
http://s3tools.org

I went to the Linode signup page and signed up with my email. Using third parties for authentication means they track you more easily, and introduces an unnecessary dependency.

The email signup procedure requires a mobile phone for SMS-based MFA. I would rather use a TOTP authenticator app instead of SMS for MFA.

I want to ensure that I limit my financial liability. One way of protecting myself is to limit the maximum amount that can be charged to the payment mechanism. Because PayPal has no maximum transaction limit, it is not a suitable choice for a service with unlimited financial liability. However, credit cards can have transaction limits set, and Google Pay has the following limits:

If you set up your Google Pay balance to make contactless payments, there are some transaction limits:

Maximum single transaction amount: $2,000 USD.
Daily maximum total transaction amount: $2,500 USD.
Up to 15 transactions per day.
Additional limits on the dollar amount or frequency of transactions may be imposed in accordance with the Google Pay Terms of Service. – From Google Pay Help

Here are additional limits for Google Pay. Because of prior good experiences with how credit card processors handled fraud, I decided to use a credit card with a low limit instead of Google Pay.

Generating Linode Access Keys

I followed the directions:

Logged into the Cloud Manager.
Selected the Object Storage menu item in the sidebar and clicked on the Access Keys tab.
Clicked on the Create Access Key button, which displays the Create Access Key panel.
Typed in the name mslinn as the label for the new access key.
Clicked the Submit button to create the access key.
The new access key and its secret key are displayed. This is the only time that the secret key is visible. I stored the access key and the secret key in my password manager, lastpass.com.

Configuring s3cmd

s3cmd is a Python program that works with all S3-compatible APIs, such as those provided by AWS, Linode Storage, Google Cloud Storage and DreamHost DreamObjects. I intend to use its sync subcommand to synchronize the www.mslinn.com bucket contents in Linode Storage with the most recently generated version of this website by Jekyll.

s3cmd needs to be configured for Linode before it can be used. Some configuration settings for Linode are non-obvious. The following should work for most users, with the caution that the access key and secret key are of course unique for every user.

Shell

$ s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
  Refer to user manual for detailed description of all options.

  Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
  Access Key [asdfasdf]: asdfasdfasdf
  Secret Key [asdfasdfasdf]: asdfasdfasdf
  Default Region [US]:

  Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
  S3 Endpoint [s3.amazonaws.com]: us-east-1.linodeobjects.com

  Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
  if the target S3 system supports dns based buckets.
  DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: %(bucket)s.us-east-1.linodeobjects.com

  Encryption password is used to protect your files from reading
  by unauthorized persons while in transfer to S3
  Encryption password:
  Path to GPG program [/usr/bin/gpg]:

  When using secure HTTPS protocol all communication with Amazon S3
  servers is protected from 3rd party eavesdropping. This method is
  slower than plain HTTP, and can only be proxied with Python 2.7 or newer
  Use HTTPS protocol [Yes]:

  On some networks all internet access must go through a HTTP proxy.
  Try setting it here if you can't connect to S3 directly
  HTTP Proxy server name:

  New settings:
  Access Key: asdfasdfasdf
Secret Key: asdfasdfasdf
Default Region: US
  S3 Endpoint: https://us-east-1.linodeobjects.com
DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.us-east-1.linodeobjects.com
Encryption password:
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

  Test access with supplied credentials? [Y/n] n
  Save settings? [y/N] y
  Configuration saved to '/home/mslinn/.s3cfg'

I was unable to successfully test access. Two different errors appeared at different times:

Shell

Please wait, attempting to list all buckets...
ERROR: Test failed: 403 (SignatureDoesNotMatch)
ERROR: Test failed: [Errno -2] Name or service not known

Eventually I saved the configuration without testing as shown above. This created a file called $HOME/.s3cfg:

~/.s3cfg

[default]
access_key = asdfasdfasdf
access_token =
add_encoding_exts =
add_headers =
bucket_location = US
ca_certs_file =
cache_file =
check_ssl_certificate = True
check_ssl_hostname = True
cloudfront_host = cloudfront.amazonaws.com
connection_max_age = 5
connection_pooling = True
content_disposition =
content_type =
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encoding = UTF-8
encrypt = False
expiry_date =
expiry_days =
expiry_prefix =
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = us-east-1.linodeobjects.com
host_bucket = %(bucket)s.us-east-1.linodeobjects.com
human_readablesizes = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
kms_key =
limit = -1
limitrate = 0
list_md5 = False
log_target_prefix =
long_listing = False
max_delete = -1
mime_type =
multipart_chunksize_mb = 15
multipart_copy_chunksize_mb = 1024
multipart_max_chunks = 10000
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
public_url_use_https = False
put_continue = False
recursive = False
recv_chunk = 65536
reduced_redundancy = False
requester_pays = False
restore_days = 1
restore_priority = Standard
secret_key = asdfasdfasdf
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
signurl_use_https = False
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 300
ssl_client_cert_file =
ssl_client_key_file =
stats = False
stop_on_error = False
storage_class =
throttle_max = 100
upload_id =
urlencoding_mode = normal
use_http_expect = False
use_https = True
use_mime_magic = True
verbosity = INFO
website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
website_error =
website_index = index.html

According to the docs (Access Buckets and Files through URLs), the configuration file needs website_endpoint: http://%(bucket)s.website-[cluster-url]/. However, this is an error; instead of cluster-url, which might be https://us-east-1.linodeobjects.com, cluster-id should be used, which for me was simply us-east-1. Thus the endpoint for me, and everyone with a bucket at us-east-1 in Newark, New Jersey should be:
http://%(bucket)s.website-us-east-1.linodeobjects.com/

After editing the file to manually change the value of website_endpoint, I tried to list the buckets, and that worked:

Shell

$ s3cmd ls
2022-07-01 17:32  s3://mslinn

linode-cli

Next I tried linode-cli.

Shell

$ pip install linode-cli
Collecting linode-cli
Downloading linode_cli-5.21.0-py2.py3-none-any.whl (204 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 204.2/204.2 kB 4.8 MB/s eta 0:00:00
Collecting terminaltables
Downloading terminaltables-3.1.10-py2.py3-none-any.whl (15 kB)
Collecting PyYAML
Downloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 682.2/682.2 kB 16.8 MB/s eta 0:00:00
Requirement already satisfied: requests in /home/mslinn/venv/default/lib/python3.10/site-packages (from linode-cli) (2.28.0)
Requirement already satisfied: charset-normalizer~=2.0.0 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests->linode-cli) (2.0.12)
Requirement already satisfied: idna<4,>=2.5 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests->linode-cli) (3.3)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests->linode-cli) (1.26.9)
Requirement already satisfied: certifi>=2017.4.17 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests->linode-cli) (2022.6.15)
Installing collected packages: terminaltables, PyYAML, linode-cli
Successfully installed PyYAML-6.0 linode-cli-5.21.0 terminaltables-3.1.10

The documentation fails to mention that the first time linode-cli is executed, and every time it is invoked with the configure subcommand, it attempts to open a web browser. WSL and WSL2 will not respond appropriately unless an X client is already running, for example GWSL.

Shell

$ linode-cli configure

Welcome to the Linode CLI.
This will walk you through some initial setup.

The CLI will use its web-based authentication to log you in.
If you prefer to supply a Personal Access Token, use
linode-cli configure --token

Press enter to continue.
This will open a browser and proceed with authentication.
A browser should open directing you to this URL to authenticate:

https://login.linode.com/oauth/authorize?client_id=asdfasdfasdf&response_type=token&scopes=*&redirect_uri=http://localhost:45789

If you are not automatically directed there, please copy/paste the link into your browser
to continue..


Configuring mslinn


Default Region for operations.  Choices are:
1 - ap-west
2 - ca-central
3 - ap-southeast
4 - us-central
5 - us-west
6 - us-southeast
7 - us-east
8 - eu-west
9 - ap-south
10 - eu-central
11 - ap-northeast

Default Region (Optional): 7

Default Type of Linode to deploy.  Choices are:
1 - g6-nanode-1
2 - g6-standard-1
3 - g6-standard-2
4 - g6-standard-4
5 - g6-standard-6
6 - g6-standard-8
7 - g6-standard-16
8 - g6-standard-20
9 - g6-standard-24
10 - g6-standard-32
11 - g7-highmem-1
12 - g7-highmem-2
13 - g7-highmem-4
14 - g7-highmem-8
15 - g7-highmem-16
16 - g6-dedicated-2
17 - g6-dedicated-4
18 - g6-dedicated-8
19 - g6-dedicated-16
20 - g6-dedicated-32
21 - g6-dedicated-48
22 - g6-dedicated-50
23 - g6-dedicated-56
24 - g6-dedicated-64
25 - g1-gpu-rtx6000-1
26 - g1-gpu-rtx6000-2
27 - g1-gpu-rtx6000-3
28 - g1-gpu-rtx6000-4

Default Type of Linode (Optional):

Default Image to deploy to new Linodes.  Choices are:
1 - linode/almalinux8
2 - linode/almalinux9
3 - linode/alpine3.12
4 - linode/alpine3.13
5 - linode/alpine3.14
6 - linode/alpine3.15
7 - linode/alpine3.16
8 - linode/arch
9 - linode/centos7
10 - linode/centos-stream8
11 - linode/centos-stream9
12 - linode/debian10
13 - linode/debian11
14 - linode/debian9
15 - linode/fedora34
16 - linode/fedora35
17 - linode/fedora36
18 - linode/gentoo
19 - linode/kali
20 - linode/debian11-kube-v1.20.15
21 - linode/debian9-kube-v1.20.7
22 - linode/debian9-kube-v1.21.1
23 - linode/debian11-kube-v1.21.12
24 - linode/debian9-kube-v1.22.2
25 - linode/debian11-kube-v1.22.9
26 - linode/debian11-kube-v1.23.6
27 - linode/opensuse15.3
28 - linode/opensuse15.4
29 - linode/rocky8
30 - linode/slackware14.2
31 - linode/slackware15.0
32 - linode/ubuntu16.04lts
33 - linode/ubuntu18.04
34 - linode/ubuntu20.04
35 - linode/ubuntu21.10
36 - linode/ubuntu22.04
37 - linode/centos8
38 - linode/slackware14.1
39 - linode/ubuntu21.04

Default Image (Optional):
Active user is now mslinn

Config written to /home/mslinn/.config/linode-cli

linode-cli configuration created this file:

~/.config/linode-cli

[DEFAULT]
default-user = mslinn

[mslinn]
token = asdfasdfasdfasdfasdfasdf
region = us-east

Right after I did the above, I was surprised to get the following email from Linode:

Hello,

This is a notification to inform you that a device has been trusted to skip authentication on your Linode Account (mslinn). The request came from the following IP address: 70.53.179.203.

The device will not be prompted for a username or password for 30 days.

If this action did not originate from you, we recommend logging in and changing your password immediately. Also, as an extra layer of security, we highly recommend enabling two-factor authentication on your account. Please see the link below for more information on how to enable two-factor authentication:

Two-Factor Authentication

If you have any questions or concerns, please do not hesitate to contact us 24/7 by opening a support ticket from within the Linode Manager, giving us a call, or emailing support@linode.com.

Contact

Thank you,
The Linode.com Team

Contact Us

I began to suspect that Linode’s documentation had major deficiencies at this point. Instead of merely configuring a command-line client, my IP was whitelisted unexpectedly. I have nothing polite to say about this.

After poking around a bit, I discovered my API Tokens page on Linode. It seems that the linode-cli created a personal access token with a label containing the name of the computer used: Linode CLI @ camille.

Clicking on the View Scopes button above displays the permissions granted to the token:

This personal access token is all-powerful. The linode-cli quietly created a personal access token with all permissions enabled, and did not warn the user, and did not say how to reduce or limit the permissions.

Security is enhanced when the minimum permission necessary is provided to accomplish necessary tasks. Instead, this token silently maximizes the financial risk to the person or entity who pays for the account.

Creating the Website Bucket

The name of the bucket must exactly match the name of the subdomain that the website is served from. If you try to serve content from a misnamed bucket, you will not be able to apply an SSL certificate; instead, the certificate will be issued by linodeobjects.com.

I wanted to test using the subdomain linode.mslinn.com, and then if all was well, switch www.mslinn.com over to Linode Storage. Because buckets cannot be renamed, I had to make 2 buckets with identical contents, one called linode.mslinn.com and the other called www.mslinn.com. If I decide to stay with Linode, I will delete the bucket I am using for testing, called linode.mslinn.com, and run this website from the www.mslinn.com bucket.

I created a website-enabled bucket called www.mslinn.com as follows:

Shell

$ s3cmd mb --acl-public s3://www.mslinn.com
Bucket 's3://www.mslinn.com/' created 

$ s3cmd ls
2022-07-01 17:32  s3://www.mslinn.com 

$ s3cmd ws-create \
  --ws-index=index.html \
  --ws-error=404.html \
  s3://www.mslinn.com
Bucket 's3://www.mslinn.com/': website configuration created.

The s3cmd ws-info subcommand displays the Linode Object Storage bucket URL as the Website endpoint:

Shell

$ s3cmd ws-info s3://www.mslinn.com
Bucket s3://www.mslinn.com/: Website configuration
  Website endpoint: http://www.mslinn.com.website-us-east-1.linodeobjects.com/
  Index document:   index.html
  Error document:   404.html

Defining a CNAME for the Bucket

Although Linode provides a DNS Manager, while I am testing out Linode I wanted to continue using Namecheap for DNS, so I navigated to ap.www.namecheap.com/Domains/DomainControlPanel/mslinn.com/advancedns and created CNAMEs like this:

Shell

linode.mslinn.com  CNAME  linode.mslinn.com.website-us-east-1.linodeobjects.com
www.mslinn.com     CNAME  www.mslinn.com.website-us-east-1.linodeobjects.com

Warning: if the CNAME does not point to a URL with the word website in it, for example www.mslinn.com.us-east-1.linodeobjects.com, the default index file and the error file will not automatically be served when expected.

Making a Free SSL Certificate

Qualys / SSL Labs rates the SSL security provided by AWS CloudFront using the AWS-generated SSL certificates with a “B” grade.

Lets see if Linode Object Storage can host a more secure website when provided with a 4096-bit certificate generated by certbot/letsencrypt. The conclusion of this article will reveal the answer.

Please read Creating and Renewing Letsencrypt Wildcard SSL Certificates for details.

Linode DNS Authentication

Eventually, if I stick with Linode, I could use the certbot DNS plugin for Linode, but I will not use it right now because I am trialing Linode. Use of this plugin requires a configuration file containing Linode API credentials, obtained from your Linode account’s Applications & API Tokens page. I stored the credentials in ~/.certbot/linode.ini:

~/.certbot/linode.ini

# Linode API credentials used by Certbot
dns_linode_key = 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ64
dns_linode_version = [|3|4]

Here is how to create an SSL certificate certbot using Linode DNS authentication:

Shell


$ certbot certonly \
  --dns-linode \
  --dns-linode-credentials ~/.certbot/linode.ini \
  --rsa-key-size 4096 \
  -d mslinn.com \
  -d *.mslinn.com \
  --config-dir ~/.certbot/mslinn.com/config \
  --logs-dir ~/.certbot/mslinn.com/logs \
  --work-dir ~/.certbot/mslinn.com/work

Installing the Certificate

Now I created 2 certificates: one with the full chain of responsibility (fullchain.pem) and one containing my private key (privkey.pem). I used bash environment variables called CERT_DIR, CERT and KEY to make the incantation convenient to type in. CERT and KEY contain the contents of the files fullchain.pem and privkey.pem, respectively. As you can see, the result was a valid (wildcard) SSL certificate.

Shell

$ CERT_DIR=/home/mslinn/.certbot/mslinn.com/config/live/mslinn.com

$ CERT="$( cat $CERT_DIR/fullchain.pem )"

$ KEY="$( cat $CERT_DIR/privkey.pem )"

$ linode-cli object-storage ssl-upload \
  us-east-1 www.mslinn.com \
  --certificate "$CERT" \
  --private_key "$KEY"
┌──────┐
│ ssl  │
├──────┤
│ True │
└──────┘

Syncing the Website Bucket

I wrote a bash script to build this Jekyll-generated website. Jekyll places the generated website in a directory called _site. I synchronized the contents of the freshly generated mslinn.com website, stored in _site/, with the bucket as follows. Note the trailing slash on _site/, it is significant.

Shell

$ s3cmd sync \
  --acl-public --delete-removed --guess-mime-type --quiet \
  _site/ \
  s3://www.mslinn.com
INFO: No cache file found, creating it.
INFO: Compiling list of local files...
INFO: Running stat() and reading/calculating MD5 values on 2157 files, this may take some time...
INFO: [1000/2157]
INFO: [2000/2157]
INFO: Retrieving list of remote files for s3://www.mslinn.com/_site ...
INFO: Found 2157 local files, 0 remote files
INFO: Verifying attributes...
INFO: Summary: 2137 local files to upload, 20 files to remote copy, 0 remote files to delete
... lots more output ...
Done. Uploaded 536630474 bytes in 153.6 seconds, 3.33 MB/s.

Originally, I did not provide the --acl-public option to s3cmd sync. That meant the files in the bucket were all private – the opposite of what is required for a website. Here is the incantation for making all the files in the bucket publicly readable:

Shell

$ s3cmd setacl s3://www.mslinn.com/ \
  --acl-public --recursive --quiet

Linode’s S3 implementation faithfully mirrors a problem that AWS S3 has. CSS files are automatically assigned the MIME type text/plain, instead of text/css.

Shell

$ curl -sI http://linode.mslinn.com/assets/css/style.css | \
  grep Content-Type
Content-Type: text/plain

I re-uploaded the CSS files with the proper mime type using this incantation:

Shell

$ cd _site

$ find . -name *.css -exec \
  s3cmd put -P --mime-type=text/css {} s3://www.mslinn.com/{} \;

It should be possible to provide a MIME-type mapping somehow! Linode, if you are reading this, please provide an extension to the S3 functionality, so customers could go beyond AWS's bugs.

The Result

Success! The same content can be fetched with 3 different URLs:

https://linode.mslinn.com uses the free wildcard certificate that was created above.
http://linode.mslinn.com.website-us-east-1.linodeobjects.com works without an SSL certificate.
https://linode.mslinn.com.website-us-east-1.linodeobjects.com uses a certificate from us-east-1.linodeobjects.com, which shows as an error, as it should. The error would be prevented if the certificate was a wildcard certificate issued by linodeobjects.com instead.

Qualys / SSL Labs rated the site on Linode Storage with the Let’s Encrypt SSL certificate: and gave it the same rating as when the site was hosted on AWS S3 / CloudFront.

BTW, you can test the date range of a certificate with this incantation:

Shell

$ curl https://linode.mslinn.com -vI --stderr - | grep 'date:'
*  start date: Nov 25 17:46:07 2022 GMT
*  expire date: Feb 23 17:46:06 2023 GMT

The site feels quite responsive. 😁

Considering Microsoft Azure for Static Websites

2022-06-20T00:00:00-04:00

After realizing that AWS does not integrate security with real-time billing, and being presented with a huge bill after my account was hijacked, I decided to see if Microsoft Azure offered me better financial security.

In particular, I am looking for two things:

A more secure authentication mechanism – If AWS credentials are compromised, they can be used from anywhere in the world.
Restricting scope and scale of authorized services – With AWS, if an IAM user has a role that allows EC2 instances to be launched, any number of any size EC2 instances can be launched. There is no way to cap the number or type of EC2 instances, and AWS real-time billing is not integrated with the authentication mechanism. Furthermore, AWS has a peculiar set of busy work tasks for victims of account hijacking that appears to be targeted more towards increasing support revenue than address root issues. This means that the bad guys have no limit to the financial damage they can incur on their victims.

For PaaS vendors such as AWS, Azure, Digital Ocean, Cloudflare, ScaleWay, etc: “pay-as-you-go” is shorthand for “there is nothing you can do to limit your financial liability”.

I set out to discover if Azure also suffers from these same problems.

Azure Roles and Policies

Policies

Azure policies can be used to define the desired behavior for your organization's Windows VMs and Linux VMs. By using policies, an organization can enforce various conventions and rules throughout the enterprise. Enforcement of the desired behavior can help mitigate risk while contributing to the success of the organization.

Azure role-based access control

Using Azure role-based access control (Azure RBAC), you can segregate duties within your team and grant only the amount of access to users on your VM that they need to perform their jobs. Instead of giving everybody unrestricted permissions on the VM, you can allow only certain actions. You can configure access control for the VM in the Azure portal, using the Azure CLI, or Azure PowerShell.

– From Azure Virtual Machines Documentation

Ekran Systems publishes a good description of various flavors of RBAC, and the more complex ABAC. Okta, Inc. also publishes a good overview of RBAC and ABAC. I have no connection with either company.

I soon discovered the following passage in the Azure documentation, which appeared to exactly match the second item in the wish list at the top of this article:

Azure RBAC focuses on managing user actions at different scopes. If control of an action is required, then Azure RBAC is the correct tool to use. Even if an individual has access to perform an action, if the result is a non-compliant resource, Azure Policy still blocks the create or update.

The combination of Azure RBAC and Azure Policy provides full scope control in Azure. – From Azure documentation

AWS RBAC does not provide functionality equivalent to that provided by Microsoft Azure RBAC plus Azure Policy, even when combined with other AWS functionality.

No Limits On Other Azure Services

The financial limitations that Azure allows you to impose are only available for virtual machines. Users are subject to unlimited financial liability from other Azure services. It seems I have ~~two~~three choices:

Figure out how to set up RBAC and accept that other services might run up a huge bill.
Look for a fixed-price hosting service
Give Azure Spending Limit a try

Before I sign off today, I encountered a blocking issue with Azure that I'd like to mention.

Active Directory?

I want to update my website by running a command-line command that runs something analogous to rsync. I spent quite some time looking for how this might be done with Azure.

Before long, it seemed that Active Directory was the best way forward, however setting it up is daunting. Apparently a service principal is better suited for scripts that run on-premises, like my home office. I got the impression that I need to register an app; unsure what app might be required for hosting a web site on Azure Blob Storage with Azure CDN.

These Active Directory docs look like a lot of abstract, generalized information that is way more complex than I need. I found a streamlined article for my use case. Not sure what this means: “Only storage accounts created with the Azure Resource Manager deployment model support Azure AD authorization.” This streamlined document is not very streamlined. For me, this issue is a significant barrier to adoption.

Update: Azure Spending Limit

I just bumped into an article entitled Azure spending limit. Perhaps this might be useful. I tried to find out more from Microsoft, but at first they said I would have to subscribe to an expensive support plan before anyone would speak to me. That is not how I wish to proceed. Later, I opened a ticket inquiring about Azure customers facing unlimited financial liability.

Update 2022-06-29

The ticket was escalated twice, then Microsoft sent me the following official response (emphasis is mine):

Correct, at this moment that feature of setting limits does not exist, Azure is not able to safeguard customers from unlimited financial liability.

We are a support team that we provide help with current account or billing issues, in addition to that, I would like to mention that Azure products are constantly being updated, also Azure is constantly improving to offer better services to our customers. For this reason, I would like to invite you to leave your feedback in the Azure feedback forum (General Feedback · Community (azure.com)), so your feedback is forwarded through the right channel and to the Microsoft internal resources that work on it.

I responded by indicating that I would close my Azure account.

While the practice of imposing unlimited financial liability on customers is common today for PaaS vendors, it is unnecessary and predatory. If in the future I need to use a product or service that incurs unlimited financial liability for a client, I will require the client to accept the liability.

Creating and Renewing Letsencrypt Wildcard SSL Certificates

2022-06-15T00:00:00-04:00

The process of making wildcard SSL certificates can be hard to get right at first. However, I use wildcard SSL certificates for most of my internet domains, so it is important to me. This article distills the essence of what I have found to be important when creating these certificates.

Update – 9 months after writing this article, I wrote about a better way to generate wildcard SSL certificates: Wildcard SSL Certificates for Letsencrypt with Lego.

Making a Free Wildcard SSL Certificate

Certbot powers EFF’s Letsencrypt capability. Source code is on GitHub.

Install certbot on Debian distros such as Ubuntu as follows:

Shell

$ yes | sudo apt install certbot
Preparing to unpack .../06-python3-zope.event_4.4-3_all.deb ...
  Unpacking python3-zope.event (4.4-3) ...
  Selecting previously unselected package python3-zope.component.
  Preparing to unpack .../07-python3-zope.component_4.3.0-3_all.deb ...
  Unpacking python3-zope.component (4.3.0-3) ...
  Selecting previously unselected package python3-certbot.
  Preparing to unpack .../08-python3-certbot_1.21.0-1build1_all.deb ...
  Unpacking python3-certbot (1.21.0-1build1) ...
  Selecting previously unselected package python3-icu.
  Preparing to unpack .../09-python3-icu_2.8.1-0ubuntu2_amd64.deb ...
  Unpacking python3-icu (2.8.1-0ubuntu2) ...
  Selecting previously unselected package certbot.
  Preparing to unpack .../10-certbot_1.21.0-1build1_all.deb ...
  Unpacking certbot (1.21.0-1build1) ...
  Setting up python3-configargparse (1.5.3-1) ...
  Setting up python3-requests-toolbelt (0.9.1-1) ...
  Setting up python3-parsedatetime (2.6-2) ...
  Setting up python3-icu (2.8.1-0ubuntu2) ...
  Setting up python3-zope.event (4.4-3) ...
  Setting up python3-zope.hookable (5.1.0-1build1) ...
  Setting up python3-josepy (1.10.0-1) ...
  Setting up python3-zope.component (4.3.0-3) ...
  Setting up python3-acme (1.21.0-1) ...
  Setting up python3-certbot (1.21.0-1build1) ...
  Setting up certbot (1.21.0-1build1) ...
  Created symlink /etc/systemd/system/timers.target.wants/certbot.timer → /lib/systemd/system/certbot.timer.
  Processing triggers for man-db (2.10.2-1) ...
  Scanning processes...
  Scanning processor microcode...
  Scanning linux images...

  Failed to retrieve available kernel versions.

  Failed to check for processor microcode upgrades.

  No services need to be restarted.

  No containers need to be restarted.

  No user sessions are running outdated binaries.

  No VM guests are running outdated hypervisor (qemu) binaries on this host.

certbot has 2 subcommands of interest: certonly (used when creating a certificate for the first time), and renew (used when updating a pre-existing certificate). The creation process requires the user to do things before it can complete, so it can only be run interactively.

The files and directories creating by the process of creating a new SSL certificate should not be deleted. Contrary to what the Letsencrypt certbot documentation says, I have found that these files and directories allow the renewal process to proceed without requiring user interaction, so it can be scripted.

Certbot Help Messages

Here is the help message for the certbot certonly subcommand:

Shell

$ certbot certonly --help
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...

  Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
  it will attempt to use a webserver both for obtaining and installing the
  certificate. The most common SUBCOMMANDS and flags are:

  obtain, install, and renew certificates:
  (default) run   Obtain & install a certificate in your current webserver
  certonly        Obtain or renew a certificate, but do not install it
  renew           Renew all previously obtained certificates that are near
  expiry
  enhance         Add security enhancements to your existing configuration
  -d DOMAINS       Comma-separated list of domains to obtain a certificate for

  (the certbot apache plugin is not installed)
  --standalone      Run a standalone webserver for authentication
  (the certbot nginx plugin is not installed)
  --webroot         Place files in a server's webroot folder for authentication
  --manual          Obtain certificates interactively, or using shell script
  hooks

  -n               Run non-interactively
  --test-cert       Obtain a test certificate from a staging server
  --dry-run         Test "renew" or "certonly" without saving any certificates
  to disk

  manage certificates:
  certificates    Display information about certificates you have from Certbot
  revoke          Revoke a certificate (supply --cert-name or --cert-path)
  delete          Delete a certificate (supply --cert-name)

  manage your account:
  register        Create an ACME account
  unregister      Deactivate an ACME account
  update_account  Update an ACME account
  --agree-tos       Agree to the ACME server's Subscriber Agreement
  -m EMAIL         Email address for important account notifications

  More detailed help:

  -h, --help [TOPIC]    print this message, or detailed help on a topic;
  the available TOPICS are:

  all, automation, commands, paths, security, testing, or any of the
  subcommands or plugins (certonly, renew, install, register, nginx,
  apache, standalone, webroot, etc.)
  -h all                print a detailed help page including all topics
  --version             print the version number
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Here is the help message for the certbot certonly subcommand:

Shell

$ certbot renew --help
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...

  Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
  it will attempt to use a webserver both for obtaining and installing the
  certificate. The most common SUBCOMMANDS and flags are:

  obtain, install, and renew certificates:
  (default) run   Obtain & install a certificate in your current webserver
  certonly        Obtain or renew a certificate, but do not install it
  renew           Renew all previously obtained certificates that are near
  expiry
  enhance         Add security enhancements to your existing configuration
  -d DOMAINS       Comma-separated list of domains to obtain a certificate for

  (the certbot apache plugin is not installed)
  --standalone      Run a standalone webserver for authentication
  (the certbot nginx plugin is not installed)
  --webroot         Place files in a server's webroot folder for authentication
  --manual          Obtain certificates interactively, or using shell script
  hooks

  -n               Run non-interactively
  --test-cert       Obtain a test certificate from a staging server
  --dry-run         Test "renew" or "certonly" without saving any certificates
  to disk

  manage certificates:
  certificates    Display information about certificates you have from Certbot
  revoke          Revoke a certificate (supply --cert-name or --cert-path)
  delete          Delete a certificate (supply --cert-name)

  manage your account:
  register        Create an ACME account
  unregister      Deactivate an ACME account
  update_account  Update an ACME account
  show_account    Display account details
  --agree-tos       Agree to the ACME server's Subscriber Agreement
  -m EMAIL         Email address for important account notifications

  More detailed help:

  -h, --help [TOPIC]    print this message, or detailed help on a topic;
  the available TOPICS are:

  all, automation, commands, paths, security, testing, or any of the
  subcommands or plugins (certonly, renew, install, register, nginx,
  apache, standalone, webroot, etc.)
  -h all                print a detailed help page including all topics
  --version             print the version number
  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Complete documentation for certbot is here.

Generic DNS Authentication

I used certbot to create a special file within the website as follows, without using sudo. I want a wildcard SSL certificate, which requires certbot to use DNS authentication.

BTW, the following options include two -d options, one for the domain apex (mslinn.com) and one for the subdomains (*.mslinn.com)

Shell

$ certbot certonly \
  --manual \
  --agree-tos \
  --preferred-challenges dns-01 \
  --rsa-key-size 4096 \
  -d mslinn.com -d *.mslinn.com \
  --config-dir ~/.certbot/mslinn.com/config \
  --logs-dir ~/.certbot/mslinn.com/logs \
  --work-dir ~/.certbot/mslinn.com/work
Saving debug log to /home/mslinn/.certbot/mslinn.com/logs/letsencrypt.log
Enter email address (used for urgent renewal and security notices)
(Enter 'c' to cancel):  mslinn@mslinn.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: n
Account registered.
Requesting a certificate for mslinn.com and *.mslinn.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please deploy a DNS TXT record under the name:

_acme-challenge.mslinn.com.

with the following value:

n6o3qMw5N7qxAzV4uNQePKxJjaw0f-Bo32CybWr-lhE

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue
(This must be set up in addition to the previous challenges; do not remove,
replace, or undo the previous challenge tasks yet. Note that you might be
asked to create multiple distinct TXT records with the same name. This is
permitted by DNS standards.)

Before continuing, verify the TXT record has been deployed. Depending on the DNS
provider, this may take some time, from a few seconds to multiple minutes. You can
check if it has finished deploying with aid of online tools, such as the Google
Admin Toolbox: https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.mslinn.com.
Look for one or more bolded line(s) below the line ';ANSWER'. It should show the
value(s) you’ve just added.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue
Successfully received certificate.
Certificate is saved at: /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/fullchain.pem
Key is saved at:         /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/privkey.pem
This certificate expires on 2022-09-29.
These files will be updated when the certificate renews.

NEXT STEPS:
- This certificate will not be renewed automatically.
Autorenewal of --manual certificates requires the use of an authentication hook script
(--manual-auth-hook) but one was not provided.
To renew this certificate, repeat this same certbot command before the certificate’s expiry date.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let’s Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

I made a bash script (~/.local/bin/sslCert) to create SSL certificates, which performs the same steps as the above, but it works for any site, and can renew certificates as well as create them:

~/.local/bin/sslCert

#!/bin/bash

function help {
  echo "Creates or updates an wildcard SSL certificate

Usage: $(basename $0) DOMAIN

Example: $(basename $0) scalacourses.com

The relevant files are stored in ~/.certbot/DOMAIN/

Creating a new certificate requires an interactive user session;
updating a certificate can be done in a cron job.

Currently, the renew verb is capable of either renewing all installed
certificates that are due to be renewed or renewing a single certificate
specified by its name. If you would like to renew specific certificates
by their domains, use the certonly command instead.

The renew verb may provide other options for selecting certificates to
renew in the future.

Ask for help or search for solutions at https://community.letsencrypt.org
See the logfile /home/mslinn/.certbot/ancientwarmth.com/logs/letsencrypt.log
or re-run Certbot with -v for more details.


BTW, you can test the date range of the certificate with this incantation:
  curl https://domain.com -vI --stderr - | grep 'date:'
"
  exit 1
}

if [ -z "$1" ]; then help; fi

DOMAIN="$1"
if [ -d "$HOME/.certbot/$DOMAIN" ]; then # Renew existing certificate
  CMD=certonly # renew
  OPTIONS=--force-renew
else # Make new certificate (interactively)
  CMD=certonly
  unset OPTIONS
fi
certbot "$CMD" $OPTIONS \
  --agree-tos \
  --config-dir "$HOME/.certbot/$DOMAIN/config" \
  -d "$DOMAIN" -d "*.$DOMAIN" \
  --email mslinn@mslinn.com \
  --logs-dir "$HOME/.certbot/$DOMAIN/logs" \
  --manual \
  --preferred-challenges dns-01 \
  --rsa-key-size 4096 \
  --work-dir "$HOME/.certbot/$DOMAIN/work"

Here is the help message for the script:

Shell

$ sslCert
Creates or updates an wildcard SSL certificate

Usage: sslCert DOMAIN

Example: sslCert scalacourses.com

The relevant files are stored in ~/.certbot/DOMAIN/

Creating a new certificate requires an interactive user session;
updating a certificate can be done in a cron job.

The following crontab entry causes the script to run every 60 days.

crontab

0 0 1 */2 * .local/bin/sslCert mslinn.com

The next time the script was run, output looked like:

Shell

$ sslCert mslinn.com
Saving debug log to /home/mslinn/.certbot/mslinn.com/logs/letsencrypt.log
Renewing an existing certificate for mslinn.com and *.mslinn.com

Successfully received certificate.
Certificate is saved at: /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/fullchain.pem
Key is saved at:         /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/privkey.pem
This certificate expires on 2022-12-13.
These files will be updated when the certificate renews.

NEXT STEPS:
- This certificate will not be renewed automatically.
  Autorenewal of --manual certificates requires the use of an authentication hook script (--manual-auth-hook)
  but one was not provided. To renew this certificate, repeat this same certbot command before the
  certificate's expiry date.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
* Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
* Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

😁

Checking the Certificate With A Web Browser

Microsoft Windows caches SSL certificates, which can prevent your web browser from detecting newly updated SSL certificates. To clear the Windows SSL cache:

Press the Windows key, type Internet Options, and press Enter.
Select the Content tab.
Click the Clear SSL state button.
Click the OK button.

Considering Cloudflare R2 for Static Websites

2022-06-09T00:00:00-04:00

After deciding to close my AWS account, I started looking for alternatives to host my static websites. Without getting into my selection criteria, my short list of possible hosting companies was Microsoft Azure, Cloudflare, Digital Ocean, Linode, and Netlify. Many other options exist.

Cloudflare has a free tier that has no time limit. It includes an S3 work-alike called R2, SSL and a built-in world-wide CDN that works automatically. 250 GB of storage and 1 TB/month of transfer are provided at no charge, forever. There are also no ingress or egress charges. Cloudflare’s edge network now spans 275 cities around the world, with nearly all Internet users within 50 milliseconds of a Cloudflare server.

This article documents my experience with Cloudflare. If you don't care about details, and want to know my verdict on Cloudflare R2, skip to the end.

For PaaS vendors such as AWS, Azure, Digital Ocean, Cloudflare, ScaleWay, etc.: “pay-as-you-go” is shorthand for “there is nothing you can do to limit your financial liability”.

This is what I did

After creating an account, I followed the directions at R2 get started guide.

Wrangler

The directions told me to set up Wrangler v2, a command-line interface for transferring files to R2.

Shell

$ npm install -g wrangler
npm WARN config global `--global`, `--local` are deprecated. Use `--location=global` instead.
npm WARN deprecated rollup-plugin-inject@3.0.2: This package has been deprecated and is no longer maintained. Please use @rollup/plugin-inject.

added 56 packages, and audited 57 packages in 8s

10 high severity vulnerabilities

To address issues that do not require attention, run:
npm audit fix

To address all issues, run:
npm audit fix --force

Run `npm audit` for details.

The above messages do not inspire confidence. Wrangler is written using Node. I believe that Node has more security issues than any other computer language, particularly in package management. The OWasp recommendations do not address the fundamental security vulnerabilities in the Node package management infrastructure.

RClone

I looked for alternatives to Wrangler and found RClone. RClone is a command-line program to manage files on cloud storage. It has many subcommands, including two types of sync. Although the RClone documentation does not mention Cloudflare, the Cloudflare docs described how to set up RClone.

Limits

Platform limits are important. Not only are the technical limits important for defining inputs and outputs, users should be particularly interested in spend limits, so they are not subject to unlimited financial liability.

Workers Paid plan is separate from any other Cloudflare plan (Free, Professional, Business) you may have. If you are an Enterprise customer, reach out to your account team to confirm pricing details.

Only requests that hit a Worker will count against your limits and your bill.

– From Cloudflare Workers Pricing Fine Print

Setup

Caching

Cache control.

Pages

> Cloudflare Pages is a JAMstack platform for frontend developers to collaborate and deploy websites.

– From pages.cloudflare.com

Cloudflare Pages supports Jekyll sites. However, it looks like Cloudflare Pages builds the site in the cloud. While this might be a useful mechanism for many, my Jekyll builds need to access my local machine, and use my Jekyll plugins.

Cloudflare Workers Sites suits my use case. The Start From Existing documentation looks appropriate, except it is written for using Wrangler, which I view as a security threat.

The Verdict: No to Cloudflare

CloudFlare does not offer a spend limit for accounts on paid plans. This is unacceptable. I tried to remove my credit card, but found I could not. I then deleted my user account, and saw:

It could take up to 12 months to delete your information completely.

There is no way to frame this as an example of how Cloudflare is looking out for the best interests of their customers.

Upgrading PostgreSQL Ubuntu

2022-05-28T00:00:00-04:00

This article describes my recommended steps to follow when upgrading a Postgres database forced upon you when upgrading Ubuntu, and the new OS includes a new release of Postgres.

I use PostgreSQL as the database for ScalaCourses.com. The site is served from an Ubuntu instance, and there is a backup instance.

When upgrading to Ubuntu 22.04, PostgreSQL upgraded from v13 to v14; I used pg_upgrade, which required temporarily swapping the ports that Postgres used.

However, I used pg_upgradecluster when upgrading to Ubuntu 23.04; and PostgreSQL upgraded from v14 to v15. Unfortunately, the documented procedure to use pg_upgradecluster has no consideration for systemd, and this makes the upgrade slightly more awkward than it might otherwise be.

While both approaches have issues, I found that it is easier and faster to upgrade a Postgres installation using pg_upgradecluster than using pg_upgrade.

Upgrade Backup System First

One of the benefits of a backup system is that you can work through upgrade problems on the backup, before doing the same with the live instance.

This article suggests to upgrade the entire system to Ubuntu 23.04, then upgrade the active Postgres database cluster. The alternative would be to add a new PPA for Postgres, and just update that program before upgrading the entire system; this merely substitutes one set of potential issues for another. The article also demonstrates renaming the extra database cluster that the Postgres upgrade creates, but that just creates cruft since it is not required for anything.

I decided to delete the new database cluster instead, since upgrading the old database cluster is all that is required. As you will read in the Upgrade Procedure section, this is also the advice given during the upgrade procedure.

I decided to first upgrade the Ubuntu OS, then migrate the Postgres database cluster.

Upgrade Removed NVidia Support

After running do-release upgrade on the production system, the system worked fine, except the console had no graphics. I fixed it by automatically installing all video drivers:

Shell

$ sudo ubuntu-drivers autoinstall

Examining the Postgres Databases

pg_lsclusters displays the available PostgreSQL database clusters. Prior to upgrading to Postgres 15, I had never deleted the old versions. They had quietly accumulated over the 11 years that I had been running ScalaCourses:

Shell

$ pg_lsclusters
Ver Cluster Port Status Owner    Data directory               Log file
9.5 main    5431 online postgres /var/lib/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log
9.6 main    5433 online postgres /var/lib/postgresql/9.6/main /var/log/postgresql/postgresql-9.6-main.log
10  main    5434 online postgres /var/lib/postgresql/10/main  /var/log/postgresql/postgresql-10-main.log
12  main    5432 online postgres /var/lib/postgresql/12/main  /var/log/postgresql/postgresql-12-main.log
13  main    5435 online postgres /var/lib/postgresql/13/main  /var/log/postgresql/postgresql-13-main.log
14  main    5436 online postgres /var/lib/postgresql/14/main  /var/log/postgresql/postgresql-14-main.log

Listing the databases in the active cluster showed an unnecessary database:

Shell

$ sudo -iu postgres psql -l
List of databases
     Name     |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
--------------+----------+----------+-------------+-------------+-----------------------
 bix_dev      | postgres | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 |
 postgres     | postgres | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 |
 scalacourses | postgres | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 |
 template0    | postgres | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 | =c/postgres          +
              |          |          |             |             | postgres=CTc/postgres
 template1    | postgres | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 | =c/postgres          +
              |          |          |             |             | postgres=CTc/postgres
(5 rows)

I do not care about the bix_dev database, so I decided to delete it. The database that I care about is called scalacourses.

Shell

$ sudo -iu postgres psql -c 'drop database bix_dev;'
psql (14.7 (Ubuntu 14.7-0ubuntu0.22.10.1))
Type "help" for help.

DROP DATABASE

Upgrade Procedure

Ubuntu is normally upgraded by running do-release-upgrade. Near the end of that process, the following message appeared.

At this point postgresql-15 and postgresql-client-15 had just been installed. The message did not reflect the true state of the system, which caused me some consternation for a moment.

The PostgreSQL version 14 is obsolete, but the server or client packages are still installed. Please install the latest packages (postgresql-15 and postgresql-client-15) and upgrade the existing clusters with pg_upgradecluster (see manpage).

Please be aware that the installation of postgresql-15 will automatically create a default cluster 15/main. If you want to upgrade the 14/main cluster, you need to remove the already existing 15 cluster (pg_dropcluster --stop 15 main, see manpage for details).

The old server and client packages are no longer supported. After the existing clusters are upgraded, the postgresql-14 and postgresql-client-14 packages should be removed.

Please see /usr/share/doc/postgresql-common/README.Debian.gz for details.

Following is /usr/share/doc/postgresql-common/README.Debian.gz, which was referenced by the above message. Most of that file is uninteresting, however two sections are important, and we will look at them.

/usr/share/doc/postgresql-common/README.Debian.gz

PostgreSQL for Debian
=====================

PostgreSQL is a fully featured object-relational database management system. It
supports a large part of the SQL standard and is designed to be extensible by
users in many aspects. Its features include ACID transactions, foreign keys,
views, sequences, subqueries, triggers, outer joins, multiversion concurrency
control, and user-defined types and functions.

Since the on-disk data format of all major PostgreSQL versions (like 9.6,
11, etc.) is incompatible to each other, Debian's PostgreSQL packaging
architecture is designed to maintain clusters of different major versions in
parallel.

This postgresql-common package provides the common infrastructure and all
frontend programs that users and administrators use. The version specific
server and client programs are shipped in postgresql-*-<version> packages.

For a detailed description of the architecture, please see

/usr/share/doc/postgresql-common/README.md.gz

First steps for the impatient
-----------------------------
Eventually you will not get around reading at least some parts of the manual,
but if you want to get straight into playing SQL, here are the steps to create
a database user and a database for the Unix user 'joe':

1. Install a database server with the major version of your choice
('postgresql-XY', e. g. 'postgresql-11').  Preferably the latest
version, which you can get by installing the metapackage
'postgresql'. This will automatically create a default cluster
'main' with the database superuser 'postgres'.

2. Get a shell for the database superuser 'postgres'. If your system
has an active root user, use su:

# su -s /bin/bash postgres

If your system uses sudo to get administrative rights, use sudo instead:

joe$ sudo -u postgres bash

3. In this postgres shell, create a database user with the same name as your
Unix login:

$ createuser -DRS joe

For details about the options, see createuser(1).

4. Create a database "joework" which is owned by "joe":

$ createdb -O joe joework

For details about the options, see createdb(1).

5. Exit the postgres shell.

6. As user joe, you should now be able to connect to your database with

$ psql joework

Cluster management
------------------
For managing clusters, the following commands are provided (each with its own
manual page):

pg_createcluster  - Create a new cluster or integrate an existing one into
the postgresql-common architecture.
pg_dropcluster    - Completely remove a cluster.
pg_ctlcluster     - Control the server process of a cluster (start, stop,
restart).
pg_lsclusters     - Show a list of all existing clusters and their status.
pg_upgradecluster - Migrate a cluster from one major version to another one.
pg_renamecluster  - Rename a cluster.

Please note that you can of course also use the upstream tools for
creating clusters, such as initdb(1). However, please note that in
this case you cannot expect *any* of above pg_* tools to work, since
they use different configuration settings (SSL, data directories,
etc.) and file locations (e. g.
/etc/postgresql/11/main/postgresql.conf). If in doubt, then do *not*
use initdb, but only pg_createcluster. Since merely installing
postgresql-NN will already set up a default cluster which is ready to
work, most people do not need to bother about initdb or
pg_createcluster at all.

Port assignment
---------------
Please note that the pg_* tools automatically manage the server ports
unless you specify them manually. The first cluster which is ever
created (by any major version) will run on the default port 5432, and
each new cluster will use the next higher free one.

E. g. if you first install "postgresql-11" on a clean system, the
default 11/main cluster will run on port 5432. If you then create
another 11 cluster, or install the "postgresql-12" package, that new
one will run on 5433.

Please use "pg_lsclusters" for displaying the cluster <-> port
mapping, and please have a look at the pg_createcluster manpage (the
--port option) for details.

Default clusters and upgrading
------------------------------
When installing a postgresql-NN package from scratch, a default
cluster 'main' will automatically be created. This operation is
equivalent to doing 'pg_createcluster NN main --start'.

Due to this default cluster, an immediate attempt to upgrade an
earlier 'main' cluster to a new version will fail and you need to
remove the newer default cluster first. E. g., if you have
postgresql-9.6 installed and want to upgrade to 11, you first install
postgresql-11:

apt-get install postgresql-11

Then drop the default 11 cluster that was just created:

pg_dropcluster 11 main --stop

And then upgrade the 9.6 cluster to the latest installed version (e. g. 11):

pg_upgradecluster 9.6 main

SSL
---
The PostgreSQL server packages support SSL, which provides encrypted and
authenticated network communication. SSL should be used if you have an
untrusted network between a database server and a client and these exchange
security sensitive data like passwords or confidential database contents.

When a cluster is created with pg_createcluster, SSL support will automatically
be enabled. postgresql-common makes use of the 'snakeoil' SSL certificate that
is generated by the ssl-cert package, so that SSL works out of the box
(ssl_cert_file, ssl_key_file). In addition, if /etc/postgresql-common/root.crt
exists, it will be used as CA certificate file (ssl_ca_file).

/etc/postgresql-common/root.crt is a dummy file by default, so that
client-side authentication is not performed. To enable it, you should
add some root certificates to it. A reasonable choice is to just
symlink the file to /etc/ssl/certs/ssl-cert-snakeoil.pem; in this
case, client certificates need to be signed by the snakeoil
certificate, which might be desirable in many cases. See

/usr/share/doc/postgresql-doc-11/html/ssl-tcp.html

for details (in package postgresql-doc).

Further documentation
---------------------
All commands shipped by postgresql-common have detailed manpages. See
postgresql-common(7) for the documentation of the database client program
wrapping, and user_clusters(5) and postgresqlrc(5) for the cluster
configuration.

The documentation of the database server and client functions, SQL commands,
modules, etc.  documented is shipped in the per-version packages
postgresql-doc-<version>.

The Cluster management section in the above file provides some background information for our purposes, so I repeat it here:

Cluster management

For managing clusters, the following commands are provided (each with
its own manual page):

pg_createcluster  - Create a new cluster or integrate an existing one
                    into the postgresql-common architecture.
pg_dropcluster    - Completely remove a cluster.
pg_ctlcluster     - Control the server process of a cluster
                    (start, stop, restart).
pg_lsclusters     - Show a list of all existing clusters and their status.
pg_upgradecluster - Migrate a cluster from one major version to another one.
pg_renamecluster  - Rename a cluster.

Please note that you can of course also use the upstream tools for
creating clusters, such as initdb(1). However, please note that in
this case you cannot expect *any* of above pg_* tools to work, since
they use different configuration settings (SSL, data directories,
etc.) and file locations (e. g.
/etc/postgresql/11/main/postgresql.conf). If in doubt, then do *not*
use initdb, but only pg_createcluster. Since merely installing
postgresql-NN will already set up a default cluster which is ready to
work, most people do not need to bother about initdb or
pg_createcluster at all.

The Default clusters and upgrading section is more important, however it contained errors. I fixed the errors in the following instructions.

Default clusters and upgrading

When installing a postgresql-NN package from scratch, a default
cluster 'main' will automatically be created. This operation is
equivalent to doing 'pg_createcluster NN main --start'.

Due to this default cluster, an immediate attempt to upgrade an
earlier 'main' cluster to a new version will fail and you need to
remove the newer default cluster first. E. g., if you have
postgresql-9.6 installed and want to upgrade to 11, you first install
postgresql-11:

yes | sudo apt install postgresql-11

Then drop the default 11 cluster that was just created:

sudo -iu postgres pg_dropcluster 11 main --stop

And then upgrade the 9.6 cluster to the latest installed version (e. g. 11):

sudo -iu postgres pg_upgradecluster 9.6 main

Systemd can significantly affect the upgrade process, and unfortunately it is completely ignored by the above instructions. I muddled through, as you will see.

Transcript

The following is a transcript of what I actually did:

Shell

$ sudo -iu postgres pg_dropcluster --stop 15 main
Warning: stopping the cluster using pg_ctlcluster will mark the systemd
unit as failed. Consider using systemctl:
  sudo systemctl stop postgresql@15-main
Warning: systemd was not informed about the removed cluster yet. Operations like
"service postgresql start" might fail. To fix, run:
  sudo systemctl daemon-reload 

$ sudo -iu postgres pg_upgradecluster 14 main
Stopping old cluster...
Warning: stopping the cluster using pg_ctlcluster will mark the systemd unit as failed.
Consider using systemctl:
  sudo systemctl stop postgresql@14-main
Restarting old cluster with restricted connections...
Notice: extra pg_ctl/postgres options given, bypassing systemctl for start operation
Creating new PostgreSQL cluster 15/main ...
/usr/lib/postgresql/15/bin/initdb -D /var/lib/postgresql/15/main --auth-local peer --auth-host scram-sha-256 --no-instructions --encoding UTF8 --lc-collate en_CA.UTF-8 --lc-ctype en_CA.UTF-8
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_CA.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/15/main ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... America/Toronto
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Warning: systemd does not know about the new cluster yet. Operations like
"service postgresql start" will not handle it. To fix, run:
  sudo systemctl daemon-reload

Copying old configuration files...
Copying old start.conf...
Copying old pg_ctl.conf...
Starting new cluster...
Notice: extra pg_ctl/postgres options given, bypassing systemctl for start operation
Roles, databases, schemas, ACLs...
 set_config
------------

(1 row)

set_config
------------

(1 row)

set_config
------------

(1 row)

set_config
------------

(1 row)

Fixing hardcoded library paths for stored procedures...
Upgrading database template1...
Analyzing database template1...
Fixing hardcoded library paths for stored procedures...
Upgrading database postgres...
Analyzing database postgres...
Fixing hardcoded library paths for stored procedures...
Upgrading database scalacourses...
Analyzing database scalacourses...
Stopping target cluster...
Stopping old cluster...
Disabling automatic startup of old cluster...
Starting upgraded cluster on port 5432...
Warning: the cluster will not be running as a systemd service.
Consider using systemctl:
  sudo systemctl start postgresql@15-main

Success. Please check that the upgraded cluster works. If it does,
you can remove the old cluster with
     pg_dropcluster 14 main

Ver Cluster Port Status Owner    Data directory              Log file
14  main    5433 down   postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log
Ver Cluster Port Status Owner    Data directory              Log file
15  main    5432 online postgres /var/lib/postgresql/15/main /var/log/postgresql/postgresql-15-main.log

When I attempted to following the above instructions for systemd, errors appeared:

Shell

$ sudo systemctl daemon-reload

$ sudo systemctl start postgresql@15-main
Job for postgresql@15-main.service failed because the service did not take the steps required by its unit configuration.
See "systemctl status postgresql@15-main.service" and "journalctl -xeu postgresql@15-main.service" for details.

Great, an obscure failure message. Others have fussed with this problem. I expected that rebooting would be much faster than trying to fix it, and that did prove to be the case:

Shell

$ sudo reboot

After rebooting, the upgraded Postgres cluster worked fine:

Shell

$ pg_lsclusters
Ver Cluster Port Status Owner    Data directory              Log file
14  main    5433 down   postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log
15  main    5432 online postgres /var/lib/postgresql/15/main /var/log/postgresql/postgresql-15-main.log

$ sudo -iu postgres psql -l
List of databases
    Name     |    Owner     | Encoding |   Collate   |    Ctype    | ICU Locale | Locale Provider |   Access privileges
--------------+--------------+----------+-------------+-------------+------------+-----------------+-----------------------
postgres     | postgres     | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 |            | libc            |
scalacourses | scalacourses | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 |            | libc            |
template0    | postgres     | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 |            | libc            | =c/postgres          +
              |              |          |             |             |            |                 | postgres=CTc/postgres
template1    | postgres     | UTF8     | en_CA.UTF-8 | en_CA.UTF-8 |            | libc            | =c/postgres          +
              |              |          |             |             |            |                 | postgres=CTc/postgres
 (4 rows)

Here is how I deleted the older database clusters:

Shell

$ for X in 9.5 9.6 10 12 13; do
  sudo -iu postgres pg_dropcluster --stop $X main
end

😁

All done!

Montréal International vs. Bill 96

2022-05-27T00:00:00-04:00

Last night I attended the 25th Annual General Meeting of Montréal International, an energetic organization dedicated to attracting foreign business to establish a presence in the greater Montréal region. My key takeaways were:

I spoke with about 20 Montréal International employees; all of them were bilingual or trilingual, intelligent, motivated and well-informed. Their talent and commitment is palpable. Clearly, the hiring process is successful.
The event was marred by long-winded, self-congratulatory speeches by senior management, and former senior managers who were invited to a panel discussion. This lack of discipline caused the event to significantly run overtime.
The proceedings were conducted entirely in French. For an organization that calls itself “Montréal International”, this is unforgivable.
I asked every employee I spoke with if Quebec’s new Bill 96 impacted their goals. All of them expressed grave concern, and said that their mandate to attract foreign business had become extremely difficult as a result. However, not one mention was made by anyone on stage about Bill 96.

Montréal International’s mandate is severely impacted by Quebec's Bill 96. If any organization has an intrinsic mandate to demonstrate public leadership towards repealing or modifying Bill 96 so it respects basic human rights, it would be Montréal International. Instead, Montréal International’s leadership, which consists entirely of old white French-speaking alpha males, is complicit in their silence, and by their actions.

Limit Your Financial Vulnerability From AWS Account Hijacking

2022-05-26T00:00:00-04:00

I am a free agent, independent and critical. You can buy my time as a software expert, but not my opinion. Just because Amazon / AWS hired me in the past to help them against a patent troll does not mean they own me. Quite the contrary.

In the time you take a quick shower, your AWS account can be highjacked and tens of thousands of dollars in fees can be incurred. EC2 is the service that provides the greatest financial risk. You can take steps to limit your liability, and this article shows you how.

Out-of-Control Cloud Costs

Out-of-control costs are causing cloud customers to reduce or eliminate their cloud use. The cloud backlash has begun: Why big data is pulling compute back on premises – Thomas Robinson, published by TechCrunch 2023-03-20.

Why we’re leaving the cloud, by David Heinemeier Hansson, creator of Ruby on Rails and CTO of 37signals. Mr. Hansson also wrote We stand to save $7m over five years from our cloud exit.

AWS IAM Users and Roles Have No Budget Limitations

AWS customers are exposed to unlimited financial liability

I encountered two main issues when attempting to secure against AWS account hijacking:

AWS does not provide a mechanism to guard against launching expensive services. For example, if an IAM user has a role that allows EC2 instances to be launched, then that user can launch an unlimited number of EC2 instances of any size.

The most expensive Amazon EC2 is currently the p4de.24xlarge, which costs $40.96 USD per hour on-demand in the US East (N. Virginia) region. The instance comes with 96 vCPUs, 1152 GiB of RAM and eight 1TB NVMe SSDs, and has eight NVIDIA A100 Tensor Core GPUs.

Using stolen credentials that allow EC2 instances to be launched, an attacker using scripts could spin up an armada of p4de.24xlarge instances and incur eye-popping costs in minutes.
AWS Budgets provides a convenient way to shut down pre-designated services when a total budgetary amount is exceeded. However, the AWS AMI security model is not integrated into real-time cost monitoring. Thus, there is no way to automatically shut down newly launched service instances that exceed a pre-authorized budgetary amount.

Vercel’s Pricing Policy Incorporates Limits

We provide customers with tools to observe, control, and alert on their infrastructure spend with Spend Management. You can define a spend amount (e.g. $40) and receive email, web, and SMS notifications as you reach that amount. When reaching 100%, you can optionally automatically pause all projects with a hard limit. Over the past year, we've added features to the platform based on your feedback to help automatically prevent runaway spend, including recursion protection, improved function defaults, hard spend limits, Attack Challenge Mode, and more.

AWS, Azure, Google App Engine, IBM Cloud and Linode all need a pricing policy like Vercel’s.

Shared Security Responsibility

The AWS Customer Agreement and Shared Responsibility Model describe how AWS shares responsibility for security with users. If you follow the AWS security recommendations, described next, you will not be held accountable for extra charges if your AWS account is hijacked.

The AWS Security Blog published a nice overview, entitled Getting Started: Follow Security Best Practices as You Configure Your AWS Resources. Some of the same information is also provided in Security best practices in IAM.

Even though the AWS instructions do not prevent bad guys taking over your account, follow them anyway so you won't be held liable for the costs incurred if your AWS account is hijacked.

AWS Security Recommendations

Security needs to have depth. Any single measure can fail, and given enough scale, all measures will eventually fail. Layer your security measures so that one failure, no matter how grave, will not be fatal.

AWS recommends the following. I have highlighted what AWS personnel have described to me as being the quickest and easiest path; this article walks through those steps to the extent that I have been able.

Set up at least two of the following services to monitor cost and usage:
2. .
3. CloudTrail User Guide.
4. Web Application Firewall (WAF).
5. Trusted Advisor. I found that Trusted Advisor was somewhat easier to use than CloudWatch.
For more information about managing your AWS cost and usage, see the AWS Cost Management User Guide.
Set up at least one of the following security best practices:
1. .
2. AWS Security Hub.
3. Amazon GuardDuty.

Do The Easiest Things First

I usually prefer to do the easiest things first. Not everyone agrees. Doing even one highlighted item above provides a significant security improvement, so why wait?

The best is the enemy of the good.

– Voltaire

Better a diamond with a flaw than a pebble without.

– Confucius

Striving to better, oft we mar what’s well.

– Shakespeare

Root Credentials

If a bad guy has your root credentials, they can change budget limits. For greatest security, delete your root credentials by selecting Access Keys (access key ID and secret access key) as shown in the image below. However, if you do so, many things become impossible.

Enabling MFA

Yubikey NFC

The easiest highlighted item above is to enable multifactor authentication (MFA).

I use a virtual MFA device, enabled for my root account Google Authenticator for Android and iOS. Google Authenticator features a RFC 6238 standards-based TOTP (time-based one-time password) algorithm.

I use a YubiKey to provide MFA for the AMI user ID that I use to log into the AWS console for everyday work.

Only use root credentials when absolutely necessary.

Trusted Advisor

After adding MFA to root credentials, I found that using AWS Trusted Advisor was the next easiest thing to try to make my AWS account secure.

Once you visit the Trusted Advisor Console, and agree to enable Trusted Advisor, IAM permissions are added to the AWS IAM user that you are logged in as.

Usage seems straightforward; however, I found the information for securing S3 buckets puzzling at first:

The column labeled ACL Allows List means that buckets flagged with Yes have an ACL that allows objects to be listed. AWS recommends that S3 buckets not allow objects to be listed by the public. To correct this:

Click on the bucket name.
Click on the Permissions tab in the page that opens next.
Scroll down to Access Control List.
Click on the Edit button.
Disable the items marked in red, as shown below.

Referring back to the previous screenshot, the Policy Allows Access column flags all S3 buckets used to serve webpages as insecure. To serve HTML pages and assets stored in an S3 bucket, the permissions must be set to allow objects to be read by everyone. Trusted Advisor flags these buckets as insecure, which does not make sense to me.

Perhaps there is a better security policy for when webpages are served via CloudFront, but as is often the case with AWS documentation, it is difficult to piece together how to optimally configure S3 permissions with CloudFront options.

Best Practices Evolve With Technology

The AWS documentation is written as if the products described have always existed in their present state. In the course of researching this article, I discovered that AWS provides revision histories for their services. For example, the revision history of S3 is documented in the AWS S3 User Guide, and the CloudFront revision history is documented in the AWS CloudFront Developer Guide. Revision histories are useful for experienced technologists to review periodically, so they can keep up with best practices as they evolve.

I started using AWS S3 to serve websites in 2013, and at that time, CloudFront did not exist yet. As the two services matured, best practices, which were at first unknown, evolved and interactions between them became more complex. In particular, CloudFront http/https promotion, how origins are specified, and S3 permissions, all interact in ways that were not well documented for several years. This was a source of security problems and unwanted downtime for me.

I have 15 buckets that are used to serve web pages. On my AWS Support Center console, Trusted Advisor shows that “You have a yellow check affecting 15 of 24 resources”; however, the Trusted Advisor console displays those items as red triangles with exclamation marks.

This is confusing, and became frustrating when an AWS security support technician suggested I disregard anything that seemed annoying. For an IT/security person, that advice is the quickest way to hell that I know.

Rough Spots That Need Love

While researching this article I found:

Under Origin Settings, for Origin Domain Name, choose the Amazon S3 bucket that you created previously. For S3 bucket access, select Yes, use OAI (bucket can restrict access to only CloudFront). For the Origin access identity, you can choose from the list, or choose Create new OAI (both will work). For Bucket policy, select Yes, update the bucket policy.

– From Use an Amazon CloudFront distribution to serve a static website

However, I found conflicting information near the top of this article:

Important

If you use an Amazon S3 bucket configured as a website endpoint, you must set it up with CloudFront as a custom origin. You can’t use the origin access identity feature described in this topic. However, you can restrict access to content on a custom origin by setting up custom headers and configuring your origin to require them. For more information, see Restricting access to files on custom origins.

– From Restricting access to Amazon S3 content by using an origin access identity (OAI)

As if this is not confusing enough, Using various origins with CloudFront distributions states that in order for Amazon S3 redirects to work, a non-standard format for the origin must be used:

http://bucket-name.s3-website-region.amazonaws.com

As I discussed in a previous article, AWS S3 buckets support two types of redirects, but there is no indication of which type might or might not work with the non-standard origin format.

Also, there is no mention of whether https would work or not, or if these disparate instructions even work together. As usual, there is lots of uncertainty about how things actually work on AWS, and for how long before things change.

As per standard operating procedure with AWS, one would have to muck about extensively to determine which of the conflicting statements above is generally prevalent, and under what circumstances. I expect that one article is newer than the other, and it likely supercedes the older article somewhat to some degree under as-yet unknown circumstances. If I get lucky, the Trusted Advisor warnings will go away. Trusted Advisor should reference whichever article is currently correct.

Update 2022-05-31

I received the following email from Trusted Advisor. Unfortunately, the contents of the email were different from what I found on the Trusted Advisor console. Perhaps this is because a higher-level, and more expensive, service contract would be required to see those messages. If so, then this is not the way to gain happy customers. The cost of the higher service level would be 4 times more than my total AWS costs at present, clearly not something that I would want to do.

Dear AWS customer:

AWS Trusted Advisor currently shows alerts for 3 checks (1 red and 2 yellow) and $0 of potential monthly savings based on your usage.

Here is a summary of status changes for this week:

The alert severity of these checks has improved:

From red to yellow:
   Amazon S3 Bucket Permissions

From red to green:
   VPC
   Security Groups - Specific Ports Unrestricted

From yellow to green:
   VPC Internet Gateways

The alert status of 5 checks has not changed. For details and recommendations, visit AWS Trusted Advisor.

Sign in to the Trusted Advisor console to view your check results.

Best,

AWS Support
Was this report helpful? Send us your feedback.

To add insult to injury, the final "Send us your feedback" just links to a generic form for AWS sales, where one can pound sand.

There is abundant evidence to demonstrate that whereas amazon.com is consumer-centric, AWS marches to a very different drummer. Perhaps small businesses are not presently viewed by AWS as a significant growth opportunity. My personal experience is that the current risk/value proposition provided by AWS is no longer attractive for that demographic.

Update 2022-06-03

The credit card I used to pay for my AWS usage had a very high limit. In retrospect that was unwise.

AWS took weeks to agree to refund the charge on my credit card, and then they said it would take another week for the reversed charges to appear. This meant that if I only paid for the other charges on that card, the month-to-month credit card interest on the fraudulent AWS charges would have been problematic.

Just before the credit card billing period ended, I called Visa and asked what my options were. Apparently I am not the first person who had this problem with AWS. The answer was immediate: report the charges as fraud. OK, that made sense. This meant my card was immediately cancelled, the charges were forgiven by Visa, and a new card with a new number would arrive in about a week.

Managing Costs With AWS Budgets

Short-lived hijacking is the most serious financial threat, and AWS Budgets was seemingly not designed to guard against that. Ticking off this item will enable AWS to absolve you of financial responsibility for being hacked, but you should not expect that it will do much to slow down an attacker.

AWS Budgets are easy to set up in a passive manner, but they are clumsy to work with and not effective for preventing new services from being launched that would exceed the budget. The cost limiting functionality is not automatic, it must be set up manually. AWS documentation does not provide detailed how-to instructions for common scenarios. Tens of thousands of words of documentation must be digested before the full capabilities can be harnessed.

Another significant issue is that only three types of actions are supported:

IAM Policy – I am unclear on how to set this up so the bad guys cannot launch new services, and I am not certain this is possible.
Service Control Policy – Hopefully this could prevent new services from being launched that would exceed the budget; however, my head exploded when I researched this.

Attempting to use this option to prevent new, expensive services from being launched would seem to introduce lots of complexity. Perhaps AWS did not consider this use case, or deemed it unimportant.
Automate Instances to Stop for EC2 or RDS – If you run a t3.nano instance, for example, bad guys can only incur a certain amount of financial damage. Instead of shutting down the EC2 or RDS instances that I want to run, my main concern is to prevent bad guys from launching expensive new services. This option is therefore unsuitable for my use case.

If AWS Budgets provides an easy way of preventing new services to be launched that would exceed the budget, I could not find it after several hours of reading.

Passively Monitoring Budget Spend

Simply follow the directions and accept the defaults. Daily budgets do not support enabling forecasted alerts, or daily budget planning, so select Monthly Budgets. There is no need to set up SNS alerts or AWS Chatbot alerts, email notification works just as well for most users.

Attaching Actions

Following are my notes from my attempt to enable cost limiting. I was unable to attach an action that prevented expensive new services from being started, and it is unclear if resolving the problem will, in fact, provide the desired benefit. Perhaps someone reading these notes will be able to tell me about an approach that might work for a reasonable amount of effort.

Because AWS will absolve you of financial liability if you create a Budget without any actions, don't stress about the pointless exercise involved in making a Budget that cannot protect you. Just do it and give AWS more time to figure out a better way of preventing account hijacking.

You can stop reading this article now, unless the details interest you.

For PaaS vendors such as AWS, Azure, Digital Ocean, Cloudflare, ScaleWay, etc: “pay-as-you-go” is shorthand for “there is nothing you can do to limit your financial liability”.

When you get to the step labeled Attach actions - Optional, choose Add Action.
You must choose between using an existing IAM role for AWS Budgets to use, or you can create a new IAM role expressly for this purpose. IAM roles are free. I like the idea of a dedicated IAM role, so I clicked on manually create an IAM role.
A new browser tab opened, and I clicked on the blue Create role button.
Now a confusing page appeared:
I selected the default Trusted entity type, AWS Service, then from the pulldown menu labeled Use cases for other AWS services: I selected Budgets.
A new radio button appeared underneath the pull-down, also labeled Budgets, and I clicked on that. This does not win any usability awards.
I clicked on the button labeled Next.
Managed policy name: AWSBudgetsActionsRolePolicyForResourceAdministrationWithSSM

This managed policy is focused on specific actions that AWS Budgets takes on your behalf when completing a specific action. This policy gives AWS Budgets broad permission to control AWS resources. For example, starts and stops Amazon EC2 or Amazon RDS instances by running AWS Systems Manager (SSM) scripts.
– From Using identity-based policies (IAM policies) for AWS Cost Management

On the page, I applied AWSBudgetsActionsRolePolicyForResourceAdministrationWithSSM, then I clicked on Next.
On the final page, I named the IAM role Budget and clicked on Create role.
Back in the Billing Management Console, I refreshed the list of available IAM roles, then selected the new role called Budget.
I selected Service Control Policy from the pull-down that appeared next, labeled Which action type should be applied when the budget threshold has been exceeded?. I have no idea how to proceed, or even if going in this direction might provide the desired results.

Microsoft Clarity Lets Me Watch You Click and Scroll

2022-03-31T00:00:00-04:00

I spent a fair bit of time designing the plugins for this Jekyll-powered website for SEO. These plugins are published as open source; click on the word Jekyll in the sentence above this paragraph to learn more. You are welcome!

I do not use Google Analytics because it slows down page load time dramatically. I do use Google Search Console, however, and recently started trying out Microsoft Bing Webmaster Tools.

Microsoft Clarity and Hotjar

The day before April Fool's Day in 2022, I stumbled across Microsoft Clarity, a free product that I knew nothing about. I was curious to know what benefit it might provide.

I learned that one of the things it can do is provide videos of actual user sessions as they interact with the website. Those videos are fantastic.

As a writer of a lot of online content, I have spent hours watching you, dear readers. Individually yet anonymously. Watching people read teaches you a lot about what is important to them. As a writer, knowing your audience allows you to be more relevant. BTW, I do not know who or where my readers are, just the country they are in.

Update 2023-07-10

I no longer need to watch the videos.
Microsoft Clarity's AI generates summaries.
Often it yields incredibly actionable information, but the Key Takeaways section is generally way off the mark. YMMV.

Today I noticed the insights tab, visible when viewing an entry for a ‘hit’. Seems like Microsoft has employed AI to generate the following impressive summary of the user's behavior. I added punctuation, a light edit and a whimsical pad of notepaper.

Visited URL matches regex: ^https://www\.mslinn\.com/softwareexpert/index\.html(\?.*)?$
Last 7 days

Session insights
Some key takeaways from this session are:

The user was interested in the software expert witness and computer expert services offered by Mike Slinn, as they visited the homepage from Google and spent about four minutes there.

The user was also curious about the technology expert article series, especially the ones related to Git and version control systems. They visited the article index page multiple times and clicked on several articles, spending about 15 minutes in total on this topic.

The user was most engaged with the article on libgit2, a library that provides low-level access to Git operations. They spent about one and a half minutes on this article and clicked on a link to git-fame, a tool that shows the contribution statistics of a Git repository.

The user frequently resized their browser window, which may indicate that they were using a mobile device or adjusting their screen for better readability. The user also switched between tabs or applications often, as indicated by the page hidden and page visible events. This may suggest that they were multitasking or comparing information from different sources.

01:47 / 47:31

Update 2023-07-18

Today I stumbled across the Clarity Live plugin for the Google Chrome web browser. There is no such plugin for Firefox, sadly.

View instant heatmaps right on your live site and watch recent session recordings for any page you are on with our extension.

GDPR & CCPA ready
No sampling
Built on open source

– From Microsoft Clarity

Update 2023-10-04

I just stumbled across the Clarity Client API.

You can quickly get started with Clarity without coding but by interacting with the Clarity client API. This API can help you access advanced features as described in this reference. Add the following calls to Clarity APIs to the HTML or JavaScript of your webpage to access these features.

Note

Your Clarity ID serves as your API key. No other client API key is necessary, and there is no cost for using Clarity client APIs.

Check out this video!

This is one of the first user sessions that Microsoft Clarity recorded for this website.

As you can see, Microsoft Clarity lets me watch movies of users clicking and scrolling through my website; spooky yet very informative.

Clarity is open source. Here is the GitHub project.

I have since learned that Hotjar is similar to Microsoft Clarity.

Online Behavior Matches Real-World Behavior

The user in the above video read a bit about my experience as a software expert witness, then straightaway downloaded my resume. They were on the website for just over one minute. Although they did call me, their online behavior showed a lack of urgency, and their ‘real-world’ behavior mirrored what I saw in the movie.

A week later another party visited my site, and spent 80 minutes carefully reading 3 pages, among others. Their ‘real-world’ behavior also matched their online behavior, in that they exhibited a sense of urgency towards engaging a software expert.

Tracking Downloads And Other Behavior

Microsoft Clarity does not consider a download as a click. It does not even notice downloads. For me, downloads are what I care about most. If you never download my resume, you probably are not a candidate for hiring me as a software expert. I want to track downloads, not clicks. Clicks are nice, but there is a direct relationship between resume downloads and signing contracts with legal firms who represent new clients.

AWS CloudTrail and CloudWatch can provide download details and much more. For example, user IP addresses and geographic location can be captured when a monitored event occurs.

Many Websites Perform Surveillance

Wired Magazine published an article on a similar type of surveillance (keyloggers) on May 11, 2022. I paraphrased two sentences from that article:

1.8% of websites studied gathered an EU user's email address without their consent, and a staggering 2.95% logged a US user's email.
For US users, 8.4% of sites may have been leaking data to Meta, Facebook’s parent company, and 7.4% of sites may be impacted for EU users.

– From Thousands of Popular Websites See What You Type—Before You Hit Submit

Yours truly does nothing of the sort.

Popular Pages

The data that Clarity manages can be downloaded with the Clarity Data Export API. API calls are limited to 10 calls per project, per day; that is not a problem. The endpoint is www.clarity.ms/export-data/api/v1/project-live-insights.

An access token is required to use the API. To get one, open the Clarity project and select Settings -> Data Export -> Generate new API token. I called my token clarity_data_export and stored it in an environment variable called CLARITY_DATA

Here is how I retrieved the information about the most popular pages in the last 30 days using the bash command line:

Shell

$ DAYS=30

$ DIM1="PopularPages"

$ PARAMS="numOfDays=$DAYS&dimension1=$DIM1"

$ URL="https://www.clarity.ms/export-data/api/v1/project-live-insights"

$ curl \
  --location "$URL?$PARAMS" \
  --header 'Content-Type: application/json' \
  --header "Authorization: Bearer $CLARITY_DATA"
$

Looking Ahead

I would very much like to receive a continuous data feed of the above AI-generated per-visit summary, in real time as it becomes available. An email stream would be a quick way to start. A streaming api for data would be a logical next step.

Perhaps the Clarity API might be a place to start.

Make a Visual Studio Code Extension

2022-03-01T00:00:00-05:00

I want to be able to move pages in my Jekyll-powered website without breaking links from the outside. The jekyll-redirect-from Jekyll plugin generates little HTML files that contain http-meta-refresh client-side redirects as desired. I wanted an easy way to inject the names of those redirect pages into the Jekyll front matter. Since I usually author my website using Visual Studio Code, a custom Visual Studio Code extension seems like a good way to create the redirects.

Extension Requirements

All the extension has to do is discover the URL path component of the page currently being edited, and write an entry into the front matter. For example, I have set up Jekyll such that given a page at collections/_posts/2022/2022-02-21-jekyll-debugging.html, it would deploy to /blog/2022/02/21/jekyll-debugging.html. The required front matter would be:

Shell

redirect_from:
  - /blog/2022/02/21/jekyll-debugging.html

With that in mind, if I want to move a published article to another location without breaking it, the extension should:

Present a new menu option when a right-click on a file in the sidebar, or a right-click on a file name tab in the editor.
When the menu item is selected, obtain the relative path to the file within the project directory (for example: collections/_posts/2022/2022-02-21-jekyll-debugging.html)
Convert the relative path to the deployed relative path (for example, /blog/2022/02/21/jekyll-debugging.html)
Write the entry into the front matter of the file. For example:
Shell
```
redirect_from:
  - /blog/2022/02/21/jekyll-debugging.html
```

Background

The Microsoft documentation describes how to write a Visual Studio Code extension.

Setup

Visual Studio Code extensions are written in JavaScript (actually, node.js) or TypeScript. Ensure that a version of node.js has been installed:

Shell

$ which node
/home/mslinn/.nvm/versions/node/v17.3.1/bin/node

Ensure you have set up your global package manager for node.js, then type:

Shell

$ npm install -g yo generator-code
  npm WARN deprecated har-validator@5.1.5: this library is no longer supported
  npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
  npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142

  added 872 packages, and audited 873 packages in 57s

  15 vulnerabilities (13 moderate, 2 high)

  To address issues that do not require attention, run:
    npm audit fix

  To address all issues (including breaking changes), run:
    npm audit fix --force

  Run `npm audit` for details.

The yo module is the culprit; lets address its vulnerabilities:

Shell

$ npm install -g npm-check-updates
added 270 packages, and audited 271 packages in 9s

found 0 vulnerabilities

Generating the Skeleton

I decided to use TypeScript instead of JavaScript for the extension. TypeScript code converts to JavaScript, however it uses type inference and integrates better with some editors.

Shell

$ mkdir redirect_generator

$ cd redirect_generator/

$ $ yo code
? ==========================================================================
We’re constantly looking for ways to make yo better!
May we anonymously report usage statistics to improve the tool over time?
More info: https://github.com/yeoman/insight & http://yeoman.io
==========================================================================  No


     _-----_     ╭──────────────────────────╮
    |       |    │   Welcome to the Visual  │
    |--(o)--|    │   Studio Code Extension  │
   `---------´   │        generator!        │
    ( _´U`_ )    ╰──────────────────────────╯
    /___A___\   /
     |  ~  |
   __'.___.'__
 ´   `  |° ´ Y `

? What type of extension do you want to create?  New Extension (TypeScript)
? What's the name of your extension?  redirect_generator
? What's the identifier of your extension?  redirect-generator
? What's the description of your extension? Injects the URL of a redirect page into Jekyll front matter
? Initialize a git repository? Yes
? Bundle the source code with webpack? No
? Which package manager to use? npm
Writing in /mnt/f/work/jekyll/redirect_generator/redirect-generator...
    create redirect-generator/.vscode/extensions.json
    create redirect-generator/.vscode/launch.json
    create redirect-generator/.vscode/settings.json
    create redirect-generator/.vscode/tasks.json
    create redirect-generator/package.json
    create redirect-generator/tsconfig.json
    create redirect-generator/.vscodeignore
    create redirect-generator/vsc-extension-quickstart.md
    create redirect-generator/README.md
    create redirect-generator/CHANGELOG.md
    create redirect-generator/src/extension.ts
    create redirect-generator/src/test/runTest.ts
    create redirect-generator/src/test/suite/extension.test.ts
    create redirect-generator/src/test/suite/index.ts
    create redirect-generator/.eslintrc.json

Changes to package.json were detected.

Running npm install for you to install the required dependencies.

added 203 packages, and audited 204 packages in 29s

found 0 vulnerabilities

Your extension redirect-generator has been created!

To start editing with Visual Studio Code, use the following commands:

     code redirect-generator

Open vsc-extension-quickstart.md inside the new extension for further instructions
on how to modify, test and publish your extension.

For more information, also visit http://code.visualstudio.com and follow us @code.


? Do you want to open the new folder with Visual Studio Code? Open with `code`

     _-----_     ╭───────────────────────╮
    |       |    │      Bye from us!     │
    |--(o)--|    │       Chat soon.      │
   `---------´   │      Yeoman team      │
    ( _´U`_ )    │    http://yeoman.io   │
    /___A___\   /╰───────────────────────╯
     |  ~  |
   __'.___.'__
 ´   `  |° ´ Y `

The instructions in the web page assume the happy path is the only possibility. Instead:

Do not work on the extension in a workspace that has any other projects in it.
The online instructions assume that TypeScript was used, not JavaScript, so the file types are assumed to be .ts.

The instructions in vsc-extension-quickstart.md do not make assumptions about TypeScript.

Debugging the Extension

I found the procedure awkward at first:

Press F5 to activate the first defined launcher
Once the new debug VSCode instance settles down, click in it and type CTRL-Shift-P, then type redirect.
Any breakpoints in the original VSCode instance will trigger.
Once the breakpoint is cleared, a message saying Add redirecct from redirect_generator! will appear in the debugged instance of VSCode.

Extension Anatomy

I moved on to the next part of the tutorial, Extension Anatomy.

Node.js, NVM, NPM and Yarn

2022-03-01T00:00:00-05:00

Node.js, a JavaScript runtime, is not my favorite programming environment. Originally released 15 years ago by Ryan Dahl, a software engineer at Google, it’s historical disregard for security and stability is problematic.

However, node.js has significant traction, especially amongst Millennial software technologists, and is used by many Ruby and Scala projects for HTML asset pipelines.

I put together these notes for installing and maintaining node.js on Ubuntu/WSL using a virtualized version manager.

Why Virtualize Node.js?

It is better to use virtualized user- and project-specific node.js instances, instead of working with a system-wide installation of node.js. This allows you to install and upgrade node.js packages without using supervisor privileges. Also, virtualized instances allows you to work on many different independent node.js projects at the same time, without package version collisions.

Docker is over-sold. It adds unnecessary complexity to software projects. Instead of virtualizing the entire software environment, as docker attempts to do, virtualizing the programming environment with node.js nvm, Python venv, or Ruby rbenv are much easier and more productive approaches.

I think docker has been pushed hard in the media because it is a gateway technology to PaSS. This is a trend that PaSS vendors like AWS and Azure want to encourage, but customers are pushing back.

Nvm

Nvm, the Node Version Manager, makes it easy to install multiple virtualized node.js instances, and to easily switch between them. Nvm retains a unique set of installed packages for each node.js instance. The node.js version in each instance is distinct.

Nvm is installed and updated as follows:

Shell

$ curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14926  100 14926    0     0  84806      0 --:--:-- --:--:-- --:--:-- 85291
=> nvm is already installed in /home/mslinn/.nvm, trying to update using git
=> => Compressing and cleaning up git repository

=> nvm source string already in /home/mslinn/.bashrc
=> bash_completion source string already in /home/mslinn/.bashrc
=> Close and reopen your terminal to start using nvm or run the following to use it now:

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion

View the currently installed versions of node.js:

Shell

$ nvm list
->       system
iojs -> N/A (default)
node -> stable (-> N/A) (default)
unstable -> N/A (default)

View the very long list of available versions of node.js like this:

Shell

$ nvm list-remote
          v17.7.0
        v17.7.1
        v17.7.2
        v17.8.0
        v17.9.0
        v17.9.1
        v18.0.0
        v18.1.0
        v18.2.0
        v18.3.0
        v18.4.0
        v18.5.0
        v18.6.0
        v18.7.0
        v18.8.0
        v18.9.0
        v18.9.1
       v18.10.0
       v18.11.0
       v18.12.0   (LTS: Hydrogen)
       v18.12.1   (LTS: Hydrogen)
       v18.13.0   (Latest LTS: Hydrogen)
        v19.0.0
        v19.0.1
        v19.1.0
        v19.2.0
        v19.3.0
        v19.4.0
        v19.5.0

Using Nvm to Install Node.js

Install the latest release of node.js using nvm:

Shell

$ nvm install node
Downloading and installing node v19.5.0...
Downloading https://nodejs.org/dist/v19.5.0/node-v19.5.0-linux-x64.tar.xz...
##################################################################### 100.0%
Computing checksum with sha256sum
Checksums matched!
Now using node v19.5.0 (npm v9.3.1)
Creating default alias: default -> node (-> v19.5.0)

In the above example, node is an alias for “the latest version of node.js”. To install a specific version of node.js, for example 18.13.0:

Shell

$ nvm install 18.13.0

Yarn

Needle and thread under an electron microscope

Whether you use nvm as your node.js virtualization mechanism, you also need a node.js package manager to install and maintain project dependencies.

Yarn supposedly stands for Yet Another Resource Negotiator, and it is a package manager like npm, described next. It was developed by Facebook and is now open-source. Yarn was developed to address npm’s performance and security issues.

The name actually suggests the major advantage yarn has over its predecessor, npm: multi-threading instead of single-threading.

Yarn generates a yarn.lock file, which helps easy merges. The merges are predictable. Yarn caches every package it has downloaded, so it never needs to download the same package again. It also does almost everything concurrently to maximize resource utilization. This means faster installs. Yarn guarantees that any installation that works on one system will work exactly the same on another system, unlike npm.

The notes for the yarn package state that Yarn was inspired by Bundler and Cargo, and is nearly command-line compatible with npm.

Yarn introduces the zero-install concept, which means that a project should be able to be used as soon as it is cloned. It uses Plug’n’Play to resolve dependencies via the yarn cache folder and not from node_modules. The cache folder is by default stored within your project folder, in .yarn/cache.

Installing Yarn

To install yarn on Ubuntu:

Shell

$ curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | \
  sudo apt-key add -

$ echo "deb https://dl.yarnpkg.com/debian/ stable main" | \
  sudo tee /etc/apt/sources.list.d/yarn.list

$ sudo apt update && sudo apt install yarn

Configuring a Project With Yarn

The yarn init command initiates an interactive session that creates a package.json file.

Shell

$ cd /path/to/my/node/project

$ yarn init
yarn init v1.22.19
question name (node): blah
question version (1.0.0): 0.1.0
question description: Just a little song I wrote
question entry point (index.js):
question repository url: https://github.com/mslinn/blah
question author: Mike Slinn
question license (MIT):
question private:
success Saved package.json
Done in 48.29s.

The above dialog creates package.json in the current directory:

package.json

{
  "name": "blah",
  "version": "0.1.0",
  "description": "Just a little song I wrote",
  "main": "index.js",
  "repository": "https://github.com/mslinn/blah",
  "author": "Mike Slinn",
  "license": "MIT"
}

Adding a Dependency to a Project

Yarn stores dependencies locally. If the proper version is present locally, it is fetched from the disk during a yarn add command, otherwise it is downloaded. This is the general format of the command to add a dependency to a node.js project using yarn:

Shell

$ yarn add package_name

To add a specific version of a package:

Shell

$ yarn add package_name@version_number

To install a global package, the general formats are:

Shell

$ yarn global add package_name

$ yarn global add package_name@version_number

Yarn and Git

The official Yarn docs say to add the following to .gitignore for git projects that use yarn:

.gitignore

.yarn/*
!.yarn/cache
!.yarn/patches
!.yarn/plugins
!.yarn/releases
!.yarn/sdks
!.yarn/versions

Installing Dependencies

To install all the dependencies in a node.js project, type yarn or yarn install:

Shell

$ cd /path/to/my/project

$ yarn

Updating Dependencies

To upgrade all the dependencies in a node.js project, use yarn upgrade:

Shell

$ cd /path/to/my/project

$ yarn upgrade

Npm

Npm is the original package manager for the family of JavaScript programming languages, and it is still the default package manager for node.js. This will likely change as yarn matures. Npm helps install libraries, plugins, frameworks and applications. It consists of a command-line client, also called npm, and an online database of public and paid-for private packages called the npm registry.

If you are using yarn then you do not need to use npm.

Npm fetches dependencies from the npm registry for every npm install command.

Npm generates a package-lock.json file. The layout of this file is a trade-off between determinism and simplicity. The same node_modules/ folder will be generated by every version of npm. Every dependency will have a version number associated with it in the package-lock file.

Npm vs. Yarn

When using npm install, dependencies are installed sequentially, one after another. The output logs in the terminal are informative but a bit hard to read. To install the packages with Yarn, run the yarn command. Yarn installs packages in parallel, which is one of the reasons it is quicker than npm.

For more information, see Difference between npm and yarn.

Install Npm With a Node Version Manager

Follow these instructions. In summary:

Shell

$ mkdir ~/.npm-global

$ npm config set prefix ~/.npm-global

$ cat >> ~/.bashrc <<EOF
export PATH="$HOME/.npm-global/bin:$PATH"

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" 1>&2  # Loads nvm
EOF

$ source ~/.bashrc

$ curl -fsSL https://deb.nodesource.com/setup_21.x | \
  sudo -E bash - && \
  sudo apt-get install -y nodejs

Install A Global Package

To install a global package, the command template for npm is:

Shell

$ npm install -g package_name[@version_number]

For example:

Shell

$ npm install -g broken-link-checker
npm WARN deprecated calmcard@0.1.1: no longer maintained
npm WARN deprecated nopter@0.3.0: try optionator
npm WARN deprecated uuid@2.0.3: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated urlobj@0.0.11: use universal-url, minurl, relateurl, url-relation

added 104 packages, and audited 105 packages in 5s

Fun With Python Enums

2022-02-10T00:00:00-05:00

This article demonstrates how to define additional properties for Python 3 enums. Defining an additional property in a Python enum can provide a simple way to provide string values. The concept is then expanded to demonstrate composition, an important concept for functional programming. This article concludes with a demonstration of dynamic dispatch in Python, by further extending an enum.

Adding Properties to Python Enums

Searching for python enum string value yields some complex and arcane ways to approach the problem.

Below is a short example of a Python enum that demonstrates a simple way to provide lower-case string values for enum constants: a new property, to_s, is defined. This property provides the string representation that is required. You could define other properties and methods to suit the needs of other projects.

cad_enums.py

"""Defines enums"""

from enum import Enum, auto

class EntityType(Enum):
    """Types of entities"""
    SITE = auto()
    GROUP = auto()
    COURSE = auto()
    SECTION = auto()
    LECTURE = auto()

    @property
    def to_s(self) -> str:
        """:return: lower-case name of this instance"""
        return self.name.lower()

Adding the following to the bottom of the program allows us to demonstrate it:

cad_enums.py (part 2)

if __name__ == "__main__":  # Just for demonstration
    print("Specifying individual values:")
    print(f"  {EntityType.SITE.value}: {EntityType.SITE.to_s}")
    print(f"  {EntityType.GROUP.value}: {EntityType.GROUP.to_s}")
    print(f"  {EntityType.COURSE.value}: {EntityType.COURSE.to_s}")
    print(f"  {EntityType.SECTION.value}: {EntityType.SECTION.to_s}")
    print(f"  {EntityType.LECTURE.value}: {EntityType.LECTURE.to_s}")

    print("\nIterating through all values:")
    for entity_type in EntityType:
        print(f"  {entity_type.value}: {entity_type.to_s}")

Running the program produces this output:

Shell

$ cad_enums.py
Specifying individual values:
  1: site
  2: group
  3: course
  4: section
  5: lecture

Iterating through all values:
  1: site
  2: group
  3: course
  4: section
  5: lecture

😁

Easy!

Constructing Enums

Enum constructors work the same as other Python class constructors. There are several ways to make a new instance of a Python enum. Let's try two ways by using the Python interpreter. Throughout this article I've inserted a blank line between Python interpreter prompts for readability.

Python

$ python
Python 3.9.7 (default, Sep 10 2021, 14:59:43)
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> from cad_enums import EntityType

>>> # Specify the desired enum constant value symbolically
>>> gtype = EntityType.GROUP
>>> print(gtype)
EntityType.GROUP 

>>> # Specify the desired enum constant value numerically
>>> stype = EntityType(1)
>>> print(stype)
EntityType.SITE

Enum Ordering

A program I am working on needs to obtain the parent EntityType. By 'parent' I mean the EntityType with the next lowest numeric value. For example, the parent of EntityType.GROUP is EntityType.SITE. We can obtain a parent enum by computing its numeric value by adding the following method to the EntityType class definition.

Python

@property
def parent(self) -> 'EntityType':
    """:return: entity type of parent; site has no parent"""
    return EntityType(max(self.value - 1, 1))

The return type above is enclosed in quotes ('EntityType') to keep Python's type checker happy, because this is a forward reference. This is a forward reference because the type is referenced before it is fully compiled.

The complete enum class definition is now:

cad_enums.py

"""Defines enums"""

from enum import Enum, auto

class EntityType(Enum):
    """Types of entities"""
    SITE = auto()
    GROUP = auto()
    COURSE = auto()
    SECTION = auto()
    LECTURE = auto()

    @property
    def to_s(self) -> str:
        """:return: lower-case name of this instance"""
        return self.name.lower()

    @property
    def parent(self) -> 'EntityType':
        """:return: entity type of parent; site has no parent"""
        return EntityType(max(self.value - 1, 1))

Lets try out the new parent property in the Python interpreter.

Python

>>> EntityType.LECTURE.parent
<EntityType.SECTION: 4> 

>>> EntityType.SECTION.parent
<EntityType.COURSE: 3> 

>>> EntityType.COURSE.parent
<EntityType.GROUP: 2> 

>>> EntityType.GROUP.parent
<EntityType.SITE: 1> 

>>> EntityType.SITE.parent
<EntityType.SITE: 1>

Enum Composition

Like methods and properties in all other Python classes, enum methods and properties compose if they return an instance of the class. Composition is also known as method chaining, and also can apply to class properties. Composition is an essential practice of a functional programming style.

The parent property returns an instance of the EntityType enum class, so it can be composed with any other property or method of that class, for example the to_s property shown earlier.

Python

>>> EntityType.LECTURE.parent.to_s
'section' 

>>> EntityType.SECTION.parent.to_s
'course' 

>>> EntityType.COURSE.parent.to_s
'group' 

>>> EntityType.GROUP.parent.to_s
'site' 

>>> EntityType.SITE.parent.to_s
'site'

Dynamic Dispatch

The Python documentation might lead someone to assume that writing dynamic dispatch code is more complex than it actually is.

To summarize the documentation, all Python classes, methods and instances are callable. Callable functions have type Callable[[InputArg1Type, InputArg2Type], ReturnType]. If you do not want any type checking, write Callable[..., Any]. However, this is not very helpful information for dynamic dispatch. Fortunately, working with Callable is very simple.

You can pass around any Python class, constructor, function or method, and later provide it with the usual arguments. Invocation just works.

Let me show you how easy it is to write dynamic dispatch code in Python, let's construct one of five classes, depending on the value of an enum. First, we need a class definition for each enum value:

Python

# pylint: disable=too-few-public-methods

class BaseClass():
    """Demo only"""

class TestLecture(BaseClass):
    """This constructor has type Callable[[int, str], TestLecture]"""
    def __init__(self, id_: int, action: str):
        print(f"CadLecture constructor called with id {id_} and action {action}")

class TestSection(BaseClass):
    """This constructor has type Callable[[int, str], TestSection]"""
    def __init__(self, id_: int, action: str):
        print(f"CadSection constructor called with id {id_} and action {action}")

class TestCourse(BaseClass):
    """This constructor has type Callable[[int, str], TestCourse]"""
    def __init__(self, id_: int, action: str):
        print(f"CadCourse constructor called with id {id_} and action {action}")

class TestGroup(BaseClass):
    """This constructor has type Callable[[int, str], TestGroup]"""
    def __init__(self, id_: int, action: str):
        print(f"CadGroup constructor called with id {id_} and action {action}")

class TestSite(BaseClass):
    """This constructor has type Callable[[int, str], TestSite]"""
    def __init__(self, id_: int, action: str):
        print(f"CadSite constructor called with id {id_} and action {action}")

Now lets add another method, called construct, to EntityType that invokes the appropriate constructor according to the value of an EntityType instance:

Python

@property
def construct(self) -> Callable:
    """:return: the appropriate Callable for each enum value"""
    if self == EntityType.LECTURE:
        return TestLecture
    if self == EntityType.SECTION:
        return TestSection
    if self == EntityType.COURSE:
        return TestCourse
    if self == EntityType.GROUP:
        return TestGroup
    return TestSite

Using named arguments makes your code resistant to problems that might sneak in due to parameters changing over time.

I favor using named arguments at all times; it avoids many problems. As code evolves, arguments might be added or removed, or even reordered.

Let's test out dynamic dispatch in the Python interpreter. A class specific to each EntityType value is constructed by invoking the appropriate Callable and passing it named arguments id_ and action.

Python

>>> EntityType.LECTURE.construct(id_=55, action="gimme_lecture")
TestLecture constructor called with id 55 and action gimme_lecture
<entity_types.TestLecture object at 0x7f9aac690070> 

>>> EntityType.SECTION.construct(id_=13, action="gimme_section")
TestSection constructor called with id 13 and action gimme_section
<entity_types.TestSection object at 0x7f9aac5c1730> 

>>> EntityType.COURSE.construct(id_=40, action="gimme_course")
TestCourse constructor called with id 40 and action gimme_course
<entity_types.TestCourse object at 0x7f9aac6900a0> 

>>> EntityType.GROUP.construct(id_=103, action="gimme_group")
TestGroup constructor called with id 103 and action gimme_group
<entity_types.TestGroup object at 0x7f9aac4c6b20> 

>>> EntityType.SITE.construct(id_=1, action="gimme_site")
TestSite constructor called with id 1 and action gimme_site
<entity_types.TestSite object at 0x7f9aac5c1730>

Because these factory methods return the newly created instance of the desired type, the string representation is printed on the console after the method finishes outputting its processing results, for example: <entity_types.TestLecture object at 0x7f9aac690070>.

Using enums to construct class instances and/or invoke methods (aka dynamic dispatch) is super powerful. It rather resembles generics, actually, even though Python’s support for generics is still in its infancy.

The complete Python program discussed in this article is here.

Linking Directories on NTFS and Ext4 Volumes

2022-02-07T00:00:00-05:00

Sometimes I need to insert some code into a program that depends on the type of format that a drive volume has. For example, today I need to either make a Windows junction to connect two directories on NTFS volumes. On the other hand, if one or both directories were on other types of volumes, I would have to connect the directories using a Linux symlink.

All of the bash scripts shown in this article are meant to run in a Bash shell running on WSL or WSL2.

Use Windows Junctions When Possible

When working on WSL, Windows junctions are more desirable than Linux hard links and symlinks because junctions are visible in Windows and also in WSL. Unlike Linux hard links, which only work within a single volume, Windows junctions can span two volumes. Linux symlinks are only visible from WSL; a symlinked directory only appears as a useless file when viewed from Windows.

Both directories need to be on NTFS volumes to make a Windows junction between them. Junctions are permitted within a single NTFS volume, or between two NTFS volumes. Linux symlinks can be used on all volume types, but only work properly when viewed from Linux.

Windows junctions are shown with a small arrow icon in Windows File Manager. In the image above, the curriculum directory is a junction.

Determining a Volume Type

I wrote the volumeType bash function to obtain the type of the volume that contains a file or directory. Linux ext4 volumes have partition type ext4. NTFS volumes have partition type 9p.

Shell

function volumeType {
  # Usually returns volume types ext4 or 9p (for NTFS)
  df -Th "$1" | tail -n 1 | awk '{print $2}'
}

Here are examples of using volumeType:

Shell

$ volumeType /mnt/c  # Under WSL/WSL2 this is usually NTFS
9p 

$ volumeType /       # For Ubuntu this defaults to ext4
ext4

All of the remaining scripts on this page either return a value (indicating true), or they do not return anything (indicating false).

Two more bash functions test if a file or directory is part of an NTFS or ext4 volume:

Shell

function isNTFS {
  if [ "$( volumeType "$1" )" == 9p ]; then echo yes; fi
}

function isExt4 {
  if [ "$( volumeType "$1" )" == ext4 ]; then echo yes; fi
}

Here are examples of using isNTFS and isExt4:

Shell

$ isNTFS /mnt/c
yes 

$ isExt4 /mnt/c

$ isNTFS /

$ isExt4 /
yes

Windows Junctions

The bothOnNTFS bash function indicates if both of the paths passed to it are on NTFS volumes.

Shell

function bothOnNTFS {
  if [ "$( isNTFS "$1" )" ] && [ "$( isNTFS "$2" )" ]; then echo yes; fi
}

Let's try out bothOnNTFS.

Shell

$ bothOnNTFS /mnt/c /mnt/f
yes 

$ bothOnNTFS /mnt/c /

bothOnNTFS lets us decide how to connect two directories. If they are both on NTFS volumes, we can connect them using a Windows junction; otherwise we'll need to symlink them.

Connecting Via a Windows Junction or Linux Symlink

We could either make a Windows junction using the mklink command, or we could make a Linux symlink using the ln -s command. Notice how the order of parameters between mklink is the reverse of the order of the Linux ln command.

Shell

if [ $( bothOnNTFS "$cadenzaCurriculum" . ) ]; then
    WINDOWS_PATH="$( wslpath -w "$cadenzaCurriculum/site_$TITLE" )"
    echo "Making Windows junction from $cadenzaCurriculum/site_$TITLE to curriculum/"
    cmd.exe /C mklink /j curriculum "$WINDOWS_PATH"
else
  echo "Symlinking $cadenzaCurriculum/site_$TITLE to curriculum/"
  ln -s "$cadenzaCurriculum/site_$TITLE" curriculum
fi

When the above code ran it produced:

Shell

Making Windows junction from /mnt/f/work/cadenzaHome/cadenzaCurriculum/site_ScalaCourses.com to curriculum/
Junction created for curriculum <<===>> F:\work\cadenzaHome\cadenzaCurriculum\site_ScalaCourses.com

Windows Junction for Windows Home in Ubuntu

Neither Linux nor Java appreciate spaces in directories. Spaces in home directories are by nature problematic. If you have a space in your home directory on Windows, you can use the PowerShell New-Item commandlet to create an equivalent junction to a new directory node without spaces in the name.

The following uses the PowerShell New-Item commandlet to create a new directory called C:\Users\mslinn, accessible from WSL as /mnt/c/mslinn, which points to the same directory as C:\Users\Mike Slinn.

PowerShell

PS C:\Users\Mike Slinn> ni "C:\Users\mslinn" -i SymbolicLink -ta "C:\Users\Mike Slinn"

Handcrafted Dynamic DNS for AWS Route53 and Namecheap

2022-01-30T00:00:00-05:00

Now that I have fiber-optic internet service in my apartment, with 500 GB/s upload and download, I thought I would save money by hosting my scalacourses.com website on an Ubuntu server that runs here, instead of AWS. My home IP address is quite stable, and only changes when the fiber modem boots up. The modem is branded as a Bell Home Hub 4000, but I believe it is actually made by Arris (formerly known as Motorola).

It makes little sense to pay the commercial cost of dedicated dynamic DNS services (typically $55 USD / year) when it is so easy to automate, and the operational cost is less than one cent per year.

I wrote two little scripts that automatically check my public IP address, and modifies the DNS record for my home IP address whenever the IP address changes. One script is for sites that use DNS provided by AWS Route53, and the other is for DNS provided by Namecheap.

The approach shown here could be used for all DNS servers that have a command-line interface. This was originally written for AWS Route53, but when I moved off AWS I made a version for Namecheap. Alternative DNS providers include Azure DNS, Cloudflare DNS, DNSMadeEasy, DNSimple, Google Cloud DNS, and UltraDNS.

Forwarding HTTP Requests

I added some entries to the modem so incoming HTTP traffic on ports 80 and 443 would be forwarded to ports 9000 and 9443 on my home server, gojira.

Using the Scripts

The version for AWS Route53 is called dynamicDnsAws, and the version for Namecheap is called dynamicDnsNamecheap.

The scripts save the IP address to a file, and periodically compare the saved value to the current value. Then the scripts modify the DNS record for a specified subdomain whenever the value of the public IP address changes.

Using the AWS Script

Here is the help information for the script:

Shell

$ dynamicDnsAws
dynamicDnsAws - Maintains a dynamic DNS record in AWS Route53

Saves data in '/home/mslinn/.dynamicDnsAws'

Syntax:
  dynamicDnsAws [OPTIONS] SUB_DOMAIN DOMAIN

OPTIONS:
  -v Verbose mode

Example usage:
  dynamicDnsAws www scalacourses.com
  dynamicDnsAws -v www scalacourses.com

Here is a sample usage:

Shell

$ dynamicDnsAws www scalacourses.com
{
    "ChangeInfo": {
        "Id": "/change/C075751811HI18SH4L8L0",
        "Status": "PENDING",
        "SubmittedAt": "2022-01-30T21:10:09.261Z",
        "Comment": "UPSERT a record for www.scalacourses.com"
    }
}

Using the Namecheap Script

Here is the help information for the script:

Shell

$ dynamicDnsNamecheap
dynamicDnsNamecheap - Maintains two Namecheap dynamic DNS records

  Saves data in '/home/mslinn/.dynamicDns'

  Syntax:
    dynamicDnsNamecheap [OPTIONS] DOMAIN PASSWORD

  OPTIONS:
    -v Verbose mode

  Example usage:
    dynamicDnsNamecheap mydomain.com  asdfasdfasdfasdfasdf
    dynamicDnsNamecheap -v mydomain.com asdfasdfasdfasdf

Here is sample usage:

Shell

$ dynamicDnsNamecheap scalacourses.com asdfasdfasdfasdfasdf
<?xml version="1.0" encoding="utf-16"?>
<interface-response>
  <Command>SETDNSHOST</Command>
  <Language>eng</Language>
  <IP>142.126.4.220</IP>
  <ErrCount>0</ErrCount>
  <errors />
  <ResponseCount>0</ResponseCount>
  <responses />
  <Done>true</Done>
  <debug><![CDATA[]]></debug>
</interface-response><?xml version="1.0" encoding="utf-16"?>
<interface-response>
  <Command>SETDNSHOST</Command>
  <Language>eng</Language>
  <IP>142.126.4.220</IP>
  <ErrCount>0</ErrCount>
  <errors />
  <ResponseCount>0</ResponseCount>
  <responses />
  <Done>true</Done>
  <debug><![CDATA[]]></debug>
</interface-response>

Invoking the Scripts from Crontab

A personal crontab can be modified by typing:

Shell

$ crontab -e

I pasted in the following into crontab on my Ubuntu server, running at home. These lines invoke the dynamicDnsNamecheap script via crontab every 5 minutes.

Shell

*/5 * * * * /path/to/dynamicDnsNamecheap my_domain.com asdfasdfasdfasdfasdf

One the above is saved, crontab will run the script every 5 minutes.

Script Source Codes

Here are the bash scripts:

dynamicDnsAws

#!/bin/bash

# Author: Mike Slinn mslinn@mslinn.com
# Written 2022-01-30

export SAVE_FILE_NAME="$HOME/.dynamicDns"

function help {
  echo "$( basename $0 ) - Maintains a dynamic DNS record in AWS Route53

Saves data in '$SAVE_FILE_NAME'

Syntax:
  $( basename $0) [OPTIONS] SUB_DOMAIN DOMAIN

OPTIONS:
  -v Verbose mode

Example usage:
  $( basename $0) my_subdomain mydomain.com
  $( basename $0) -v my_subdomain mydomain.com

"  
  exit 1
}

function upsert {
  export HOSTED_ZONES="$(
    aws route53 list-hosted-zones
  )"

  export HOSTED_ZONE_RECORD="$(
    jq -r ".HostedZones[] | select(.Name == \"$DOMAIN.\")" <<< "$HOSTED_ZONES"
  )"

  export HOSTED_ZONE_RECORD_ID="$(
    jq -r .Id <<< "$HOSTED_ZONE_RECORD"
  )"

  aws route53 change-resource-record-sets \
    --hosted-zone-id "$HOSTED_ZONE_RECORD_ID" \
    --change-batch "{
      \"Comment\": \"UPSERT a record for $SUBDOMAIN.$DOMAIN\",
      \"Changes\": [{
      \"Action\": \"UPSERT\",
        \"ResourceRecordSet\": {
          \"Name\": \"$SUBDOMAIN.$DOMAIN\",
          \"Type\": \"A\",
          \"TTL\": 300,
          \"ResourceRecords\": [{ \"Value\": \"$IP\"}]
        }
      }]
    }"

  echo "$IP" > "$SAVE_FILE_NAME"
}

if [ "$1" == -v ]; then
  export VERBOSE=true
  shift
fi

if [ -z "$2" ]; then help; fi

set -e 

export SUBDOMAIN="$1"
export DOMAIN="$2"
export IP="$( dig +short myip.opendns.com @resolver1.opendns.com )"

if [ ! -f "$SAVE_FILE_NAME" ]; then
  if [ "$VERBOSE" ]; then echo "Creating $SAVE_FILE_NAME"; fi
  upsert; 
elif [ $( cat "$SAVE_FILE_NAME" ) != "$IP" ]; then 
  if [ "$VERBOSE" ]; then 
    echo "Updating $SAVE_FILE_NAME"
    echo "'$IP' was not equal to '$( cat "$SAVE_FILE_NAME" )'"
  fi
  upsert; 
else
  if [ "$VERBOSE" ]; then echo "No change necessary for $SAVE_FILE_NAME"; fi
fi

dynamicDnsNamecheap

#!/bin/bash

# Author: Mike Slinn mslinn@mslinn.com
# Modified from AWS version (dynameicDnsAws) 2022-06-30
# See https://www.namecheap.com/support/knowledgebase/article.aspx/36/11/how-do-i-start-using-dynamic-dns/

export SAVE_FILE_NAME="$HOME/.dynamicDns"

function help {
  echo "$( basename $0 ) - Maintains two Namecheap dynamic DNS records

Saves data in '$SAVE_FILE_NAME'

Syntax:
  $( basename $0) [OPTIONS] DOMAIN PASSWORD

OPTIONS:
  -v Verbose mode

Example usage:
  $( basename $0) mydomain.com  asdfasdfasdfasdfasdf
  $( basename $0) -v mydomain.com asdfasdfasdfasdf

"
  exit 1
}

function upsert {
  curl "https://dynamicdns.park-your-domain.com/update?host=@&domain=$DOMAIN&password=$PASSWORD&ip=$IP"
  curl "https://dynamicdns.park-your-domain.com/update?host=www&domain=$DOMAIN&password=$PASSWORD&ip=$IP"
  echo "$IP" > "$SAVE_FILE_NAME"
  echo ""
}

if [ "$1" == -v ]; then
  export VERBOSE=true
  shift
fi

if [ -z "$2" ]; then help; fi

set -e

export DOMAIN="$1"
export PASSWORD="$2"
export IP="$( dig +short myip.opendns.com @resolver1.opendns.com )"

if [ ! -f "$SAVE_FILE_NAME" ]; then
  if [ "$VERBOSE" ]; then echo "Creating $SAVE_FILE_NAME"; fi
  upsert;
elif [ "$( cat "$SAVE_FILE_NAME" )" != "$IP" ]; then
  if [ "$VERBOSE" ]; then
    echo "Updating $SAVE_FILE_NAME"
    echo "'$IP' was not equal to '$( cat "$SAVE_FILE_NAME" )'"
  fi
  upsert;
else
  if [ "$VERBOSE" ]; then echo "No change necessary for $SAVE_FILE_NAME"; fi
fi

Windows Diskpart Cooperates With Diskmgmt

2022-01-14T00:00:00-05:00

Recently, I wanted to use some old hard drives as backup media. That meant scrubbing all the partitions off the drives, and installing new partitions, which of course would be empty.

Warning – Working with a command line program for system-level operations, without having backed up the system, is like walking on a tightrope without a net.

A mistake could inadvertently wipe out a different hard drive on the computer than you intended. Without the ability to restore the system disk from backup, your computer could become inoperable.

However, I was unable to repartition one of those old drives using the GUI-based Windows diskmgmt.msc disk manager. The drive had soft errors. For some reason, those errors made it impossible for diskmgmt.msc to scan the drive.

The more powerful Windows diskpart, a command line program, was able to get the job done.

This article ends by demonstrating how you can get benefit from the diskmgmt GUI even when you are using the diskpart command line interface. This is possible because every command you type into diskpart causes a Windows system event to be published, and because diskmgmt subscribes to those events, it is able to display the results as they happen.

Starting Diskpart

Run diskpart as administrator as follows:

Press the Windows key. Do not hold it down, just depress it once, as you would do for any other key, and let it go.
Type diskpart.
Use the mouse or arrow keys to select Run as administrator.

A window should open up labeled Microsoft DiskPart. Let's start by listing all the diskpart commands.

Microsoft DiskPart

Microsoft DiskPart version 10.0.19041.964

Copyright (C) Microsoft Corporation.
On computer: BEAR

DISKPART> help

Microsoft DiskPart version 10.0.19041.964

ACTIVE      - Mark the selected partition as active.
ADD         - Add a mirror to a simple volume.
ASSIGN      - Assign a drive letter or mount point to the selected volume.
ATTRIBUTES  - Manipulate volume or disk attributes.
ATTACH      - Attaches a virtual disk file.
AUTOMOUNT   - Enable and disable automatic mounting of basic volumes.
BREAK       - Break a mirror set.
CLEAN       - Clear the configuration information, or all information, off the
              disk.
COMPACT     - Attempts to reduce the physical size of the file.
CONVERT     - Convert between different disk formats.
CREATE      - Create a volume, partition or virtual disk.
DELETE      - Delete an object.
DETAIL      - Provide details about an object.
DETACH      - Detaches a virtual disk file.
EXIT        - Exit DiskPart.
EXTEND      - Extend a volume.
EXPAND      - Expands the maximum size available on a virtual disk.
FILESYSTEMS - Display current and supported file systems on the volume.
FORMAT      - Format the volume or partition.
GPT         - Assign attributes to the selected GPT partition.
HELP        - Display a list of commands.
IMPORT      - Import a disk group.
INACTIVE    - Mark the selected partition as inactive.
LIST        - Display a list of objects.
MERGE       - Merges a child disk with its parents.
ONLINE      - Online an object that is currently marked as offline.
OFFLINE     - Offline an object that is currently marked as online.
RECOVER     - Refreshes the state of all disks in the selected pack.
              Attempts recovery on disks in the invalid pack, and
              resynchronizes mirrored volumes and RAID5 volumes
              that have stale plex or parity data.
REM         - Does nothing. This is used to comment scripts.
REMOVE      - Remove a drive letter or mount point assignment.
REPAIR      - Repair a RAID-5 volume with a failed member.
RESCAN      - Rescan the computer looking for disks and volumes.
RETAIN      - Place a retained partition under a simple volume.
SAN         - Display or set the SAN policy for the currently booted OS.
SELECT      - Shift the focus to an object.
SETID       - Change the partition type.
SHRINK      - Reduce the size of the selected volume.
UNIQUEID    - Displays or sets the GUID partition table (GPT) identifier or
              master boot record (MBR) signature of a disk.

Listing Drives, Partitions and Volumes

Listing the drives is generally a good first step. Let's discover the command for that:

Microsoft DiskPart

DISKPART> list

Microsoft DiskPart version 10.0.19041.964

DISK        - Display a list of disks. For example, LIST DISK.
PARTITION   - Display a list of partitions on the selected disk.
              For example, LIST PARTITION.
VOLUME      - Display a list of volumes. For example, LIST VOLUME.
VDISK       - Displays a list of virtual disks.

OK, we can list disks, partitions, volumes and virtual disks. Let's list the disk drives.

Microsoft DiskPart

DISKPART> list disk

Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online         1863 GB  1024 KB        *
  Disk 1    Online          465 GB  1024 KB
  Disk 2    Online         1863 GB  1024 KB        *
  Disk 3    Online         1863 GB      0 B
* Disk 5    Online          931 GB   931 GB        *

NewerTech Voyager S3 caddy

At this point, I inserted the old 5.25" SATA drive that I wanted to repurpose into a NewerTech Voyager S3 caddy, connected to my computer via a USB 3 cable, and Windows automatically mounted it. The caddy also accepts 2.5" SATA drives, such as those commonly found in laptops.

Now I told diskpart to rescan the drives, and then I listed the volumes on all disks.

Microsoft DiskPart

DISKPART> rescan

Please wait while DiskPart scans your configuration...

DiskPart has finished scanning your configuration.

DISKPART> list volume

Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
  ----------  ---  -----------  -----  ----------  -------  ---------  --------
  Volume 0     D                       DVD-ROM         0 B  No Media
  Volume 1     C   BEAR_C       NTFS   Partition   1861 GB  Healthy    Boot
  Volume 2                      FAT32  Partition    100 MB  Healthy    System
  Volume 3                      NTFS   Partition    539 MB  Healthy    Hidden
  Volume 4                      NTFS   Partition    450 MB  Healthy    Hidden
  Volume 5     F   Work         NTFS   Partition   1863 GB  Healthy
  Volume 6     E   BEAR_E       NTFS   Partition   1863 GB  Healthy
  Volume 8                      FAT32  Partition    512 MB  Healthy    Hidden

Diskpart displayed the hidden volume (#8) in the drive in the caddy. This drive has readonly status set, which prevents its contents from being modified or deleted. That would be good if I wanted to use this drive as an archive, but instead I want to scrub it and write new information on it. The currently existing partitions on this drive cannot be erased until readonly is cleared. Lets remove readonly status from the drive now:

Microsoft DiskPart

DISKPART> select volume 8
Volume 8 is the selected volume.

DISKPART> attributes disk clear readonly
Disk attributes cleared successfully.

DISKPART> rescan
Please wait while DiskPart scans your configuration...
DiskPart has finished scanning your configuration.

DISKPART> list volume
Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
  ----------  ---  -----------  -----  ----------  -------  ---------  --------
  Volume 0     D                       DVD-ROM         0 B  No Media
  Volume 1     C   BEAR_C       NTFS   Partition   1861 GB  Healthy    Boot
  Volume 2                      FAT32  Partition    100 MB  Healthy    System
  Volume 3                      NTFS   Partition    539 MB  Healthy    Hidden
  Volume 4                      NTFS   Partition    450 MB  Healthy    Hidden
  Volume 5     F   Work         NTFS   Partition   1863 GB  Healthy
  Volume 6     E   BEAR_E       NTFS   Partition   1863 GB  Healthy
  Volume 8                      FAT32  Partition    512 MB  Healthy    Hidden

Wiping the Drive

Now it is time to wipe the drive clean, which removes all partitions.

Microsoft DiskPart

DISKPART> select disk 5
Disk 5 is now the selected disk.

DISKPART> list disk
Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online         1863 GB  1024 KB        *
  Disk 1    Online          465 GB  1024 KB
  Disk 2    Online         1863 GB  1024 KB        *
  Disk 3    Online         1863 GB      0 B
* Disk 5    Online          931 GB   931 GB        *

DISKPART> clean
DiskPart succeeded in cleaning the disk.

Archiving a Drive

If instead of wiping the drive, I wanted to archive the drive, I would want to set the read-only status. To do that, first select the drive as before, then type:

Microsoft DiskPart

DISKPART> list disk
# Output as shown above 

DISKPART> select disk N
Disk N is now the selected disk. 

DISKPART> attributes disk set readonly

Now the drive's contents could not accidently be erased or modified.

Create A New Partition

We need to create a new partition that spans the entire disk on the now-empty selected drive. The create command can do that. Let's look at the help before using the command:

Microsoft DiskPart

DISKPART> create
Microsoft DiskPart version 10.0.19041.964

PARTITION   - Create a partition.
VOLUME      - Create a volume.
VDISK       - Creates a virtual disk file.

DISKPART> create partition

Microsoft DiskPart version 10.0.19041.964

EFI         - Create an EFI system partition.
EXTENDED    - Create an extended partition.
LOGICAL     - Create a logical drive.
MSR         - Create a Microsoft Reserved partition.
PRIMARY     - Create a primary partition.

To create a new partition that spans the entire disk on the now-empty selected drive and make it active:

Microsoft DiskPart

DISKPART> create partition primary
DiskPart succeeded in creating the specified partition.

DISKPART> select partition 1
Partition 1 is now the selected partition.

DISKPART> active
DiskPart marked the current partition as active.

Format A Volume

Let's format the entire selected drive as one volume. Like the clean command, the format command operates on the currently selected disk. First, let's look at the help:

Microsoft DiskPart

DISKPART> help format

Formats the specified volume for use with Windows.

Syntax:  FORMAT [[FS=<FS>] [REVISION=<X.XX>] | RECOMMENDED] [LABEL=<"label">]
                [UNIT=<N>] [QUICK] [COMPRESS] [OVERRIDE] [DUPLICATE] [NOWAIT]
                [NOERR]

    FS=     Specifies the type of file system. If no file system is given,
                the default file system displayed by the FILESYSTEMS command is
                used.

    REVISION=<X.XX>

                Specifies the file system revision (if applicable).

    RECOMMENDED If specified, use the recommended file system and revision
                instead of the default if a recommendation exists. The
                recommended file system (if one exists) is displayed by the
                FILESYSTEMS command.

    LABEL=<"label">

                Specifies the volume label.

    UNIT=<N>    Overrides the default allocation unit size. Default settings
                are strongly recommended for general use. The default
                allocation unit size for a particular file system is displayed
                by the FILESYSTEMS command.

                NTFS compression is not supported for allocation unit sizes
                above 4096.

    QUICK       Performs a quick format.

    COMPRESS    NTFS only: Files created on the new volume will be compressed
                by default.

    OVERRIDE    Forces the file system to dismount first if necessary. All
                opened handles to the volume would no longer be valid.

    DUPLICATE   UDF Only: This flag applies to UDF format, version 2.5 or
                higher.
                This flag instructs the format operation to duplicate the file
                system meta-data to a second set of sectors on the disk. The
                duplicate meta-data is used by applications, for example repair
                or recovery applications. If the primary meta-data sectors are
                found to be corrupted, the file system meta-data will be read
                from the duplicate sectors.

    NOWAIT      Forces the command to return immediately while the format
                process is still in progress. If NOWAIT is not specified,
                DiskPart will display format progress in percentage.

    NOERR       For scripting only. When an error is encountered, DiskPart
                continues to process commands as if the error did not occur.
                Without the NOERR parameter, an error causes DiskPart to exit
                with an error code.

    A volume must be selected for this operation to succeed.

Examples:

    FORMAT FS=NTFS LABEL="New Volume" QUICK COMPRESS
    FORMAT RECOMMENDED OVERRIDE

Diskpart automatically chooses the optimal file system for the selected drive if you do not specify it. Choices include FAT, FAT32 and NTFS. To quick format the selected drive using the optimal file system:

Microsoft DiskPart

DISKPART> format quick

The default is to fully format the selected drive, which is what you want if the selected drive is suspect:

Microsoft DiskPart

DISKPART> format

You could specify multiple parameters, for example:

Microsoft DiskPart

DISKPART> format fs=ntfs label="My Drive" quick

Assigning a Drive Letter

Once the drive was formatted, I assigned the selected drive the letter G. You do not normally need to perform this step, unless you want to define a default letter for this drive.

Microsoft DiskPart

DISKPART> assign letter=G

Diskmgmt GUI Shows Progress

Having a GUI continuously report the current state of the drive maintenance you are performing using a command-line interface is a good practice.

The GUI-based Windows disk manager, diskmgmt.msc, can show the instantaneous progress of all diskpart commands, working on every drive, including creating and deleting partitions, volumes, formatting and much more. For example, formatting progress can be seen here:

Run diskmgmt.msc by:

Press the Windows key once and let go.
Type diskmgmt
Press Enter

Exit

To exit diskpart, either type Exit or press CTRL-C.

Microsoft DiskPart

DISKPART> exit

The drive is ready for its next assignment!

WSL / WSL 2 Backup and Restore

2022-01-10T00:00:00-05:00

I needed to back up my WSL2 installation before reinstalling Windows 10, as one must do every 6 months if you work your machine like a developer. It had been years since I had last refreshed this machine, and it was now bluescreening several times a day. Reboots took forever.

This article presents the best WSL backup approaches I found

I wanted to retain my WSL2 instance. When refreshing Windows you must reinstall all your programs, and you lose all the WSL/WSL2 instances. The refresh decommissions them, then moves them into a hidden directory tree, along with the rest of the stuff stored in your old Windows 10 profile directory tree.

I also wanted to be able to replicate my WSL2 instance reliably and easily. It was very important to me to be able to administer WSL instances separately from my Windows 10 profile.

Location, Location, Location

Only recently has it become possible to work with WSL/WSL2 images on non-system drives. Most online documentation is now out of date in this regard. Running Windows off a different drive than the drive that a WSL/WSL2 client OS runs from can provide a dramatic performance boost for some applications.

Reinstalling Windows is less traumatic if the WSL/WSL2 image elsewhere in the filesystem

Dell’s Ignorance Causes Pain

I use Dell laptops, because I love their onsite warranty price and product features. However, sometimes it feels like Dell’s solution to every problem is to replace the motherboard.

When Windows 10 encounters a new motherboard, it takes anti-piracy measures, and refuses to do much of anything useful.

Refreshing Windows solves that problem, but doing that blows away the standard WSL/WSL2 image. Again. And again. And again!

Dell replaced the motherboard 4 times in 2 years for one of my laptops. They simply do not understand how replacing the motherboard creates long-term issues.

Test Your Backup

Backing up WSL/WSL2 can be problematic. Restoring it is even more delicate. Knowing this, I decided to backup and test the restoration process before refreshing Windows and potentially losing my working system.

I have tried several times before to make this work. As I said, it had been years since I allowed the OS in my main workstation to be refreshed. The reason I resisted doing proper maintenance was because I wanted to preserve my WSL2 Ubuntu image. Until today, I met with frustrating failures every time I attempted to use the standard tools. Today I succeeded, via new software, hence the publication of my notes. Who wants to read stories about all the things that do not work?

Following is my experience. I began by following the directions in How to back up a Windows Subsystem for Linux (WSL) distribution, published 18 Feb 2021 by Windows Central. It is a nice story, but I've tried this stuff on a variety of computers, and this is not Microsoft’s most robust code base. Read on and I will tell you of reality as I found it.

WSL Command-Line Options

First lets note the software versions I am using:

cmd.exe

Microsoft Windows [Version 10.0.19045.4412]
(c) Microsoft Corporation. All rights reserved. 

C:\Users\Mike Slinn>wsl -v
WSL version: 2.1.3.0
Kernel version: 5.15.146.1-2
WSLg version: 1.0.60
MSRDC version: 1.2.5105
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.19045.4412

Now lets look at the command-line options for the Native Windows wsl command (which is actually wsl.exe).

cmd.exe (continued)

C:\Users\Mike Slinn>wsl --help
Copyright (c) Microsoft Corporation. All rights reserved.
For privacy information about this product please visit https://aka.ms/privacy.

Usage: wsl.exe [Argument] [Options...] [CommandLine]

Arguments for running Linux binaries:

    If no command line is provided, wsl.exe launches the default shell.

    --exec, -e 
        Execute the specified command without using the default Linux shell.

    --shell-type 
        Execute the specified command with the provided shell type.

    --
        Pass the remaining command line as-is.

Options:
    --cd 
        Sets the specified directory as the current working directory.
        If ~ is used the Linux user's home path will be used. If the path begins
        with a / character, it will be interpreted as an absolute Linux path.
        Otherwise, the value must be an absolute Windows path.

    --distribution, -d 
        Run the specified distribution.

    --user, -u 
        Run as the specified user.

    --system
        Launches a shell for the system distribution.

Arguments for managing Windows Subsystem for Linux:

    --help
        Display usage information.

    --debug-shell
        Open a WSL2 debug shell for diagnostics purposes.

    --install [Distro] [Options...]
        Install a Windows Subsystem for Linux distribution.
        For a list of valid distributions, use 'wsl.exe --list --online'.

        Options:
            --no-launch, -n
                Do not launch the distribution after install.

            --web-download
                Download the distribution from the internet instead of the Microsoft Store.

            --no-distribution
                Only install the required optional components, does not install a distribution.

            --enable-wsl1
                Enable WSL1 support.

    --manage  
        Changes distro specific options.

        Options:
            --set-sparse, -s 
                Set the vhdx of distro to be sparse, allowing disk space to be automatically reclaimed.

    --mount 
        Attaches and mounts a physical or virtual disk in all WSL 2 distributions.

        Options:
            --vhd
                Specifies that  refers to a virtual hard disk.

            --bare
                Attach the disk to WSL2, but don't mount it.

            --name 
                Mount the disk using a custom name for the mountpoint.

            --type 
                Filesystem to use when mounting a disk, if not specified defaults to ext4.

            --options 
                Additional mount options.

            --partition 
                Index of the partition to mount, if not specified defaults to the whole disk.

    --set-default-version 
        Changes the default install version for new distributions.

    --shutdown
        Immediately terminates all running distributions and the WSL 2
        lightweight utility virtual machine.

    --status
        Show the status of Windows Subsystem for Linux.

    --unmount [Disk]
        Unmounts and detaches a disk from all WSL2 distributions.
        Unmounts and detaches all disks if called without argument.

    --update
        Update the Windows Subsystem for Linux package.

        Options:
            --pre-release
                Download a pre-release version if available.

    --version, -v
        Display version information.

Arguments for managing distributions in Windows Subsystem for Linux:

    --export   [Options]
        Exports the distribution to a tar file.
        The filename can be - for stdout.

        Options:
            --vhd
                Specifies that the distribution should be exported as a .vhdx file.

    --import    [Options]
        Imports the specified tar file as a new distribution.
        The filename can be - for stdin.

        Options:
            --version 
                Specifies the version to use for the new distribution.

            --vhd
                Specifies that the provided file is a .vhdx file, not a tar file.
                This operation makes a copy of the .vhdx file at the specified install location.

    --import-in-place  
        Imports the specified .vhdx file as a new distribution.
        This virtual hard disk must be formatted with the ext4 filesystem type.

    --list, -l [Options]
        Lists distributions.

        Options:
            --all
                List all distributions, including distributions that are
                currently being installed or uninstalled.

            --running
                List only distributions that are currently running.

            --quiet, -q
                Only show distribution names.

            --verbose, -v
                Show detailed information about all distributions.

            --online, -o
                Displays a list of available distributions for install with 'wsl.exe --install'.

    --set-default, -s 
        Sets the distribution as the default.

    --set-version  
        Changes the version of the specified distribution.

    --terminate, -t 
        Terminates the specified distribution.

    --unregister 
        Unregisters the distribution and deletes the root filesystem.

Backing Up With WSL

Now let's use the wsl command to back up the Ubuntu VM in WSL2.

cmd.exe

C:\Users\Mike Slinn>wsl --export Ubuntu ubuntuBear_2021-01-10.tar

Well, well, it backed up without any problem in about an hour! Color me ~~surprised~~happy. File size was about 50 GB.

Alright, let's try importing the image now. We'll locate the new image at f:\ubuntuBear. It used to be that WSL did not support moving or installing a distro to non-system drives. No longer!

cmd.exe

C:\Users\Mike Slinn>wsl --import Ubuntu f:\ubuntuBear ubuntuBear_2021-01-10.tar
A distribution with the supplied name already exists.

The error message A distribution with the supplied name already exists makes sense because I backed up a WSL2 instance called Ubuntu and instance names must be unique. Let's import under the name UbuntuBear:

cmd.exe

C:\Users\Mike Slinn>wsl --import UbuntuBear f:\ubuntuBear ubuntuBear_2021-01-10.tar
Unspecified error

Ahh, the dreaded Unspecified error that wsl import is infamous for. I found a few potential solutions, detailed below; I show the most desirable solution first.

Solution 1: Importing the VDHX File

Importing the original vhdx file directly instead of creating and importing the tar file has been reported to work by several people. This is a more desirable solution because it requires less work and is faster. Note that the ^ character is a DOS command-line continuation character, much like the Bash \ character.

cmd.exe

C:\Users\Mike Slinn>wsl --import Ubuntu-20.04-d^
  D:\WSL\Ubuntu-22.04\^
  %HOMEDRIVE%%HOMEPATH%\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu20.04onWindows_79rhkp1fndgsc\LocalState\ext4.vhdx^
  --vhd

Solution 2: Updating Linux Kernel

The following was written by ken-cdit:

Had the same issue. I exported two WSL 2 distros and then did a clean install of my Windows OS. "Unspecified error" came up when I tried to import them.

I fixed it by downloading the Linux kernel update package referred to at https://docs.microsoft.com/en-us/windows/wsl/install-win10 and then running the command wsl --set-default-version 2.

After this, my two distros imported without error.

I had the whole thing still in the console so here is a screencap...

Solution 3: LxRunOffline

Installing LxRunOffline

Google brought me to a potential solution, LxRunOffline, which provided me limited success.

I downloaded the binaries in LxRunOffline-v3.5.0-msvc.zip directly from GitHub into C:\Program Files\LxRunOffline.

Now add C:\Program Files\LxRunOffline to the Windows PATH:

cmd.exe

C:\Users\Mike Slinn>setx PATH "%PATH%;C:\Program Files\LxRunOffline"

SUCCESS: Specified value was saved. %}

I then ran the following in an administrative shell to register the DLL:

cmd.exe (continued)

C:\Users\Mike Slinn>regsvr32 "C:\Program Files\LxRunOffline\LxRunOfflineShellExt.dll"

LxRunOffline Help Info

Let's progressively discover how this command-line program can be used. First I'll just type the command name, which causes the program to list its top-level actions.

cmd.exe

C:\Users\Mike Slinn>LxRunOffline
[ERROR] No action is specified.

Supported actions are:
    l, list            List all installed distributions.
    gd, get-default    Get the default distribution, which is used by bash.exe.
    sd, set-default    Set the default distribution, which is used by bash.exe.
    i, install         Install a new distribution.
    ui, uninstall      Uninstall a distribution.
    rg, register       Register an existing installation directory.
    ur, unregister     Unregister a distribution but not delete the installation directory.
    m, move            Move a distribution to a new directory.
    d, duplicate       Duplicate an existing distribution in a new directory.
    e, export          Export a distribution"s filesystem to a .tar.gz file, which can be imported by the "install" command.
    r, run             Run a command in a distribution.
    di, get-dir        Get the installation directory of a distribution.
    gv, get-version    Get the filesystem version of a distribution.
    ge, get-env        Get the default environment variables of a distribution.
    se, set-env        Set the default environment variables of a distribution.
    ae, add-env        Add to the default environment variables of a distribution.
    re, remove-env     Remove from the default environment variables of a distribution.
    gu, get-uid        Get the UID of the default user of a distribution.
    su, set-uid        Set the UID of the default user of a distribution.
    gk, get-kernelcmd  Get the default kernel command line of a distribution.
    sk, set-kernelcmd  Set the default kernel command line of a distribution.
    gf, get-flags      Get some flags of a distribution. See https://docs.microsoft.com/en-us/previous-versions/windows/desktop/api/wslapi/ne-wslapi-wsl_distribution_flags for details.
    sf, set-flags      Set some flags of a distribution. See https://docs.microsoft.com/en-us/previous-versions/windows/desktop/api/wslapi/ne-wslapi-wsl_distribution_flags for details.
    s, shortcut        Create a shortcut to launch a distribution.
    ec, export-config  Export configuration of a distribution to an XML file.
    ic, import-config  Import configuration of a distribution from an XML file.
    sm, summary        Get general information of a distribution.
    version            Get version information about this LxRunOffline.exe.

Alright, let's get information about the i (install) option.

cmd.exe (continued)

C:\Users\Mike Slinn>LxRunOffline i
[ERROR] the option '-d' is required but missing

Options:
  -n arg                Name of the distribution
  -d arg                The directory to install the distribution into.
  -f arg                The tar file containing the root filesystem of the
                        distribution to be installed. If a file of the same
                        name with a .xml extension exists and "-c" isn"t
                        specified, that file will be imported as a config file.
  -r arg                The directory in the tar file to extract. This argument
                        is optional.
  -c arg                The config file to use. This argument is optional.
  -v arg (=2)           The version of filesystem to use, latest available one
                        if not specified.
  -s                    Create a shortcut for this distribution on Desktop.

I'll use the -d option to specify the directory to install into, as well as the -f and -n options.

LxRunOffline Import

I feel brave, let's try importing for real now:

cmd.exe

C:\Users\Mike Slinn>LxRunOffline i -d f:/ubuntuBear -n UbuntuBear -f ubuntuBear_2021-01-10.tar
[ERROR] The distro "UbuntuBear" already exists.

Some debris remains from the failed import (remember the Unspecified error a moment ago?). I will delete the b0rked UbuntuBear instance in a moment. Let's call this newly cloned linux instance UbuntuBear2.

cmd.exe (continued)

C:\Users\Mike Slinn>LxRunOffline i -d f:/ubuntuBear -n UbuntuBear2 -f ubuntuBear_2021-01-10.tar
[WARNING] Ignoring an unsupported file "var/lib/docker/volumes/backingFsBlockDev" of type 0060000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-17041-8893592-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10716-846285-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1899-75308257-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10151-8749190-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-18789-129803847-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-811-127664322-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12447-115564106-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-24480-65839207-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-23230-107496852-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-23230-107496852-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-24480-65839207-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-14211-5836039-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19778-110293600-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1969-34761241-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-17041-8893592-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1899-75308257-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-7642-17996-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10151-8749190-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19082-8772885-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12857-5829248-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-26692-70009588-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12447-115564106-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-22582-1181861-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-14211-5836039-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1969-34761241-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-7642-17996-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10716-846285-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-11424-57667328-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-28497-49814437-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-18789-129803847-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-2234-80045-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19082-8772885-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12857-5829248-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-22582-1181861-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-26692-70009588-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10990-5814132-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-811-127664322-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10990-5814132-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-11424-57667328-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-2234-80045-out" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19778-110293600-in" of type 0010000.
[WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-28497-49814437-in" of type 0010000.
[WARNING] Ignoring an unsupported file "dev/random" of type 0020000.
[WARNING] Ignoring an unsupported file "dev/tty" of type 0020000.
[WARNING] Ignoring an unsupported file "dev/full" of type 0020000.
[WARNING] Ignoring an unsupported file "dev/urandom" of type 0020000.
[WARNING] Ignoring an unsupported file "dev/ptmx" of type 0020000.
[WARNING] Ignoring an unsupported file "dev/zero" of type 0020000.
[WARNING] Ignoring an unsupported file "dev/console" of type 0020000.
[WARNING] Ignoring an unsupported file "dev/null" of type 0020000.
[WARNING] Ignoring an unsupported file "dev/mapper/control" of type 0020000.
[ERROR] Couldn"t create the file "\\?\f:\ubuntuBear\rootfs\home\mslinn\.atom\packages\markdown-preview-plus\spec\fixtures\subdir\�cc�nt�d.md".
Reason: The file exists.

Most of the warnings do not seem to be important. I do not care about Docker anyway. Also, now I know that temporary files and dev nodes should all be deleted before running this program. Even better, the program that created the tar should be modified to not attempt to replicate any temporary files.

I don't understand the problem with the atom package, but I'll try to delete all of the atom settings from the tar, along with the other problematic directories, and then retry.

Cleaning Up WSL

First let's see the VMs registered with WSL:

cmd.exe (continued)

C:\Users\Mike Slinn>wsl -l
Windows Subsystem for Linux Distributions:
Ubuntu (Default)
UbuntuBear
UbuntuBear2

Let's delete the debris remaining from the failed UbuntuBear and UbuntuBear2 imports:

cmd.exe (continued)

C:\Users\Mike Slinn>wsl --unregister UbuntuBear
Unregistering... 

C:\Users\Mike Slinn> wsl --unregister UbuntuBear2
Unregistering...

Attempting to delete the directory created by LxRunOffline hit a corrupted directory (scroll the output below to see the error).

cmd.exe (continued)

C:\Users\Mike Slinn>del /s /q f:\ubuntuBear
Deleted file - f:\ubuntuBear\rootfs\init
Deleted file - f:\ubuntuBear\rootfs\lib
Deleted file - f:\ubuntuBear\rootfs\lib32
Deleted file - f:\ubuntuBear\rootfs\lib64
Deleted file - f:\ubuntuBear\rootfs\libx32
Deleted file - f:\ubuntuBear\rootfs\sbin
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.wget-hsts
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.Xauthority
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zcompdump
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zcompdump-Bear-5.8
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zcompdump-localhost-5.8
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zshenv
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zshrc
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zshrc.pre-oh-my-zsh
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zsh_history
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\ancient_warmth_workspace.code-workspace
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\bear2.zip
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\bear3.zip
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\bearDirs.tar
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\dead.letter
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\django_bash_completion
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\jekyll_workspace.code-workspace
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\msp.txt
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\nodesource_setup.sh
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\package-lock.json
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\package.json
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\worldPeaceMusicCollective.png
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\worldPeaceMusicCollectiveBordered.png
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.vscode-server\extensions\wix.vscode-import-cost-2.15.0\package.json
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.vscode-server\extensions\wix.vscode-import-cost-2.15.0\pom.xml
Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.vscode-server\extensions\wix.vscode-import-cost-2.15.0\README.md
The file or directory is corrupted and unreadable.

The file or directory is corrupted and unreadable.

That needs to be dealt with right away! I fixed the directory errors in drive F like this:

cmd.exe

C:\Program Files>chkdsk /F F:
The type of the file system is NTFS.

Chkdsk cannot run because the volume is in use by another
process.  Chkdsk may run if this volume is dismounted first.
ALL OPENED HANDLES TO THIS VOLUME WOULD THEN BE INVALID. y

Chkdsk cannot dismount the volume because it is a system drive or
there is an active paging file on it.  Would you like to schedule
this volume to be checked the next time the system restarts? (Y/N) y

This volume will be checked the next time the system restarts.

I rebooted the system, and it fixed the errors on drive F:.

Pruning the tar

First lets back up the tar that we want to prune.

Shell

$ pushd '/mnt/c/Users/Mike Slinn/'

$ ls -alF *.tar
-rwxr--r-- 1 mslinn mslinn   524482560 Jan 10 18:45 'bear_ubuntu_2021-01-10.tar'*
-rwxr--r-- 1 mslinn mslinn 51464028160 Jan 10 21:11 'ubuntuBear_2021-01-10.tar'* 

$ cp ubuntuBear_2021-01-10.tar ubuntuBearPruned_2021-01-10.tar

Now lets try to prune out all the problematic, and unnecessary, files.

Shell

$ tar -f ubuntuBearPruned_2021-01-10.tar \
  --delete dev/* \
  --delete tmp/* \
  --delete home/mslinn/.atom/* \
  --delete var/lib/docker/* \
  --delete var/sitesUbuntu/* \
  --delete var/work/* \
  --delete var/tmp/*
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.fuseoverlayfs.opaque'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.fuseoverlayfs.opaque'
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: dev/*: Not found in archive
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: tmp/*: Not found in archive
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: home/mslinn/.atom/*: Not found in archive
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: var/lib/docker/*: Not found in archive
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: var/sitesUbuntu/*: Not found in archive
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: var/work/*: Not found in archive
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: var/tmp/*: Not found in archive
tar: Exiting with failure status due to previous errors

I think the error messages just indicate that the tar was made by BSD-TAR, while Ubuntu uses GNU-TAR. I installed bsdtar like this:

Shell

$ yes | sudo apt-get install libarchive-tools
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  libarchive-tools
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 57.1 kB of archives.
After this operation, 207 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu hirsute/universe amd64 libarchive-tools amd64 3.4.3-2 [57.1 kB]
Fetched 57.1 kB in 0s (167 kB/s)
Selecting previously unselected package libarchive-tools.
(Reading database ... 244498 files and directories currently installed.)
Preparing to unpack .../libarchive-tools_3.4.3-2_amd64.deb ...
Unpacking libarchive-tools (3.4.3-2) ...
Setting up libarchive-tools (3.4.3-2) ...
Processing triggers for man-db (2.9.4-2) ...

I tried again using bsdtar, which does not support GNU tar's --delete option. Unfortunately, I only got a 1KB file out, no matter what I did.

Shell

$ bsdtar -cvf ubuntuBearPruned2_2021-01-10.tar \
  --exclude 'dev/*' \
  --exclude 'tmp/*' \
  --exclude 'home/mslinn/.atom/*' \
  --exclude 'var/lib/docker/*' \
  --exclude 'var/sitesUbuntu/*' \
  --exclude 'var/work/*' \
  --exclude 'var/tmp/*' \
  @ubuntuBear_2021-01-10.tar

Maybe there is a bug. Maybe there is bad documentation. I do not think I made an error. Life is short. Bugs are rampant. Illegitimi non carborundum!

I give up on this direction. Let's try to win some other way!

Success Duplicating the Ubuntu Instance

Let’s try duplicating the Ubuntu instance with LxRunOffline. First the Ubuntu VM must be terminated.

cmd.exe

C:\Users\Mike Slinn> wsl -t Ubuntu

C:\Users\Mike Slinn> LxRunOffline duplicate -n Ubuntu -N UbuntuWsl2 -d f:\UbuntuWsl2

The above ran for a couple of hours and concluded without error. LxRunOffline duplicate automatically registers the new instance.

The F:\UbuntuWsl2directory was 239 GB. That's a fair-sized VM. The directory only contains one file, called ext4.vhdx.

Let's see the Ubuntu instances that are currently set:

cmd.exe

C:\Users\Mike Slinn> wsl --list
Windows Subsystem for Linux Distributions:
Ubuntu (Default)
UbuntuWsl2

😁

That is what I wanted to see!

I started the new UbuntuWsl2 Ubuntu instance like this:

cmd.exe

C:\Users\Mike Slinn> wsl -d UbuntuWsl2

To set the default Ubuntu instance I typed:

cmd.exe

C:\Users\Mike Slinn> wsl --setdefault UbuntuWsl2

Let's verify that worked:

cmd.exe

C:\Users\Mike Slinn> wsl --list
Windows Subsystem for Linux Distributions:
UbuntuWsl2 (Default)
Ubuntu

I then deleted the huge tar files that I had created in earlier steps. I never needed them because I used LxRunOffline duplicate.

Ubuntu on 500GB USB3 SSD

I have a Sandisk Extreme 500GB USB3 SSD drive. This tiny, light, and very portable storage device should be perfect for holding my Ubuntu development system. How cool it would be to be able to drop it into a small pocket!

Better yet, the performance of portable USB3 SSD drives blows away traditional hard drives.

For example, assume that your SSD drive appears as drive X. Also assume that LxRunOffline.exe and LxRunOffline.dll have been copied to the root directory of the SSD drive. To register your portable Ubuntu you would simply type:

cmd.exe

C:\> X:LxRunOffline register X:\MyUbuntu -n UbuntuMSlinn

You can walk up to any recently updated Windows 10 machine, plug your tiny USB3 SSD drive into a USB3 or USB3.1 port, run the registration command, and boom! you are productive with great storage performance.

😁

Duplicating to SSD

Again, the Ubuntu VM must be terminated.

cmd.exe

C:\Users\Mike Slinn> wsl -t Ubuntu

C:\Users\Mike Slinn> LxRunOffline duplicate -n Ubuntu -N UbuntuWsl2Extreme -d I:\UbuntuWsl2Extreme

The above ran for a couple of hours and concluded without error. LxRunOffline duplicate automatically registers the new instance.

Let's see the Ubuntu instances that are currently set:

cmd.exe

C:\Users\Mike Slinn> wsl --list
Windows Subsystem for Linux Distributions:
Ubuntu (Default)
UbuntuWsl2
UbuntuWsl2Extreme

😁

LxRunOffline Update 2023-04-25

Arman Mazloumzadeh posted the following:

I finally resolved this amiss during the following steps (Windows 10.0.19045 Build 19045):

Running wsl.exe --import Ubuntu A:\wsl\Ubuntu A:\wsl\Ubuntu.tar = > Unspecified error

Running LxRunOffline.exe install -n Ubuntu -d A:\wsl\Ubuntu -f A:\wsl\Ubuntu.tar => [ERROR] Couldn't get the value "DistributionName" of the registry key "Software\Microsoft\Windows\CurrentVersion\Lxss\AppxInstallerCache"

Removing the Software\Microsoft\Windows\CurrentVersion\Lxss\AppxInstallerCache registry key
Running LxRunOffline.exe install -n Ubuntu -d A:\wsl\Ubuntu -f A:\wsl\Ubuntu.tar => [ERROR] Couldn't get the value "DistributionName" of the registry key "Software\Microsoft\Windows\CurrentVersion\Lxss\TryStoreWSL"

Removing the Software\Microsoft\Windows\CurrentVersion\Lxss\TryStoreWSL registry key
Running LxRunOffline.exe install -n Ubuntu -d A:\wsl\Ubuntu -f A:\wsl\Ubuntu.tar => ✔

Running ubuntu.exe config --default-user arman => ✔

done 🎉

Starting and Stopping Portable VMs

You can start the WSL VM called UbuntuMSlinn by using wsl -d:

cmd.exe

C:\> wsl -d UbuntuMSlinn

You can stop it by using wsl -t:

cmd.exe

C:\> wsl -t UbuntuMSlinn

Microsoft WSL Issue Reporting

In the course of writing this article I found the web page for reporting a WSL issue, so that Microsoft can look at it.

AI / ML System Behavior Reflects the Society That Produced It

2021-12-26T00:00:00-05:00

Artificial intelligence systems, including machine learning systems, are primarily software-driven. Like all software, AI and ML systems reflect the societies that produced them. Conway’s Law explains how the internal organization of software reflects the organization that created it.

I postulate that the behavior of AI & ML systems reflect the society that the systems are embedded in. Do you find the AI systems are instrusive or exploitive? Look in the collective mirror for the society it was created by.

If the populace is viewed as merely something for corporations to exploit, and this view is accepted at enough levels in society, that becomes the status quo. In my opinion, the USA in 2022 has definitely reached that point.

Conway’s Law

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.

– Melvin E. Conway, 1967

Allan Kelly’s Corollary

Organisational design is system design.

– Allan Kelly, 2005

The The Mirroring Hypothesis: Theory, Evidence and Exceptions by Lyra J. Colfer and Carliss Y. Baldwin was published by the Harvard Business School in 2016.

A large-scale system is one that stretches across time and space. A large organization similarly stretches across time and space. Conway’s Law ties the two. How we organize defines how we think collectively, and thus what we make collectively...

... To build software, first build a community...

Software is essentially about people, and the larger-scale we make it, the more that truth becomes visible. One program reflects how one person thinks. A large-scale application reflects how many people think together.

– Pieter Hintjes, Sex in Title, and Other Stories, Dec 16, 2013

Do you even care? If so, please be the change you want to see in the world.

We but mirror the world. All the tendencies present in the outer world are to be found in the world of our body. If we could change ourselves, the tendencies in the world would also change. As a man changes his own nature, so does the attitude of the world change towards him. This is the divine mystery supreme. A wonderful thing it is and the source of our happiness. We need not wait to see what others do. – From Mahatma Gandhi

Spring-Breezifier: Solving COVID-19 With HVAC

2021-12-22T00:00:00-05:00

When the COVID-19 pandemic began, the world was unprepared and no one knew how the virus was transmitted. As the medical profession lurched awkwardly into action, it gave advice that later turned out to be incorrect. As time went by, important details about how the virus is transmitted were revealed, and the previous medical advice had become politicized. The new advice has not yet been generally adopted as policy and is currently unknown to most people.

For every complex problem there is an answer that is clear, simple, and wrong.

– H. L. Mencken 1880-1956.

About This Article

This article's intent is to show how the COVID-19 pandemic could be dealt with simply, cheaply, and with a minimum of aggravation by raising awareness of the potential role of HVAC (heating, ventilation and air conditioning) equipment.

The author is an electrical engineer with no health-related or HVAC-related qualifications. However, he can read and write, ideas often come to mind when presented with new information.

After initially publishing this article, an old friend showed me a YouTube video made by a consulting civil engineer who specializes in this topic. You can skip to it if you are impatient and want to know in-depth technical specifics.

All the new information referenced in this article is generally available, from qualified and reputable sources, and this article just disseminates it.

This article concludes with actionable suggestions.

Update 2023-06-08

[The] federal agency has set a target - five air changes per hour - for how much rooms and buildings should be ventilated...

it’s easy to see the guidance only in the context of Covid-19, it will help with many other airborne hazards like wildfire smoke, allergens and other infectious diseases, such as the flu...

“If they had broadcast and implemented these changes at the beginning, there never would have been a pandemic”, said Kimberly Prather, an atmospheric chemist at the University of California at San Diego and the Scripps Institution of Oceanography.

– From CDC sets first target for indoor air ventilation to prevent spread of Covid-19

COVID-19 Is Primarily Transmitted Via Aerosols

The most significant bit of information about COVID-19, which was not generally accepted at the beginning of the pandemic, is that it is mostly transmitted via aerosols; that is, via small airborne droplets only a few microns in diameter. (One micron is a millionth of a meter, or one twenty-five thousandth of an inch). Aerosol particles can hang in the air for hours and travel hundreds of feet.

The Centers for Disease Control and Prevention (CDC) officially recently officially recognized that SARS-CoV-2 (the virus that causes COVID-19) is airborne, meaning it is highly transmissible through the air.

Exposure occurs in three principal ways:

inhalation of very fine respiratory droplets and aerosol particles
deposition of respiratory droplets and particles on exposed mucous membranes in the mouth, nose, or eye by direct splashes and sprays
touching mucous membranes with hands that have been soiled either directly by virus-containing respiratory fluids or indirectly by touching surfaces with virus on them.

People release respiratory fluids during exhalation (e.g., quiet breathing, speaking, singing, exercise, coughing, sneezing) in the form of droplets across a spectrum of sizes. These droplets carry virus and transmit infection. To stay healthy, avoid these droplets.

The largest droplets settle out of the air rapidly, within seconds to minutes. The smallest very fine droplets, and aerosol particles formed when these fine droplets rapidly dry, are small enough that they can remain suspended in the air for minutes to hours.

–From the CDC Scientific Brief: SARS-CoV-2 Transmission, Updated May 7, 2021.

The idea that physical distancing of 6 feet (2 meters) might help keep people healthy is an example of bad science from a study 80 years ago that was not debunked until recently.

Transmission of COVID-19 from inhalation of virus in the air can occur at distances greater than six feet. Particles from an infected person can move throughout an entire room or indoor space. The particles can also linger in the air after a person has left the room – they can remain airborne for hours in some cases.

–From the US Environmental Protection Agency (EPA): Indoor Air and Coronavirus (COVID-19), web page was updated December 15, 2021.

Distance between people is not a significant transmissibility factor. Imagine two people, back to back, one (downwind) very sick with COVID-19, and the other (healthy) upwind, and the wind is blowing at 20 mph (32 km/h). The sick person will not infect the healthy person, unless they exchange positions. Just control the air that is breathed, and all airborne viruses such as COVID-19 will be controled.

What we learned during the pandemic is that aerosols were one of the main drivers in spreading the COVID-19 virus and that their importance in the transmission of many other respiratory pathogens has been systematically underappreciated.

– Dr. Robert Schooley, Professor in the Department of Medicine at the University of California San Diego (November 22, 2021), quoted from COVID Gets Airborne – UC San Diego develops computer model to aid understanding of how viruses travel through the air

Controlling Virus Infiltration

We would do well to remember that the COVID-19 virus causes sickness. Without exposure to the virus people would not get sick. The risk that a person might catch the disease is directly related to the number of viruses inhaled per unit of time. For example, the more viruses someone inhales in an hour, the more likely they will get sick.

The amount of virus necessary to make a person sick is called the infectious dose. All the countermeasures that have been used (hand washing, physical distancing, masks, curfews, etc.) are intended to reduce the number of viruses people are exposed to per unit time. Pandemic policy treats countermeasures as proxies for virus transmission vectors. By not updating pandemic policy as new information becomes available, we loose the ability to control the spread of the virus.

HEPA Filters

HEPA filters on HVAC equipment are designed to remove these aerosols. They come in a huge variety of sizes, shapes and air flow capacities. If you could just breathe the pure air emitted from a fan that had a HEPA filter, you would not get sick from any airborne virus. Any number of people could gather safely, if each of them received an adequate supply of pure air in their face at all times.

HEPA is a type of pleated mechanical air filter. It is an acronym for “high efficiency particulate air filter” (as officially defined by the U.S. Dept. of Energy). This type of air filter can theoretically remove at least 99.97% of dust, pollen, mold, bacteria, and any airborne particles with a size of 0.3 microns (µm).

–US Environment Protection Agency

HEPA filters are inexpensive and are readily available. They are not new. In fact, HEPA filter technology was created in the 1940s by the US Army Chemical Corps and National Defense Research Committee as part of the Manhattan Project. HEPA technology was declassified after World War II and became available for commercial and personal use. HEPA filters play a key role in the research and development of modern pharmaceuticals, aerospace engineering, and computer chip manufacturing.

HEPA air cleaners, which remain little-used in Canadian hospitals, are a cheap and easy way to reduce risk from airborne pathogens.

–From Nature Magazine, October 6, 2021: Real-world data show that filters clean COVID-causing virus from air – An inexpensive type of portable filter efficiently screened SARS-CoV-2 and other disease-causing organisms from hospital air.

It is easy to attach a HEPA filter to a fan, or to replace an existing HVAC filter with a HEPA filter. Here is a very scientific approach to a home-made solution.

Dr. Jeffrey E. Terrell, director of the Michigan Sinus Center, demonstrates how to build an air purifier with a HEPA filter for about $25 with parts from your local hardware store.

One of the comments on the above video, from Rick Rude in 2014, was:

Better to pull air thru the filter, if you push air through the filter dust that builds up will get blown off and go back into the room. Also, if the fan is pulling air through, the filter will stick to the fan and won't need as much tape to hold it on. Always handle used filters with care because handling them rough will release concentrated dust back into your air. If possible, always take air cleaners or vacuum cleaners outside to change the filters or bags.

MERV Filters

MERV is a graduated standard; MERV 7 is the minimum standard for furnaces, while MERV 13 is the highest.

MERV 13 filters are more efficient at removing large particles from the air, while HEPA filters are more efficient at removing small particles from the air. MERV 13 filters can remove up to 99.97% of particles from the air, while HEPA filters can remove up to 99.99% of particles from the air.

How to Build a Corsi-Rosenthal Box.

How to Identify and Rectify Poorly Ventilated Indoor Spaces Using Engineering Controls

David Elfstrom, P. Eng., is a independent civil engineer based in Simcoe, Ontario who consults on the efficient use of energy in buildings. During the pandemic he has been drawing attention to the importance of ventilation, filtration, and overall indoor environmental quality. Earlier this year Mr. Elfstrom completed a ventilation and air pathways assessment of an apartment building in outbreak for a public health unit. This presentation was part of Passive Buildings Canada's 2021 Annual General Meeting.

Mr. Elfstrom also co-authored the Masks4Canada Room Ventilation/Filtration Guide and Tip Sheet.

N95, KF94, FFP2, and 9152 Masks and Their Cousins

If you need to move about in a crowd, or enter a space that contained people a few hours ago, you need to wear a properly fitting face mask that filters out the tiny aerosol particles that contain the COVID-19 virus. N95 masks and their cousins (KN95, KF94, etc.) do the job because they filter particles as small as 0.3 microns. This is similar to the particle size filtered by HEPA filters. Caution: KN95 is a self-reported test standard, and lacks strict government regulation by China, resulting in many underperforming and often flat-out fake masks.

The US National Personal Protective Technology Laboratory (NPPTL), which is part of the US National Institute for Occupational Safety and Health (NIOSH), which itself is part of the CDC, evaluated various masks and published the results as NPPTL Respirator Assessments to Support the COVID-19 Response.

The following types of masks do not reliably filter tiny aerosols, so they do not adequately protect you from COVID-19 variants such as Omicron:

Surgical masks (what most people wear these days)
Masks with activated charcoal
Vented masks
Cloth masks
Gaiters
Ill-fitting masks that do not cover and seal the mouth and nose

Inoculations and Pills

Inoculations, including boosters, have proven to be very helpful. Pills from Pfizer Inc. and Merck & Co. for people sick with COVID-19 will also be very helpful once they become available. However, pills will only help sick people get better, they will not prevent getting sick.

Until everyone in the entire world has somehow acquired an effective level of antibodies, COVID-19 will continue to mutate.

The Only Solution Available Today

Inoculations and masks are good, but they are not perfect preventative measures against COVID-19. Protection against infection is not 100%. At present, the only preventative measure against COVID-19 that would allow people to safely mingle in person would be to guarantee continuous streams of pure air directed individually at each person's face.

This measure would also prevent the spread of all other diseases and their variants that are primarily spread via aerosols, including coronoviruses, colds, flus, tuberculosis, MERS-CoV, measles, ebola and chickenpox. Pollen, dust and other particulates would also be removed, so athsma sufffers would feel relief from extended periods breathing pure air.

The solution to living with COVID-19 is simple, safe, inexpensive and is not disruptive:

Live your social life with a pure breeze in your face.

Life as we knew it before the pandemic started 2 years ago could resume, mostly.

HVAC Upgrades Are Key

The engineering group that encompasses HVAC is ASHRAE. ASHRAE was formed as the American Society of Heating, Refrigerating and Air-Conditioning Engineers by the merger in 1959 of American Society of Heating and Air-Conditioning Engineers (ASHAE), founded in 1894 and The American Society of Refrigerating Engineers (ASRE), founded in 1904.

ASHRAE offers Core Recommendations for Reducing Airborne Infectious Aerosol Exposure. MERV 13 or better levels of performance are recommended, and HEPA filters surpass that specifation. In fact, all HEPA filters have a rating of a MERV 17 or higher.

Entrepreneurs

This represents an opportunity for entrepreneurs.

Restaurants

Imagine a restaurant that stays open with near-normal seating capacity because each table is equipped with a fan and ducting that blows a gentle breeze of pure air directly into the face of every patron. The air would recirculate within the restaurant, so there would be no need to upgrade the existing HVAC system because the HEPA filters located at each table would continuously clean the air within the restaurant. Those filters would need to be cleaned and disinfected daily. The wait staff would all wear N95 or similar masks, however each cashier could instead enjoy their own gentle breeeze of pure air.

Cost to equip each seat in the restaurant could be less than $50. For do-it-yourself owners, the cost could approach $10 per seat. For example, a 100-seat restaurant might be able to retrofit 75 of those seats at a cost of less than $3750. They would never have to close again due to any pandemic caused by airborne aerosols... provided, of course, that local regulations recognized this approach.

Retail Stores and Office Buildings

Enclosing sales clerks in stores behind clear plastic walls is wrong because it decreases air circulation. Instead, each those staff members should have a dedicated ducted fan that blows a gentle breeze of pure air into their face at all times.

Similarly, queues of people awaiting their turn at checking out should have fans blowing pure air at their heads. The air would recirculate within the store, there would be no need to upgrade the existing HVAC system because the HEPA filters would continuously clean the air within the building.

Spring-Breezifier Is Offered Into the Public Domain

I offer this idea to the world; I am an idea machine so I cannot act on most of them. Just for fun, let's call this idea the Spring-Breezifier.

Designing and building Spring-Breezifiers seems like it might be a good high school or youth group project. HVAC installers in particular should be able to make short work of this idea; go ahead and make lots of money providing pure airflows by building custom Spring-Breezifiers for your customers, we all thank you!

If anyone would like to talk to me about their Spring-Breezifier project, I would be happy to speak with them.

Next Steps

Medical HVAC specialists could suggest the necessary cubic feet per minute of air required per person sufficient to guarantee a continuous stream of pure air, and provide guidelines for designing ductwork to direct that air effectively.
Build and test prototypes. Most of the effort will be in designing and building the ducting and/or the scaffolding. Adding inner baffles would raise the cost and weight, lower the noise and make the airflow less turbulent. Perhaps Spring-Breezifiers could be built entirely from off-the-shelf parts for some or even most installations. Large-format 3D printers for special circumstances might be helpful.
Propose the inclusion of pure air streams into public spaces as a matter of public health policy.

Disappointing Scala 3 Installation Experience

2021-05-19T00:00:00-04:00

I run ScalaCourses.com, an online Scala training web site. After years of hype, Scala 3 is now available.

Scala 3: Not Yet Ready for Production

Scala 3 was eight years in the making. You would never know that from the horrible installation process and the disappointing installation instructions.

It is going to take quite a while before Scala 3, which was known as Dotty before it was released, can be trusted in production. According to the Dotty GitHub project, the only published future milestone is v3.1.0, which has no due date. Given that Scala 3 uses an entirely new build process, and an entirely new (and nonstandard) installation process, and that the internals of the Scala compiler were almost completely replaced, I doubt that version will be stable enough for use on production projects.

Installation Choices

Installation cholices include:

Command-line Scala has limited use cases, but is nice to have around for occassional experimentation.
SBT (which features an enhanced Scala REPL) is very helpful for interactively developing code, as well as for building and testing.
IntelliJ provides the best Scala coding productivity and code quality.
VSCode has been playing catch-up but is not full-featured yet.

Installation Transcript

The following is the transcript of how I installed command-line Scala on Ubuntu 20.10 running under WSL2.

Remove Scala 2

This step is not required. Scala 2 and Scala 3 can easily co-exist on the same system because their names are different.

Shell

$ sudo apt remove scala
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  scala
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 666 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 236024 files and directories currently installed.)
Removing scala (2.13.4-400) ...
Processing triggers for man-db (2.9.3-2) ...

Install Coursier

Coursier is a multithreaded downloader for project dependencies, and it now also downloads Scala 3. SBT uses coursier internally.

Shell

$ curl -fLo cs https://git.io/coursier-cli-"$(uname | tr LD ld)"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   144  100   144    0     0    285      0 --:--:-- --:--:-- --:--:--  6000
100 57.1M  100 57.1M    0     0  3656k      0  0:00:15  0:00:15 --:--:-- 4092k 

$ mv cs ~/.local/bin/

$ chmod a+x ~/.local/bin/cs

$ cs install cs
https://repo1.maven.org/maven2/io/get-coursier/apps/maven-metadata.xml
  100.0% [##########] 1.8 KiB (8.5 KiB / s)
https://repo1.maven.org/maven2/io/get-coursier/coursier-cli_2.12/maven-metadata.xml
  No new update since 2021-03-23 14:35:16
Wrote cs
Warning: /home/mslinn/.local/share/coursier/bin is not in your PATH
To fix that, add the following line to ~/.bashrc

export PATH="$PATH:/home/mslinn/.local/share/coursier/bin" 

$  export PATH="$PATH:/home/mslinn/.local/share/coursier/bin"

Now let's test Coursier:

Shell

$ cs
Coursier 2.0.16
Usage: cs [options] [command] [command-options]

Available commands: bootstrap, channel, complete, fetch, get, install, java, java-home, launch, list, publish, resolve, setup, uninstall, update, search

Type  cs command --help  for help on an individual command

This is the Coursier help message for the install subcommand:

Shell

$ cs install --help
Command: install
Usage: cs install
--cache  <string?>
Cache directory (defaults to environment variable COURSIER_CACHE, or ~/.cache/coursier/v1 on Linux and ~/Library/Caches/Coursier/v1 on Mac)
--mode | -m  <offline|update-changing|update|missing|force>
Download mode (default: missing, that is fetch things missing from cache)
--ttl | -l  <duration>
TTL duration (e.g. "24 hours")
--parallel | -n  <int>
Maximum number of parallel downloads (default: 6)
--checksum  <checksum1,checksum2,...>
Checksum types to check - end with none to allow for no checksum validation if no checksum is available, example: SHA-256,SHA-1,none
--retry-count  <int>
Retry limit for Checksum error when fetching a file
--cache-file-artifacts | --cfa  <bool>
Flag that specifies if a local artifact should be cached.
--follow-http-to-https-redirect  <bool>
Whether to follow http to https redirections
--credentials  <host(realm) user:pass|host user:pass>
Credentials to be used when fetching metadata or artifacts. Specify multiple times to pass multiple credentials. Alternatively, use the COURSIER_CREDENTIALS environment variable
--credential-file  <string*>
Path to credential files to read credentials from
--use-env-credentials  <bool>
Whether to read credentials from COURSIER_CREDENTIALS (env) or coursier.credentials (Java property), along those passed with --credentials and --credential-file
--quiet | -q  <counter>
Quiet output
--verbose | -v  <counter>
Increase verbosity (specify several times to increase more)
--progress | -P  <bool>
Force display of progress bars
--log-changing  <bool>
Log changing artifacts
--log-channel-version | --log-index-version | --log-jvm-index-version  <bool>
Log app channel or JVM index version
--graalvm-home  <string?>
--graalvm-option  <string*>
--graalvm-default-version  <string?>
--install-dir | --dir  <string?>
--install-platform  <string?>
Platform for prebuilt binaries (e.g. "x86_64-pc-linux", "x86_64-apple-darwin", "x86_64-pc-win32")
--install-prefer-prebuilt  <bool>
--only-prebuilt  <bool>
Require prebuilt artifacts for native applications, don't try to build native executable ourselves
--repository | -r  <maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|ivy:$pattern>
Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots)
--default-repositories  <bool>
--proguarded  <bool?>
--channel  <org:name>
Channel for apps
--default-channels  <bool>
Add default channels
--contrib  <bool>
Add contrib channel
--file-channels  <bool>
Add channels read from the configuration directory
--jvm  <string?>
--jvm-dir  <string?>
--system-jvm  <bool?>
--local-only  <bool>
--update  <bool>
--jvm-index  <string?>
--repository | -r  <maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|scala-integration|scala-nightlies|ivy:$pattern|jitpack|clojars|jcenter|apache:$repo>
Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots)
--no-default  <bool>
Do not add default repositories (~/.ivy2/local, and Central)
--sbt-plugin-hack  <bool>
Modify names in Maven repository paths for sbt plugins
--drop-info-attr  <bool>
Drop module attributes starting with 'info.' - these are sometimes used by projects built with sbt
--channel  <org:name>
Channel for apps
--default-channels  <bool>
Add default channels
--contrib  <bool>
Add contrib channel
--file-channels  <bool>
Add channels read from the configuration directory
--repository | -r  <maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|scala-integration|scala-nightlies|ivy:$pattern|jitpack|clojars|jcenter|apache:$repo>
Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots)
--no-default  <bool>
Do not add default repositories (~/.ivy2/local, and Central)
--sbt-plugin-hack  <bool>
Modify names in Maven repository paths for sbt plugins
--drop-info-attr  <bool>
Drop module attributes starting with 'info.' - these are sometimes used by projects built with sbt
--channel  <org:name>
Channel for apps
--default-channels  <bool>
Add default channels
--contrib  <bool>
Add contrib channel
--file-channels  <bool>
Add channels read from the configuration directory
--env  <bool>
--disable-env | --disable  <bool>
--setup  <bool>
--user-home  <string?>
--add-channel  <string*>
(deprecated)
--force | -f  <bool>

Install Scala 3

The Scala 3 compiler and REPL are separate programs: scala3-compiler and scala3-repl.

Shell

$ cs install scala3-compiler
https://repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.0.0/scala3-compiler_3-3.0.0.pom
  100.0% [##########] 4.8 KiB (79.7 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.0.0/tasty-core_3-3.0.0.pom
  100.0% [##########] 3.5 KiB (69.5 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.0.0/scala3-library_3-3.0.0.pom
  100.0% [##########] 3.6 KiB (53.9 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.0.0/scala3-interfaces-3.0.0.pom
  100.0% [##########] 3.4 KiB (65.9 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.0.0/scala3-interfaces-3.0.0.jar
  100.0% [##########] 3.4 KiB (113.9 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.0.0/tasty-core_3-3.0.0.jar
  100.0% [##########] 71.9 KiB (192.7 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.0.0/scala3-library_3-3.0.0.jar
  100.0% [##########] 1.1 MiB (1.8 MiB / s)
https://repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.0.0/scala3-compiler_3-3.0.0.jar
  100.0% [##########] 14.7 MiB (3.7 MiB / s)
Wrote scala3-compiler 

$ cs install scala3-repl
https://repo1.maven.org/maven2/io/get-coursier/apps/maven-metadata.xml
  No new update since 2021-05-14 04:42:19
Wrote scala3-repl

Run Scala 3 REPL

The part is easy!

Shell

$ scala3-repl --version
Scala code runner version 3.0.0 -- Copyright 2002-2021, LAMP/EPFL 

$ scala3-repl
scala>

Easily Run Scala REPL With SBT

If you do not mind directories called project/ and target/ being created in your current directory, and you have already installed sbt, you can get a REPL powered by Scala 3 like this:

Shell

$ sbt "-Dsbt.version=1.5.2" ++3.0.0! console
[info] welcome to sbt 1.5.2 (Ubuntu Java 11.0.11)
  [info] loading global plugins from /home/mslinn/.sbt/1.0/plugins
  [info] loading project definition from /var/work/ancientWarmth/ancientWarmth/project
  [info] set current project to ancientwarmth (in build file:/var/work/ancientWarmth/ancientWarmth/)
  [info] Forcing Scala version to 3.0.0 on all projects.
  [info] Reapplying settings...
  [info] set current project to ancientwarmth (in build file:/var/work/ancientWarmth/ancientWarmth/)
  [info] Updating
  [info] Resolved  dependencies
  [info] Updating
  https://repo1.maven.org/maven2/org/scala-lang/scaladoc_3/3.0.0/scaladoc_3-3.0.0.pom
    100.0% [##########] 6.1 KiB (82.6 KiB / s)
  https://repo1.maven.org/maven2/org/scala-lang/scala3-tasty-inspector_3/3.0.0/scala3-tasty-inspector_3-3.0.0.pom
    100.0% [##########] 3.6 KiB (80.8 KiB / s)
  [info] Resolved  dependencies
  [info] Fetching artifacts of
  [info] Fetched artifacts of
  [info] Fetching artifacts of
  https://repo1.maven.org/maven2/org/scala-lang/scala3-tasty-inspector_3/3.0.0/scala3-tasty-inspector_3-3.0.0.jar
    100.0% [##########] 16.6 KiB (338.1 KiB / s)
  https://repo1.maven.org/maven2/org/scala-lang/scaladoc_3/3.0.0/scaladoc_3-3.0.0.jar
    100.0% [##########] 1.5 MiB (3.1 MiB / s)
  [info] Fetched artifacts of

  scala>

Thanks to @renghen for this tip.

ScalaCourses

If you want to learn how to work effectively with Scala for functional and object-oriented programming, ScalaCourses.com is your best option. The course material is suitable for Scala 2 and Scala 3. Visit ScalaCourses.com to learn how to become a proficient Scala programmer.

OCI / Docker / AWS Lambda / Django / Buildah / podman

2021-04-29T00:00:00-04:00

This article is a work in progress. Some of it may be incorrect, and some thoughts might lead nowhere. I am publicly posting it in this state so I can discuss it with others. This article will be improved as information becomes available.

Goal

As previously discussed, Buildah is a drop-in replacement for using docker build and a Dockerfile. Buildah’s build-using-dockerfile, or bud argument makes it behave just like docker build does.

The goal of this article is to use Buildah / podman to create an Open Container Initiative (OCI) container image with a Django app, including the Python 3.8 runtime installed. The Django app will start when the container is created. The code for the Django app will be stored on the local machine where its source code can be edited, and it will be mapped into the container from the host system. Changes made to the code from the host system will be immediately visible inside the container.

TODO

Background: AWS publishes Deploying Python with an AWS base image, but that does not discuss running or testing. Create a Lambda function with the console is a more complete article, but is focused on using the web browser console, using Docker, and Node.js. So many differences from the desired goal make the articles difficult to translate to AWS CLI, Buildah / podman and Python.

Talk about the AWS Lambda Runtime Interface Emulator, compare and contrast with the AWS Lambda Python Runtime Interface Client.

Compare these AWS Lambda Runtimes with other, equivalant runtimes.

OCI images are swapped in when AWS Lambda is invoked. Do larger images cost more to use? If so, discuss.

Deploy Python Lambda function with Container Image

Consider this Dockerfile, which launches a Python 3.8 command-line application in a manner compatible with AWS Lambda:

Dockerfile

FROM public.ecr.aws/lambda/python:3.8

COPY app.py ./
CMD ["app.handler"]

Following is a small Python app called app.py, which will be launched by the Dockerfile. The Python app can be run as an AWS Lambda program because it implements the handler entry point.

app.py

import sys

def handler(event, context):
    return f"Hello from AWS Lambda using Python {sys.version}!"

Build Image

Buildah builds the image, just the same way that Docker would:

Shell

$ buildah bud -t hello .
STEP 1: FROM public.ecr.aws/lambda/python:3.8
Getting image source signatures
Copying blob 03ac043af787 skipped: already exists
Copying blob 420e64b38334 done
Copying blob ff259f25b075 done
Copying blob 3ff716981d54 done
Copying blob 6b6e623a48a8 done
Copying blob 9aa8f1e66d54 done
Copying config 67dc3a2a54 done
Writing manifest to image destination
Storing signatures
STEP 2: COPY app.py   ./
STEP 3: CMD ["app.handler"]
STEP 4: COMMIT hello
Getting image source signatures
Copying blob 683073d39306 skipped: already exists
Copying blob 658871a69e1f skipped: already exists
Copying blob 6fa16f35d11e skipped: already exists
Copying blob d6fa53d6caa6 skipped: already exists
Copying blob 61c062506436 skipped: already exists
Copying blob 1c1d66a5fd95 skipped: already exists
Copying blob 33af9dc6463a done
Copying config 98862dfd20 done
Writing manifest to image destination
Storing signatures
--> 98862dfd208
98862dfd2087152ee821553d6cb1c033e735af06e5f11c814bcc9300fb65584e

Test Lambda function Locally

Before calling the Lambda API from a local container, first run the container. Containers default to running in the foreground, but the -d option causes a container to be run as a background process. This container is given the name hello, the external HTTP endpoint at 9000 is mapped to internal port 8080, and the latest version of the hello lambda function is run in the container.

Shell

$ podman run \
  -d \
  --name hello \
  -p 9000:8080 \
  hello:latest
d4d296e4c91d01c98d312e3f79599dca53990d95218e94bbdfbbac6a43cde9e8

Call the local version of the Lambda API:

Shell

$ curl \
  -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" \
  -d '{}'
"Hello from AWS Lambda using Python 3.8.9 (default, Apr 20 2021, 13:58:54) \n[GCC 7.3.1 20180712 (Red Hat 7.3.1-12)]!"

Stop the container called hello.

Shell

$ podman stop hello
96cc1b1ed92368a1165d6a6ad0b1e5544d4ac751b64e94df33bf2322e6d7b30c

Create AWS ECR Repository

AWS provides a registry for OCI-compatible image repositories called the AWS Elastic Container Registry (ECR).

Shell

$ aws ecr create-repository help
CREATE-REPOSITORY() CREATE-REPOSITORY()

NAME
create-repository -

DESCRIPTION
Creates a repository. For more information, see Amazon ECR Repositories
in the Amazon Elastic Container Registry User Guide .

See also: AWS API Documentation

See 'aws help' for descriptions of global parameters.

SYNOPSIS
create-repository
--repository-name <value>
[--tags <value>]
[--image-tag-mutability <value>]
[--image-scanning-configuration <value>]
[--cli-input-json <value>]
[--generate-cli-skeleton <value>]

OPTIONS
--repository-name (string)
The name to use for the repository. The repository name may be spec-
ified on its own (such as nginx-web-app ) or it can be prepended
with a namespace to group the repository into a category (such as
project-a/nginx-web-app ).

--tags (list)
The metadata that you apply to the repository to help you categorize
and organize them. Each tag consists of a key and an optional value,
both of which you define. Tag keys can have a maximum character
length of 128 characters, and tag values can have a maximum length
of 256 characters.

(structure)
The metadata that you apply to a resource to help you categorize
and organize them. Each tag consists of a key and an optional
value, both of which you define. Tag keys can have a maximum
character length of 128 characters, and tag values can have a
maximum length of 256 characters.

Key -> (string)
One part of a key-value pair that make up a tag. A key is a
general label that acts like a category for more specific tag
values.

Value -> (string)
The optional part of a key-value pair that make up a tag. A
value acts as a descriptor within a tag category (key).

Shorthand Syntax:

Key=string,Value=string ...

JSON Syntax:

[
{
"Key": "string",
"Value": "string"
}
...
]

--image-tag-mutability (string)
The tag mutability setting for the repository. If this parameter is
omitted, the default setting of MUTABLE will be used which will al-
low image tags to be overwritten. If IMMUTABLE is specified, all im-
age tags within the repository will be immutable which will prevent
them from being overwritten.

Possible values:

o MUTABLE

o IMMUTABLE

--image-scanning-configuration (structure)
The image scanning configuration for the repository. This setting
determines whether images are scanned for known vulnerabilities af-
ter being pushed to the repository.

scanOnPush -> (boolean)
The setting that determines whether images are scanned after be-
ing pushed to a repository. If set to true , images will be
scanned after being pushed. If this parameter is not specified,
it will default to false and images will not be scanned unless a
scan is manually started with the StartImageScan API.

Shorthand Syntax:

scanOnPush=boolean

JSON Syntax:

{
"scanOnPush": true|false
}

--cli-input-json (string) Performs service operation based on the JSON
string provided. The JSON string follows the format provided by --gen-
erate-cli-skeleton. If other arguments are provided on the command
line, the CLI values will override the JSON-provided values. It is not
possible to pass arbitrary binary values using a JSON-provided value as
the string will be taken literally.

--generate-cli-skeleton (string) Prints a JSON skeleton to standard
output without sending an API request. If provided with no value or the
value input, prints a sample input JSON that can be used as an argument
for --cli-input-json. If provided with the value output, it validates
the command inputs and returns a sample output JSON for that command.

See 'aws help' for descriptions of global parameters.

EXAMPLES
Example 1: To create a repository

The following create-repository example creates a repository inside the
specified namespace in the default registry for an account.

aws ecr create-repository \
--repository-name project-a/nginx-web-app

Output:

{
"repository": {
"registryId": "123456789012",
"repositoryName": "sample-repo",
"repositoryArn": "arn:aws:ecr:us-west-2:123456789012:repository/project-a/nginx-web-app"
}
}

For more information, see Creating a Repository in the Amazon ECR User
Guide.

Example 2: To create a repository configured with image tag immutabil-
ity

The following create-repository example creates a repository configured
for tag immutability in the default registry for an account.

aws ecr create-repository \
--repository-name sample-repo \
--image-tag-mutability IMMUTABLE

Output:

{
"repository": {
"registryId": "123456789012",
"repositoryName": "sample-repo",
"repositoryArn": "arn:aws:ecr:us-west-2:123456789012:repository/sample-repo",
"imageTagMutability": "IMMUTABLE"
}
}

For more information, see Image Tag Mutability in the Amazon ECR User
Guide.

Example 3: To create a repository configured with a scanning configura-
tion

The following create-repository example creates a repository configured
to perform a vulnerability scan on image push in the default registry
for an account.

aws ecr create-repository \
--repository-name sample-repo \
--image-scanning-configuration scanOnPush=true

Output:

{
"repository": {
"registryId": "123456789012",
"repositoryName": "sample-repo",
"repositoryArn": "arn:aws:ecr:us-west-2:123456789012:repository/sample-repo",
"imageScanningConfiguration": {
"scanOnPush": true
}
}
}

For more information, see Image Scanning in the Amazon ECR User Guide.

OUTPUT
repository -> (structure)
The repository that was created.

repositoryArn -> (string)
The Amazon Resource Name (ARN) that identifies the repository.
The ARN contains the arn:aws:ecr namespace, followed by the re-
gion of the repository, AWS account ID of the repository owner,
repository namespace, and repository name. For example,
arn:aws:ecr:region:012345678910:repository/test .

registryId -> (string)
The AWS account ID associated with the registry that contains
the repository.

repositoryName -> (string)
The name of the repository.

repositoryUri -> (string)
The URI for the repository. You can use this URI for Docker push
or pull operations.

createdAt -> (timestamp)
The date and time, in JavaScript date format, when the reposi-
tory was created.

imageTagMutability -> (string)
The tag mutability setting for the repository.

imageScanningConfiguration -> (structure)
The image scanning configuration for a repository.

scanOnPush -> (boolean)
The setting that determines whether images are scanned after
being pushed to a repository. If set to true , images will be
scanned after being pushed. If this parameter is not speci-
fied, it will default to false and images will not be scanned
unless a scan is manually started with the StartImageScan
API.



CREATE-REPOSITORY()

The following creates an AWS ECR image repository in called hello within the test namespace. Images are scanned for known vulnerabilities after they are pushed to the repository.

Shell

$ aws ecr create-repository \
  --repository-name test/hello \
  --image-scanning-configuration scanOnPush=true
{
  "repository": {
      "repositoryArn": "arn:aws:ecr:us-east-1:031372724784:repository/test/hello",
      "registryId": "031372724784",
      "repositoryName": "test/hello",
      "repositoryUri": "031372724784.dkr.ecr.us-east-1.amazonaws.com/test/hello",
      "createdAt": 1620232146.0,
      "imageTagMutability": "MUTABLE",
      "imageScanningConfiguration": {
          "scanOnPush": true
      }
  }
}

Tag Image

podman tag – Assigns a new image name to an existing image. A full name refers to the entire image name, including the optional tag after the :. If there is no tag provided, then podman will default to latest for both the image and the target-name. – From man podman-tag.

Shell

$ IMAGE_NAME=hello

$ IMAGE_VERSION=0.1

$ podman tag $IMAGE_NAME:$IMAGE_VERSION \
  $REGISTRY/$IMAGE_NAME:$IMAGE_VERSION

$ podman images
REPOSITORY                                                  TAG              IMAGE ID      CREATED         SIZE
localhost/hello                                             0.1              98862dfd2087  39 minutes ago  622 MB
752246127823.dkr.ecr.us-east-1.amazonaws.com/hello          latest           98862dfd2087  39 minutes ago  622 MB
public.ecr.aws/lambda/python                                3.8              67dc3a2a54fb  25 hours ago    622 MB
752246127823.dkr.ecr.us-east-1.amazonaws.com/ancientwarmth  latest           5d18ea34fc30  28 hours ago    2.03 GB
localhost/ancientwarmth                                     latest           5d18ea34fc30  28 hours ago    2.03 GB
<none>                                                      <none>           40ef32b39cf4  5 days ago      622 MB
docker.io/library/amazonlinux                               latest           53ef897d731f  5 days ago      170 MB
docker.io/amazon/aws-lambda-python                          3.8              e12ea62c5582  9 days ago      622 MB
docker.io/library/alpine                                    latest           6dbb9cc54074  2 weeks ago     5.88 MB
docker.io/lambci/lambda                                     build-python3.8  714c659c9f6f  3 months ago    2.03 GB

Push Image to ECR

Podman will use the IAM credentials for the dev profile in ~/.aws/credentials to log into that AWS account:

~/.aws/credentials

[default]
aws_access_key_id = ********************
aws_secret_access_key = ****************************************
region = us-east-1

[dev]
aws_access_key_id = ********************
aws_secret_access_key = ****************************************
region = us-east-1

Shell

$ export AWS_PROFILE=dev

$ AWS_ACCOUNT="$( aws sts get-caller-identity \
  --query Account \
  --output text
)"

$ AWS_REGION="$( aws configure get region )"

$ REGISTRY="$AWS_ACCOUNT.dkr.ecr.$AWS_REGION.amazonaws.com"

$ aws ecr get-login-password \
    --region "$AWS_REGION" | \
  podman login \
    --password-stdin \
    --username AWS \
    "$REGISTRY"
Login Succeeded!

Now that podman is logged into AWS, use podman push the image to AWS ECR:

Shell

$ podman push test/$IMAGE_NAME \
  $REGISTRY/$IMAGE_NAME:$IMAGE_VERSION
Getting image source signatures
Copying blob 692590faf2d1 [--------------------------------------] 8.0b / 8.2MiB
Copying blob 397718cff58d [--------------------------------------] 8.0b / 206.2MiB
Copying blob 9ca787b1c91c [--------------------------------------] 8.0b / 93.1MiB
Copying blob ef26f5221b79 [--------------------------------------] 8.0b / 196.7MiB
Copying blob 0a3f69c27a89 [--------------------------------------] 8.0b / 316.4MiB
Copying blob 5b3cbb76df75 [--------------------------------------] 8.0b / 1.1GiB
Copying blob e9cad39831b0 [--------------------------------------] 8.0b / 3.5KiB
Error: Error copying image to the remote destination:
Error writing blob: Error initiating layer upload to /v2/ancientwarmth/blobs/uploads/ in
752246127823.dkr.ecr.us-east-1.amazonaws.com:
name unknown: The repository with name 'hello' does not exist in the registry with id '752246127823'

The results of an image scan for the new repository can be retrieved as follows:

Shell

$ aws ecr describe-image-scan-findings \
  --repository-name test/hello \
  --image-id imageTag=tag_name

Deploy Python Lambda function with Container Image

Podman can invoke the app using an OCI container with Amazon Linux 2 and Python 3.8:

Shell

$ podman container run -ti \
  public.ecr.aws/lambda/python:3.8 \
  blog/docker/podman/app.py
Trying to pull public.ecr.aws/lambda/python:3.8...
Getting image source signatures
Copying blob 1de4740de1c2 done
Copying blob 03ac043af787 done
Copying blob 2e2bb77ae2dc done
Copying blob 842c9dce67e8 done
Copying blob df513d38f4d9 done
Copying blob 031c6369fb2b done
Copying config e12ea62c55 done
Writing manifest to image destination
Storing signatures
time="2021-05-02T23:38:30.971" level=info msg="exec '/var/runtime/bootstrap' (cwd=/var/task, handler=)"

Docker, OCI Images, Buildah and podman

2021-04-28T00:00:00-04:00

There are many ways to create and run Docker-compatible images. Docker is probably the worst option, mostly because it runs as a daemon, and all *nix daemons run with root privileges. Also, the docker-ce package lists iptables as a dependency, which needs systemd to be running normally, and WSL2 only partially supports systemd.

A Comprehensive Container Runtime Comparison provides helpful background information and an interesting historical viewpoint.

Open Container Initiative (OCI)

The latest evolution of Docker-compatible images, OCI image format (not to be confused with Oracle Cloud Infrastructure), is compatible with:

Supported OCI formats include:

Docker containers schema 1
Docker containers schema 2
Pods (groups of containers)
Images
Volumes

Buildah, podman and skopeo

This article discusses 3 related open source projects from RedHat / IBM that provide an alternative to Docker: Buildah, podman and skopeo. These 3 projects share a common source code base, and are daemonless tools for managing Open Container Initiative (OCI) images.

Paraphrasing the reasons expressed in Podman and Buildah for Docker Users for using podman instead of Docker, wherever podman is mentioned, read “podman, Buildah and skopeo”:

The Podman approach is simply to directly interact with the image registry, with the container and image storage, and with the Linux kernel through the runC container runtime process (not a daemon).

Running Podman as a normal user means that Podman will, by default, store images and containers in the user’s home directory. Podman uses a repository in the user’s home directory: ~/.local/share/containers (instead of /var/lib/docker).

Despite the new locations for the local repositories, the images created by Docker and Podman are compatible with the OCI standard. Podman can push to and pull from popular container registries like Quay.io and Docker hub, as well as private registries.

Buildah vs. podman

Podman can build OCI containers interactively or in batch mode. You can either build using a Dockerfile using podman build (batch mode), or you can interactively run a container, make changes to the running image, and then podman commit those changes to a new image tag.

Buildah was written before podman. Some of Buildah's source code for creating and managing container images was ported to podman. The podman build command is a subset of Buildah’s functionality.

However, apparently the differences between the two programs are important:

Buildah builds OCI images. Confusingly, podman build can also be used to build Docker images also, but it’s incredibly slow and used up a lot of disk space by using the vfs storage driver by default. buildah bud (‘build using Dockerfile’) was much faster for me, and uses the overlay storage driver.

– From Goodbye Docker: Purging is Such Sweet Sorrow by Ian Miell.

podman

Podman supports developing, managing, and running OCI Containers on Linux systems, including WSL, without requiring root privilege.

shell Installation on Ubuntu

$ yes | sudo apt install buildah podman skopeo

Podman commands are very nearly the same as Docker’s.

Because podman is a drop-in replacement for docker, the following alias enables the docker command to invoke podman:

~/.bash_aliases

alias docker=podman

As described in How to Install and Use Podman on Ubuntu 20.04, I added 'registry.access.redhat.com' to the list of registries in /etc/containers/registries.conf. I also added gallery.ecr.aws and gcr.io.

/etc/containers/registries.conf

[registries.search]
registries = ['docker.io', 'gallery.ecr.aws', 'gcr.io', 'quay.io', 'registry.access.redhat.com']

podman Help

Shell

$ podman --help
Manage pods, containers and images

  Usage:
    podman [flags]
    podman [command]

  Available Commands:
    attach      Attach to a running container
    auto-update Auto update containers according to their auto-update policy
    build       Build an image using instructions from Containerfiles
    commit      Create new image based on the changed container
    container   Manage containers
    cp          Copy files/folders between a container and the local filesystem
    create      Create but do not start a container
    diff        Display the changes to the object's file system
    events      Show podman events
    exec        Run a process in a running container
    export      Export container's filesystem contents as a tar archive
    generate    Generate structured data based on containers and pods.
    healthcheck Manage health checks on containers
    help        Help about any command
    history     Show history of a specified image
    image       Manage images
    images      List images in local storage
    import      Import a tarball to create a filesystem image
    info        Display podman system information
    init        Initialize one or more containers
    inspect     Display the configuration of object denoted by ID
    kill        Kill one or more running containers with a specific signal
    load        Load an image from container archive
    login       Login to a container registry
    logout      Logout of a container registry
    logs        Fetch the logs of one or more containers
    manifest    Manipulate manifest lists and image indexes
    mount       Mount a working container's root filesystem
    network     Manage networks
    pause       Pause all the processes in one or more containers
    play        Play a pod and its containers from a structured file.
    pod         Manage pods
    port        List port mappings or a specific mapping for the container
    ps          List containers
    pull        Pull an image from a registry
    push        Push an image to a specified destination
    restart     Restart one or more containers
    rm          Remove one or more containers
    rmi         Removes one or more images from local storage
    run         Run a command in a new container
    save        Save image to an archive
    search      Search registry for image
    start       Start one or more containers
    stats       Display a live stream of container resource usage statistics
    stop        Stop one or more containers
    system      Manage podman
    tag         Add an additional name to a local image
    top         Display the running processes of a container
    unmount     Unmounts working container's root filesystem
    unpause     Unpause the processes in one or more containers
    unshare     Run a command in a modified user namespace
    untag       Remove a name from a local image
    version     Display the Podman Version Information
    volume      Manage volumes
    wait        Block on one or more containers

  Flags:
        --cgroup-manager string     Cgroup manager to use ("cgroupfs"|"systemd") (default "cgroupfs")
        --cni-config-dir string     Path of the configuration directory for CNI networks
        --conmon string             Path of the conmon binary
    -c, --connection string         Connection to use for remote Podman service
        --events-backend string     Events backend to use ("file"|"journald"|"none") (default "file")
        --help                      Help for podman
        --hooks-dir strings         Set the OCI hooks directory path (may be set multiple times) (default [/usr/share/containers/oci/hooks.d])
        --identity string           path to SSH identity file, (CONTAINER_SSHKEY)
        --log-level string          Log messages above specified level (debug, info, warn, error, fatal, panic) (default "error")
        --namespace string          Set the libpod namespace, used to create separate views of the containers and pods on the system
        --network-cmd-path string   Path to the command for configuring the network
    -r, --remote                    Access remote Podman service (default false)
        --root string               Path to the root directory in which data, including images, is stored
        --runroot string            Path to the 'run directory' where all state information is stored
        --runtime string            Path to the OCI-compatible binary used to run containers, default is /usr/bin/runc
        --storage-driver string     Select which storage driver is used to manage storage of images and containers (default is overlay)
        --storage-opt stringArray   Used to pass an option to the storage driver
        --syslog                    Output logging information to syslog as well as the console (default false)
        --tmpdir string             Path to the tmp directory for libpod state content.

                                    Note: use the environment variable 'TMPDIR' to change the temporary storage location for container images, '/var/tmp'.

        --url string                URL to access Podman service (CONTAINER_HOST) (default "unix:/home/mslinn/.docker/run/podman/podman.sock")
    -v, --version                   Version of Podman

  Use "podman [command] --help" for more information about a command.

podman info

Shell

$ podman info
host:
  arch: amd64
  buildahVersion: 1.15.2
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: unknown'
  cpus: 8
  distribution:
    distribution: ubuntu
    version: "20.10"
  eventLogger: file
  hostname: Bear
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.72-microsoft-standard-WSL2
  linkmode: dynamic
  memFree: 897724416
  memTotal: 6231638016
  ociRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc93
      commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
      spec: 1.0.2-dev
      go: go1.13.15
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    path: /home/mslinn/.docker/run/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.3.1
  swapFree: 0
  swapTotal: 0
  uptime: 306h 23m 6.37s (Approximately 12.75 days)
registries:
  search:
  - quay.io
  - docker.io
  - gallery.ecr.aws
  - registry.access.redhat.com
store:
  configFile: /home/mslinn/.config/containers/storage.conf
  containerStore:
    number: 8
    paused: 0
    running: 0
    stopped: 8
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.0.0
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/mslinn/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 4
  runRoot: /home/mslinn/.docker/run/containers
  volumePath: /home/mslinn/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Wed Dec 31 19:00:00 1969
  GitCommit: ""
  GoVersion: go1.14.7
  OsArch: linux/amd64
  Version: 2.0.6

podman container Help

Shell

$ man podman-container
podman-container(1)                                   General Commands Manual                                   podman-container(1)

NAME
       podman-container - Manage containers

SYNOPSIS
       podman container subcommand

DESCRIPTION
       The container command allows you to manage containers

COMMANDS
       ┌───────────┬────────────────────────────────┬───────────────────────────────────┐
       │Command    │ Man Page                       │ Description                       │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │attach     │ podman-attach(1)               │ Attach to a running container.    │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │checkpoint │ podman-container-checkpoint(1) │ Checkpoints one or more running   │
       │           │                                │ containers.                       │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │cleanup    │ podman-container-cleanup(1)    │ Cleanup the container's network   │
       │           │                                │ and mountpoints.                  │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │commit     │ podman-commit(1)               │ Create new image based on the     │
       │           │                                │ changed container.                │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │cp         │ podman-cp(1)                   │ Copy files/folders between a      │
       │           │                                │ container and the local           │
       │           │                                │ filesystem.                       │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │create     │ podman-create(1)               │ Create a new container.           │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │diff       │ podman-diff(1)                 │ Inspect changes on a container or │
       │           │                                │ image's filesystem.               │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │exec       │ podman-exec(1)                 │ Execute a command in a running    │
       │           │                                │ container.                        │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │exists     │ podman-container-exists(1)     │ Check if a container exists in    │
       │           │                                │ local storage                     │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │export     │ podman-export(1)               │ Export a container's filesystem   │
       │           │                                │ contents as a tar archive.        │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │init       │ podman-init(1)                 │ Initialize a container            │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │inspect    │ podman-inspect(1)              │ Display a container or image's    │
       │           │                                │ configuration.                    │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │kill       │ podman-kill(1)                 │ Kill the main process in one or   │
       │           │                                │ more containers.                  │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │list       │ podman-ps(1)                   │ List the containers on the        │
       │           │                                │ system.(alias ls)                 │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │logs       │ podman-logs(1)                 │ Display the logs of a container.  │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │mount      │ podman-mount(1)                │ Mount a working container's root  │
       │           │                                │ filesystem.                       │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │pause      │ podman-pause(1)                │ Pause one or more containers.     │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │port       │ podman-port(1)                 │ List port mappings for the        │
       │           │                                │ container.                        │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │prune      │ podman-container-prune(1)      │ Remove all stopped containers     │
       │           │                                │ from local storage.               │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │restart    │ podman-restart(1)              │ Restart one or more containers.   │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │restore    │ podman-container-restore(1)    │ Restores one or more containers   │
       │           │                                │ from a checkpoint.                │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │rm         │ podman-rm(1)                   │ Remove one or more containers.    │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │run        │ podman-run(1)                  │ Run a command in a container.     │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │runlabel   │ podman-container-runlabel(1)   │ Executes a command as described   │
       │           │                                │ by a container image label.       │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │start      │ podman-start(1)                │ Starts one or more containers.    │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │stats      │ podman-stats(1)                │ Display a live stream of one or   │
       │           │                                │ more container's resource usage   │
       │           │                                │ statistics.                       │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │stop       │ podman-stop(1)                 │ Stop one or more running          │
       │           │                                │ containers.                       │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │top        │ podman-top(1)                  │ Display the running processes of  │
       │           │                                │ a container.                      │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │unmount    │ podman-unmount(1)              │ Unmount a working container's     │
       │           │                                │ root filesystem.(Alias unmount)   │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │unpause    │ podman-unpause(1)              │ Unpause one or more containers.   │
       ├───────────┼────────────────────────────────┼───────────────────────────────────┤
       │wait       │ podman-wait(1)                 │ Wait on one or more containers to │
       │           │                                │ stop and print their exit codes.  │
       └───────────┴────────────────────────────────┴───────────────────────────────────┘

SEE ALSO
       podman, podman-exec, podman-run

                                                                                                                podman-container(1)

Podman Run Help

Shell

$ podman container run --help
Run a command in a new container

Description:
  Runs a command in a new container from the given image

Usage:
  podman container run [flags] IMAGE [COMMAND [ARG...]]

Examples:
  podman container run imageID ls -alF /etc
        podman container run --network=host imageID dnf -y install java
        podman container run --volume /var/hostdir:/var/ctrdir -i -t fedora /bin/bash

Flags:
      --add-host strings                         Add a custom host-to-IP mapping (host:ip) (default [])
      --annotation strings                       Add annotations to container (key:value)
  -a, --attach strings                           Attach to STDIN, STDOUT or STDERR
      --authfile string                          Path of the authentication file. Use REGISTRY_AUTH_FILE environment variable to override
      --blkio-weight string                      Block IO weight (relative weight) accepts a weight value between 10 and 1000.
      --blkio-weight-device DEVICE_NAME:WEIGHT   Block IO weight (relative device weight, format: DEVICE_NAME:WEIGHT)
      --cap-add strings                          Add capabilities to the container
      --cap-drop strings                         Drop capabilities from the container
      --cgroup-parent string                     Optional parent cgroup for the container
      --cgroupns string                          cgroup namespace to use
      --cgroups string                           control container cgroup configuration ("enabled"|"disabled"|"no-conmon") (default "enabled")
      --cidfile string                           Write the container ID to the file
      --conmon-pidfile string                    Path to the file that will receive the PID of conmon
      --cpu-period uint                          Limit the CPU CFS (Completely Fair Scheduler) period
      --cpu-quota int                            Limit the CPU CFS (Completely Fair Scheduler) quota
      --cpu-rt-period uint                       Limit the CPU real-time period in microseconds
      --cpu-rt-runtime int                       Limit the CPU real-time runtime in microseconds
      --cpu-shares uint                          CPU shares (relative weight)
      --cpus float                               Number of CPUs. The default is 0.000 which means no limit
      --cpuset-cpus string                       CPUs in which to allow execution (0-3, 0,1)
      --cpuset-mems string                       Memory nodes (MEMs) in which to allow execution (0-3, 0,1). Only effective on NUMA systems.
  -d, --detach                                   Run container in background and print container ID
      --detach-keys [a-Z]                        Override the key sequence for detaching a container. Format is a single character [a-Z] or a comma separated sequence of `ctrl-<value>`, where `<value>` is one of: `a-cf`, `@`, `^`, `[`, `\`, `]`, `^` or `_` (default "ctrl-p,ctrl-q")
      --device strings                           Add a host device to the container
      --device-cgroup-rule strings               Add a rule to the cgroup allowed devices list
      --device-read-bps strings                  Limit read rate (bytes per second) from a device (e.g. --device-read-bps=/dev/sda:1mb)
      --device-read-iops strings                 Limit read rate (IO per second) from a device (e.g. --device-read-iops=/dev/sda:1000)
      --device-write-bps strings                 Limit write rate (bytes per second) to a device (e.g. --device-write-bps=/dev/sda:1mb)
      --device-write-iops strings                Limit write rate (IO per second) to a device (e.g. --device-write-iops=/dev/sda:1000)
      --disable-content-trust                    This is a Docker specific option and is a NOOP
      --dns strings                              Set custom DNS servers
      --dns-opt strings                          Set custom DNS options
      --dns-search strings                       Set custom DNS search domains
      --entrypoint string                        Overwrite the default ENTRYPOINT of the image
  -e, --env stringArray                          Set environment variables in container (default [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,TERM=xterm])
      --env-file strings                         Read in a file of environment variables
      --env-host                                 Use all current host environment variables in container
      --expose strings                           Expose a port or a range of ports
      --gidmap strings                           GID map to use for the user namespace
      --group-add strings                        Add additional groups to join
      --health-cmd string                        set a healthcheck command for the container ('none' disables the existing healthcheck)
      --health-interval string                   set an interval for the healthchecks (a value of disable results in no automatic timer setup) (default "30s")
      --health-retries uint                      the number of retries allowed before a healthcheck is considered to be unhealthy (default 3)
      --health-start-period string               the initialization time needed for a container to bootstrap (default "0s")
      --health-timeout string                    the maximum time allowed to complete the healthcheck before an interval is considered failed (default "30s")
  -h, --hostname string                          Set container hostname
      --http-proxy                               Set proxy environment variables in the container based on the host proxy vars (default true)
      --image-volume string                      Tells podman how to handle the builtin image volumes ("bind"|"tmpfs"|"ignore") (default "bind")
      --init                                     Run an init binary inside the container that forwards signals and reaps processes
      --init-path string                         Path to the container-init binary
  -i, --interactive                              Keep STDIN open even if not attached
      --ip string                                Specify a static IPv4 address for the container
      --ipc string                               IPC namespace to use
      --kernel-memory <number>[<unit>]           Kernel memory limit (format: <number>[<unit>], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes))
  -l, --label stringArray                        Set metadata on container
      --label-file strings                       Read in a line delimited file of labels
      --log-driver string                        Logging driver for the container
      --log-opt strings                          Logging driver options
      --mac-address string                       Container MAC address (e.g. 92:d0:c6:0a:29:33)
  -m, --memory <number>[<unit>]                  Memory limit (format: <number>[<unit>], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes))
      --memory-reservation <number>[<unit>]      Memory soft limit (format: <number>[<unit>], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes))
      --memory-swap string                       Swap limit equal to memory plus swap: '-1' to enable unlimited swap
      --memory-swappiness int                    Tune container memory swappiness (0 to 100, or -1 for system default) (default -1)
      --mount stringArray                        Attach a filesystem mount to the container
      --name string                              Assign a name to the container
      --network string                           Connect a container to a network (default "slirp4netns")
      --no-healthcheck                           Disable healthchecks on container
      --no-hosts                                 Do not create /etc/hosts within the container, instead use the version from the image
      --oom-kill-disable                         Disable OOM Killer
      --oom-score-adj int                        Tune the host's OOM preferences (-1000 to 1000)
      --pid string                               PID namespace to use
      --pids-limit int                           Tune container pids limit (set 0 for unlimited, -1 for server defaults)
      --pod string                               Run container in an existing pod
      --pod-id-file string                       Read the pod ID from the file
      --privileged                               Give extended privileges to container
  -p, --publish strings                          Publish a container's port, or a range of ports, to the host (default [])
  -P, --publish-all                              Publish all exposed ports to random ports on the host interface
      --pull string                              Pull image before creating ("always"|"missing"|"never") (default "missing")
  -q, --quiet                                    Suppress output information when pulling images
      --read-only                                Make containers root filesystem read-only
      --read-only-tmpfs                          When running containers in read-only mode mount a read-write tmpfs on /run, /tmp and /var/tmp (default true)
      --replace                                  If a container with the same name exists, replace it
      --restart string                           Restart policy to apply when a container exits ("always"|"no"|"on-failure")
      --rm                                       Remove container (and pod if created) after exit
      --rmi                                      Remove container image unless used by other containers
      --rootfs                                   The first argument is not an image but the rootfs to the exploded container
      --seccomp-policy string                    Policy for selecting a seccomp profile (experimental) (default "default")
      --security-opt stringArray                 Security Options
      --shm-size <number>[<unit>]                Size of /dev/shm (format: <number>[<unit>], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes)) (default "65536k")
      --sig-proxy                                Proxy received signals to the process (default true)
      --stop-signal string                       Signal to stop a container. Default is SIGTERM
      --stop-timeout uint                        Timeout (in seconds) to stop a container. Default is 10 (default 10)
      --subgidname string                        Name of range listed in /etc/subgid for use in user namespace
      --subuidname string                        Name of range listed in /etc/subuid for use in user namespace
      --sysctl strings                           Sysctl options
      --systemd string                           Run container in systemd mode ("true"|"false"|"always") (default "true")
      --tmpfs tmpfs                              Mount a temporary filesystem (tmpfs) into a container
  -t, --tty                                      Allocate a pseudo-TTY for container
      --uidmap strings                           UID map to use for the user namespace
      --ulimit strings                           Ulimit options
  -u, --user string                              Username or UID (format: <name|uid>[:<group|gid>])
      --userns string                            User namespace to use
      --uts string                               UTS namespace to use
  -v, --volume stringArray                       Bind mount a volume into the container
      --volumes-from strings                     Mount volumes from the specified container(s)
  -w, --workdir string                           Working directory inside the container

podman run

From Building and Deploying Lambdas from a Docker Container by Keith Gregory:

Shell

$ podman run \
  -it \
  --entrypoint /bin/bash \
  --rm \
  -v /tmp:/mnt \
  amazon/aws-lambda-python:3.8
Trying to pull quay.io/amazon/aws-lambda-python:3.8...
  Requesting bear token: invalid status code from registry 405 (Method Not Allowed)
Trying to pull docker.io/amazon/aws-lambda-python:3.8...
Getting image source signatures
Copying blob df513d38f4d9 skipped: already exists
Copying blob 2e2bb77ae2dc skipped: already exists
Copying blob 031c6369fb2b skipped: already exists
Copying blob 03ac043af787 skipped: already exists
Copying blob 842c9dce67e8 skipped: already exists
Copying blob 1de4740de1c2 [--------------------------------------] 0.0b / 0.0b
Copying config e12ea62c55 done
Writing manifest to image destination
Storing signatures
bash-4.2# pwd
/var/task

Cleaning Up a Container

podman container cleanup is a good command to know about.

Shell

$ podman container cleanup --help
Cleanup network and mountpoints of one or more containers

Description:

   podman container cleanup

   Cleans up mount points and network stacks on one or more containers
   from the host. The container name or ID can be used. This command is
   used internally when running containers, but can also be used if
   container cleanup has failed when a container exits.


Usage:
  podman container cleanup [options] CONTAINER [CONTAINER...]

Examples:
  podman container cleanup --latest
  podman container cleanup ctrID1 ctrID2 ctrID3
  podman container cleanup --all

Options:
  -a, --all           Cleans up all containers
      --exec string   Clean up the given exec session instead of the container
  -l, --latest        Act on the latest container podman is aware of
                      Not supported with the "--remote" flag
      --rm            After cleanup, remove the container entirely
      --rmi           After cleanup, remove the image entirely

Buildah

Buildah is a drop-in replacement for using docker build and a Dockerfile.

Where Buildah really shines is in its native commands, which you can use to interact with container builds. Rather than using build-using-dockerfile/bud for each build, Buildah has commands to actually interact with the temporary container created during the build process. (Docker uses temporary, or intermediate containers, too, but you don’t really interact with them while the image is being built.)

Unlike docker build, Buildah doesn’t commit changes to a layer automatically for every instruction in the Dockerfile – it builds everything from top to bottom, every time. On the positive side, this means non-cached builds (for example, those you would do with automation or build pipelines) end up being somewhat faster than their Docker build counterparts, especially if there are many instructions.

– From Getting started with Buildah., published by opensource.com

Some key Buildah subcommands:

buildah bud: Buildah’s build-using-dockerfile, or bud argument makes it behave just like docker build does.
buildah from: Build up a container root filesystem from an image or from scratch.
buildah config: Adjust defaults in the image's configuration blob.
buildah run: buildah run is for running commands that build a container image. This is similar to RUN in a Dockerfile, and unlike docker run.
buildah commit: Commit changes to the container to a new image.
buildah push: Push images to registries (such a Quay) or a local dockerd instance.

Buildah Help

Shell

$ buildah -h
A tool that facilitates building OCI images

Usage:
  buildah [flags]
  buildah [command]

Available Commands:
  add                    Add content to the container
  build-using-dockerfile Build an image using instructions in a Dockerfile
  commit                 Create an image from a working container
  config                 Update image configuration settings
  containers             List working containers and their base images
  copy                   Copy content into the container
  from                   Create a working container based on an image
  help                   Help about any command
  images                 List images in local storage
  info                   Display Buildah system information
  inspect                Inspect the configuration of a container or image
  login                  Login to a container registry
  logout                 Logout of a container registry
  manifest               Manipulate manifest lists and image indexes
  mount                  Mount a working container's root filesystem
  pull                   Pull an image from the specified location
  push                   Push an image to a specified destination
  rename                 Rename a container
  rm                     Remove one or more working containers
  rmi                    Remove one or more images from local storage
  run                    Run a command inside of the container
  tag                    Add an additional name to a local image
  umount                 Unmount the root file system of the specified working containers
  unshare                Run a command in a modified user namespace
  version                Display the Buildah version information

Flags:
  -h, --help                                 help for buildah
      --log-level string                     The log level to be used. Either "debug", "info", "warn" or "error". (default "error")
      --registries-conf string               path to registries.conf file (not usually used)
      --registries-conf-dir string           path to registries.conf.d directory (not usually used)
      --root string                          storage root dir (default "/var/lib/containers/storage")
      --runroot string                       storage state dir (default "/var/run/containers/storage")
      --storage-driver string                storage-driver
      --storage-opt strings                  storage driver option
      --userns-gid-map ctrID:hostID:length   default ctrID:hostID:length GID mapping to use
      --userns-uid-map ctrID:hostID:length   default ctrID:hostID:length UID mapping to use
  -v, --version                              version for buildah

Use "buildah [command] --help" for more information about a command.

Buildah / Dockerfile Compatibility

Buildah can create an image from a Dockerfile by typing:

Shell

$ buildah bud -t hello .

…instead of:

Shell

$ sudo docker build -t hello .

Buildah can create an image called hello from the Dockerfile and the Python app by typing:

Shell

$ buildah bud -t hello .
STEP 1: FROM public.ecr.aws/lambda/python:3.8
  Getting image source signatures
  Copying blob 1de4740de1c2 done
  Copying blob 2e2bb77ae2dc done
  Copying blob df513d38f4d9 done
  Copying blob 03ac043af787 done
  Copying blob 031c6369fb2b done
  Copying blob 842c9dce67e8 done
  Copying config e12ea62c55 done
  Writing manifest to image destination
  Storing signatures
  STEP 2: COPY app.py   ./
  STEP 3: CMD ["app.handler"]
  STEP 4: COMMIT hello
  Getting image source signatures
  Copying blob 109f575f8e6a skipped: already exists
  Copying blob ff64b4f854ad skipped: already exists
  Copying blob dd66ad8702f4 skipped: already exists
  Copying blob d6fa53d6caa6 skipped: already exists
  Copying blob 80166c3283e5 skipped: already exists
  Copying blob 61f74564c3aa skipped: already exists
  Copying blob d95ebdc79761 done
  Copying config 40ef32b39c done
  Writing manifest to image destination
  Storing signatures
  --> 40ef32b39cf
  40ef32b39cf4ffd3d2e4e3426bec4a5ea168524f7f3fcfe863a378abd9794270

Once the build is complete, the new image can be displayed with the buildah images command:

Shell

$ buildah images
REPOSITORY                     TAG      IMAGE ID       CREATED          SIZE
localhost/hello                latest   40ef32b39cf4   56 seconds ago   622 MB

The new image, tagged hello:latest, can be pushed to a remote image registry. This is easily accomplished with the buildah push command.

buildah push Help

Shell

$ man buildah-push
buildah-push(1)                                   General Commands Manual                                  buildah-push(1)

NAME
       buildah-push - Push an image from local storage to elsewhere.

SYNOPSIS
       buildah push [options] image [destination]

DESCRIPTION
       Pushes an image from local storage to a specified destination, decompressing and recompessing layers as needed.

imageID
       Image stored in local container/storage

DESTINATION
       The  DESTINATION  is a location to store container images. If omitted, the source image parameter will be reused as
       destination.

       The Image "DESTINATION" uses a "transport":"details" format. Multiple transports are supported:

       dir:path
         An existing local directory path storing the manifest, layer tarballs and signatures as individual files. This is
       a non-standardized format, primarily useful for debugging or noninvasive container inspection.

       docker://docker-reference
         An  image  in a registry implementing the "Docker Registry HTTP API V2". By default, uses the authorization state
       in $XDG\_RUNTIME\_DIR/containers/auth.json, which is set using (buildah login). If the authorization state  is  not
       found there, $HOME/.docker/config.json is checked, which is set using (docker login).
         If  docker-reference  does  not include a registry name, the image will be pushed to a registry running on local‐
       host.

       docker-archive:path[:docker-reference]
         An image is stored in the docker save formatted file.  docker-reference is only used when creating such  a  file,
       and it must not contain a digest.

       docker-daemon:docker-reference
         An image _dockerreference stored in the docker daemon internal storage. If _dockerreference does not begin with a
       valid registry name (a domain name containing "." or the reserved name "localhost") then the default registry  name
       "docker.io"  will be prepended. _dockerreference must contain either a tag or a digest. Alternatively, when reading
       images, the format can also be docker-daemon:algo:digest (an image ID).

       oci:path:tag
         An image tag in a directory compliant with "Open Container Image Layout Specification" at path.

       oci-archive:path:tag
         An image tag in a tar archive compliant with "Open Container Image Layout Specification" at path.

       If the transport part of DESTINATION is omitted, "docker://" is assumed.

OPTIONS
       --authfile path

       Path of the authentication file. Default is ${XDG_RUNTIME_DIR}/containers/auth.json, which is set using buildah lo‐
       gin.   If  the  authorization  state  is  not found there, $HOME/.docker/config.json is checked, which is set using
       docker login.

       --cert-dir path

       Use certificates at path (*.crt, *.cert, *.key) to connect to the  registry.   Default  certificates  directory  is
       /etc/containers/certs.d.

       --creds creds

       The [username[:password]] to use to authenticate with the registry if required.  If one or both values are not sup‐
       plied, a command line prompt will appear and the value can be entered.  The password is entered without echo.

       --digestfile Digestfile

       After copying the image, write the digest of the resulting image to the file.

       --disable-compression, -D

       Don't compress copies of filesystem layers which will be pushed.

       --encryption-key key

       The [protocol:keyfile] specifies the encryption protocol, which can be JWE  (RFC7516),  PGP  (RFC4880),  and  PKCS7
       (RFC2315) and the key material required for image encryption. For instance, jwe:/path/to/key.pem or pgp:admin@exam‐
       ple.com or pkcs7:/path/to/x509-file.

       --format, -f

       Manifest Type (oci, v2s1, or v2s2) to use when saving image to directory using the  'dir:'  transport  (default  is
       manifest type of source)

       --quiet, -q

       When writing the output image, suppress progress output.

       --remove-signatures

       Don't copy signatures when pushing images.

       --sign-by fingerprint

       Sign the pushed image using the GPG key that matches the specified fingerprint.

       --tls-verify bool-value

       Require HTTPS and verify certificates when talking to container registries (defaults to true)

EXAMPLE
       This example pushes the image specified by the imageID to a local directory in docker format.

       # buildah push imageID dir:/path/to/image

       This example pushes the image specified by the imageID to a local directory in oci format.

       # buildah push imageID oci:/path/to/layout:image:tag

       This example pushes the image specified by the imageID to a tar archive in oci format.

       # buildah push imageID oci-archive:/path/to/archive:image:tag

       This example pushes the image specified by the imageID to a container registry named registry.example.com.

       # buildah push imageID docker://registry.example.com/repository:tag

       This example pushes the image specified by the imageID to a container registry named registry.example.com and saves
       the digest in the specified digestfile.

       # buildah push --digestfile=/tmp/mydigest imageID docker://registry.example.com/repository:tag

       This example works like docker push, assuming registry.example.com/my_image is a local image.

       # buildah push registry.example.com/my_image

       This example pushes the image specified by the imageID to a private container registry  named  registry.example.com
       with authentication from /tmp/auths/myauths.json.

       # buildah push --authfile /tmp/auths/myauths.json imageID docker://registry.example.com/repository:tag

       This example pushes the image specified by the imageID and puts into the local docker container store.

       # buildah push imageID docker-daemon:image:tag

       This example pushes the image specified by the imageID and puts it into the registry on the localhost while turning
       off tls verification.
        # buildah push --tls-verify=false imageID docker://localhost:5000/my-imageID

       This example pushes the image specified by the imageID and puts it into the registry on the localhost using creden‐
       tials and certificates for authentication.
        #   buildah   push   --cert-dir      /auth  --tls-verify=true  --creds=username:password  imageID  docker://local‐
       host:5000/my-imageID

ENVIRONMENT
       BUILD_REGISTRY_SOURCES

       BUILD_REGISTRY_SOURCES, if set, is treated as a JSON object which contains lists of registry names under  the  keys
       insecureRegistries, blockedRegistries, and allowedRegistries.

       When pushing an image to a registry, if the portion of the destination image name that corresponds to a registry is
       compared to the items in the blockedRegistries list, and if it matches any of them, the push attempt is denied.  If
       there are registries in the allowedRegistries list, and the portion of the name that corresponds to the registry is
       not in the list, the push attempt is denied.

       TMPDIR The TMPDIR environment variable allows the user to specify where temporary files are  stored  while  pulling
       and pushing images.  Defaults to '/var/tmp'.

FILES
       registries.conf (/etc/containers/registries.conf)

       registries.conf  is the configuration file which specifies which container registries should be consulted when com‐
       pleting image names which do not include a registry or domain portion.

       policy.json (/etc/containers/policy.json)

       Signature policy file.  This defines the trust policy for container images.  Controls  which  container  registries
       can be used for image, and whether or not the tool should trust the images.

SEE ALSO
       buildah(1), buildah-login(1), containers-policy.json(5), docker-login(1), containers-registries.conf(5)

buildah                                                  June 2017                                         buildah-push(1)

Shell

$ buildah run \
  --entrypoint /var/lang/bin/pip \
  --rm \
  --user "$(id -u):$(id -g)" \
  -v "$(pwd):/mnt" \
  amazon/aws-lambda-python:3.8 \
  install --target /mnt/build --upgrade psycopg2-binary

How To

The following was inspired by Recap: images and containers from Docker for beginners. The equivalent commands for Docker alternatives are shown.

Check software version

Shell

$ docker -v
Docker version 20.10.2, build 20.10.2-0ubuntu1~20.10.1

Shell

$ buildah -v
buildah version 1.15.2 (image-spec 1.0.1, runtime-spec 1.0.2-dev)

Shell

$ podman -v
podman version 2.0.6

Download the Amazon Linux 2 image

AWS Lambda functions run under Amazon Linux.

Each of these 3 commands does a very similar task, downloading a specific image. Docker uses different subdirectories for images than Buildah and podman do.

Shell

$ sudo docker pull amazonlinux
Using default tag: latest
latest: Pulling from library/amazonlinux
3c2c91c7c431: Pull complete
Digest: sha256:06b9e2433e4e563e1d75bc8c71d32b76dc49a2841e9253746eefc8ca40b80b5e
Status: Downloaded newer image for amazonlinux:latest
docker.io/library/amazonlinux:latest

Buildah works without complaint.

Shell

$ buildah pull amazonlinux
53ef897d731f9a5673c083d0e86d7911f85d6e63bb2be2346b17bdbacdc58637

podman seems to hiccup and then complete successfully.

Shell

$ podman pull amazonlinux
Trying to pull quay.io/amazonlinux...
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n"
Trying to pull docker.io/library/amazonlinux...
Getting image source signatures
Copying blob 3c2c91c7c431 [--------------------------------------] 0.0b / 0.0b
Copying config 53ef897d73 done
Writing manifest to image destination
Storing signatures
53ef897d731f9a5673c083d0e86d7911f85d6e63bb2be2346b17bdbacdc58637

Run a Bash Command in an OCI Container

Again, Docker must be run as root for this operation, this represents an unnecessary security risk.

Shell

$ sudo docker container run amazonlinux echo 'Hello World!'
Hello World!

Shell

$ podman container run amazonlinux cat /etc/os-release
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Show All Locally Available Images

Again, Docker must be run as root for this operation, this represents an unnecessary security risk.

Shell

$ sudo docker images
REPOSITORY    TAG       IMAGE ID       CREATED        SIZE
amazonlinux   latest    53ef897d731f   21 hours ago   163MB

Shell

$ podman images
REPOSITORY                     TAG     IMAGE ID      CREATED       SIZE
localhost/hello                latest  40ef32b39cf4  5 hours ago   622 MB
docker.io/library/amazonlinux  latest  53ef897d731f  21 hours ago  170 MB

List OCI Containers

Shell

$ sudo docker container ls
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

Shell

$ podman container ls
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

View All OCI Containers (Running or Not)

Shell

$ sudo docker container ls -a
CONTAINER ID   IMAGE         COMMAND                 CREATED          STATUS                      PORTS     NAMES
250f56d9aced   amazonlinux   "echo 'Hello World!'"   14 minutes ago   Exited (0) 14 minutes ago             competent_einstein

Shell

$ podman container ls -a
CONTAINER ID  IMAGE                                 COMMAND            CREATED            STATUS                        PORTS   NAMES
  0f8203e9d3b8  docker.io/library/amazonlinux:latest  echo Hello world!  36 minutes ago     Exited (0) 36 minutes ago
    beautiful_mestorf
  14282ace8978  docker.io/library/amazonlinux:latest  echo Hello world!  36 minutes ago     Exited (0) 36 minutes ago
    beautiful_goldwasser
  1b9a8db52fb9  docker.io/library/alpine:latest       echo Hello World!  About an hour ago  Exited (0) About an hour ago          zealous_easley
  6444ee144488  docker.io/library/amazonlinux:latest  echo Hello World!  12 minutes ago     Exited (0) 12 minutes ago
    frosty_ritchie
  7444122cbc59  docker.io/library/alpine:latest       cat /etc/motd      About an hour ago  Exited (0) About an hour ago          elated_sammet
  aef84973d6ad  docker.io/library/amazonlinux:latest  echo Hello world!  About an hour ago  Exited (0) About an hour ago          lucid_sinoussi
  e210f74bc209  docker.io/library/amazonlinux:latest  cat /etc/motd      About an hour ago  Exited (0) About an hour ago          jovial_borg

List Running OCI containers

Shell

$ sudo docker container ps -a
CONTAINER ID   IMAGE         COMMAND                 CREATED         STATUS                     PORTS     NAMES
250f56d9aced   amazonlinux   "echo 'Hello World!'"   5 minutes ago   Exited (0) 5 minutes ago             competent_einstein

podman has a problem with the container ps sub-subcommand.

Shell

$ podman container ps -a
Error: unrecognized command `podman container ps`
Try 'podman container --help' for more information.

buildah push to Docker Daemon

Shell

$ buildah push hello:latest docker-daemon:hello:latest
Getting image source signatures
  Copying blob sha256:72fcdba8cff9f105a61370d930d7f184702eeea634ac986da0105d8422a17028
   247.02 MiB / 247.02 MiB [==================================================] 2s
  Copying blob sha256:e567905cf805891b514af250400cc75db3cb47d61219750e0db047c5308bd916
   144.75 MiB / 144.75 MiB [==================================================] 1s
  Copying config sha256:6d54bef73e638f2e2dd8b7bf1c4dfa26e7ed1188f1113ee787893e23151ff3ff
   1.59 KiB / 1.59 KiB [======================================================] 0s
  Writing manifest to image destination
  Storing signatures 

$ buildah images | head -n2
REPOSITORY              TAG             IMAGE ID        CREATED                 SIZE
docker.io/hello      latest       6d54bef73e63  2 minutes ago   398 MB 

$ buildah run -t hello:latest
Hello, world!

Delete an OCI Image

Delete an OCI image in Buildah's ~/.local/share/container directory with the rmi subcommand:

Shell

$ buildah rmi e12ea62c5582
e12ea62c5582f91a2228e3e284ea957f2df4f1cdb150fd2c189ef8f11d7633ce

Stack Overflow Culture: Zero-Sum, Authoritarian and Hormonally Imbalanced

2021-04-18T00:00:00-04:00

This article describes some problems that significantly impact Stack Overflow users, and offers suggestions for improvement. If you are unfamiliar with Stack Overflow, or would like to read a summary of what this article is based on, Kevin Workman wrote a terrific summary in March 2019 entitled “The Stack Overflow Culture Wars”. Very little has changed since then.

Introduction

Stack Overflow is the premier website world-wide for programmers to help each other by asking and answering questions. It has a defined protocol for this type of interaction, however new user on-boarding is often ineffective, so newcomers are not properly informed, and old-timers often do not exhibit appropriate people skills. The protocol is somewhat misguided and appropriate tools for better interaction are not provided.

As a result, this website has developed a well-documented reputation for being “a valuable resource, but a scary place to contribute due to potential hostility.”

Sometimes, loving something means caring enough to admit that it has a problem.

– Jay Hanlon, writing about Stack Overflow when he was EVP of Culture and Experience.

Stack Overflow suffers from militant moderators who close and delete reasonable submissions and answers due to Draconian rules.

– sleavey commenting on a Hacker News thread entitled Stack Overflow Culture.

Change Starts At the Top

Prashanth Chandrasekar, CEO

Prashanth Chandrasekar became CEO of Stack Overflow in September, 2019. Inside Intercom interviewed Mr. Chandrasekar in August 2020. This puff piece made no mention of any cultural problems. The focus was on the brilliance of Stack Overflow’s technology. If Mr. Chandrasekar has a vision for how to guide social change, he did not mention it. Instead, in the article and his publications since then he only speaks publicly about transitioning Stack Overflow to a product-led SaaS company.

Shortly after becoming CEO, Mr. Chandrasekar published “Scripting the Future of Stack Overflow, in which he wrote:

We learned that we needed much better channels to listen to our moderators and community members. We have not evolved the existing channels of engagement for power users in our community, like Meta, or articulated how we intended to make improvements going forward. This has caused friction as our user base and business have rapidly grown. We acknowledge these issues, apologize for our mistakes, and have plans for improving in the future.

Later in the article, he mentioned improvements to the code of conduct, a survey and establishing a moderator council. More than 2 years later, none of this has made the slightest difference in Stack Overflow's culture.

Mihir Pathak — EVP, Strategy & Transformation

Prior to his employment at Stack Overflow, Mr. Pathak was a derivatives strategist McKinsey & Company, a world-renouned change management company. That is to say, although Mr. Pathak worked at McKinsey, he was not there to assist other companies make structural changes; instead, he was responsible for pricing methodologies and hedging techniques underlying financial derivative products and options trading strategies – a heads-down money manager.

Mr. Pathak’s page at StackOverflow is no longer available. Google cached it. No announcement has been made about his departure or if there will be a replacement.

Mr. Pathak's job description is still visible at themuse.com:

Mihir is responsible for the long-term business strategy of Stack Overflow, which includes forming partnerships with like-minded organizations and understanding how to best serve the needs of future developers.

Mr. Pathak was employed from September 2016 at Stack Overflow as a business development executive, not a change management champion.

Teresa Dietrich, Chief Product and Technology Officer

In January 2020 Teresa Dietrich was made Chief Product and Technology Officer. She also came from McKinsey & Company, where she was Global Head of Product & Engineering. Two months after she took the job she wrote “Sharing our first quarter 2020 community roadmap”. That rather bland article did not seem to indicate any recognition of serious problems within the Stack Overflow culture, and I could not find any publicly available results of the studies that were mentioned.

16 months after Ms. Dietrich started her job, I do not see any cultural change. Has Ms. Dietrich been tasked with leading such a change? If so:

Does she have board-level support?
Does she have what she needs to make significant cultural changes?
Why has she not made any public acknowledgement of a cultural problem? One possible answer is that her boss, Mr. Chandrasekar, has not done so.
Is Ms. Dietrich the right person for the job? This is not a purely technical problem, it is a social problem. I believe that women in general possess more highly developed social skills, but the skills necessary to climb to the top are not the skills required to make this type of cultural shift.

Are Mr. Chandrasekar and Ms. Dietrich part of the cultural problem, or part of the solution?

Where Am I Coming From?

I have no agenda, no investment in the status quo, nothing to prove, no contacts at the company, I am not an undercover journalist, and I am not a competitor or investor. I am just Joe User... and I am not shy when I believe I have something that I think needs to be heard.

If I mispeak, please tell me. If I missed something, again please tell me. If there is a bigger picture I would like to learn about it.

I have used Stack Overflow and its sibling websites for more than 10 years. Until a few weeks ago, I contributed a few answers here and there, and otherwise did not spend much time on the sites. All contributors are volunteers, so the only reasons to contribute are for prestige, social bonding and altruism.

For almost 30 days, in my spare time, I helped people who asked questions on Stack Overflow. I am writing this article because although I enjoy helping others, the enjoyment I experienced while doing so within the Stack Overflow environment was overshadowed by the regressive behavior tolerated and even enforced by other contributors. I should also say that I did encounter certain other contributors with whom interaction was very pleasant. However, interactions with alpha contributors with the highest scores seemed more often than not to be quite unpleasant. Since I first published this article, I have mostly not interacted on the site. I remain open to contributing to improvements in the culture.

The next image is provided so readers know that I am able to effectively participate in the current Stack Overflow website.

The volume of accepted and upvoted answers put me in the top 0.14% of Stack Overflow answerers in under 30 days.

Over 100 of the answers I offered in a 30-day period were accepted as the preferred answer. In fact, most of my answers were selected as the preferred answer. That means many more alternative answers were not accepted.

After losing, sometimes a contributor will delete their post. I have done it myself, when the winning post was significantly better by all measures.

Some of the other contributors who had provided alternative answers that were not selected clearly felt they had lost a competition. This structure, and others that I describe below, define a system designed to generate hostility. Stack Overflow, as currently implemented, promotes dominance behavior, which for most primates (other than bonobos) is patriarchal.

Gamification

Stack Overflow's success has be in part due to its successful gamification of the interaction between questioners and answerers. Gamification is powerful and addictive. Unfortunately, the model chosen resembles FPS (first-person shooter) games, instead of co-operative games.

Jeff Atwood, one of the two original authors of Stack Overflow, wrote an article in 2011 entitled The Gamification, in which he writes:

Gaming elements are not tacked on to the Stack Exchange Q&A engine, but a natural and essential element of the design.

Just below that sentence, Mr. Atwood shows a screenshot from an FPS video game:

FPS Game screenshot from 'The Gamification'.

I haven't ever quite come out and said it this way, but … I played a lot of Counter-Strike from 1998 to 2001, and Stack Overflow is in many ways my personal Counter-Strike.

– Jeff Atwood, from “The Gamification”

FPS games are structured so the player only wins by killing others. This is an entirely different motivational structure than a scenario where a person only wins if others succeed.

Terminology

I use a few non-standard terms in this article:

Questioner: One who provides a question
Answerer: One who provides an answer
Downvoter: One who downvotes another person's contribution
Downvotee: One who has their contribution downvoted

Dialog Improves Most Questions

A small percentage of questions asked on Stack Overflow are unambiguous, contain all the necessary information, and are phrased well enough to be understood. For these questions, answers can be posted without any interaction between questioner and potential answerers.

However, most questions involve a dialog between potential answerers and the questioner. In the dialog, the question is refined, and the questioner's code and any other relevant data is elicited and provided. The tools provided for such a dialog are unfortunately problematic.

The only two mechanisms for interaction between questioner and potential answerers are comments and answers. Comments have severe limitations that greatly reduce their effectiveness for eliciting information from a questioner:

Comments must be very short
Comments cannot be formatted properly
Comments cannot be edited for more than a few minutes

This means that answerers who are trying to explain something to the questioner to elicit more information, or guide the questioner towards understanding their problem better, must resort to posting an incomplete answer. Posting an incomplete answer is risky; other potential answerers can attack the answer by downvoting it.

Downvotes typically last forever

Some Incentives Promote Hostility

Many long-time users have completely objectified other users, and act as if Stack Overflow is a video game. Points are accumulated, and at any given time there are a finite number of questions to answer. A person's reputation on Stack Overflow is represented by a single number, which is the number of points they have accumulated. This single number is the structural source of many problems. A more nuanced reputation score would be a giant step forward.

Many of these long-term answerers have come to view their participation on Stack Overflow as a zero-sum competition; they can only win (that is, have their answer accepted) if everyone else loses (that is, no-one else provides an answer that is accepted).

Some answerers employ intimidation order to suppress competition. Downvotes are often used in the same way against other answerers as a brush back pitch in baseball.

Downvoting has no negative consequences for the downvoter

Standard operating procedure for competitively-minded answerers is to downvote answers from others at every opportunity. There is no risk in downvoting:

Downvotes are untraceable; there is no public record of who downvoted or when a downvote was cast.
Downvotes are free to downvoters; this encourages liberal downvoting.

This scenario incentivizes competitively-minded answerers to strafe the competition with downvotes at every opportunity.

If downvotes cost the downvoter the same number of points as they penalize the downvotee, then downvotes would become much rarer.

Downvotes typically last forever. Yes, a downvoter could theoretically reverse a downvote, but it is awkward for them to find their old downvotes, and there is no incentive to do so.

Questions from newcomers are also frequently downvoted, without discussion, or along with hostile remarks. That leaves a permanent impression, and tends to select for new members who are comfortable with dominance-based hostility. This is a self-perpetuating, and highly toxic, social order.

Toxic Masculinity Harms Men and Society As A Whole, from Focus for Health

Suggestion: Gamified Onboarding

The only information for new users that is directly accessible from the Stack Overflow front page, is the Help Center, under the question mark icon. It is obvious from the often very polite and tentative inquiries made by new users when they ask their first question that they never noticed that information, or if they did it did not seem relevant. ... and then those new users are mercilessly slammed.

New users should be presented with a short instructional question and answer-style introduction, where information is provided on how to be a good Stack Overflow user. This should happen before they get the opportunity to post their first question. Different levels of users should be explained. Although Stack Overflow is all-English, the onboarding should be multilingual.

Suggestion: Multilingual Support

A high percentage of users do not speak English very well. They really struggle, and tolerance is low on Stack Overflow for bad English. Other sites, for example Facebook and LinkedIn, have a translation facility built right in. I think Facebook did a particularly good job. Why not do something similar on StackOverflow.com? English would be the official language, but everyone world-wide would be able to interact much more effectively.

This site is multilingual.
It is not that hard to do!

Machine translation is really quite good. I have it on this site. Want to view this site in one of over 100 languages? Just go to the top of this page and click on this pull-down menu labeled Select Language:

You will then see the list of languages that you can view this website in:

Morally speaking, not to provide a multilingual site discriminates against non-English-speaking people.

Suggestion: More Nuanced Reputation Score

Instead of a single metric, answerers should be rated along several dimensions. Economists and psychologists both use multidimensional diagrams. The following diagram represents data in 4 dimensions. More dimensions can easily be shown in this type of diagram.

Multi-dimensional data can easily be visualized by outlined shapes

Instead of "bigger is better" (a single number indicating status, with the high score indicating alpha status), more information would allow for more detail, so a fuzzy diagram would show little interaction, while a highly detailed and intricate design would indicate a lot of participation. Some answerers might be stronger regarding some metrics, while being weaker on other metrics.

Instead of displaying a person's score, the shape of their participation would be shown. Some people prefer to be seen as well-rounded, others prefer to be the best on selected aspects and ignore the others. One size does not fit all.

People would start to give pet names for various shapes. Jokes would be made about the shapes.

HR personnel would start to hire teams based on how well these shapes meshed together. Money will be made by timely entrepreneurs because these shapes will quickly be adopted industry-wide. Some will aspire to change their shape.

An entire industry will spring up servicing those who wish to modify their shape.

Suggestion: Encourage and Highlight Elicitation

Elicitation is a difficult skill to master. The current high-scorers have no incentives to employ elicitation, and they act as if on a campaign to eradicate it.

Introduce an Elicitation Phase

A button should be introduced that lets everyone who visits the question page that a potential answerer would like to elicit information. While at least one such button is enabled, no downvotes are possible relating to the question, and the question cannot be closed or moved to another forum by anyone, regardless of their privilege level.

The elicitation mode would time out, but not suddenly or unexpectedly. Instead, it would back off, rather like the Ethernet back-off algorithm for collision resolution used in random access MAC protocols.

Both the potential answerer and the questioner would be given cues that they have a question or a response waiting, and after a period of inactivity the special status ends. I leave the details of the timing undefined for others to discuss.

Suggestion: Restrict Downvoting

Downvoting needs to incorporate:

Accountability (no more anonymous downvoting)
Cost (no more drive-by shootings without consequences)
Time window (vote after the dust settles, not during the elicitation phase)
Public displays of user profiles should prominently display that user's downvotes and upvotes, with links
Aggregate statistics on user profiles of their percentage downvotes and upvotes, trends (absolute and relative), etc.

Serverless E-Commerce

2021-04-14T00:00:00-04:00

As readers of this blog know, I have been chronicling my adventure into Python-powered e-commerce for several months. I have been focusing on Django in general, and Django-Oscar in particular. Webapps made with this technology are almost exclusively run on dedicated real or virtual machines. Serverless computing is a method of providing backend services on an as-used basis. AWS Lambda is the best-known example of serverless computing, and it combines nicely with a CDN like AWS CloudFront.

This article discusses 3 goals for an e-commerce system. Two goals are provided by the technology behind serverless webapps:

Enormous and instantaneous scalability.
Pay-as-you-go without an up-front cost commitment.

I have one more goal: very low latency for online shoppers.

The Big Picture

AWS Lambda consists of two main parts: the Lambda service which manages the execution requests, and the Amazon Linux micro virtual machines provisioned using AWS Firecracker, which actually runs the code.

A Firecracker VM is started the first time a given Lambda function receives an execution request (the so-called “Cold Start”), and as soon as the VM starts, it begins to poll the Lambda service for messages. When the VM receives a message, it runs your function code handler, passing the received message JSON to the function as the event object.

Thus every time the Lambda service receives a Lambda execution request, it checks if there is a Firecracker microVM available to manage the execution request. If so, it delivers the message to the VM to be executed.

In contrast, if no available Firecracker VM is found, it starts a new VM to manage the message. Each VM executes one message at a time, so if a lot of concurrent requests are sent to the Lambda service, for example due to a traffic spike received by an API gateway, several new Firecracker VMs will be started to manage the requests and the average latency of the requests will be higher since each VM takes roughly a second to start.

– From How to run any programming language on AWS Lambda: Custom Runtimes by Matteo Moroni.

AWS Lambda Limits

AWS Lambda programs have access to considerable resources, enough for most e-commerce stores. The AWS Lambda runtime environment has the following limitations, some of which can be improved upon with some work:

The disk space (ephemeral) is limited to 512 MB.
The default deployment package size is 50 MB.
The memory range is from 128 to 3008 MB.
The maximum execution timeout for a function is 15 minutes.
Request and response (synchronous calls) body payload size can be up to 6 MB.
Event request (asynchronous calls) body can be up to 128 KB.

CloudFront

Brian Caffey wrote Building a Django application on AWS with Cloud Development Kit (CDK). The website Mr. Caffey's article discusses does not use Lambda, instead his website is always running. So, this option is quite informative and well-thought-out, but it is AWS-specific and does not discuss serverless architecture.

For me, the most interesting part about Mr. Caffey's article is it mentions using 3 origins with AWS CloudFront: (1) an origin for ALB (for hosting the Django API), (2) an origin for the S3 website (static Vue.js site), and (3) an S3 origin for Django assets. Mr. Caffey does not say why he used 3 origins, but feeding one CloudFront distribution from multiple origins would mean that all of their content would appear on the same Internet subdomain.

This means that the extra HTTP handshaking required for certain CORS (cross-origin HTTP requests) requests between subdomains would be avoided; specifically, there would be no need for pre-flight requests. This would make the website seem noticeably faster if users did lots of content editing and/or transactions with the website. My own pet project has users creating and modifying content, and purchasing product, so taking the requirement for the CORS handshakes away would be a win, plus the end user's web browser could reuse the origin HTTP connection, speeding up even non-cacheable requests.

Tamás Sallai wrote How to route to multiple origins with CloudFront – Set up path-based routing with Terraform. Mr. Sallai is a prolific writer!

Edge Computing

Performing computations and serving assets from a nearby point of presence minimizes latency for end users. E-commerce customers much prefer online stores that respond quickly. Edge computing can deliver that experience world-wide, and developers can deploy their work from wherever they are.

AWS Lambda@Edge (console) runs the Lambda computation in one of 13 regional AWS points of presence, one hop removed from the CloudFront edge locations, or at least in the same availability zone at the CloudFront point of presence. Distributed database issues would need to be addressed before significant benefits would accrue from this implementing this decentralized architecture. Unfortunately, Lambda@Edge has some significant restrictions that prevent it from running nontrivial Django apps.

Lambda@Edge Restrictions

From requirements and restrictions on using Lambda functions with CloudFront, it is apparent that it is not possible to run non-trivial Django apps securely at the edge with good performance.

You can add triggers only for functions in the US East (N. Virginia) Region.
You can’t configure your Lambda function to access resources inside your VPC.
AWS Lambda environment variables are not supported.
Lambda functions with AWS Lambda layers are not supported.
Using AWS X-Ray is not supported.
AWS Lambda reserved concurrency and provisioned concurrency are not supported.
Lambda functions defined as container images are not supported.

Until such time as Lambda@Edge removes the above restrictions, Django webapps will continue to be deployed as centralized webapps, which means that ultra-low latency is not possible world-wide.

CloudFront Functions

CloudFront Functions are closer to the user, but have even more restrictions than Lambda@Edge. Alas, CloudFront Functions do not seem likely to be able to support significant computation any time soon.

From “Introducing CloudFront Functions – Run Your Code at the Edge with Low Latency at Any Scale”

Infrastructure as Code (IaC)

For anything bigger than a toy cloud application, Infrastructure as Code (IaC) is table stakes. You’d be hard-pressed to find someone managing anything of scale who thinks letting folks point and click in the console is the optimal route.

– From CloudFormation, Terraform, or CDK? A guide to IaC on AWS by Jared Short, published by acloudguru.com.

Terraform, AWS CloudFormation, Packer, Pulumi, and GeoEngineer are the most popular tools in the category "Infrastructure Build Tools".
– from Stackshare.io

Infographic: Lambda Framework Comparison

Yan Cui at Lumigo.io made this terrific infographic, which compares 9 serverless application frameworks and infrastructure management tools according to opinionatedness and customizability. This article discusses some of those technologies.

From 'AWS Lambda Deployment Frameworks', by Yan Cui at lumigo.io

The trade-off between customizability and opinionatedness is that highly customizable frameworks require more code to do things that opinionated frameworks do more succinctly. On the other hand, very opinionated frameworks are more limited in their abilities. A classic example of an opinionated framework is Ruby on Rails, which is specifically designed for master/detail applications. Other types of applications should use a different framework, or no framework at all.

Two of the technologies on the above infographic are Zappa and Terraform, both of which I discuss in this article. Zappa is rather opinionated, while Terraform is very customizable.

AWS Cloud Development Kit (CDK)

AWS CDK provides a programmatic interface for modeling and provisioning cloud resources. Languages supported include Java, JavaScript, .NET, Node.js, Python and Typescript.

Even if AWS is not directly the service provider, awareness of the AWS CDK is important because some other options, for example the Cloud Development Kit for Terraform (cdktf), are based on AWS CDK.

Chalice – Serverless Django on AWS

Chalice is an AWS open-source project that has good traction. This Python serverless microframework for AWS allows applications that use Amazon API Gateway and AWS Lambda to be quickly created and deployed.

The name and logo of this project are suggestive of the Holy Grail. I found the thinly veiled references to Christianity to be off-putting. Religious references have no place in a professional environment. Programmers who work with this project have religious icons, words and phrases continuously presented to them, and they must write words that are strongly identified with Christian doctrine for them to write software. This is forced indoctrination.

Django w/ Zappa & AWS Lambda

Zappa is a popular library for serverless web hosting of Python webapps. Zappa allows Python WSGI webapps like Django to run on AWS Lambda instead of from within a container like AWS EC2. I am particularly interested in using Zappa to package and run django-oscar for AWS Lambda and CloudFront.

Zappa can perform two primary functions:

Packaging – Zappa can build a Django webapp into an AWS Lambda package. The package can be delivered via other mechanisms, for example mechanisms that are not even Python aware.
Deploying – Zappa can deploy and Django webapp to AWS Lambda, and configure several AWS services to feed events to the Django webapp.

Zappa does not provide a means to define additional resources as part of the overall infrastructure. It is also somewhat rigid in how it defines certain resources which can lead to friction when incorporating Zappa within organizations with more rigid requirements on cloud resource management. With Zappa, you are better off allowing it to manage all the pieces needed for your web application on its own and manage other resources with a separate tool such as stacker or Terraform.

… or use Zappa's package command to create an archive that is ready for upload to lambda and utilize the other helpful functions the project provides for use after code is deployed.

– from The Evolution of Maintainable Lambda Development Pt 2 by JBS Custom Software Solutions.

The Zappa documentation is excellent. The project has some rough edges, but the new regime coming on board seem competent and fired up. They have some work ahead to set things straight, but the technical path seems clear.

I think this project deserves special attention. Lots of moldy issues and PRs need to be processed, which a small team could get done fairly quickly. The project might also benefit from someone to hone the messaging. I opened an issue on the Zappa GitHub microsite to discuss this.

This seminal project has been around several years, and other well-known projects that have been developed since Zappa was first released have acknowledged that Zappa provided inspiration. Time to brush it up and set it straight again; its best days lie ahead!

Edgar Roman wrote this helpful document: Guide to using Django with Zappa.

I've messing around with Zappa, will report back.

Videos

Videos of Zappa exist.

This video has got all the right technologies mixed together for me: Serverless Deployment of a Django Project with AWS Lambda, Zappa, S3 and PostgreSQL.

Djambda / AWS Lambda / Terraform

Terraform does not impose a runtime dependency unless the realtime orchestration features are used.

Djambda is an example project setting up Django application in AWS Lambda managed by Terraform. I intend to play with it and write up my experience right here Real Soon Now.

This project uses GitHub Actions to create environments for the master branch and pull requests. I wonder if this project can be used without GitHub actions?

[Terraform] does not provide an abstraction layer for the AWS, Azure, or Google Cloud. It does that deliberately, as you should embrace all aspects when using cloud - not extract a common denominator from the services delivered by the cloud provider.

– From AWS CDK? Why not Terraform? by Wojciech Gawroński.

Serverless Framework with WSGI

The docs describe Serverless WSGI as:

Serverless plugin to deploy WSGI applications (Flask/Django/Pyramid etc.) and bundle Python packages.

I am concerned that the Serverless architecture requires an ongoing runtime dependency on the viability and good will of Serverless, Inc. Any hiccup on their part will immediately be felt by all their users. It would make me nervous to base daily operational infrastructure on this.

Bintray and JCenter Went Poof!

I do not want to rely upon online services from a software tool vendor to run my builds. The Scala community is still recovering from Bintray and JCenter shutting down. I had dozens of Scala libraries on Bintray. I do not plan to migrate them, they are gone from public access.

On February 3, 2021, JFrog announced that they will be shutting down Bintray and JCenter. A complete shutdown is planned for February 2022.

– JCenter shutdown impact on Gradle builds

Trading Autonomy for Minimal Convenience is a Poor Trade

Remember that free products are usually subject to change or termination without notice. Examples abound of many companies whose free (and non-free) products suddenly ceased. There is no need to assume this type of vulnerability, so I block my metaphoric ears to the siren sound that tempts trusting souls into assuming unnecessary dependencies, and I chose tooling that is completely under my control.

What happens if I exceed the fair use policy?

From the Serverless Pricing and Terms page.

We want to offer a lot of value for free so you can get your idea off the ground, before introducing any infrastructure cost. The intent of the fair use policy is to ensure that we can provide a high quality of service without incurring significant infrastructure costs. The great majority of users will fall well within range of the typical usage guidelines. While we reserve the right to throttle services if usage exceeds the fair use policy, we do not intend to do so as long as we can deliver a high quality of service without significant infrastructure costs.

If you anticipate your project will exceed these guidelines, please contact our support team. We’ll work with you on a plan which scales well.

AWS AppRunner

AWS just announced AppRunner. I wonder how suitable it is...

Merging a Remote File with a Local File

2021-04-12T00:00:00-04:00

Today I am once again re-installing WSL2 on one of my laptops. Seems that a Windows 10 installation’s half-life is measured in months, after which time a reset is required. The reset preserves data, but not installed programs and not the WSL setup.

When I set up an OS I often use a pre-existing system’s files as templates for the new system’s files.

Meld

Meld is a fantastic, F/OSS file and directory merge tool. 2-way and 3-way merges are supported. Meld uses X Windows for its user interface. GWSL makes it easy to run X apps on WSL and WSL2.

Shell

$ yes | sudo apt install meld

Merging a remote file with a local file using Meld is easy once you know how. Unless the remote file system is mounted locally, Meld cannot be used to modify remote files and directories, just local files and directories.

Following is the incantation I used to display my local .profile and interactively merge it with my profile on an Ubuntu Linux machine called gojira.

Shell

$ meld ~/.profile <(ssh mslinn@gojira cat .profile)&

The above runs ssh in a subshell, logs in as mslinn to the machine called gojira and then displays the contents of .profile on gojira. Meld compares the output of cat with the local copy of ~/.profile, and displays the differences:

😁

Meld makes it easy to reconcile file versions.

Visual Studio Code Workspace Settings

2021-04-11T00:00:00-04:00

For me, the killer feature that Visual Studio Code is how it integrates the Windows user interface with working on WSL and WSL2. Programs residing on the active WSL OS image execute natively on that OS, while VSCode continues to run as a native Windows application. This is possible because VSCode installs a proxy on the target OS. The proxy does the bidding of the Windows executable.

Getting a project to execute on the target OS instead of the host OS can be tricky. I have found that using a workspace to hold a collection of VSCode projects is very helpful, because the definition of the collection also defines how they are handled.

WSL projects have different types of VSCode workspace entries than Windows entries do. They are easy to recognize and change once you know what to look for. The two possibile types of VSCode workspace project entries in a .workspace file are:

WSL Project – "uri": "vscode-remote://wsl+ubuntu/path/to/vscode/project"
Windows Project – "path": "C:\\path\\to\\vscode\\project"

The following VSCode workspace file has both types of entries. For me, this is an error; I only want WSL projects. My task is to change the yellow highlighted Windows project and make it look like the other WSL projects.

aw.workspace.code-workspace

{
  "folders": [
          {
                  "uri": "vscode-remote://wsl+ubuntu/var/sitesUbuntu/www.ancientwarmth.com"
          },
          {
                  "uri": "vscode-remote://wsl+ubuntu/var/work/django/django"
          },
          {
                  "uri": "vscode-remote://wsl+ubuntu/var/work/django/oscar"
          },
          {
                  "uri": "vscode-remote://wsl+ubuntu/var/work/ancientWarmth/ancientWarmth"
          },
          {
                  "path": "../../var/work/django/main"
          }
  ],
  "remoteAuthority": "wsl+Ubuntu",
  "settings": {
          "liveServer.settings.multiRootWorkspaceName": "www.mslinn.com",
          "python.pythonPath": "/var/work/django/oscar/bin/python",
          "git.ignoreLimitWarning": true,
          "sqltools.connections": [
                  {
                          "previewLimit": 50,
                          "server": "localhost",
                          "port": 5432,
                          "driver": "PostgreSQL",
                          "name": "Ancient Warmth on Camille",
                          "database": "ancient_warmth",
                          "username": "postgres",
                          "password": "hithere"
                  }
          ]
  }
}

All I need to do is change this entry:

aw.workspace.code-workspace snippet

"path": "../../var/work/django/main"

To:

aw.workspace.code-workspace snippet

"uri": "vscode-remote://wsl+ubuntu/var/var/work/django/main"

😁

The modified entry will cause VSCode to launch the project from WSL, instead of Windows.

A Python Virtual Environment For Every Project

2021-04-09T00:00:00-04:00

Python virtual environments are cheap to make and use – unless you are unfortunate enough to program in native Windows. I have adopted the habit of making a Python virtual environment (venv) for each significant Python project, plus a default venv for trivial Python work.

Why Virtualize Python?

It is better to use virtualized user- and project-specific Python instances, instead of working with a system-wide installation of Python. This allows you to install and upgrade Python packages without using supervisor privileges. Also, virtualized instances allows you to work on many different independent Python projects at the same time, without package version collisions.

Docker is over-sold. It adds unnecessary complexity to software projects. Instead of virtualizing the entire software environment, as docker attempts to do, virtualizing the Python programming environment as described in this article, Ruby rbenv, or node.js nvm are much easier and more productive approaches.

I think docker has been pushed hard in the media because it is a gateway technology to PaSS. This is a trend that PaSS vendors like AWS and Azure want to encourage, but customers are pushing back.

Python’s venv Virtualization Module

Venv is a tool to create isolated virtual Python environments. It has been included with Python since Python v3.3, which was released 11 years ago.

PEP 405 specifies Python’s venv virtualization module.

Deprecated Virtualization Modules

Python 3.6 was released 7 years ago. It deprecated the other virtualization modules, pyenv and virtualenv. Instead, use venv, as described in this article.

Extra Installation Steps for Ubuntu

Venv was included with Python on Ubuntu until Ubuntu 23.04 It virtualizes all the common Python executables.

Debian changed pip’s behavior as a result of PEP 668 – Marking Python base environments as “externally managed”. This affects Ubuntu because it is downstream. Starting with Ubuntu 23.04, you need to install venv by typing:

Shell

$ yes | sudo apt install python3.xx-venv

... where xx is the minor version number of Python that is installed.

For Python 3.12, provided with Ubuntu 24.04 (Noble Numbat), the following incantation is required:

Shell

$ yes | sudo apt install python3.12-venv

For Python 3.11, provided with Ubuntu 23.10 (Mantic Minotaur), the following incantation is required:

Shell

$ yes | sudo apt install python3.11-venv

Some software projects use meta-packages to specify version-agnostic dependencies. Perhaps Python on Ubuntu will do so in the future.

In order to virtualize pip, venv needs to invoke ensurepip, which is not installed by default on Ubuntu. On Ubuntu systems, the ensurepip command is provided by a package called python3-pip.

Shell

$ yes | sudo apt install python3-pip

You could install both of the above packages together, of course:

Shell

$ yes | sudo apt install python3.12-venv python3-pip

Manually Creating a VEnv

The following demonstrates how to create a new virtual python environment in the ~/venv/default/ directory. Intermediate directories, such as venv in this example, will be created as required.

Shell

$ python3 -m venv ~/venv/default

At this point, the virtual environment just contained executable images for Python.

Shell

$ ls ~/venv/default/**
/home/mslinn/venv/default/lib64@  /home/mslinn/venv/default/pyvenv.cfg

/home/mslinn/venv/default/bin:
Activate.ps1  activate  activate.csh  activate.fish  pip*  pip3*  pip3.10*  python@  python3@  python3.10@

/home/mslinn/venv/default/include:

/home/mslinn/venv/default/lib:
python3.10/

Upgrading Python

Re-installing the venv at a later date will update the version of Python in the venv. You will also need to install the appropriate python3-venv package when upgrading Python.

Each time you (re)install a virtual environment, all pip packages are removed. To prevent the loss of your installed pip packages, use pip freeze before upgrading Ubuntu, so the pip packages are memorialized in requirements.txt. After upgrading Python, run pip install to re-install the pip packages.

The following incantation runs pip freeze on every directory under ~/venv, and creates a file called requirements.txt in each venv directory:

Shell

$ find ~/venv -maxdepth 1 -mindepth 1 -type d -print0 |
  while IFS= read -r -d '' DIR; do
    source "$DIR/bin/activate"
    pip freeze > "$DIR/requirements.txt"
  done

The above incantation should be performed for each of your Python venvs before you upgrade Ubuntu, because sometimes Python venvs do not work after an upgrade. The next step should fix Python venvs that broke during an Ubuntu upgrade.

Here is an example showing the commands that must be typed to upgrade Python from v3.11 to v3.12, which you will have to do when upgrading from Ubuntu 23.10 to Ubuntu 24.04. Scroll the code window to see all the commands:

Shell

$ yes | sudo apt install python3.12-venv
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  python3.12-venv
0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
Need to get 5676 B of archives.
After this operation, 28.7 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu noble/universe amd64 python3.12-venv amd64 3.12.3-1 [5676 B]
Fetched 5676 B in 0s (31.1 kB/s)
Selecting previously unselected package python3.12-venv.
(Reading database ... 444385 files and directories currently installed.)
Preparing to unpack .../python3.12-venv_3.12.3-1_amd64.deb ...
Unpacking python3.12-venv (3.12.3-1) ...
Setting up python3.12-venv (3.12.3-1) ...
Scanning processes...

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host. 

$ find ~/venv -maxdepth 1 -type d -print0 |
  while IFS= read -r -d '' DIR; do
    python3 -m venv "$DIR"
    source "$DIR/bin/activate"
    pip install -r "$DIR/requirements.txt"
  done

VEnvs are Nearly Free

The cost of a venv is virtually free for all OSes except native Windows. This is because on all OSes except native Windows, the executable images are linked by default, so they do not require much storage space.

The ls command below shows that the python program in the default venv is linked to /usr/bin/python3.10.

Shell

$ ls -go ~/venv/default/bin/python
lrwxrwxrwx 1 18 Apr 9 06:01 /home/mslinn/venv/default/bin/python -> python3*

Create a VEnv for Every Python Project

My projects are stored under the directory pointed to by $work.

My standard procedure when making a Python project called $work/blah is to also create a venv for it at within the project, at $work/blah/.venv/. I add an entry to .gitignore that merely consists of a line that says .venv/.

A bash alias could be defined called blah that makes the project directory current and activates the venv:

~/.bash_aliases

alias blah="cd $work/blah; source ./venv/bin/activate"

😁

Now you could type blah at a shell prompt, and you would be working on that project. Boom!

Deactivate a VEnv

Stop using venvs with deactivate. Notice that the prompt changes.

Shell

(aw) $ deactivate

$

Directory-Locked Python Virtualization

After setting up a Python virtual environment, a quick examination of the pip script shows that it is hard-coded to the directory that it was made for:

Shell

$ head -n 1 ~/venv/aw/bin/pip
#!/home/mslinn/venv/aw/bin/python

For virtualized environments, such as Docker, this means that a Python virtual environment created without Docker can only be used within a Docker image if the path to it is the same from within the Docker image as when it was created.

Poetry and Python Virtual Environments

Python venvs are a good way to isolate Python programs from each other, so their dependencies do not clash.

The two best-known package managers for Python are PIP and Poetry. PIP is bundled with Python, while Poetry needs to be explicitly installed.

Python venvs use links to provide virtualized Python commands. Inside each venv, you will find links to programs like python, pip and library dependencies for every project that installs dependencies while the venv is active.

Poetry can:

Create a venv in .venv
Install specified library dependencies
Run the program using the venv dependencies
Publish the Python package.

Installing Poetry

If you are not careful, installing Poetry could be a chicken-and-egg race condition. You do not want Poetry to be a dependency of the venv that it manages, because it is designed to manage the libraries within the venv.

Poetry has a custom installer that creates a dedicated virtual environment for itself, and operates independently from other Python-related mechanisms. This isolated environment prevents unintentional upgrades or removals of dependencies, allowing Poetry to manage its own dependencies properly.

While it is possible to install Poetry using PIP within a venv, it is better to install Poetry system-wide. The Setup section below discusses how to do that.

PIP dependencies are often specified in a file called requirements.txt. This file ensures that a Python project uses the correct versions of dependencies by listing all the packages and their versions. Poetry uses a file called poetry.lock for the same purpose. In addition to the names and versions of the dependencies, poetry.lock also specifies the hashes of the dependant source files.

Poetry Help Messages

The top-level help message for Poetry is:

Shell

$ poetry --help
Description:
  Lists commands.

Usage:
  list [options] [--] [<namespace>]

Arguments:
  namespace                  The namespace name

Options:
  -h, --help                 Display help for the given command. When no command is given display help for the list command.
  -q, --quiet                Do not output any message.
  -V, --version              Display this application version.
      --ansi                 Force ANSI output.
      --no-ansi              Disable ANSI output.
  -n, --no-interaction       Do not ask any interactive question.
      --no-plugins           Disables plugins.
      --no-cache             Disables Poetry source caches.
  -C, --directory=DIRECTORY  The working directory for the Poetry command (defaults to the current working directory).
  -v|vv|vvv, --verbose       Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug.

Help:
  The list command lists all commands:

    poetry list

  You can also display the commands for a specific namespace:

    poetry list test

The Poetry commands are:

Shell

$ poetry list
Poetry (version 1.7.1)

Usage:
  command [options] [arguments]

Options:
  -h, --help                 Display help for the given command. When no command is given display help for the list command.
  -q, --quiet                Do not output any message.
  -V, --version              Display this application version.
      --ansi                 Force ANSI output.
      --no-ansi              Disable ANSI output.
  -n, --no-interaction       Do not ask any interactive question.
      --no-plugins           Disables plugins.
      --no-cache             Disables Poetry source caches.
  -C, --directory=DIRECTORY  The working directory for the Poetry command (defaults to the current working directory).
  -v|vv|vvv, --verbose       Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug.

Available commands:
  about              Shows information about Poetry.
  add                Adds a new dependency to pyproject.toml.
  build              Builds a package, as a tarball and a wheel by default.
  check              Validates the content of the pyproject.toml file and its consistency with the poetry.lock file.
  config             Manages configuration settings.
  export             Exports the lock file to alternative formats.
  help               Displays help for a command.
  init               Creates a basic pyproject.toml file in the current directory.
  install            Installs the project dependencies.
  list               Lists commands.
  lock               Locks the project dependencies.
  new                Creates a new Python project at <path>.
  publish            Publishes a package to a remote repository.
  remove             Removes a package from the project dependencies.
  run                Runs a command in the appropriate environment.
  search             Searches for packages on remote repositories.
  shell              Spawns a shell within the virtual environment.
  show               Shows information about packages.
  update             Update the dependencies as according to the pyproject.toml file.
  version            Shows the version of the project or bumps it when a valid bump rule is provided.

  cache
  cache clear        Clears a Poetry cache by name.
  cache list         List Poetry's caches.

  debug
  debug info         Shows debug information.
  debug resolve      Debugs dependency resolution.

  env
  env info           Displays information about the current environment.
  env list           Lists all virtualenvs associated with the current project.
  env remove         Remove virtual environments associated with the project.
  env use            Activates or creates a new virtualenv for the current project.

  self
  self add           Add additional packages to Poetry's runtime environment.
  self install       Install locked packages (incl. addons) required by this Poetry installation.
  self lock          Lock the Poetry installation's system requirements.
  self remove        Remove additional packages from Poetry's runtime environment.
  self show          Show packages from Poetry's runtime environment.
  self show plugins  Shows information about the currently installed plugins.
  self update        Updates Poetry to the latest version.

  source
  source add         Add source configuration for project.
  source remove      Remove source configured for the project.
  source show        Show information about sources configured for the project.

Detecting a Venv

Active virtual environments suppress Poetry’s behavior.

For example:

Shell

$ poetry shell
Virtual environment already activated: /home/mslinn/venv/default

If you want to use Poetry on a project, deactivate any active virtual environment before attempting to work on the project.

Shell

$ deactivate

Here is a fragment of a script that detects if a venv is active, and disables it if so:

Bash script fragment

# Disable any activated virtual environment
VENV="$( dirname "$( which virtualenv)" )"
if [ -d "$VENV" ]; then source "$VENV/activate" && deactivate; fi

Setup

The optimal process for setting up Poetry for use in a project is:

Install Poetry, preferably system-wide. I used apt to install the poetry package.
Shell
```
$ yes | sudo apt install python3-poetry
```
Configure Poetry to manage the libraries within the venv. The poetry config help message is shown below.
Use Poetry to install the library dependencies.
Use Poetry to run the program using the dependencies in the venv, and/or to spawn a command-line subshell that provides access to the dependencies managed by Poetry.

The help information for the poetry config command is:

Shell

$ poetry config --help
Description:
  Manages configuration settings.

Usage:
  config [options] [--] [<key> [<value>...]]

Arguments:
  key                        Setting key.
  value                      Setting value.

Options:
      --list                 List configuration settings.
      --unset                Unset configuration setting.
      --local                Set/Get from the project's local configuration.
  -h, --help                 Display help for the given command. When no command is given display help for the list command.
  -q, --quiet                Do not output any message.
  -V, --version              Display this application version.
      --ansi                 Force ANSI output.
      --no-ansi              Disable ANSI output.
  -n, --no-interaction       Do not ask any interactive question.
      --no-plugins           Disables plugins.
      --no-cache             Disables Poetry source caches.
  -C, --directory=DIRECTORY  The working directory for the Poetry command (defaults to the current working directory).
  -v|vv|vvv, --verbose       Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug.

Help:
  This command allows you to edit the poetry config settings and repositories.

  To add a repository:

      poetry config repositories.foo https://bar.com/simple/

  To remove a repository (repo is a short alias for repositories):

      poetry config --unset repo.foo

Initializing the Poetry .venv

Imagine you have a Python project stored in $my_project.

Shell

$ cd $my_project

poetry install

The poetry install subcommand installs dependencies specified in poetry.lock, within the venv.

The help information for the poetry install command is:

Shell

$ poetry install --help
Description:
  Installs the project dependencies.

Usage:
  install [options]

Options:
      --without=WITHOUT      The dependency groups to ignore. (multiple values allowed)
      --with=WITH            The optional dependency groups to include. (multiple values allowed)
      --only=ONLY            The only dependency groups to include. (multiple values allowed)
      --no-dev               Do not install the development dependencies. (Deprecated)
      --sync                 Synchronize the environment with the locked packages and the specified groups.
      --no-root              Do not install the root package (the current project).
      --no-directory         Do not install any directory path dependencies; useful to install dependencies without source code, e.g. for caching of Docker layers)
      --dry-run              Output the operations but do not execute anything (implicitly enables --verbose).
      --remove-untracked     Removes packages not present in the lock file. (Deprecated)
  -E, --extras=EXTRAS        Extra sets of dependencies to install. (multiple values allowed)
      --all-extras           Install all extra dependencies.
      --only-root            Exclude all dependencies.
      --compile              Compile Python source files to bytecode. (This option has no effect if modern-installation is disabled because the old installer always compiles.)
  -h, --help                 Display help for the given command. When no command is given display help for the list command.
  -q, --quiet                Do not output any message.
  -V, --version              Display this application version.
      --ansi                 Force ANSI output.
      --no-ansi              Disable ANSI output.
  -n, --no-interaction       Do not ask any interactive question.
      --no-plugins           Disables plugins.
      --no-cache             Disables Poetry source caches.
  -C, --directory=DIRECTORY  The working directory for the Poetry command (defaults to the current working directory).
  -v|vv|vvv, --verbose       Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug.

Help:
  The install command reads the poetry.lock file from
  the current directory, processes it, and downloads and installs all the
  libraries and dependencies outlined in that file. If the file does not
  exist it will look for pyproject.toml and do the same.

  poetry install

  By default, the above command will also install the current project. To install only the
  dependencies and not including the current project, run the command with the
  --no-root option like below:

    poetry install --no-root

Here is an example of using poetry install.

Shell

$ poetry install --with local
Creating virtualenv private-gpt in /mnt/f/work/my_project/.venv

Created Files

The first time that Poetry does something, e.g. when the poetry install or poetry shell subcommands are run, Poetry creates a file in the project directory called poetry.toml. As an example, if your Python project is stored in /mnt/c/work/my_project/, poetry.toml will contain:

/mnt/c/work/my_project/poetry.toml

[virtualenvs]
create = true
in-project = true
path = "/mnt/c/work/my_project/.venv"

A file called .venv/pyvenv.cfg is also created, which looks like this:

/mnt/c/work/my_project/.venv/pyvenv.cfg

home = /usr/bin
implementation = CPython
version_info = 3.11.6.final.0
virtualenv = 20.24.1+ds
include-system-site-packages = false
base-prefix = /usr
base-exec-prefix = /usr
base-executable = /usr/bin/python3
prompt = my_project-py3.11

poetry run

The poetry run subcommand runs Python programs and scripts with the virtual environment managed by Poetry.

Shell

$ poetry run python my_program

If the first line of a script is the shebang for running Python, like this:

scripts/setup shebang

#!/usr/bin/env python3

... then we do not need to specify python or python3 on the command line.

Shell

$ poetry run my_script_containg_shebang

poetry shell

You could spawn a subshell that provides access to the dependencies managed by Poetry with the poetry shell subcommand:

Shell

$ poetry shell
Creating virtualenv my_project in /mnt/f/work/evelyn/my_project/.venv
Spawning shell within /mnt/f/work/my_project/.venv
. /mnt/f/work/my_project/.venv/bin/activate
. /mnt/f/work/my_project/.venv/bin/activate

To deactivate the virtual environment, and exit this new shell, type CTRL-D or exit.

poetry show --tree

You can view all the dependencies for a Poetry project like this:

Shell

$ poetry show --tree
black 22.12.0 The uncompromising code formatter.
├── click >=8.0.0
│   └── colorama *
├── mypy-extensions >=0.4.3
├── pathspec >=0.9.0
└── platformdirs >=2
boto3 1.34.2 The AWS SDK for Python
├── botocore >=1.34.2,<1.35.0
│   ├── jmespath >=0.7.1,<2.0.0
│   ├── python-dateutil >=2.1,<3.0.0
│   │   └── six >=1.5
│   └── urllib3 >=1.25.4,<2.1
├── jmespath >=0.7.1,<2.0.0
└── s3transfer >=0.9.0,<0.10.0
    └── botocore >=1.33.2,<2.0a.0
        ├── jmespath >=0.7.1,<2.0.0
        ├── python-dateutil >=2.1,<3.0.0
        │   └── six >=1.5
        └── urllib3 >=1.25.4,<2.1
chromadb 0.4.20 Chroma.
├── bcrypt >=4.0.1
├── chroma-hnswlib 0.7.3
│   └── numpy *
├── fastapi >=0.95.2
│   ├── anyio >=3.7.1,<4.0.0
│   │   ├── idna >=2.8
│   │   └── sniffio >=1.1
│   ├── email-validator >=2.0.0
│   │   ├── dnspython >=2.0.0
│   │   └── idna >=2.0.0 (circular dependency aborted here)
│   ├── httpx >=0.23.0
│   │   ├── anyio * (circular dependency aborted here)
│   │   ├── certifi *
│   │   ├── h2 >=3,<5
│   │   │   ├── hpack >=4.0,<5
│   │   │   └── hyperframe >=6.0,<7
│   │   ├── httpcore ==1.*
│   │   │   ├── certifi * (circular dependency aborted here)
│   │   │   └── h11 >=0.13,<0.15
│   │   ├── idna * (circular dependency aborted here)
│   │   └── sniffio * (circular dependency aborted here)
│   ├── itsdangerous >=1.1.0
│   ├── jinja2 >=2.11.2
│   │   └── markupsafe >=2.0
│   ├── orjson >=3.2.1
│   ├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<2.0.0 || >2.0.0,<2.0.1 || >2.0.1,<2.1.0 || >2.1.0,<3.0.0
│   │   ├── annotated-types >=0.4.0
│   │   ├── pydantic-core 2.14.5
│   │   │   └── typing-extensions >=4.6.0,<4.7.0 || >4.7.0
│   │   └── typing-extensions >=4.6.1 (circular dependency aborted here)
│   ├── pydantic-extra-types >=2.0.0
│   │   └── pydantic >=2.0.3 (circular dependency aborted here)
│   ├── pydantic-settings >=2.0.0
│   │   ├── pydantic >=2.3.0 (circular dependency aborted here)
│   │   └── python-dotenv >=0.21.0
│   ├── python-multipart >=0.0.5
│   ├── pyyaml >=5.3.1
│   ├── starlette >=0.27.0,<0.28.0
│   │   └── anyio >=3.4.0,<5 (circular dependency aborted here)
│   ├── typing-extensions >=4.5.0 (circular dependency aborted here)
│   ├── ujson >=4.0.1,<4.0.2 || >4.0.2,<4.1.0 || >4.1.0,<4.2.0 || >4.2.0,<4.3.0 || >4.3.0,<5.0.0 || >5.0.0,<5.1.0 || >5.1.0
│   └── uvicorn >=0.12.0
│       ├── click >=7.0
│       │   └── colorama *
│       ├── colorama >=0.4 (circular dependency aborted here)
│       ├── h11 >=0.8 (circular dependency aborted here)
│       ├── httptools >=0.5.0
│       ├── python-dotenv >=0.13 (circular dependency aborted here)
│       ├── pyyaml >=5.1 (circular dependency aborted here)
│       ├── uvloop >=0.14.0,<0.15.0 || >0.15.0,<0.15.1 || >0.15.1
│       ├── watchfiles >=0.13
│       │   └── anyio >=3.0.0 (circular dependency aborted here)
│       └── websockets >=10.4
├── grpcio >=1.58.0
├── importlib-resources *
├── kubernetes >=28.1.0
│   ├── certifi >=14.05.14
│   ├── google-auth >=1.0.1
│   │   ├── cachetools >=2.0.0,<6.0
│   │   ├── pyasn1-modules >=0.2.1
│   │   │   └── pyasn1 >=0.4.6,<0.6.0
│   │   └── rsa >=3.1.4,<5
│   │       └── pyasn1 >=0.1.3 (circular dependency aborted here)
│   ├── oauthlib >=3.2.2
│   ├── python-dateutil >=2.5.3
│   │   └── six >=1.5
│   ├── pyyaml >=5.4.1
│   ├── requests *
│   │   ├── certifi >=2017.4.17 (circular dependency aborted here)
│   │   ├── charset-normalizer >=2,<4
│   │   ├── idna >=2.5,<4
│   │   └── urllib3 >=1.21.1,<3
│   ├── requests-oauthlib *
│   │   ├── oauthlib >=3.0.0 (circular dependency aborted here)
│   │   └── requests >=2.0.0 (circular dependency aborted here)
│   ├── six >=1.9.0 (circular dependency aborted here)
│   ├── urllib3 >=1.24.2,<2.0 (circular dependency aborted here)
│   └── websocket-client >=0.32.0,<0.40.0 || >0.40.0,<0.41.dev0 || >=0.43.dev0
├── mmh3 >=4.0.1
├── numpy >=1.22.5
├── onnxruntime >=1.14.1
│   ├── coloredlogs *
│   │   └── humanfriendly >=9.1
│   │       └── pyreadline3 *
│   ├── flatbuffers *
│   ├── numpy >=1.21.6
│   ├── packaging *
│   ├── protobuf *
│   └── sympy *
│       └── mpmath >=0.19
├── opentelemetry-api >=1.2.0
│   ├── deprecated >=1.2.6
│   │   └── wrapt >=1.10,<2
│   └── importlib-metadata >=6.0,<7.0
│       └── zipp >=0.5
├── opentelemetry-exporter-otlp-proto-grpc >=1.2.0
│   ├── backoff >=1.10.0,<3.0.0
│   ├── deprecated >=1.2.6
│   │   └── wrapt >=1.10,<2
│   ├── googleapis-common-protos >=1.52,<2.0
│   │   └── protobuf >=3.19.5,<3.20.0 || >3.20.0,<3.20.1 || >3.20.1,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<5.0.0.dev0
│   ├── grpcio >=1.0.0,<2.0.0
│   ├── opentelemetry-api >=1.15,<2.0
│   │   ├── deprecated >=1.2.6 (circular dependency aborted here)
│   │   └── importlib-metadata >=6.0,<7.0
│   │       └── zipp >=0.5
│   ├── opentelemetry-exporter-otlp-proto-common 1.21.0
│   │   ├── backoff >=1.10.0,<3.0.0 (circular dependency aborted here)
│   │   └── opentelemetry-proto 1.21.0
│   │       └── protobuf >=3.19,<5.0 (circular dependency aborted here)
│   ├── opentelemetry-proto 1.21.0 (circular dependency aborted here)
│   └── opentelemetry-sdk >=1.21.0,<1.22.0
│       ├── opentelemetry-api 1.21.0 (circular dependency aborted here)
│       ├── opentelemetry-semantic-conventions 0.42b0
│       └── typing-extensions >=3.7.4
├── opentelemetry-instrumentation-fastapi >=0.41b0
│   ├── opentelemetry-api >=1.12,<2.0
│   │   ├── deprecated >=1.2.6
│   │   │   └── wrapt >=1.10,<2
│   │   └── importlib-metadata >=6.0,<7.0
│   │       └── zipp >=0.5
│   ├── opentelemetry-instrumentation 0.42b0
│   │   ├── opentelemetry-api >=1.4,<2.0 (circular dependency aborted here)
│   │   ├── setuptools >=16.0
│   │   └── wrapt >=1.0.0,<2.0.0 (circular dependency aborted here)
│   ├── opentelemetry-instrumentation-asgi 0.42b0
│   │   ├── asgiref >=3.0,<4.0
│   │   ├── opentelemetry-api >=1.12,<2.0 (circular dependency aborted here)
│   │   ├── opentelemetry-instrumentation 0.42b0 (circular dependency aborted here)
│   │   ├── opentelemetry-semantic-conventions 0.42b0
│   │   └── opentelemetry-util-http 0.42b0
│   ├── opentelemetry-semantic-conventions 0.42b0 (circular dependency aborted here)
│   └── opentelemetry-util-http 0.42b0 (circular dependency aborted here)
├── opentelemetry-sdk >=1.2.0
│   ├── opentelemetry-api 1.21.0
│   │   ├── deprecated >=1.2.6
│   │   │   └── wrapt >=1.10,<2
│   │   └── importlib-metadata >=6.0,<7.0
│   │       └── zipp >=0.5
│   ├── opentelemetry-semantic-conventions 0.42b0
│   └── typing-extensions >=3.7.4
├── overrides >=7.3.1
├── posthog >=2.4.0
│   ├── backoff >=1.10.0
│   ├── monotonic >=1.5
│   ├── python-dateutil >2.1
│   │   └── six >=1.5
│   ├── requests >=2.7,<3.0
│   │   ├── certifi >=2017.4.17
│   │   ├── charset-normalizer >=2,<4
│   │   ├── idna >=2.5,<4
│   │   └── urllib3 >=1.21.1,<3
│   └── six >=1.5 (circular dependency aborted here)
├── pulsar-client >=3.1.0
│   └── certifi *
├── pydantic >=1.9
│   ├── annotated-types >=0.4.0
│   ├── pydantic-core 2.14.5
│   │   └── typing-extensions >=4.6.0,<4.7.0 || >4.7.0
│   └── typing-extensions >=4.6.1 (circular dependency aborted here)
├── pypika >=0.48.9
├── pyyaml >=6.0.0
├── requests >=2.28
│   ├── certifi >=2017.4.17
│   ├── charset-normalizer >=2,<4
│   ├── idna >=2.5,<4
│   └── urllib3 >=1.21.1,<3
├── tenacity >=8.2.3
├── tokenizers >=0.13.2
│   └── huggingface-hub >=0.16.4,<1.0
│       ├── filelock *
│       ├── fsspec >=2023.5.0
│       │   ├── aiohttp <4.0.0a0 || >4.0.0a0,<4.0.0a1 || >4.0.0a1
│       │   │   ├── aiosignal >=1.1.2
│       │   │   │   └── frozenlist >=1.1.0
│       │   │   ├── attrs >=17.3.0
│       │   │   ├── frozenlist >=1.1.1 (circular dependency aborted here)
│       │   │   ├── multidict >=4.5,<7.0
│       │   │   └── yarl >=1.0,<2.0
│       │   │       ├── idna >=2.0
│       │   │       └── multidict >=4.0 (circular dependency aborted here)
│       │   └── requests *
│       │       ├── certifi >=2017.4.17
│       │       ├── charset-normalizer >=2,<4
│       │       ├── idna >=2.5,<4 (circular dependency aborted here)
│       │       └── urllib3 >=1.21.1,<3
│       ├── packaging >=20.9
│       ├── pyyaml >=5.1
│       ├── requests * (circular dependency aborted here)
│       ├── tqdm >=4.42.1
│       │   └── colorama *
│       └── typing-extensions >=3.7.4.3
├── tqdm >=4.65.0
│   └── colorama *
├── typer >=0.9.0
│   ├── click >=7.1.1,<9.0.0
│   │   └── colorama *
│   ├── colorama >=0.4.3,<0.5.0 (circular dependency aborted here)
│   ├── rich >=10.11.0,<14.0.0
│   │   ├── markdown-it-py >=2.2.0
│   │   │   └── mdurl >=0.1,<1.0
│   │   └── pygments >=2.13.0,<3.0.0
│   ├── shellingham >=1.3.0,<2.0.0
│   └── typing-extensions >=3.7.4.3
├── typing-extensions >=4.5.0
└── uvicorn >=0.18.3
    ├── click >=7.0
    │   └── colorama *
    ├── colorama >=0.4 (circular dependency aborted here)
    ├── h11 >=0.8
    ├── httptools >=0.5.0
    ├── python-dotenv >=0.13
    ├── pyyaml >=5.1
    ├── uvloop >=0.14.0,<0.15.0 || >0.15.0,<0.15.1 || >0.15.1
    ├── watchfiles >=0.13
    │   └── anyio >=3.0.0
    │       ├── idna >=2.8
    │       └── sniffio >=1.1
    └── websockets >=10.4
fastapi 0.103.2 FastAPI framework, high performance, easy to learn, fast to code, ready for production
├── anyio >=3.7.1,<4.0.0
│   ├── idna >=2.8
│   └── sniffio >=1.1
├── email-validator >=2.0.0
│   ├── dnspython >=2.0.0
│   └── idna >=2.0.0
├── httpx >=0.23.0
│   ├── anyio *
│   │   ├── idna >=2.8
│   │   └── sniffio >=1.1
│   ├── certifi *
│   ├── h2 >=3,<5
│   │   ├── hpack >=4.0,<5
│   │   └── hyperframe >=6.0,<7
│   ├── httpcore ==1.*
│   │   ├── certifi * (circular dependency aborted here)
│   │   └── h11 >=0.13,<0.15
│   ├── idna * (circular dependency aborted here)
│   └── sniffio * (circular dependency aborted here)
├── itsdangerous >=1.1.0
├── jinja2 >=2.11.2
│   └── markupsafe >=2.0
├── orjson >=3.2.1
├── pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<2.0.0 || >2.0.0,<2.0.1 || >2.0.1,<2.1.0 || >2.1.0,<3.0.0
│   ├── annotated-types >=0.4.0
│   ├── pydantic-core 2.14.5
│   │   └── typing-extensions >=4.6.0,<4.7.0 || >4.7.0
│   └── typing-extensions >=4.6.1 (circular dependency aborted here)
├── pydantic-extra-types >=2.0.0
│   └── pydantic >=2.0.3
│       ├── annotated-types >=0.4.0
│       ├── pydantic-core 2.14.5
│       │   └── typing-extensions >=4.6.0,<4.7.0 || >4.7.0
│       └── typing-extensions >=4.6.1 (circular dependency aborted here)
├── pydantic-settings >=2.0.0
│   ├── pydantic >=2.3.0
│   │   ├── annotated-types >=0.4.0
│   │   ├── pydantic-core 2.14.5
│   │   │   └── typing-extensions >=4.6.0,<4.7.0 || >4.7.0
│   │   └── typing-extensions >=4.6.1 (circular dependency aborted here)
│   └── python-dotenv >=0.21.0
├── python-multipart >=0.0.5
├── pyyaml >=5.3.1
├── starlette >=0.27.0,<0.28.0
│   └── anyio >=3.4.0,<5
│       ├── idna >=2.8
│       └── sniffio >=1.1
├── typing-extensions >=4.5.0
├── ujson >=4.0.1,<4.0.2 || >4.0.2,<4.1.0 || >4.1.0,<4.2.0 || >4.2.0,<4.3.0 || >4.3.0,<5.0.0 || >5.0.0,<5.1.0 || >5.1.0
└── uvicorn >=0.12.0
    ├── click >=7.0
    │   └── colorama *
    ├── colorama >=0.4 (circular dependency aborted here)
    ├── h11 >=0.8
    ├── httptools >=0.5.0
    ├── python-dotenv >=0.13
    ├── pyyaml >=5.1
    ├── uvloop >=0.14.0,<0.15.0 || >0.15.0,<0.15.1 || >0.15.1
    ├── watchfiles >=0.13
    │   └── anyio >=3.0.0
    │       ├── idna >=2.8
    │       └── sniffio >=1.1
    └── websockets >=10.4
injector 0.21.0 Injector - Python dependency injection framework, inspired by Guice
llama-index 0.9.3 Interface between LLMs and your data
├── aiohttp >=3.8.6,<4.0.0
│   ├── aiosignal >=1.1.2
│   │   └── frozenlist >=1.1.0
│   ├── attrs >=17.3.0
│   ├── frozenlist >=1.1.1 (circular dependency aborted here)
│   ├── multidict >=4.5,<7.0
│   └── yarl >=1.0,<2.0
│       ├── idna >=2.0
│       └── multidict >=4.0 (circular dependency aborted here)
├── aiostream >=0.5.2,<0.6.0
│   └── typing-extensions *
├── beautifulsoup4 >=4.12.2,<5.0.0
│   └── soupsieve >1.2
├── dataclasses-json >=0.5.7,<0.6.0
│   ├── marshmallow >=3.18.0,<4.0.0
│   │   └── packaging >=17.0
│   └── typing-inspect >=0.4.0,<1
│       ├── mypy-extensions >=0.3.0
│       └── typing-extensions >=3.7.4
├── deprecated >=1.2.9.3
│   └── wrapt >=1.10,<2
├── fsspec >=2023.5.0
│   ├── aiohttp <4.0.0a0 || >4.0.0a0,<4.0.0a1 || >4.0.0a1
│   │   ├── aiosignal >=1.1.2
│   │   │   └── frozenlist >=1.1.0
│   │   ├── attrs >=17.3.0
│   │   ├── frozenlist >=1.1.1 (circular dependency aborted here)
│   │   ├── multidict >=4.5,<7.0
│   │   └── yarl >=1.0,<2.0
│   │       ├── idna >=2.0
│   │       └── multidict >=4.0 (circular dependency aborted here)
│   └── requests *
│       ├── certifi >=2017.4.17
│       ├── charset-normalizer >=2,<4
│       ├── idna >=2.5,<4 (circular dependency aborted here)
│       └── urllib3 >=1.21.1,<3
├── httpx *
│   ├── anyio *
│   │   ├── idna >=2.8
│   │   └── sniffio >=1.1
│   ├── certifi *
│   ├── h2 >=3,<5
│   │   ├── hpack >=4.0,<5
│   │   └── hyperframe >=6.0,<7
│   ├── httpcore ==1.*
│   │   ├── certifi * (circular dependency aborted here)
│   │   └── h11 >=0.13,<0.15
│   ├── idna * (circular dependency aborted here)
│   └── sniffio * (circular dependency aborted here)
├── nest-asyncio >=1.5.8,<2.0.0
├── nltk >=3.8.1,<4.0.0
│   ├── click *
│   │   └── colorama *
│   ├── joblib *
│   ├── regex >=2021.8.3
│   └── tqdm *
│       └── colorama * (circular dependency aborted here)
├── numpy *
├── openai >=1.1.0
│   ├── anyio >=3.5.0,<5
│   │   ├── idna >=2.8
│   │   └── sniffio >=1.1
│   ├── distro >=1.7.0,<2
│   ├── httpx >=0.23.0,<1
│   │   ├── anyio * (circular dependency aborted here)
│   │   ├── certifi *
│   │   ├── h2 >=3,<5
│   │   │   ├── hpack >=4.0,<5
│   │   │   └── hyperframe >=6.0,<7
│   │   ├── httpcore ==1.*
│   │   │   ├── certifi * (circular dependency aborted here)
│   │   │   └── h11 >=0.13,<0.15
│   │   ├── idna * (circular dependency aborted here)
│   │   └── sniffio * (circular dependency aborted here)
│   ├── pydantic >=1.9.0,<3
│   │   ├── annotated-types >=0.4.0
│   │   ├── pydantic-core 2.14.5
│   │   │   └── typing-extensions >=4.6.0,<4.7.0 || >4.7.0
│   │   └── typing-extensions >=4.6.1 (circular dependency aborted here)
│   ├── sniffio * (circular dependency aborted here)
│   ├── tqdm >4
│   │   └── colorama *
│   └── typing-extensions >=4.5,<5 (circular dependency aborted here)
├── optimum >=1.13.2,<2.0.0
│   ├── coloredlogs *
│   │   └── humanfriendly >=9.1
│   │       └── pyreadline3 *
│   ├── datasets *
│   │   ├── aiohttp *
│   │   │   ├── aiosignal >=1.1.2
│   │   │   │   └── frozenlist >=1.1.0
│   │   │   ├── attrs >=17.3.0
│   │   │   ├── frozenlist >=1.1.1 (circular dependency aborted here)
│   │   │   ├── multidict >=4.5,<7.0
│   │   │   └── yarl >=1.0,<2.0
│   │   │       ├── idna >=2.0
│   │   │       └── multidict >=4.0 (circular dependency aborted here)
│   │   ├── dill >=0.3.0,<0.3.8
│   │   ├── fsspec >=2021.11.1
│   │   │   ├── aiohttp <4.0.0a0 || >4.0.0a0,<4.0.0a1 || >4.0.0a1 (circular dependency aborted here)
│   │   │   └── requests *
│   │   │       ├── certifi >=2017.4.17
│   │   │       ├── charset-normalizer >=2,<4
│   │   │       ├── idna >=2.5,<4 (circular dependency aborted here)
│   │   │       └── urllib3 >=1.21.1,<3
│   │   ├── huggingface-hub >=0.14.0,<1.0.0
│   │   │   ├── filelock *
│   │   │   ├── fsspec >=2023.5.0 (circular dependency aborted here)
│   │   │   ├── packaging >=20.9
│   │   │   ├── pyyaml >=5.1
│   │   │   ├── requests * (circular dependency aborted here)
│   │   │   ├── tqdm >=4.42.1
│   │   │   │   └── colorama *
│   │   │   └── typing-extensions >=3.7.4.3
│   │   ├── multiprocess *
│   │   │   └── dill >=0.3.7 (circular dependency aborted here)
│   │   ├── numpy >=1.17
│   │   ├── packaging * (circular dependency aborted here)
│   │   ├── pandas *
│   │   │   ├── numpy >=1.23.2,<2 (circular dependency aborted here)
│   │   │   ├── python-dateutil >=2.8.2
│   │   │   │   └── six >=1.5
│   │   │   ├── pytz >=2020.1
│   │   │   └── tzdata >=2022.1
│   │   ├── pyarrow >=8.0.0
│   │   │   └── numpy >=1.16.6 (circular dependency aborted here)
│   │   ├── pyyaml >=5.1 (circular dependency aborted here)
│   │   ├── requests >=2.19.0 (circular dependency aborted here)
│   │   ├── tqdm >=4.62.1 (circular dependency aborted here)
│   │   └── xxhash *
│   ├── datasets >=1.2.1 (circular dependency aborted here)
│   ├── evaluate *
│   │   ├── datasets >=2.0.0 (circular dependency aborted here)
│   │   ├── dill * (circular dependency aborted here)
│   │   ├── fsspec >=2021.05.0 (circular dependency aborted here)
│   │   ├── huggingface-hub >=0.7.0 (circular dependency aborted here)
│   │   ├── multiprocess * (circular dependency aborted here)
│   │   ├── numpy >=1.17 (circular dependency aborted here)
│   │   ├── packaging * (circular dependency aborted here)
│   │   ├── pandas * (circular dependency aborted here)
│   │   ├── requests >=2.19.0 (circular dependency aborted here)
│   │   ├── responses <0.19
│   │   │   ├── requests >=2.0,<3.0 (circular dependency aborted here)
│   │   │   └── urllib3 >=1.25.10 (circular dependency aborted here)
│   │   ├── tqdm >=4.62.1 (circular dependency aborted here)
│   │   └── xxhash * (circular dependency aborted here)
│   ├── huggingface-hub >=0.8.0 (circular dependency aborted here)
│   ├── numpy * (circular dependency aborted here)
│   ├── onnx *
│   │   ├── numpy * (circular dependency aborted here)
│   │   └── protobuf >=3.20.2
│   ├── onnxruntime >=1.11.0
│   │   ├── coloredlogs * (circular dependency aborted here)
│   │   ├── flatbuffers *
│   │   ├── numpy >=1.21.6 (circular dependency aborted here)
│   │   ├── packaging * (circular dependency aborted here)
│   │   ├── protobuf * (circular dependency aborted here)
│   │   └── sympy *
│   │       └── mpmath >=0.19
│   ├── packaging * (circular dependency aborted here)
│   ├── protobuf >=3.20.1 (circular dependency aborted here)
│   ├── sympy * (circular dependency aborted here)
│   ├── torch >=1.9
│   │   ├── filelock * (circular dependency aborted here)
│   │   ├── fsspec * (circular dependency aborted here)
│   │   ├── jinja2 *
│   │   │   └── markupsafe >=2.0
│   │   ├── networkx *
│   │   ├── nvidia-cublas-cu12 12.1.3.1
│   │   ├── nvidia-cuda-cupti-cu12 12.1.105
│   │   ├── nvidia-cuda-nvrtc-cu12 12.1.105
│   │   ├── nvidia-cuda-runtime-cu12 12.1.105
│   │   ├── nvidia-cudnn-cu12 8.9.2.26
│   │   │   └── nvidia-cublas-cu12 * (circular dependency aborted here)
│   │   ├── nvidia-cufft-cu12 11.0.2.54
│   │   ├── nvidia-curand-cu12 10.3.2.106
│   │   ├── nvidia-cusolver-cu12 11.4.5.107
│   │   │   ├── nvidia-cublas-cu12 * (circular dependency aborted here)
│   │   │   ├── nvidia-cusparse-cu12 *
│   │   │   │   └── nvidia-nvjitlink-cu12 *
│   │   │   └── nvidia-nvjitlink-cu12 * (circular dependency aborted here)
│   │   ├── nvidia-cusparse-cu12 12.1.0.106 (circular dependency aborted here)
│   │   ├── nvidia-nccl-cu12 2.18.1
│   │   ├── nvidia-nvtx-cu12 12.1.105
│   │   ├── sympy * (circular dependency aborted here)
│   │   ├── triton 2.1.0
│   │   │   └── filelock * (circular dependency aborted here)
│   │   └── typing-extensions * (circular dependency aborted here)
│   └── transformers >=4.26.0
│       ├── accelerate >=0.21.0
│       │   ├── huggingface-hub * (circular dependency aborted here)
│       │   ├── numpy >=1.17 (circular dependency aborted here)
│       │   ├── packaging >=20.0 (circular dependency aborted here)
│       │   ├── psutil *
│       │   ├── pyyaml * (circular dependency aborted here)
│       │   ├── safetensors >=0.3.1
│       │   └── torch >=1.10.0 (circular dependency aborted here)
│       ├── filelock * (circular dependency aborted here)
│       ├── huggingface-hub >=0.19.3,<1.0 (circular dependency aborted here)
│       ├── numpy >=1.17 (circular dependency aborted here)
│       ├── packaging >=20.0 (circular dependency aborted here)
│       ├── protobuf * (circular dependency aborted here)
│       ├── pyyaml >=5.1 (circular dependency aborted here)
│       ├── regex !=2019.12.17
│       ├── requests * (circular dependency aborted here)
│       ├── safetensors >=0.3.1 (circular dependency aborted here)
│       ├── sentencepiece >=0.1.91,<0.1.92 || >0.1.92
│       ├── tokenizers >=0.14,<0.19
│       │   └── huggingface-hub >=0.16.4,<1.0 (circular dependency aborted here)
│       ├── torch >=1.10,<1.12.0 || >1.12.0 (circular dependency aborted here)
│       └── tqdm >=4.27 (circular dependency aborted here)
├── pandas *
│   ├── numpy >=1.23.2,<2
│   ├── python-dateutil >=2.8.2
│   │   └── six >=1.5
│   ├── pytz >=2020.1
│   └── tzdata >=2022.1
├── sentencepiece >=0.1.99,<0.2.0
├── sqlalchemy >=1.4.49
│   ├── greenlet !=0.4.17
│   └── typing-extensions >=4.2.0
├── tenacity >=8.2.0,<9.0.0
├── tiktoken >=0.3.3
│   ├── regex >=2022.1.18
│   └── requests >=2.26.0
│       ├── certifi >=2017.4.17
│       ├── charset-normalizer >=2,<4
│       ├── idna >=2.5,<4
│       └── urllib3 >=1.21.1,<3
├── transformers >=4.34.0,<5.0.0
│   ├── accelerate >=0.21.0
│   │   ├── huggingface-hub *
│   │   │   ├── filelock *
│   │   │   ├── fsspec >=2023.5.0
│   │   │   │   ├── aiohttp <4.0.0a0 || >4.0.0a0,<4.0.0a1 || >4.0.0a1
│   │   │   │   │   ├── aiosignal >=1.1.2
│   │   │   │   │   │   └── frozenlist >=1.1.0
│   │   │   │   │   ├── attrs >=17.3.0
│   │   │   │   │   ├── frozenlist >=1.1.1 (circular dependency aborted here)
│   │   │   │   │   ├── multidict >=4.5,<7.0
│   │   │   │   │   └── yarl >=1.0,<2.0
│   │   │   │   │       ├── idna >=2.0
│   │   │   │   │       └── multidict >=4.0 (circular dependency aborted here)
│   │   │   │   └── requests *
│   │   │   │       ├── certifi >=2017.4.17
│   │   │   │       ├── charset-normalizer >=2,<4
│   │   │   │       ├── idna >=2.5,<4 (circular dependency aborted here)
│   │   │   │       └── urllib3 >=1.21.1,<3
│   │   │   ├── packaging >=20.9
│   │   │   ├── pyyaml >=5.1
│   │   │   ├── requests * (circular dependency aborted here)
│   │   │   ├── tqdm >=4.42.1
│   │   │   │   └── colorama *
│   │   │   └── typing-extensions >=3.7.4.3
│   │   ├── numpy >=1.17
│   │   ├── packaging >=20.0 (circular dependency aborted here)
│   │   ├── psutil *
│   │   ├── pyyaml * (circular dependency aborted here)
│   │   ├── safetensors >=0.3.1
│   │   └── torch >=1.10.0
│   │       ├── filelock * (circular dependency aborted here)
│   │       ├── fsspec * (circular dependency aborted here)
│   │       ├── jinja2 *
│   │       │   └── markupsafe >=2.0
│   │       ├── networkx *
│   │       ├── nvidia-cublas-cu12 12.1.3.1
│   │       ├── nvidia-cuda-cupti-cu12 12.1.105
│   │       ├── nvidia-cuda-nvrtc-cu12 12.1.105
│   │       ├── nvidia-cuda-runtime-cu12 12.1.105
│   │       ├── nvidia-cudnn-cu12 8.9.2.26
│   │       │   └── nvidia-cublas-cu12 * (circular dependency aborted here)
│   │       ├── nvidia-cufft-cu12 11.0.2.54
│   │       ├── nvidia-curand-cu12 10.3.2.106
│   │       ├── nvidia-cusolver-cu12 11.4.5.107
│   │       │   ├── nvidia-cublas-cu12 * (circular dependency aborted here)
│   │       │   ├── nvidia-cusparse-cu12 *
│   │       │   │   └── nvidia-nvjitlink-cu12 *
│   │       │   └── nvidia-nvjitlink-cu12 * (circular dependency aborted here)
│   │       ├── nvidia-cusparse-cu12 12.1.0.106 (circular dependency aborted here)
│   │       ├── nvidia-nccl-cu12 2.18.1
│   │       ├── nvidia-nvtx-cu12 12.1.105
│   │       ├── sympy *
│   │       │   └── mpmath >=0.19
│   │       ├── triton 2.1.0
│   │       │   └── filelock * (circular dependency aborted here)
│   │       └── typing-extensions * (circular dependency aborted here)
│   ├── filelock * (circular dependency aborted here)
│   ├── huggingface-hub >=0.19.3,<1.0 (circular dependency aborted here)
│   ├── numpy >=1.17 (circular dependency aborted here)
│   ├── packaging >=20.0 (circular dependency aborted here)
│   ├── protobuf *
│   ├── pyyaml >=5.1 (circular dependency aborted here)
│   ├── regex !=2019.12.17
│   ├── requests * (circular dependency aborted here)
│   ├── safetensors >=0.3.1 (circular dependency aborted here)
│   ├── sentencepiece >=0.1.91,<0.1.92 || >0.1.92
│   ├── tokenizers >=0.14,<0.19
│   │   └── huggingface-hub >=0.16.4,<1.0 (circular dependency aborted here)
│   ├── torch >=1.10,<1.12.0 || >1.12.0 (circular dependency aborted here)
│   └── tqdm >=4.27 (circular dependency aborted here)
├── typing-extensions >=4.5.0
├── typing-inspect >=0.8.0
│   ├── mypy-extensions >=0.3.0
│   └── typing-extensions >=3.7.4
└── urllib3 <2
mypy 1.7.1 Optional static typing for Python
├── mypy-extensions >=1.0.0
└── typing-extensions >=4.1.0
pre-commit 2.21.0 A framework for managing and maintaining multi-language pre-commit hooks.
├── cfgv >=2.0.0
├── identify >=1.0.0
├── nodeenv >=0.11.1
│   └── setuptools *
├── pyyaml >=5.1
└── virtualenv >=20.10.0
    ├── distlib >=0.3.7,<1
    ├── filelock >=3.12.2,<4
    └── platformdirs >=3.9.1,<5
pypdf 3.17.2 A pure-python PDF library capable of splitting, merging, cropping, and transforming PDF files
pytest 7.4.3 pytest: simple powerful testing with Python
├── colorama *
├── iniconfig *
├── packaging *
└── pluggy >=0.12,<2.0
pytest-asyncio 0.21.1 Pytest support for asyncio
└── pytest >=7.0.0
    ├── colorama *
    ├── iniconfig *
    ├── packaging *
    └── pluggy >=0.12,<2.0
pytest-cov 3.0.0 Pytest plugin for measuring coverage.
├── coverage >=5.2.1
└── pytest >=4.6
    ├── colorama *
    ├── iniconfig *
    ├── packaging *
    └── pluggy >=0.12,<2.0
python-multipart 0.0.6 A streaming multipart parser for Python
pyyaml 6.0.1 YAML parser and emitter for Python
qdrant-client 1.7.0 Client library for the Qdrant vector search engine
├── grpcio >=1.41.0
├── grpcio-tools >=1.41.0
│   ├── grpcio >=1.60.0
│   ├── protobuf >=4.21.6,<5.0dev
│   └── setuptools *
├── httpx >=0.14.0
│   ├── anyio *
│   │   ├── idna >=2.8
│   │   └── sniffio >=1.1
│   ├── certifi *
│   ├── h2 >=3,<5
│   │   ├── hpack >=4.0,<5
│   │   └── hyperframe >=6.0,<7
│   ├── httpcore ==1.*
│   │   ├── certifi * (circular dependency aborted here)
│   │   └── h11 >=0.13,<0.15
│   ├── idna * (circular dependency aborted here)
│   └── sniffio * (circular dependency aborted here)
├── numpy >=1.21
├── portalocker >=2.7.0,<3.0.0
│   └── pywin32 >=226
├── pydantic >=1.10.8
│   ├── annotated-types >=0.4.0
│   ├── pydantic-core 2.14.5
│   │   └── typing-extensions >=4.6.0,<4.7.0 || >4.7.0
│   └── typing-extensions >=4.6.1 (circular dependency aborted here)
└── urllib3 >=1.26.14,<2.0.0
ruff 0.1.8 An extremely fast Python linter and code formatter, written in Rust.
types-pyyaml 6.0.12.12 Typing stubs for PyYAML
watchdog 3.0.0 Filesystem events monitoring

poetry update

You can update all the dependencies for a Poetry project like this:

Shell

$ poetry update

For Further Reading

Python Best Practices for a New Project in 2021. This article is already becoming dated. YMMV.

Summary

Demonstrated how to make an alias for working with Python virtual environments (venvs) that are coupled with Python projects.
Deactivating the current venv was demonstrated using the deactivate command, provided with every venv.
Locked directories mean that Python virtual environments should normally only be created in the same environment they are intended to be used.

Escaping HTML on Clipboard From a Windows Hot Key via WSL

2021-04-03T00:00:00-04:00

I frequently show HTML source code when I write. That HTML must be escaped prior to displaying it on a web page.

Script to Apply HTML Escape to Clipboard

This bash script applies an HTML escape conversion to the contents of the system clipboard. If you have WSL on your machine, you could store it on the WSL file system, for example in ~/.local/bin/escapeHtml.

escapeHtml

#!/bin/bash

# SPDX-License-Identifier: Apache-2.0

function help {
  echo "$(basename $0) - Escapes HTML with entities.
Reads from STDIN or pipe, or converts the clipboard.
Result is copied to the clipboard.
"
  exit 1
}

function filesAreLinked {
  "$1" -ef "$2"
}

function checkDependencies {
  if [ -z `which recode` ]; then yes | sudo apt install recode; fi

  # See https://github.com/sindresorhus/clipboard-cli
  if [ -z `which clipboard` ]; then
    if [ "$( filesAreLinked /bin/npm /usr/local/lib/node_modules/npm/bin/npm-cli.js )" ]; then
      # No nodejs venv
      sudo -H npm install --global clipboard-cli
    else  # nodejs venv
      npm install --global clipboard-cli
    fi
  fi
}


if [ "$1" == -h ]; then help; fi
checkDependencies

if [ -t 0 ]; then  # Not reading from a terminal
  HTML="$( clipboard )"
else  # Reading from stdin or pipe
  # See https://stackoverflow.com/a/32365596/553865
  HTML=$(cat; echo x)
  HTML=${HTML%x}   # Strip the trailing x
fi
RESULT="$( recode utf8..html <<< "$HTML" )"
echo "$RESULT" | sed "s/&#13;//g" | sed "s/'/\&#39;/g" | clipboard
echo "$( echo "$RESULT" | wc -l ) lines have been placed on the clipboard."

Using the Script

Select some text in any document or anywhere that text can be selected.
Run escapeHtml on the same machine. If you have Windows with WSL you can run the script there, or run it in native Windows, does not matter.
Paste escaped HTML into your target document.

Hot Key Trigger

Trigger the script with a hot key via your OS's facilities. This section just discusses how native Windows hot keys can be used to trigger this script running in WSL.

Right-click in a folder
Select New / Shortcut
For Type the location of the item, type:
Type the location of the item
```
%windir%\System32\wsl.exe ~/.local/bin/escapeHtml/escapeHtml
```
Click Change icon and select a retro icon for this shortcut.
Click on Apply.
Click on Next.
When prompted for Type a name for this shortcut, save as HTML Escape Clip to Clip.
Click on Finish.
Right-click on the new shortcut and click in Shortcut key
I used CTRL+ALT+A.
Click on OK.

😁

You can now quickly copy HTML from any source to the clipboard, apply an HTML escape conversion to the clipboard contents, and then paste the escaped HTML into an editor.

Microsoft Visual Studio Code Notes

2021-03-22T00:00:00-04:00

Launching VSCode

Opening A File

To open a file, simply provide the file name on the command line. For example, to open abd/xyz/my_func.c, type:

Shell

$ code abd/xyz/my_func.c

If VSCode has at least one window open, the top-most window will open a new tab containing the desired file. Otherwise, VSCode will launch, with the desired file displayed in an editor pane.

This works with relative and absolute file names.

You can type the incantation into any command console, such as CMD, PowerShell or bash. You can also use the console built into VSCode, which can be toggled with CTRL-~, or you can use an external console such as Microsoft Terminal.

Opening A File At a Line

To open a file at a given line number, use the -g option on the command line, and follow the file name with a colon and the line number. For example, to open abd/xyz/my_func.c at line 123, type:

Shell

$ code -g abd/xyz/my_func.c:123

Useful Default Key Bindings

CTRL+~

Toggle visibility of the current Terminal pane in the lower panel.

CTRL+B

Toggle visibility of the Explorer view, which contains the files and directories for the VSCode project.

CTRL+J

Toggle visibility of the lower panel, containing Problems, Output, Debug Console, Terminal, GitLens and Watch.

CTRL+K M

Set the language mode for the file displayed in the editor.

CTRL+K CTRL+S

Display / edit the Keyboard Shortcuts definitions.
You can filter the keybindings by pressing ALT+K or clicking the icon of the little keyboard at the top right of the Keyboard Shortcut page, then press the keys that you want to see the key binding for. \ The keyboard icon starts keystroke recording mode. Recording mode is sticky; each time you revisit the Keyboard Shortcuts tab you can just press the keys you are interested in to see their bindings. Step by step:

When you press the CTRL key you will see "ctrl" displayed, and recording mode continues to listen to what you type. Don't do this right now, but FYI, if you toggle keystroke recording mode now, and then remove the quotes around "ctrl", you will see a sorted list of all the key chords bound to CTRL.
Next, when you add the Shift key to the key chord, you then see "ctrl+shift" displayed. Don't do this right now, but FYI, if you toggle keystroke recording mode now, and then remove the quotes, you will see a sorted list of all the key chords bound to CTRL+Shift.
Finally, adding the = key to the key chord shows all the commands bound to CTRL+Shift+=.

CTRL+K+0 (zero)

Completely fold the active editor contents.

CTRL+K+1 (one)

Fold level 1 the active editor contents.

CTRL+K+2

Fold level 2 the active editor contents.

CTRL+B

Toggle side bar visibility.

CTRL+P

Show names of recently opened tabs, which might contain files to edit, or might be VSCode settings, or VSCode key bindings, etc. Click on a tab name to open it. This key binding is bound to Go to File, which is somewhat logical but not the most descriptive name.

CTRL+Shift+P

Open the Command Palette.

CTRL+L G

Open the active (currently edited) file on GitHub in the default web browser. Requires the Open in GitHub extension.

CTRL+, (comma)

Open the Settings tab.

CTRL+Shift+T

Reopen the most recently closed editor tab.

ALT+Shift+F

Format the current file.

Annoying Side Bar Auto Reveal

Problem

When editing a file, if you CTRL+click on a function, method or class name defined in a dependency, the dependency's folder will be expanded in the side bar. Some dependencies are deeply nested, which means that the side bar expands quite a lot. In order to close the folders in the side bar it is necessary to go all the way back to the top of that folder, which is annoying and wastes time.

Solution

In settings, look for Explorer: Auto Reveal, which controls whether the explorer should automatically reveal and select files when opening them.

To do that, bring up settings with CTRL+, (comma), and then type reveal ex into the filter.

The default value is true. Change the value to false.

Bonus: Reveal Active File in Side Bar

To scroll to a file that you are editing in the list of files in the side bar, right-click on the file's tab, then select Reveal in side bar.

Even better, define a keyboard shortcut to do this:

Bring up the Keyboard Shortcuts definitions by typing CTRL+K CTRL+S.
Type reveal side into the search bar.
Double-click on File: Reveal Active File in Side Bar.
For my desired key shortcut, I pressed CTRL+Shift+ALT+R, then pressed Enter.

Extensions

Extension Directory

Extensions are stored in ~/.vscode/extensions/.

Settings

User settings are stored in %AppData%\Code\User\ on Windows, in files called settings.json, keybindings.json, syncLocalSettings.json and tasks.json.

Project settings are stored in .vscode/ on all platforms, in files called launch.json, settings.json and tasks.json.

Workspace settings are stored in files with a .code-workspace filetype. They can be painful to work with because paths in them are OS-dependent.

Open Windows Directory From File Explorer

The ability to have Visual Studio Code open a Windows directory from File Explorer is useful. When you install VS Code for Windows, you can add an Open with Code action to the Windows Explorer file context menu by enabling the two options that are highlighted in the image below.

To add the ability without reinstalling VSCode, save the following file, then double-click on it.

vscode_open_dir.reg

Windows Registry Editor Version 5.00

; Open files [HKEY_CLASSES_ROOT\*\shell\Open with VS Code] @="Edit with VS Code"
"Icon"="%LocalAppData%\\Programs\\Microsoft VS Code\\Code.exe,0"

[HKEY_CLASSES_ROOT\*\shell\Open with VS Code\command]
@="\"%LocalAppData%\\Programs\\Microsoft VS Code\\Code.exe\" \"%1\""

; This will make it appear when you right click ON a folder
; The "Icon" line can be removed if you don't want the icon to appear
[HKEY_CLASSES_ROOT\Directory\shell\vscode]
@="Open in VS Code"
"Icon"="\"%LocalAppData%\\Programs\\Microsoft VS Code\\Code.exe\",0"

[HKEY_CLASSES_ROOT\Directory\shell\vscode\command]
@="\"%LocalAppData%\\Programs\\Microsoft VS Code\\Code.exe\" \"%1\""

; This will make it appear when you right click INSIDE a folder
; The "Icon" line can be removed if you don't want the icon to appear
[HKEY_CLASSES_ROOT\Directory\Background\shell\vscode]
@="Open in VS Code"
"Icon"="\"%LocalAppData%\\Programs\\Microsoft VS Code\\Code.exe\",0"

[HKEY_CLASSES_ROOT\Directory\Background\shell\vscode\command]
@="\"%LocalAppData%\\Programs\\Microsoft VS Code\\Code.exe\" \"%V\""

There is no need to reboot your computer. Now you should see a new entry when you right-click on a directory:

Thanks to Erick Petrucelli and Walid Bousseta on StackOverflow for this procedure.

Command-Line AWS Utilities

2021-03-22T00:00:00-04:00

Here are some command-line utilities I have written for AWS. They are dependent on aws cli. You can download all of these utilities in tar format. Extract them into the current directory like this:

Shell

$ tar xf mslinn_aws.tar

awsCfInvalidate

Given a CloudFront distribution ID, invalidate the distribution.

awsCfInvalidate

#!/bin/bash

function help {
  printf "$1$(basename $0) - Invalidate the CloudFront distribution for the given ID.
If no distribution with the given ID exists, the empty string is returned and the return code is 2.
A message is printed asynchronously to the console when the invalidation completes.

Syntax: $(basename $0) distId

Syntax: awsCfS3Dist www.mslinn.com | $(basename $0)
"
  exit 1
}

function waitForInvalidation {
  echo "Waiting for invalidation $2 to complete."
  aws cloudfront wait invalidation-completed \
    --distribution-id "$1" \
    --id "$2"
  echo "Invalidation $2 has completed."
}


if [ "$1" == -h ]; then help; fi

if [ "$1" ]; then
  DIST_ID="$1"
  shift
elif [ ! -t 0 ]; then
  read -r DIST_ID
fi
if [ -z "$DIST_ID" ]; then help 'Error: No CloudFront distribution ID was specified.\n\n'; fi

if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi

JSON="$( aws cloudfront create-invalidation \
  --distribution-id "$DIST_ID" \
  --paths "/*"
)"

INVALIDATION_ID="$( jq -r .Invalidation.Id <<< "$JSON" )"
waitForInvalidation "$DIST_ID" "$INVALIDATION_ID" &

Example usages:

Shell

$ awsCfInvalidate E2P5S6OYKQNB6B
Waiting for invalidation IFOPKECU4YYHD to complete. 

... do other things ... 

$ Invalidation IFOPKECU4YYHD has completed.

Shell

$ awsCfS3Dist www.mslinn.com | awsCfInvalidate
Waiting for invalidation IFOPKECU4YYHD to complete. 

... do other things ... 

$ Invalidation IFOPKECU4YYHD has completed.

awsCfS3Dist

Given an S3 bucket name, return the CloudFront distribution JSON.

awsCfS3Dist

#!/bin/bash

function help {
  printf "$1$(basename $0) - Obtain the CloudFront distribution JSON for an S3 bucket.
If no S3 bucket with the given name exists, the empty string is returned and the return code is 2.

Syntax: $(basename $0) bucketName

Syntax: echo bucketName | $(basename $0)
"
  exit 1
}

if [ "$1" == -h ]; then help; fi

if [ "$1" ]; then
  BUCKET_NAME="$1"
  shift
elif [ ! -t 0 ]; then
  read -r BUCKET_NAME
fi
if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi

if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi

if [ "$( aws s3api head-bucket --bucket $BUCKET_NAME 2> >(grep -i 'Not Found') )" ]; then
  >&2 echo "Error: Bucket $BUCKET_NAME does not exist."
  exit 2
fi

DIST_ID="$( awsCfS3DistId "$BUCKET_NAME" )"

if [ -z "$DIST_ID" ]; then exit 2; fi

aws cloudfront get-distribution-config --id "$DIST_ID"

Example usages:

Shell

$ awsCfS3Dist www.mslinn.com
{
  "ETag": "E1DIZUSLMOLXKP",
  "DistributionConfig": {
      "CallerReference": "1454487160038",
      "Aliases": {
          "Quantity": 2,
          "Items": [
              "www.mslinn.com",
              "mslinn.com"
          ]
      },
      "DefaultRootObject": "index.html",
      "Origins": {
          "Quantity": 1,
          "Items": [
              {
                  "Id": "S3-www.mslinn.com",
                  "DomainName": "www.mslinn.com.s3-website-us-east-1.amazonaws.com",
                  "OriginPath": "",
                  "CustomHeaders": {
                      "Quantity": 0
                  },
                  "CustomOriginConfig": {
                      "HTTPPort": 80,
                      "HTTPSPort": 443,
                      "OriginProtocolPolicy": "http-only",
                      "OriginSslProtocols": {
                          "Quantity": 3,
                          "Items": [
                              "TLSv1",
                              "TLSv1.1",
                              "TLSv1.2"
                          ]
                      },
                      "OriginReadTimeout": 30,
                      "OriginKeepaliveTimeout": 5
                  },
                  "ConnectionAttempts": 3,
                  "ConnectionTimeout": 10
              }
          ]
      },
      "OriginGroups": {
          "Quantity": 0
      },
      "DefaultCacheBehavior": {
          "TargetOriginId": "S3-www.mslinn.com",
          "TrustedSigners": {
              "Enabled": false,
              "Quantity": 0
          },
          "ViewerProtocolPolicy": "redirect-to-https",
          "AllowedMethods": {
              "Quantity": 2,
              "Items": [
                  "HEAD",
                  "GET"
              ],
              "CachedMethods": {
                  "Quantity": 2,
                  "Items": [
                      "HEAD",
                      "GET"
                  ]
              }
          },
          "SmoothStreaming": false,
          "Compress": true,
          "LambdaFunctionAssociations": {
              "Quantity": 0
          },
          "FieldLevelEncryptionId": "",
          "CachePolicyId": "658327ea-f89d-4fab-a63d-7e88639e58f6"
      },
      "CacheBehaviors": {
          "Quantity": 0
      },
      "CustomErrorResponses": {
          "Quantity": 2,
          "Items": [
              {
                  "ErrorCode": 403,
                  "ResponsePagePath": "",
                  "ResponseCode": "",
                  "ErrorCachingMinTTL": 60
              },
              {
                  "ErrorCode": 404,
                  "ResponsePagePath": "",
                  "ResponseCode": "",
                  "ErrorCachingMinTTL": 60
              }
          ]
      },
      "Comment": "",
      "Logging": {
          "Enabled": false,
          "IncludeCookies": false,
          "Bucket": "",
          "Prefix": ""
      },
      "PriceClass": "PriceClass_All",
      "Enabled": true,
      "ViewerCertificate": {
          "ACMCertificateArn": "arn:aws:acm:us-east-1:031372724784:certificate/2be42926-829c-4db9-be7d-a72e951256d4",
          "SSLSupportMethod": "sni-only",
          "MinimumProtocolVersion": "TLSv1",
          "Certificate": "arn:aws:acm:us-east-1:031372724784:certificate/2be42926-829c-4db9-be7d-a72e951256d4",
          "CertificateSource": "acm"
      },
      "Restrictions": {
          "GeoRestriction": {
              "RestrictionType": "none",
              "Quantity": 0
          }
      },
      "WebACLId": "",
      "HttpVersion": "http1.1",
      "IsIPV6Enabled": false
  }
}

Shell

$ echo www.mslinn.com | awsCfS3Dist
{
  "ETag": "E1DIZUSLMOLXKP",
  "DistributionConfig": {
      "CallerReference": "1454487160038",
      "Aliases": {
          "Quantity": 2,
          "Items": [
              "www.mslinn.com",
              "mslinn.com"
          ]
      },
      "DefaultRootObject": "index.html",
      "Origins": {
          "Quantity": 1,
          "Items": [
              {
                  "Id": "S3-www.mslinn.com",
                  "DomainName": "www.mslinn.com.s3-website-us-east-1.amazonaws.com",
                  "OriginPath": "",
                  "CustomHeaders": {
                      "Quantity": 0
                  },
                  "CustomOriginConfig": {
                      "HTTPPort": 80,
                      "HTTPSPort": 443,
                      "OriginProtocolPolicy": "http-only",
                      "OriginSslProtocols": {
                          "Quantity": 3,
                          "Items": [
                              "TLSv1",
                              "TLSv1.1",
                              "TLSv1.2"
                          ]
                      },
                      "OriginReadTimeout": 30,
                      "OriginKeepaliveTimeout": 5
                  },
                  "ConnectionAttempts": 3,
                  "ConnectionTimeout": 10
              }
          ]
      },
      "OriginGroups": {
          "Quantity": 0
      },
      "DefaultCacheBehavior": {
          "TargetOriginId": "S3-www.mslinn.com",
          "TrustedSigners": {
              "Enabled": false,
              "Quantity": 0
          },
          "ViewerProtocolPolicy": "redirect-to-https",
          "AllowedMethods": {
              "Quantity": 2,
              "Items": [
                  "HEAD",
                  "GET"
              ],
              "CachedMethods": {
                  "Quantity": 2,
                  "Items": [
                      "HEAD",
                      "GET"
                  ]
              }
          },
          "SmoothStreaming": false,
          "Compress": true,
          "LambdaFunctionAssociations": {
              "Quantity": 0
          },
          "FieldLevelEncryptionId": "",
          "CachePolicyId": "658327ea-f89d-4fab-a63d-7e88639e58f6"
      },
      "CacheBehaviors": {
          "Quantity": 0
      },
      "CustomErrorResponses": {
          "Quantity": 2,
          "Items": [
              {
                  "ErrorCode": 403,
                  "ResponsePagePath": "",
                  "ResponseCode": "",
                  "ErrorCachingMinTTL": 60
              },
              {
                  "ErrorCode": 404,
                  "ResponsePagePath": "",
                  "ResponseCode": "",
                  "ErrorCachingMinTTL": 60
              }
          ]
      },
      "Comment": "",
      "Logging": {
          "Enabled": false,
          "IncludeCookies": false,
          "Bucket": "",
          "Prefix": ""
      },
      "PriceClass": "PriceClass_All",
      "Enabled": true,
      "ViewerCertificate": {
          "ACMCertificateArn": "arn:aws:acm:us-east-1:031372724784:certificate/2be42926-829c-4db9-be7d-a72e951256d4",
          "SSLSupportMethod": "sni-only",
          "MinimumProtocolVersion": "TLSv1",
          "Certificate": "arn:aws:acm:us-east-1:031372724784:certificate/2be42926-829c-4db9-be7d-a72e951256d4",
          "CertificateSource": "acm"
      },
      "Restrictions": {
          "GeoRestriction": {
              "RestrictionType": "none",
              "Quantity": 0
          }
      },
      "WebACLId": "",
      "HttpVersion": "http1.1",
      "IsIPV6Enabled": false
  }
}

awsCfS3DistId

Given an S3 bucket name, return the CloudFront distribution ID.

awsCfS3DistId

#!/bin/bash

function help {
  printf "$1$(basename $0) - Obtain the CloudFront distribution ID for an S3 bucket.
If no S3 bucket with the given name exists, the empty string is returned and the return code is 2.

Syntax: $(basename $0) bucketName

Syntax: echo bucketName | $(basename $0)
"
  exit 1
}

if [ "$1" == -h ]; then help; fi

if [ "$1" ]; then
  BUCKET_NAME="$1"
  shift
elif [ ! -t 0 ]; then
  read -r BUCKET_NAME
fi
if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi

if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi

if [ "$( aws s3api head-bucket --bucket $BUCKET_NAME 2> >(grep -i 'Not Found') )" ]; then
  >&2 echo "Error: Bucket $BUCKET_NAME does not exist."
  exit 2
fi

DIST_ID="$(
  aws cloudfront list-distributions \
  --query "DistributionList.Items[*].{id:Id,origin:Origins.Items[0].Id}[?origin=='S3-$BUCKET_NAME'].id" \
  --output text
)"

if [ -z "$DIST_ID" ]; then exit 2; fi
echo "$DIST_ID"

Example usages:

Shell

$ awsCfS3DistId www.mslinn.com
E2P5S6OYKQNB6B

Shell

$ echo www.mslinn.com | awsCfS3DistId
E2P5S6OYKQNB6B

awsCfS3MakeDist

Creates a CloudFront distribution for the given bucket name. Returns the new distribution's ID.

awsCfS3MakeDist

#!/bin/bash

function help {
  printf "$1$(basename $0) - Make a new CloudFront distribution for the given S3 bucket name.

Returns the new distribution's ID.

Syntax: $(basename $0) bucketName

Syntax: echo bucketName | $(basename $0)
"
  exit 1
}

function doesDistributionExist {
  DIST_ID="$( awsCfS3Dist "$BUCKET_NAME" )"
  if [ "$DIST_ID" ]; then echo true; fi
}

function createDist {
  read -r -d '' NEW_DIST_JSON <<EOF
{
  "CallerReference": "$BUCKET_NAME",
  "Aliases": {
    "Quantity": 0
  },
  "DefaultRootObject": "index.html",
  "Origins": {
    "Quantity": 1,
    "Items": [
      {
        "Id": "$BUCKET_NAME",
        "DomainName": "$BUCKET_NAME.s3.amazonaws.com",
        "S3OriginConfig": {
          "OriginAccessIdentity": ""
        }
      }
    ]
  },
  "DefaultCacheBehavior": {
    "TargetOriginId": "$BUCKET_NAME",
    "ForwardedValues": {
      "QueryString": true,
      "Cookies": {
        "Forward": "none"
      }
    },
    "TrustedSigners": {
      "Enabled": false,
      "Quantity": 0
    },
    "ViewerProtocolPolicy": "redirect-to-https",
    "MinTTL": 3600
  },
  "CacheBehaviors": {
    "Quantity": 0
  },
  "Comment": "",
  "Logging": {
    "Enabled": false,
    "IncludeCookies": true,
    "Bucket": "",
    "Prefix": ""
  },
  "PriceClass": "PriceClass_All",
  "Enabled": true
}
EOF

  NEW_DIST_RESULT_JSON = "$(
    aws cloudfront create-distribution --distribution-config "$NEW_DIST_JSON"
  )"

  DISTRIBUTION_ID="$( jq -r '.Distribution.Id' <<< "$NEW_DIST_RESULT_JSON" )"

  echo "$DISTRIBUTION_ID"
}


if [ "$1" == -h ]; then help; fi

if [ -t 0 ]; then
  if [ -z "$1" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi
  BUCKET_NAME="$1"
  shift
else
  read -r BUCKET_NAME
fi
if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi

if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi

if [ "$( aws s3api head-bucket --bucket $BUCKET_NAME 2> >(grep -i 'Not Found') )" ]; then
  >&2 echo "Error: Bucket $BUCKET_NAME does not exist."
  exit 2
fi

if [ "$(doesDistributionExist)" ]; then
  >&2 echo "Error: a CloudFront distibution already exists for S3 bucket $BUCKET_NAME"
  exit 3
fi

createDist

Example usages:

Shell

$ awsCfS3MakeDist my_bucket
E2P5S6OYKQNB6B

Shell

$ echo my_bucket | awsCfS3MakeDist
E2P5S6OYKQNB6B

awsS3Mb

Make a new S3 bucket with the given name in the default AWS region. If the --public-read option is provided, set the ACL to public-read

awsS3Mb

#!/bin/bash

function help {
  printf "$1$(basename $0) - Make a new S3 bucket with the given name in the default AWS region.

Syntax: $(basename $0) bucketName [OPTIONS]

Syntax: echo bucketName | $(basename $0) [OPTIONS]

Options are:
  --public-read  Set bucket ACL to public-read
"
  exit 1
}

if [ "$1" == -h ]; then help; fi

if [ "$1" == "--public-read" ]; then
  ACL="public-read"
  shift
fi

if [ "$1" ]; then
  BUCKET_NAME="$1"
  shift
elif [ ! -t 0 ]; then
  read -r BUCKET_NAME
fi
if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi

if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi

aws s3 mb s3://$BUCKET_NAME

if [ "$ACL" ]; then
  aws s3api put-bucket-acl --bucket $BUCKET_NAME --acl $ACL
fi

Example usages:

Shell

$ awsS3Mb my_bucket

Shell

$ awsS3Mb my_bucket --public-read

Shell

$ echo my_bucket | awsS3Mb --public-read

awsS3Website

Enable an S3 bucket to be a website.

awsS3Website

#!/bin/bash

function help {
  printf "$1$(basename $0) - Enable an S3 bucket to be a website.

Syntax: $(basename $0) bucketName

Syntax: echo bucketName | $(basename $0)
"
  exit 1
}

if [ "$1" == -h ]; then help; fi

if [ "$1" ]; then
  BUCKET_NAME="$1"
  shift
elif [ ! -t 0 ]; then
  read -r BUCKET_NAME
fi
if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi

if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi

aws s3 website s3://$BUCKET_NAME \
  --index-document index.html \
  --error-document 404.html

Example usages:

Shell

$ awsS3Website my_bucket

Shell

$ echo my_bucket | awsS3Website

CORS on AWS S3 and Cloudfront

2021-03-21T00:00:00-04:00

This article shows how to enable CORS on an AWS S3 bucket with AWS CLI, then modify the bucket’s CloudFront distribution. In preparing this article, I found that the AWS S3 CORS documentation needs to be read in conjunction with how AWS CloudFront can be configured to handle CORS.

I used one origin for testing.

Shell

$ ORIGIN=ancientwarmth.com

$ JSON_FILE=cors.json

The CORS configuration for the AWS S3 bucket will be stored in the file pointed to by JSON_FILE.

Define the AWS S3 Bucket CORS Configuration

This configuration (in JSON format) contains 1 rule:

Allow GET HTTP methods from anywhere.

cors.json

{
  "CORSRules": [
    {
      "AllowedHeaders": [],
      "AllowedMethods": [
          "GET"
      ],
      "AllowedOrigins": [
          "*"
      ],
      "ExposeHeaders": []
    }  ]
}

You can read about CORS configuration in the AWS documentation.

Set the AWS S3 Bucket CORS Configuration

It is easy to set the CORS configuration for an AWS S3 bucket using AWS CLI’s aws s3api put-bucket-cors subcommand:

Shell

$ BUCKET=assets.ancientwarmth.com

$ aws s3api put-bucket-cors \
  --bucket $BUCKET \
  --cors-configuration "file://$JSON_FILE"

Test the AWS S3 Bucket CORS Configuration

Now it is time to test the S3 bucket’s CORS configuration using curl. I defined a bash function to peform the test to save typing. You can use it by first copy/pasting the code below into a shell prompt, then calling the function with the proper arguments, as shown. The function requires 3 arguments: the request origin, the URL of an asset in an AWS S3 bucket, and an HTTP method (which must be in UPPPER CASE).

Shell

$ function testCors {
  if [ -z "$1" ]; then echo "Error: No origin was provided"; exit 1; fi
  if [ -z "$2" ]; then echo "Error: No URL to test was provided"; exit 1; fi
  if [ "$3" ]; then METHOD="$3"; else METHOD=GET; fi

  curl -I -X OPTIONS \
    --no-progress-meter \
    -H "Origin: $1" \
    -H "Access-Control-Request-Method: $METHOD" \
    "$2" 2>&1 | \
  grep --color=never 'Access-Control'
}

The JSON file for testing CORS was s3://$BUCKET/testCors.json:

testCors.json

{
 "key1": "value1",
 "key2": "value2"
}

We will know if CORS is set up properly by receiving a header containing Access-Control-Allow-Origin: *.

Shell

$ URL="https://s3.amazonaws.com/$BUCKET/testCors.json"

$ testCors $ORIGIN $URL GET
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method

The origin worked when the bucket is accessed via a GET method sent to its s3.amazonaws.com DNS alias (yay!).

CORScanner (discussed in a previous article) reported no issues:

Shell

$ cors -u s3.amazonaws.com/assets.ancientwarmth.com/testCors.json
Starting CORS scan...
Finished CORS scanning...

CloudFront

I have not worked through the process of using AWS CLI to obtain a JSON object describing the distribution, and then changing some properties and writing it back. So until that happy day comes, here are 2 screen shots of the AWS CloudFront web console showing the settings. The first screen shot shows the Behaviors tab of the top-level details of the assets.ancientwarmth.com CloudFront distribution.

About to click on Edit (default behavior)' class="imgImg rounded shadow" src="/blog/images/aws/cfBehaviorCors0.png" style='width: 100%; ' title='CloudFront / Edit Distribution / Behaviors
About to click on Edit (default behavior)' />

CloudFront / Edit Distribution / Behaviors
About to click on Edit (default behavior)

My application does not require users to upload anything, so everything in the S3 bucket is truly static. Thus I have no need to PUT, POST or DELETE HTTP methods for the AWS S3 content. I have not seen a good explanation of why enabling OPTIONS HTTP methods is necessary, but every person on Stack Overflow who got CORS to work with AWS S3 says this was necessary. With that in mind, I set the following for the next screen shot:

Viewer Protocol Policy: Redirect HTTP to HTTPS
Allowed HTTP Methods: GET, HEAD, OPTIONS
Cached HTTP Methods: Enable OPTIONS
Use a cache policy and origin request policy: (default is Use legacy cache settings, which is usually undesirable)
Cache Policy: Managed-CachingOptimized
Origin Request Policy: Managed-CORS-S3Origin

Editing default CloudFront distribution behavior

Managed CORS S3 Origin Poligy

AWS CloudFront's managed origin request policy called Managed-CORS-S3Origin includes the headers that enable cross-origin resource sharing (CORS) requests when the origin is an Amazon S3 bucket. This policy's settings are:

Query strings included in origin requests: None
Headers included in origin requests:
- Origin
- Access-Control-Request-Headers
- Access-Control-Request-Method
Cookies included in origin requests: None

Wait or Invalidate

Whenever you make a configuration change to a CloudFront distribution, or the contents change, the distributed assets will not reflect those changes until the next CloudFront invalidation. Automatic invalidations take 20 minutes. You can invalidate manually for near-instant gratification. I use my AWS command-line utilities to invalidate manually:

Shell

$ awsCfS3DistId $BUCKET | awsCfInvalidate

Now the grand finale:

Shell

$ testCors $ORIGIN $URL GET
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method

😁

The presence of the Access-Control-Allow-Origin header indicates that CORS allowed the data file to be transferred from the content server (AWS S3/CloudFront) to the origin server (the command line).

Cross-Origin Resource Sharing (CORS)

2021-03-20T00:00:00-04:00

Many have tried to explain CORS, but most have not provided a clear explanation. I am going to try, then I will refer to explanations by others, who also provide examples.

Origin and Origin Server

A website is delivered to web browsers from an origin server, or origin for short. The origin server is principally responsible for generating web pages.

An origin is a combination of 3 things:

A scheme (http, https, etc.)
A (sub)domain, for example localhost, blah.com or assets.blah.com.
A port, for example 80, 443, 8000, etc.

All three things must match in order for two URLs to be considered to be from the same origin. For example:

URL 1	URL 2	Same Origin?
`http://blah.com`	`https://blah.com`	No
`https://blah.com`	`https://assets.blah.com`	No
`https://blah.com`	`https://blah.com/path/page.html`	Yes

Content Servers

In this article, I use the term content server to refer to sources of online information other than the origin server. Resources referenced by a web page, such as images, JavaScript, CSS, and data might be provided by the origin server, or they might come from a content server.

Because every server has by definition a different origin, content servers always have a different origin than the origin server. Static resources (resources that do not change) are often served by content delivery networks (CDNs), which are also content servers.

The Cross-Origin Resource Sharing (CORS) standard controls if a web page can load resources from content servers. Content servers are in charge of their content; they decide which origin servers they wish to co-operate with. When CORS support is properly configured, content servers include HTTP headers into their responses that tell a web browser if those resources may be read by the web page being constructed.

Proxy servers incorporate content from other servers into their output. The content from proxied sites are incorporated into proxy server’s output.

Here is a more detailed set of definitions:

Data is a special type of resource. CORS restricts how data is exchanged between the web page delivered to the web browser from the origin server and content servers. In particular, JSON and XML data communicated to and from content servers requires CORS authorization. Furthermore, requests (from the web browser) that send JSON, XML and other data formats to content servers also require CORS authorization.

Content servers are in charge of their content; they decide which origin servers they wish to co-operate with.

Nginx Proxy Configuration

Nginx is a content server, and an nginx website can configured to allow its content to be proxied by other servers. Other nginx websites could be configured as proxy servers.

CORS headers emitted by the proxied website determine which proxy servers are allowed to consume their content.

For example, imagine a website running at http://localhost:9000. The following configuration that would allow that server’s content to be proxied by any another server. Specifically, the highlighted snippet grants permission to any other server to include the content into their web pages. All of the proxied website’s headers are allowed to be passed through by the proxy server, if it is configured to do so.

Nginx configuration for proxied server on port 9000

server {
  listen 9000;
  listen [::]:9000;

  server_name localhost;

  location / {
    root /var/www/blah/blah;
    index index.html;

    # First attempt to serve request as file, then as directory, then fall back to displaying a 404.
    try_files $uri $uri/ =404;

    add_header 'Access-Control-Allow-Origin' '*' always;
    add_header 'Access-Control-Allow-Headers' '*' always;
  }
}

Continuing our imaginary setup, imagine a publicly accessible proxy server that needs to incorporate the content from the local server on port 9000. The configuration for the proxy server would, at a minimum, require something like the following. This configuration causes the proxied content to be included into the public server’s content.

Nginx proxy server configuration

server {
  listen 80;
  listen [::]:80;
  listen 443 ssl;
  listen [::]:443;

  server_name scalacourses.com www.scalacourses.com;
  ssl_certificate         /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/fullchain.pem;
  ssl_certificate_key     /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/privkey.pem;
  ssl_trusted_certificate /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/chain.pem;
  include /etc/letsencrypt/options-ssl-nginx.conf;
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

  root /var/www/html;
  index index.html;  # This gets served if the proxied website is down

  location / {
    proxy_pass http://localhost:9000;
  }
}

For further reading, please see my article entitled Using Nginx As a Reverse Proxy With SSL.

Content-Type Header

The Content-Type header is used to indicate the media type of the resource. The old name MIME type has been replaced by media type. Here is a list of media types.

Media types with names that start with application require CORS authentication if they are delivered from content servers, for example application/json and application/javascript.

As well, a few media types with names that start with text require CORS authentication if they are delivered from content servers, for example text/xml and text/xml-external-parsed-entity.

CORScanner

CORScanner is a popular tool for detecting CORS misconfiguration. It is a Python module that can be executed as a shell command. Install CORScanner like this:

Shell

$ pip install cors

The above adds a new executable called cors in the same directory where your python command resides.

The cors documentation conflates the words URL and origin. Everywhere the word URL appears in the documentation, the word origin should be assumed.

Example: Check Domain

Use the -u option to specify an origin to test:

Shell

$ cors -u api.github.com
Starting CORS scan...
Finished CORS scanning...

To enable more debug info, use the -v option more than once. We can see that specifying https restricts testing to that scheme.

Shell

$ cors -vv -u https://api.github.com
Starting CORS scan...
2021-03-21 09:55:58 INFO Start checking reflect_origin for https://api.github.com
2021-03-21 09:55:58 INFO nothing found for {url: https://api.github.com, origin: https://evil.com, type: reflect_origin}
2021-03-21 09:55:58 INFO Start checking prefix_match for https://api.github.com
2021-03-21 09:55:58 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com.evil.com, type: prefix_match}
2021-03-21 09:55:58 INFO Start checking suffix_match for https://api.github.com
2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://evilgithub.com, type: suffix_match}
2021-03-21 09:55:59 INFO Start checking trust_null for https://api.github.com
2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: null, type: trust_null}
2021-03-21 09:55:59 INFO Start checking include_match for https://api.github.com
2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://ithub.com, type: include_match}
2021-03-21 09:55:59 INFO Start checking not_escape_dot for https://api.github.com
2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://api.githubacom, type: not_escape_dot}
2021-03-21 09:55:59 INFO Start checking custom_third_parties for https://api.github.com
2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://whatever.github.io, type: custom_third_parties}
2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: http://jsbin.com, type: custom_third_parties}
2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://codepen.io, type: custom_third_parties}
2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://jsfiddle.net, type: custom_third_parties}
2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: http://www.webdevout.net, type: custom_third_parties}
2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://repl.it, type: custom_third_parties}
2021-03-21 09:56:00 INFO Start checking special_characters_bypass for https://api.github.com
2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com_.evil.com, type: special_characters_bypass}
2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com-.evil.com, type: special_characters_bypass}
2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com".evil.com, type: special_characters_bypass}
2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com{.evil.com, type: special_characters_bypass}
2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com}.evil.com, type: special_characters_bypass}
2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass}
2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com^.evil.com, type: special_characters_bypass}
2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%60.evil.com, type: special_characters_bypass}
2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com!.evil.com, type: special_characters_bypass}
2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com~.evil.com, type: special_characters_bypass}
2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com`.evil.com, type: special_characters_bypass}
2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com;.evil.com, type: special_characters_bypass}
2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com|.evil.com, type: special_characters_bypass}
2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&.evil.com, type: special_characters_bypass}
2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com'.evil.com, type: special_characters_bypass}
2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com(.evil.com, type: special_characters_bypass}
2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com).evil.com, type: special_characters_bypass}
2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com*.evil.com, type: special_characters_bypass}
2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com,.evil.com, type: special_characters_bypass}
2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com$.evil.com, type: special_characters_bypass}
2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com=.evil.com, type: special_characters_bypass}
2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass}
2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%0b.evil.com, type: special_characters_bypass}
2021-03-21 09:56:03 INFO Start checking trust_any_subdomain for https://api.github.com
2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: https://evil.api.github.com, type: trust_any_subdomain}
2021-03-21 09:56:03 INFO Start checking https_trust_http for https://api.github.com
2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: http://api.github.com, type: https_trust_http}
Finished CORS scanning...

Example: Check Origin

To check CORS misconfigurations of an origin:

Shell

$ cors -vvu https://api.github.com/users/mslinn/repos
Starting CORS scan...
  2021-03-21 10:08:49 INFO Start checking reflect_origin for https://api.github.com
  2021-03-21 10:08:49 INFO nothing found for {url: https://api.github.com, origin: https://evil.com, type: reflect_origin}
  2021-03-21 10:08:49 INFO Start checking prefix_match for https://api.github.com
  2021-03-21 10:08:49 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com.evil.com, type: prefix_match}
  2021-03-21 10:08:49 INFO Start checking suffix_match for https://api.github.com
  2021-03-21 10:08:49 INFO nothing found for {url: https://api.github.com, origin: https://evilgithub.com, type: suffix_match}
  2021-03-21 10:08:49 INFO Start checking trust_null for https://api.github.com
  2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: null, type: trust_null}
  2021-03-21 10:08:50 INFO Start checking include_match for https://api.github.com
  2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://ithub.com, type: include_match}
  2021-03-21 10:08:50 INFO Start checking not_escape_dot for https://api.github.com
  2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://api.githubacom, type: not_escape_dot}
  2021-03-21 10:08:50 INFO Start checking custom_third_parties for https://api.github.com
  2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://whatever.github.io, type: custom_third_parties}
  2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: http://jsbin.com, type: custom_third_parties}
  2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://codepen.io, type: custom_third_parties}
  2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://jsfiddle.net, type: custom_third_parties}
  2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: http://www.webdevout.net, type: custom_third_parties}
  2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://repl.it, type: custom_third_parties}
  2021-03-21 10:08:51 INFO Start checking special_characters_bypass for https://api.github.com
  2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com_.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com-.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com".evil.com, type: special_characters_bypass}
  2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com{.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com}.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com^.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%60.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com!.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com~.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com`.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com;.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com|.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com'.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com(.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com).evil.com, type: special_characters_bypass}
  2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com*.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com,.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com$.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com=.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:54 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:54 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%0b.evil.com, type: special_characters_bypass}
  2021-03-21 10:08:54 INFO Start checking trust_any_subdomain for https://api.github.com
  2021-03-21 10:08:54 INFO nothing found for {url: https://api.github.com, origin: https://evil.api.github.com, type: trust_any_subdomain}
  2021-03-21 10:08:54 INFO Start checking https_trust_http for https://api.github.com
  2021-03-21 10:08:54 INFO nothing found for {url: https://api.github.com, origin: http://api.github.com, type: https_trust_http}
  Finished CORS scanning...

If a scheme is not specified, then both http and https are tested:

Shell

$ cors -vvu api.github.com/users/mslinn/repos
Starting CORS scan...
  2021-03-21 10:03:30 INFO Start checking reflect_origin for http://api.github.com
  2021-03-21 10:03:30 INFO Start checking reflect_origin for https://api.github.com
  2021-03-21 10:03:30 INFO nothing found for {url: https://api.github.com, origin: https://evil.com, type: reflect_origin}2021-03-21 10:03:30 INFO Start checking prefix_match for https://api.github.com
  2021-03-21 10:03:30 INFO nothing found for {url: http://api.github.com, origin: http://evil.com, type: reflect_origin}
  2021-03-21 10:03:30 INFO Start checking prefix_match for http://api.github.com
  2021-03-21 10:03:30 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com.evil.com, type: prefix_match}
  2021-03-21 10:03:30 INFO Start checking suffix_match for https://api.github.com
  2021-03-21 10:03:30 INFO nothing found for {url: https://api.github.com, origin: https://evilgithub.com, type: suffix_match}
  2021-03-21 10:03:30 INFO Start checking trust_null for https://api.github.com
  2021-03-21 10:03:30 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com.evil.com, type: prefix_match}
  2021-03-21 10:03:30 INFO Start checking suffix_match for http://api.github.com
  2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: null, type: trust_null}
  2021-03-21 10:03:31 INFO Start checking include_match for https://api.github.com
  2021-03-21 10:03:31 INFO nothing found for {url: http://api.github.com, origin: http://evilgithub.com, type: suffix_match}
  2021-03-21 10:03:31 INFO Start checking trust_null for http://api.github.com
  2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://ithub.com, type: include_match}2021-03-21 10:03:31 INFO Start checking not_escape_dot for https://api.github.com
  2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://api.githubacom, type: not_escape_dot}
  2021-03-21 10:03:31 INFO Start checking custom_third_parties for https://api.github.com
  2021-03-21 10:03:31 INFO nothing found for {url: http://api.github.com, origin: null, type: trust_null}
  2021-03-21 10:03:31 INFO Start checking include_match for http://api.github.com
  2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://whatever.github.io, type: custom_third_parties}
  2021-03-21 10:03:31 INFO nothing found for {url: http://api.github.com, origin: http://ithub.com, type: include_match}
  2021-03-21 10:03:31 INFO Start checking not_escape_dot for http://api.github.com
  2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: http://jsbin.com, type: custom_third_parties}
  2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://codepen.io, type: custom_third_parties}
  2021-03-21 10:03:31 INFO nothing found for {url: http://api.github.com, origin: http://api.githubacom, type: not_escape_dot}
  2021-03-21 10:03:31 INFO Start checking custom_third_parties for http://api.github.com
  2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://jsfiddle.net, type: custom_third_parties}
  2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: https://whatever.github.io, type: custom_third_parties}
  2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: http://www.webdevout.net, type: custom_third_parties}
  2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://repl.it, type: custom_third_parties}
  2021-03-21 10:03:32 INFO Start checking special_characters_bypass for https://api.github.com
  2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: http://jsbin.com, type: custom_third_parties}
  2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com_.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com-.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: https://codepen.io, type: custom_third_parties}
  2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com".evil.com, type: special_characters_bypass}
  2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com{.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: https://jsfiddle.net, type: custom_third_parties}
  2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com}.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: http://www.webdevout.net, type: custom_third_parties}
  2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com^.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: https://repl.it, type: custom_third_parties}
  2021-03-21 10:03:33 INFO Start checking special_characters_bypass for http://api.github.com
  2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%60.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com!.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com_.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com~.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com`.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com-.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com;.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com".evil.com, type: special_characters_bypass}
  2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com|.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com{.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&qpos;.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com(.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com}.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com).evil.com, type: special_characters_bypass}
  2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com+.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com*.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com,.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com^.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com$.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com=.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com%60.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com!.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%0b.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:34 INFO Start checking trust_any_subdomain for https://api.github.com
  2021-03-21 10:03:35 INFO nothing found for {url: https://api.github.com, origin: https://evil.api.github.com, type: trust_any_subdomain}
  2021-03-21 10:03:35 INFO Start checking https_trust_http for https://api.github.com
  2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com~.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:35 INFO nothing found for {url: https://api.github.com, origin: http://api.github.com, type: https_trust_http}
  2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com`.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com;.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com|.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com&.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:36 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com'.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:36 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com(.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:36 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com).evil.com, type: special_characters_bypass}
  2021-03-21 10:03:36 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com*.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:37 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com,.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:38 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com$.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:38 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com=.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:38 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com+.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:38 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com%0b.evil.com, type: special_characters_bypass}
  2021-03-21 10:03:38 INFO Start checking trust_any_subdomain for http://api.github.com
  2021-03-21 10:03:39 INFO nothing found for {url: http://api.github.com, origin: http://evil.api.github.com, type: trust_any_subdomain}
  Finished CORS scanning...

AWS S3 and CloudFront SSL

2021-03-19T00:00:00-04:00

SSL certificates need to match the domain they are served from.

AWS uses one of several SSL certificates, depending on the domain that an asset is requested from.

AWS S3 applies an SSL certificate for https requests. The SSL certificate chosen depends on the bucket endpoint used: s3.amazonaws.com, *.s3.amazonaws.com, or s3.region.amazonaws.com.
AWS CloudFront will apply your custom SSL certificate (for example, a wildcard certificate such as *.ancientwarmth.com) for https requests to the CNAME for that distribution, otherwise it will apply the wildcard SSL certificate for *.cloudfront.net.

Example SSL URLs

My AWS S3 bucket called assets.ancientwarmth.com is served via a CloudFront distribution with URL d1bci9l8cjf24o.cloudfront.net that applies a wildcard SSL certificate for *.ancientwarmth.com that I created using AWS Certificate Manager. I defined a CNAME called assets.ancientwarmth.com for that same CloudFront distribution using Route 53.

All of the following URLs can be used to access my content, providing the SSL certificate matches the requested domain.

URL: https://d1bci9l8cjf24o.cloudfront.net
Origin Type: CloudFront distribution
SSL certificate origin: *.cloudfront.net
Valid SSL certificate? Yes.

URL: https://assets.ancientwarmth.com
Origin Type: CloudFront distribution
SSL certificate origin: *.ancientwarmth.com
Valid SSL certificate? Yes. (I created this wildcard certificate using Route 53.)

S3 path-style URL: https://s3.us-east-1.amazonaws.com/assets.ancientwarmth.com
Origin Type: S3 bucket
SSL certificate origin: s3.us-east-1.amazonaws.com
Valid SSL certificate? Yes.

S3 dot URL: https://assets.ancientwarmth.com.s3.amazonaws.com
Origin Type: S3 bucket
SSL certificate origin: *.s3.amazonaws.com
Valid SSL certificate? No, does not match URL (wildcards only match one subdomain).

S3 dot Region URL: https://assets.ancientwarmth.com.s3.us-east-1.amazonaws.com
Origin Type: S3 bucket
SSL certificate origin: s3.amazonaws.com
Valid SSL certificate? No, does not match URL.

Testing with curl

Curl is often used to test SSL requests. In the following curl commands, the -I option just fetches the headers, and the -v option provides verbose output. You can see the SSL certificate negotation.

Fetching an asset from a CloudFront distribution using the AWS *.cloudfront.net wildcard SSL certificate:

Shell

$ curl -Iv \
  https://d1bci9l8cjf24o.cloudfront.net/js/jquery.modal.min.js
*   Trying 52.85.149.22:443...
* TCP_NODELAY set
* Connected to d1bci9l8cjf24o.cloudfront.net (52.85.149.22) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.cloudfront.net
*  start date: Feb 22 00:00:00 2021 GMT
*  expire date: Feb 21 23:59:59 2022 GMT
*  subjectAltName: host "d1bci9l8cjf24o.cloudfront.net" matched cert's "*.cloudfront.net"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global CA G2
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5585dcfaf7e0)
> HEAD /js/jquery.modal.min.js HTTP/2
> Host: d1bci9l8cjf24o.cloudfront.net
> user-agent: curl/7.68.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
HTTP/2 200
< content-type: application/javascript
content-type: application/javascript
< content-length: 4953
content-length: 4953
< date: Sat, 20 Mar 2021 14:11:34 GMT
date: Sat, 20 Mar 2021 14:11:34 GMT
< last-modified: Sat, 20 Mar 2021 03:14:08 GMT
last-modified: Sat, 20 Mar 2021 03:14:08 GMT
< etag: "c8f50397e0560719c62a35318f413e16"
etag: "c8f50397e0560719c62a35318f413e16"
< accept-ranges: bytes
accept-ranges: bytes
< server: AmazonS3
server: AmazonS3
< x-cache: Miss from cloudfront
x-cache: Miss from cloudfront
< via: 1.1 0712e4ad4264127dfcb76a114b130495.cloudfront.net (CloudFront)
via: 1.1 0712e4ad4264127dfcb76a114b130495.cloudfront.net (CloudFront)
< x-amz-cf-pop: IAD89-C3
x-amz-cf-pop: IAD89-C3
< x-amz-cf-id: hWrjwajqqkI9-rJnK1BSQqkX9DPXIlZJLfa28UaIeze7taBP5kqMNg==
x-amz-cf-id: hWrjwajqqkI9-rJnK1BSQqkX9DPXIlZJLfa28UaIeze7taBP5kqMNg==

<
* Connection #0 to host d1bci9l8cjf24o.cloudfront.net left intact

Fetching an asset from a CloudFront distribution using my *.ancientwarmth.com wildcard SSL certificate:

Shell

$ curl -Iv \
  https://assets.ancientwarmth.com/js/jquery.modal.min.js
modal.min.js>   https://assets.ancientwarmth.com/js/jquery.modal.min.js
  *   Trying 13.226.36.16:443...
  * TCP_NODELAY set
  * Connected to assets.ancientwarmth.com (13.226.36.16) port 443 (#0)
  * ALPN, offering h2
  * ALPN, offering http/1.1
  * successfully set certificate verify locations:
  *   CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: /etc/ssl/certs
  * TLSv1.3 (OUT), TLS handshake, Client hello (1):
  * TLSv1.3 (IN), TLS handshake, Server hello (2):
  * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  * TLSv1.3 (IN), TLS handshake, Certificate (11):
  * TLSv1.3 (IN), TLS handshake, CERT verify (15):
  * TLSv1.3 (IN), TLS handshake, Finished (20):
  * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  * TLSv1.3 (OUT), TLS handshake, Finished (20):
  * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
  * ALPN, server accepted to use h2
  * Server certificate:
  *  subject: CN=*.ancientwarmth.com
  *  start date: Mar 14 00:00:00 2021 GMT
  *  expire date: Apr 12 23:59:59 2022 GMT
  *  subjectAltName: host "assets.ancientwarmth.com" matched cert's "*.ancientwarmth.com"
  *  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
  *  SSL certificate verify ok.
  * Using HTTP2, server supports multi-use
  * Connection state changed (HTTP/2 confirmed)
  * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  * Using Stream ID: 1 (easy handle 0x5599720157e0)
  > GET /js/jquery.modal.min.js HTTP/2
  > Host: assets.ancientwarmth.com
  > user-agent: curl/7.68.0
  > accept: */*
  >
  * Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
  < HTTP/2 200
  < content-type: application/javascript
  < content-length: 4953
  < date: Sat, 20 Mar 2021 12:34:25 GMT
  < last-modified: Sat, 20 Mar 2021 03:14:08 GMT
  < etag: "c8f50397e0560719c62a35318f413e16"
  < accept-ranges: bytes
  < server: AmazonS3
  < x-cache: Hit from cloudfront
  < via: 1.1 4e3df844337032b56b8434990b0f76ca.cloudfront.net (CloudFront)
  < x-amz-cf-pop: EWR53-C2
  < x-amz-cf-id: 17Dxn6QqtK6JfkJwFnESVYsG-Cbzu6H-sOTWcGDpznGcpjIZbhJDRA==
  < age: 5195

  <
* Connection #0 to host assets.ancientwarmth.com left intact

Fetching an asset from an S3 bucket using an AWS SSL certificate for all S3 buckets in region us-east-1:

Shell

$ curl -Iv \
  https://s3.us-east-1.amazonaws.com/assets.ancientwarmth.com/js/jquery.modal.min.js
*   Trying 52.216.24.46:443...
* TCP_NODELAY set
* Connected to s3.us-east-1.amazonaws.com (52.216.24.46) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=s3.amazonaws.com
*  start date: Aug  4 00:00:00 2020 GMT
*  expire date: Aug  9 12:00:00 2021 GMT
*  subjectAltName: host "s3.us-east-1.amazonaws.com" matched cert's "s3.us-east-1.amazonaws.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2
*  SSL certificate verify ok.
> GET /assets.ancientwarmth.com/js/jquery.modal.min.js HTTP/1.1
> Host: s3.us-east-1.amazonaws.com
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< x-amz-id-2: xIXUHy7YBpjZaF+cpGoSAwNvC5+NrmM5pmJM8nInI6weEkbht350xSPC9+yOBJrGs9GY0hn2V7Y=
< x-amz-request-id: JM2K8HR109JNMMB1
< Date: Sat, 20 Mar 2021 14:03:56 GMT
< Last-Modified: Sat, 20 Mar 2021 03:14:08 GMT
< ETag: "c8f50397e0560719c62a35318f413e16"
< Accept-Ranges: bytes
< Content-Type: application/javascript
< Content-Length: 4953
< Server: AmazonS3

<
* Connection #0 to host s3.us-east-1.amazonaws.com left intact

Test the date range of the certificate with this incantation:

Shell

$ curl https://ancientwarmth.com -vI --stderr - | grep 'date:'
*  start date: Nov 25 17:46:07 2022 GMT
*  expire date: Feb 23 17:46:06 2023 GMT

Pretty JSON Reduces Errors and Fatigue

2021-02-23T00:00:00-05:00

I've been using jq to format my JSON for years. It is easy to format a JSON document, just pass it through jq without any options or arguments. Notice, however, that a lot of vertical space is wasted using this formatting:

Shell

$ jq < blog/colors.json
{
  "colors": [
    {
      "color": "black",
      "hex": "#000",
      "rgb": [
        0,
        0,
        0
      ]
    },
    {
      "color": "red",
      "hex": "#f00",
      "rgb": [
        255,
        0,
        0
      ]
    },
    {
      "color": "yellow",
      "hex": "#ff0",
      "rgb": [
        255,
        255,
        0
      ]
    },
    {
      "color": "green",
      "hex": "#0f0",
      "rgb": [
        0,
        255,
        0
      ]
    },
    {
      "color": "cyan",
      "hex": "#0ff",
      "rgb": [
        0,
        255,
        255
      ]
    },
    {
      "color": "blue",
      "hex": "#00f",
      "rgb": [
        0,
        0,
        255
      ]
    },
    {
      "color": "magenta",
      "hex": "#f0f",
      "rgb": [
        255,
        0,
        255
      ]
    },
    {
      "color": "white",
      "hex": "#fff",
      "rgb": [
        255,
        255,
        255
      ]
    }
  ]
}

After reading The Pretty JSON Revolution I decided to try the program the article mentioned, oj. oj is a Go program. Here is how I compiled it on Ubuntu:

Shell

$ yes | sudo apt install golang-go

$ go get github.com/ohler55/ojg

$ go get github.com/ohler55/ojg/cmd/oj

By default, compiled go projects are placed in the ~/go/bin/ directory. Here is how I added that directory the the PATH, and made an alias for invoking the program with the proper options for maximum prettiness:

Shell

$ echo "$HOME/go/bin/:$PATH" >> ~/.bashrc

$ echo "alias pprint='oj -i 2 -s -p 80.3'" >> ~/.bash_aliases

$ source ~/.bashrc

Pretty-printing the JSON in colors.json with oj is easy:

Shell

$ pprint colors.json
{
  "colors": [
    {"color": "black", "hex": "#000", "rgb": [0, 0, 0]},
    {"color": "red", "hex": "#f00", "rgb": [255, 0, 0]},
    {"color": "yellow", "hex": "#ff0", "rgb": [255, 255, 0]},
    {"color": "green", "hex": "#0f0", "rgb": [0, 255, 0]},
    {"color": "cyan", "hex": "#0ff", "rgb": [0, 255, 255]},
    {"color": "blue", "hex": "#00f", "rgb": [0, 0, 255]},
    {"color": "magenta", "hex": "#f0f", "rgb": [255, 0, 255]},
    {"color": "white", "hex": "#fff", "rgb": [255, 255, 255]}
  ]
}

I like it!

Give it to Mikey. He won't eat it. He hates everything!

I will continue to use jq for queries, but I'll use oj for pretty-printing from now on.

JavaScript Named Arguments and Class Constructors

2021-02-11T00:00:00-05:00

Named arguments make a program safe from errors caused by changes to method arguments. JavaScript named arguments can appear in any order. Default values for parameters allow an API to evolve gracefully without runtime errors.

Building on the article entitled Cool JavaScript 9: Named arguments — Functions that get and return Objects, this article shows how JavaScript class constructors can use named arguments, optionally define default values for parameters, and conveniently inflate new class instances from JSON.

In this article I use Node.js for convenience, however the code shown will run in all modern web browsers.

JavaScript Class Definition Encapsulating Properties

Let’s quickly review how to define a JavaScript class and instantiate an instance. Here is a simple JavaScript / ECMAScript 6 class that encapsulates two properties: id and parts. The constructor merely lists the names of the parameters, which happen to be the same as the names of the class properties.

Shell

$  js
Welcome to Node.js v12.18.2.
Type ".help" for more information.
> class Ingredient {
...   constructor(id, parts) {
.....     this.id = id;
.....     this.parts = parts;
.....   }
... }
undefined

New Ingredient instances can be created using this familiar syntax:

Shell

> var ingredient = new Ingredient("123", 10);
undefined 
> ingredient
Ingredient { id: '123', parts: 10 }

Object Literals

Object literals look like JSON objects, but without quotes around property names. For example, the following defines an object literal called lit with 2 properties, called id and parts, with values "123" and 10, respectively.

Shell

>  var lit = {id: "123", parts: 10};
undefined 
$ lit
{ id: '123', parts: 10 } 
> lit.id
'123' 
> lit.parts
10

Use Object Literals to Define Arguments

We can define a class similar to Ingredient, but with the arguments replaced by a something that looks like an object literal without values. For want of a better term, I call this an object name literal. The following class definition encapsulates the same two properties as before as an object name literal.

Shell

> class IngredientX {
...   constructor({id, parts}) {
.....     this.id = id;
.....     this.parts = parts;
.....   }
... }
undefined

New IngredientX instances can be created from an object literal:

Shell

> var ingredientX1 = new IngredientX({id: "123", parts: 10 });
undefined 
> ingredientX1
IngredientX { id: '123', parts: 10 }

Because the IngredientX class definition requires an object name literal (or a JSON object, more on that later) to provide constructor arguments, constructor invocations must specify the names of each parameter being passed to the constructor arguments. This has the benefit of making your software more robust in the face of changing method signatures.

Caution: new IngredientX instances cannot be created from scalar arguments. JavaScript gives no error or warning if you do not:

Shell

> var ingredientX2 = new IngredientX("123", 10);
undefined 
> ingredientX2
IngredientX { id: undefined, parts: undefined }

JSON Object Can Be Supplied Instead of Object Literals

JSON objects can be provided as arguments instead of object literals. This is extremely handy. Replacing several arguments with a JSON object would possibly be the most significant improvement in robustness that could be made to a JavaScript project. The number of runtime errors encountered as a code base evolves would be greatly reduced.

Shell

> var ingredientX3 = new IngredientX({
...     "id": "123",
...     "parts": 10
...  });
undefined 
> ingredientX3
IngredientX { id: '123', parts: 10 }

Arguments and Parameters Can Be Provided In Any Order

This definition of ingredientX4 is identical to the definition of ingredientX3, even though the order of the arguments has been reversed:

Shell

> var ingredientX4 = new IngredientX({
...     "parts": 10,
...     "id": "123"
...  });
undefined 
> ingredientX4
IngredientX { id: '123', parts: 10 }

The parameters in the function or method declaration are also insensitive to ordering:

Shell

> class IngredientXReordered {
...   constructor({parts, id}) {
.....     this.parts = parts;
.....     this.id = id;
.....   }
... }
undefined 
> var ingredientX5 = new IngredientXReordered({
...     "parts": 10,
...     "id": "123"
...  });
undefined 
> ingredientX5
IngredientXReordered { id: '123', parts: 10 }

Object Literals Can Be Used With Any Method

Object literals / named arguments can be used to define the signature of any function or method, not just class constructors. For example:

Shell

> class IngredientY {
...    constructor({id, parts}) {
.....      this.id = id;
.....      this.parts = parts;
.....    }
... 
...    mix({duration, intensity}) {
...      console.log(`Shake for ${duration} hours at intensity ${intensity}.`);
...    }
...  }
undefined 
> var ingredientY = new IngredientY({id: "123", parts: 10 });
undefined 
> ingredientY.mix({duration: 2.5, intensity: 2});
Shake for 2.5 hours at intensity 2. 
undefined

Default Values for Named Arguments

To make this example more interesting, the default value for id will be generated as a GUID. Here are some other GUID implementations, but the best implementations have dependencies and that would just make the article more complex than necessary.

Shell

> function uuidv4() {
...   return 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g,
...     function(c) {
.....     var r = Math.random() * 16 | 0, v = c == 'x' ? r : (r & 0x3 | 0x8);
.....     return v.toString(16);
.....   });
... }
undefined 
> uuidv4()
'b13137c1-1598-42ca-9498-c1502e5405ed'

A JavaScript object literal or JSON object must be passed to a method whose parameters were defined by object literal names. If a name/value pair is not provided in the argument, then the default parameter value is used. Some examples should help demonstrate how this works:

Shell

> class IngredientZ {
...    constructor({id=uuidv4(), parts=10}) {
.....      this.id = id;
.....      this.parts = parts;
.....    }
... 
...    mix({duration=1.2, intensity=6}) {
...      console.log(`Shake for ${duration} hours at intensity ${intensity}.`);
...    }
...  }
undefined 
> var ingredientZ1 = new IngredientZ({parts: 4});
undefined 
> ingredientZ1
IngredientZ { id: '4290dc1a-4f4c-4579-9e27-39b68085ad97', parts: 4 } 
undefined

Empty objects are allowed as arguments. All this means is that default values are used for all parameters of the object name literal.

Shell

> var ingredientZ2 = new IngredientZ({});
undefined 
> ingredientZ2
IngredientZ { id: '9e70dc12-1f4c-3579-6a17-49a68385bf73', parts: 10 } 
> ingredientZ2.mix({});
Shake for 2.5 hours at intensity 2.

Missing objects result in a syntax error.

Shell

> ingredientZ2.mix();
Uncaught TypeError: Cannot read property 'id' of undefined
    at new IngredientZ2 (repl:3:17)
    {% noselect undefined

For More Information

For more information, please see JavaScript for impatient programmers (ES2021 edition).

JavaScript Linter Configuration

2021-02-08T00:00:00-05:00

Using a lint tool can really help improve your code in a hurry. I am using JSHint for a project that has a big JavaScript file that needs some love.

.jshintrc

All modern web browsers support at least the version of JavaScript that conforms to ECMAScript 6th Edition, also known as ECMAScript 2015. Neither the documentation for the Atom linter-jshint plugin nor JSHint itself explicitly state that in order to work with the version of JavaScript supported by all modern web browsers, you need to provide a JSON formatted configuration file that sets the esversion property to 6, like this:

.jshintrc

{
  "esversion": 6
}

If you do not do this, then JSHint will indicate errors if it encounters class definitions, for example.

I put .jshintrc in the top-level directory of my project.

I created a pull request for the `linter-jshint` GitHub project so this documentation would be included.

Reading from stdin

The JSHint CLI docs say:

If a file path is a dash (-) then JSHint will read from standard input.

I needed to preprocess my JavaScript source files before invoking JSHint. Because JSHint can read from standard input, there is no need to write the preprocessed file contents to a temporary file.

Removing Jekyll Front Matter for JSHint

Jekyll can process any text file, including JavaScript files, if they contain front matter markers. This is useful for invoking Jekyll plugins and/or using Liquid expressions. My big JavaScript file has some information injected into it when Jekyll generates the site.

Front matter is marked (delimited by) by two lines at the top of a file, consisting of three dashes, like this:

Shell

---
---

Here is how the empty front matter can be stripped from myfile.js so JSHint can inspect the remaining lines:

Shell

$ sed '/---/d' < myfile.js | jshint -

Functional and Non-Functional E-Commerce Requirements

2021-02-02T00:00:00-05:00

In a previous article, I described how I searched for an open-source shopping cart and the disappointing software options that I found. These searches yielded software projects that had begun 20 years ago and were generally of low quality.

The businesses that manage these projects use a failed business model for open source, namely software-as-a-service (SaaS), without much added value. The hosted products have not changed much since they were first established, and they have not kept up with the relentless advances in computer technology. Those businesses derive some additional revenue from customizing the open-source projects.

I decided to be more rigorous in my requirements analysis for a shopping cart with good coupon support so that I could find more suitable options. I now know that the cart must have:

Strong discount/coupon support.
Support for SKUs defined on-the-fly by customers as they interact with the website.
Integration with various payment processors.

Before I share my reviews of candidate shopping carts with you, let’s first discuss the state of the e-commerce market, functional and non-functional requirements, the master/detail pattern, and some technology trends for business software.

E-Commerce Is Hot, Hot, Hot

Online spending was $861 billion: 21% of total retail sales' class="imgImg rounded shadow" src="/blog/images/django/usEcommerce.png" style='width: 100%; ' title='US ecommerce grows 44% in 2020
Online spending was $861 billion: 21% of total retail sales' />

US ecommerce grows 44% in 2020
Online spending was $861 billion: 21% of total retail sales

Source: Digital Commerce 360, U.S. Department of Commerce; January 2021 ' class="imgImg rounded shadow" src="/blog/images/ecommerce/ecommerceGrowth.png" style='width: 100%; ' title=' US e-commerce vs. retail sales, 2010-2020
Source: Digital Commerce 360, U.S. Department of Commerce; January 2021 ' />

US e-commerce vs. retail sales, 2010-2020
Source: Digital Commerce 360, U.S. Department of Commerce; January 2021

There are a huge number of e-commerce sites, and that market is experiencing strong growth in part due to the COVID-19 epidemic. E-commerce has entered Main Street, as per Geoffrey Moore’s technology adoption lifecycle.

“Crossing the Chasm”, which introduced the technology adoption lifecycle

Functional Requirements

Functional requirements describe what the system must or must not do, and can be thought of in terms of how the system responds to inputs. Functional requirements usually define if/then behaviors and include calculations, data input, and business processes.

Functional requirements are features that allow the system to function as intended. In other words, if the functional requirements are not met, the system will not work. Functional requirements are product features that focus on user requirements.

– Paraphrased from QRA Corp’s definition

Functionally, a shopping cart is well understood, yet to be certain that a shopping cart actually works properly, test data and test procedures would need to be designed to verify that all the corner cases were exercised. That would be a significant amount of work to do properly, but at least the work would be rather straightforward.

Non-functional requirements (NFRs)

Non-functional requirements relate to user expectations and include security, reliability, performance, maintainability, scalability, and usability.

An NFR is a requirement that specifies criteria that can be used to judge the operation of a system, rather than specific behaviors.

The plan for implementing NFRs is detailed in the system architecture because they are usually architecturally significant requirements.

Broadly, functional requirements define what a system is supposed to do, and NFRs define how a system is supposed to be.

NFRs can be divided into two main categories:

Execution qualities, such as safety, security and usability, which are observable during operation (at run time).
Evolution qualities, such as testability, maintainability, extensibility and scalability, which are embodied in the static structure of the system.

– Paraphrased excerpts from the Wikipedia article on NFRs

I need a shopping cart that has been properly tested, is flexible to configure, and is easy to use. These non-functional requirements are more important to me than the mechanics of how the cart was built or the technology that was used, especially for a proof of concept. In my reviews of shopping cart candidates, I will focus on how well they fulfill these NFRs.

I also want to use technology that is current, properly maintained, full-featured and has a future.

Master-Detail Structures, Interfaces and Reporting

Shopping carts are a classic example of a master-detail structure. The user interface of a shopping cart must exploit the master-detail paradigm effectively.

Image from Oracle Alta UI Patterns: Master-Detail

A master–detail relationship is a one-to-many type of relationship.

Examples of a master-detail relationship are:

A set of purchase orders and a set of line items belonging to each purchase order.
An expense report with a set of expense line items or a department with a list of employees belonging to it.

An application can use this master-detail relationship to enable users to navigate through the purchase order data and see the detail data for line items only related to the master purchase order selected.

– From the Wikipedia article on master/detail data models

In the book SAP BusinessObjects BI 4.0 The Complete Reference 3/E, authors Cindi Howson and Elizabeth Newbould provide this rather abstract definition of master/detail reports:

A master/detail report is a particular kind of report in which a dimension value (master) is used to group data (detail) into separate sections. Master/detail reports allow you to analyze and format data for each unique master data value.

Page 13 of Oracle® Performance Tuning for 10gR2, Second Edition by Gavin JT Powell has a more concrete definition:

First normal form removes repetition by creating one-to-many relationships. Data repeated many times in one entity is removed to a subset entity, which becomes the container for the removed repeating data. Each row in the subset entity will contain a single reference to each row in the original entity. The original entity will then contain only nonduplicated data. This one-to-many relationship is commonly known as a master-detail relationship, where repeating columns are removed to a new entity. The new entity gets a primary key consisting of a composite of the primary key in the master entity and a unique identifier (within each master primary key) on the detail entity.

Master-detail is such a common architectural pattern that most business software vendors provide support for it. Other vendors include IBM, Oracle, Salesforce, and Microsoft. Apple’s iPad is used as a client for master-detail applications so often that the iOS SDK even provides a Master-Detail Application template. The master/detail pattern for Google Android applications can be implemented using the Material Design visual language.

Master-Detail Frameworks

Shopping carts must have engaging and effective implementations for master/detail patterns throughout: in the user interface, in the design of reports, in the data structures, in the types of software modules and their interfaces to each other, and in the internal data flow. The best-known web framework explicitly designed to support the master/detail pattern is Ruby on Rails. Of course, many other web frameworks can support master/detail.

Ruby on Rails

Momentum

Web Servers

Web servers are usually placed in front of e-commerce servers for scalability and security reasons. Netcraft has been reporting on web server deployments since 1995. The Netcraft January 2021 Web Server Survey shows some interesting trends.

Apache httpd is losing ground, as developers move to Microsoft and nginx.' class="imgImg rounded shadow" src="/blog/images/ecommerce/wss-active-share.png" style='width: 100%; ' title='Web server developer market share by server type.
Apache httpd is losing ground, as developers move to Microsoft and nginx.' />

Web server developer market share by server type.
Apache httpd is losing ground, as developers move to Microsoft and nginx.

For the busiest 1 million websites, Microsoft has taken the lead.

Microsoft’s ASP.NET web framework is free, and web hosting is free (to a point), but those enticements fulfill their purpose by locking in customers to Microsoft's software stack and tools. The nopCommerce e-commerce server is free, very capable, and has enjoyed terrific adoption. This technology might be a good business decision for e-commerce sites that have typical requirements, but not if significant customization or integration with non-Microsoft services becomes important. This article was written by an experienced developer: I rebuilt the same web API using Express, Flask, and ASP.NET. Here’s what I found.

Computer Languages

Ingenuity.' class="imgImg rounded shadow" src="/blog/images/ecommerce/mars_drone.png" style='width: 100%; ' title='Python runs Mars drone Ingenuity.' />

Python runs Mars drone Ingenuity.

Of all computer languages, Python has arguably the most momentum at present. Ruby on Rails is terrific, but its market share has dropped dramatically since 2011, and without that framework, the Ruby language would probably be much less important than it is. Python has been growing in all directions for a very long time. Microsoft’s C# language, which requires a .NET compatible runtime, is also popular but has never had the momentum that Python has.

The following graph compares the popularity of the Python Django library for making web applications against the popularity of the Ruby on Rails framework for the time period 2009 to the present day. As you can see, in 2011, Ruby on Rails was at its peak popularity; at that time, it was 300% more popular than Django. Now the situation is reversed: today, Django is 400% more popular than Ruby on Rails.

Stack Overflow Trends: Django vs Ruby on Rails

The one thing about Python that I like better than all other languages is the “batteries included” feature-driven approach. This means that Python projects can aspire to more ambitious goals for a given amount of programmer effort as compared to projects implemented in other languages. That was a significant factor in my decision to focus on Python-powered semi-custom shopping carts instead of investigating shopping carts programmed with other computer languages. Ruby on Rails would doubtless provide an excellent foundation for shopping carts, but I think Python is a better strategic choice for me at now in the world of open source.

This is not a decision I would have made in years past. Python's runtime consumes a lot more electrical power than the runtimes of other computer languages. However, Python 3.11 is 25% faster than previus versions. As available computing power continues to grow year-on-year within devices everywhere, the “Python runtime tax” has become much easier to bear.

The Python Language

Beyond Open Source

Most open-source software suffers from limited funding and conflicts of interest. This seriously impacts long-term viability. I am happy to consider all appropriate technology, and I am not fixated on open-source options. Two major technical components need to be considered:

E-commerce framework: this dictates the computer language and runtime library.
Database: the e-commerce framework often dictates the choice of database.

Monolithic vs Serverless Architectures

Modular Yet Monolithic Architectures

Monoliths on Easter Island

Traditionally, software has been built by first constructing functional modules, then combining the modules into a complete program (the monolith). For e-commerce, entire programs are deployed to production by adding configuration information and integrating with external services, such as payment processors.

This works well; however, the result is a centralized computing resource that has a fixed capability and resource requirements. The expense of operating the software does not change much, regardless of the traffic volume.

To handle periods of heavy traffic, complex mechanisms are required to scale up transactional capacity. Conversely, during periods of light traffic, the minimum cost to maintain the system in an operational state can be significant, especially for a startup company.

Serverless Architecture

Martin Fowler on Serverless Architectures

One of the big architectural advances recently has been the introduction of serverless computing. It provides near-infinite scalability without paying for unused resources. CloudFlare has a good definition:

Serverless computing is a method of providing backend services on an as-needed basis. A serverless provider allows users to write and deploy code without worrying about the underlying infrastructure. A company that gets backend services from a serverless vendor is charged based on their computation and does not have to reserve and pay for a fixed amount of bandwidth or number of servers, as the service is auto-scaling. Note that despite the name serverless, physical servers are still used, but developers do not need to be aware of them.

Vendors that provide serverless computing platforms include AWS Lambda, Azure Functions, CloudFlare Workers, Google Cloud Functions

The Serverless Framework is a language- and platform-agnostic framework. Languages supported include Node.js, Python, Java, Go, C#, Ruby, Swift, Kotlin, PHP, Scala, & F#. Platforms supported include Alibaba Cloud, AWS, Microsoft Azure, Fn Project, Google Cloud Platform, Apache OpenWhisk, CloudFlare Workers, Knative, Kubeless, Spotinst and Tencent Cloud. The GitHub project has the code.

Responsive Web Pages

Image from ‘Best Practices of Responsive Web Design’ by Bradley Nice

Responsive web pages alter their layout and appearance to suit different screen widths and resolutions. Many of the shopping carts I looked at did not support responsive web pages.

Next: Check Out Django Shopping Carts

Python, and therefore Django, is winning. I admit that I like the Django slogan: “Django – The web framework for perfectionists with deadlines”. When an option dominates the competition, you would need an exceptional reason to consider other options. My functional and non-functional requirements are mainstream, so next I will check out the leading option: Django.

OpenCart - Postgres - ngnix - Ubuntu

2021-01-30T00:00:00-05:00

I needed a shopping cart that has good coupon and discount support and flexible pricing. My requirements were unique in that each item in the cart might be a custom product, with the price computed according to a formula on the server. The store had few standard SKUs.

I started to look into three options for obtaining a shopping cart with good coupon support: building it myself, using a commercial product, or customizing an open-source project. This article is the story of the ‘customize an open source’ track.

OpenCart

I wanted to evaluate the leading open-source shopping cart contender by installing it on a development machine and giving it real data. OpenCart is renowned as one of the better open-source shopping carts available today. As with many open-source projects, the company that provides the source code has a conflict of interest: if they make installing and configuring the software effortless, then their revenue would be much less than if they had a cadre of interested but frustrated developers. I looked at the hosting options and did not like their value or customizability.

OpenCart shows its age by using MySQL and its descendants, like Maria. Long ago, I moved on from MySQL to Postgres, and I have been pleased with Postgres. I decided to try to make OpenCart run on Postgres and Ubuntu. I knew before I started that OpenCart’s support for Postgres was weak.

Spoiler: OpenCart would fail a quality review

If you find watching videos of gory slow-motion accidents entertaining, please read on.

Unit Tests

The first thing I look for when evaluating software for possible incorporation into a project is unit tests.

Shell

$ git grep -Ii test \
  ":(exclude)*.css" \
  ":(exclude)*.yml" \
  ":(exclude)*.js"

$ git grep -IiE 'phptest|Codeception' \
  ":(exclude)*.css" \
  ":(exclude)*.yml" \
  ":(exclude)*.js"

There were no tests and no references to PHPUnit or Codeception. This was a big black mark against OpenCart.

Install Dependencies

If you need information about PostgreSQL, here is the mother ship. This is a good description of how to install Postgres.

I use PGAdmin when I want a graphical interface to PostgreSQL. Installation instructions are here. In a nutshell:

Shell

$ curl https://www.pgadmin.org/static/packages_pgadmin_org.pub | \
  sudo apt-key add

$ sudo sh -c \
  'echo "deb https://ftp.postgresql.org/pub/pgadmin/pgadmin4/apt/$(lsb_release -cs) pgadmin4 main" >
  /etc/apt/sources.list.d/pgadmin4.list && apt update'

$ yes | sudo apt install pgadmin4

I installed the rest of the dependencies like this:

Shell

$ yes | sudo apt install \
    postgresql postgresql-contrib \
    software-properties-common lynx \
    php7.4 php-fpm php-gd php-curl php-postgre php-zip \
    php5-pgsql

phpenmod is a Debian / Ubuntu command for enabling PHP extensions.

Shell

$ sudo phpenmod pgsql

I verified that the desired PHP extensions were installed typing php -m.

Shell

$ php -m | grep pg
pdo_pgsql
pgsql

Figuring Out Problems

I also installed the PHP debugger when I realized I needed to do some debugging. IntelliJ (which is the big brother of PHP Storm) did a terrific job of providing debug capability for command-line PHP.

Shell

$ yes | sudo apt install php-xdebug

I opened a new console and continuously viewed the nginx, PHP and PostgreSQL error logs. This was a big help whenever I needed to figure out problems.

Shell

$ tail -f \
  /var/log/nginx/*.log \
  /var/log/php*.log \
  /var/log/postgresql/postgresql-12*.log

Configure nginx

The only change I made to /etc/nginx/nginx.conf was to change the default MIME type from application/octet-stream to text/html. The change is highlighted in yellow; scroll the code to see it.

/etc/nginx/nginx.conf

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
	worker_connections 768;
	# multi_accept on;
}

http {

	##
	# Basic Settings
	##

	sendfile on;
	tcp_nopush on;
	types_hash_maxsize 2048;
	# server_tokens off;

	# server_names_hash_bucketsize 64;
	# server_name_in_redirect off;

	include /etc/nginx/mime.types;
	#default_type application/octet-stream;
	default_type text/html;

	##
	# SSL Settings
	##

	ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
	ssl_prefer_server_ciphers on;

	##
	# Logging Settings
	##

	access_log /var/log/nginx/access.log;
	error_log /var/log/nginx/error.log notice;

	##
	# Gzip Settings
	##

	gzip on;

	# gzip_vary on;
	# gzip_proxied any;
	# gzip_comp_level 6;
	# gzip_buffers 16 8k;
	# gzip_http_version 1.1;
	# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

	##
	# Virtual Host Configs
	##

	include /etc/nginx/conf.d/*.conf;
	include /etc/nginx/sites-enabled/*;
}


#mail {
#	# See sample authentication script at:
#	# http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
#	# auth_http localhost/auth.php;
#	# pop3_capabilities "TOP" "USER";
#	# imap_capabilities "IMAP4rev1" "UIDPLUS";
#
#	server {
#		listen     localhost:110;
#		protocol   pop3;
#		proxy      on;
#	}
#
#	server {
#		listen     localhost:143;
#		protocol   imap;
#		proxy      on;
#	}
#}

I deleted /etc/nginx/sites-enabled/default and replaced it with /etc/nginx/sites-enabled/php so PHP files would be parsed properly, no matter what directory they resided in.

/etc/nginx/sites-enabled/php

# See https://tecadmin.net/setup-nginx-php-fpm-on-ubuntu-20-04/

server {
        listen 80;
        root /var/www/html;
        index index.php index.html index.htm;
        server_name example.com;

        location / {
            try_files $uri $uri/ =404;
        }

        location ~ \.php$ {
            include snippets/fastcgi-php.conf;
            fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
        }
}

Now it was time to restart nginx:

Shell

$ sudo systemctl restart nginx.service

Verify Services Are Running

Shell

$ sudo systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-01-29 16:58:04 EST; 15h ago
       Docs: man:nginx(8)
   Main PID: 1584845 (nginx)
      Tasks: 9 (limit: 38389)
     Memory: 10.0M
     CGroup: /system.slice/nginx.service
             ├─1584845 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
             ├─1584846 nginx: worker process
             ├─1584847 nginx: worker process
             ├─1584848 nginx: worker process
             ├─1584849 nginx: worker process
             ├─1584850 nginx: worker process
             ├─1584851 nginx: worker process
             ├─1584852 nginx: worker process
             └─1584853 nginx: worker process

Jan 29 16:58:04 localhost systemd[1]: Starting A high performance web server and a reverse proxy server...
Jan 29 16:58:04 localhost systemd[1]: Started A high performance web server and a reverse proxy server.

Shell

$ sudo systemctl status php7.4-fpm
● php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
     Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-01-30 08:32:36 EST; 10min ago
       Docs: man:php-fpm7.4(8)
    Process: 3430785 ExecStartPost=/usr/lib/php/php-fpm-socket-helper install /run/php/php-fpm.sock /etc/php/7.4/fp>
   Main PID: 3430782 (php-fpm7.4)
     Status: "Processes active: 0, idle: 2, Requests: 0, slow: 0, Traffic: 0req/sec"
      Tasks: 3 (limit: 38389)
     Memory: 8.1M
     CGroup: /system.slice/php7.4-fpm.service
             ├─3430782 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)
             ├─3430783 php-fpm: pool www
             └─3430784 php-fpm: pool www

Jan 30 08:32:36 localhost systemd[1]: Starting The PHP 7.4 FastCGI Process Manager...
Jan 30 08:32:36 localhost systemd[1]: Started The PHP 7.4 FastCGI Process Manager.

Verifing That PHP Works

I made the file /var/www/html/info.php, which is very common when working with PHP. Just ensure that it does not appear on your production site, or hackers will know more about your website than they should.

/var/www/html/info.php

<?php phpinfo(); ?>

Now, I verified that PHP worked by viewing information about the setup. The -I option causes curl to just return HTML headers, not the HTML body.

Shell

$ curl -I http://localhost/info.php
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Sat, 30 Jan 2021 18:50:30 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive

View info.php in a web browser to see the details. Lynx is good for that from the command line:

Shell

$ lynx http://localhost/info.php

Set Up PostgreSQL

I only changed the value of listen_addresses in postgresql.conf. Again, this change is highlighted in yellow; scroll down to see it.

/etc/postgresql/12/main/postgresql.conf

# -----------------------------
# PostgreSQL configuration file
# -----------------------------
#
# This file consists of lines of the form:
#
#   name = value
#
# (The "=" is optional.)  Whitespace may be used.  Comments are introduced with
# "#" anywhere on a line.  The complete list of parameter names and allowed
# values can be found in the PostgreSQL documentation.
#
# The commented-out settings shown in this file represent the default values.
# Re-commenting a setting is NOT sufficient to revert it to the default value;
# you need to reload the server.
#
# This file is read on server startup and when the server receives a SIGHUP
# signal.  If you edit the file on a running system, you have to SIGHUP the
# server for the changes to take effect, run "pg_ctl reload", or execute
# "SELECT pg_reload_conf()".  Some parameters, which are marked below,
# require a server shutdown and restart to take effect.
#
# Any parameter can also be given as a command-line option to the server, e.g.,
# "postgres -c log_connections=on".  Some parameters can be changed at run time
# with the "SET" SQL command.
#
# Memory units:  kB = kilobytes        Time units:  ms  = milliseconds
#                MB = megabytes                     s   = seconds
#                GB = gigabytes                     min = minutes
#                TB = terabytes                     h   = hours
#                                                   d   = days


#------------------------------------------------------------------------------
# FILE LOCATIONS
#------------------------------------------------------------------------------

# The default values of these variables are driven from the -D command-line
# option or PGDATA environment variable, represented here as ConfigDir.

data_directory = '/var/lib/postgresql/12/main'          # use data in another directory
                                        # (change requires restart)
hba_file = '/etc/postgresql/12/main/pg_hba.conf'        # host-based authentication file
                                        # (change requires restart)
ident_file = '/etc/postgresql/12/main/pg_ident.conf'    # ident configuration file
                                        # (change requires restart)

# If external_pid_file is not explicitly set, no extra PID file is written.
external_pid_file = '/var/run/postgresql/12-main.pid'                   # write an extra PID file
                                        # (change requires restart)


#------------------------------------------------------------------------------
# CONNECTIONS AND AUTHENTICATION
#------------------------------------------------------------------------------

# - Connection Settings -

listen_addresses = '*'
#listen_addresses = 'localhost'         # what IP address(es) to listen on;
                                        # comma-separated list of addresses;
                                        # defaults to 'localhost'; use '*' for all
                                        # (change requires restart)
port = 5432                             # (change requires restart)
max_connections = 100                   # (change requires restart)
#superuser_reserved_connections = 3     # (change requires restart)
unix_socket_directories = '/var/run/postgresql' # comma-separated list of directories
                                        # (change requires restart)
#unix_socket_group = ''                 # (change requires restart)
#unix_socket_permissions = 0777         # begin with 0 to use octal notation
                                        # (change requires restart)
#bonjour = off                          # advertise server via Bonjour
                                        # (change requires restart)
#bonjour_name = ''                      # defaults to the computer name
                                        # (change requires restart)

# - TCP settings -
# see "man 7 tcp" for details

#tcp_keepalives_idle = 0                # TCP_KEEPIDLE, in seconds;
                                        # 0 selects the system default
#tcp_keepalives_interval = 0            # TCP_KEEPINTVL, in seconds;
                                        # 0 selects the system default
#tcp_keepalives_count = 0               # TCP_KEEPCNT;
                                        # 0 selects the system default
#tcp_user_timeout = 0                   # TCP_USER_TIMEOUT, in milliseconds;
                                        # 0 selects the system default

# - Authentication -

#authentication_timeout = 1min          # 1s-600s
#password_encryption = md5              # md5 or scram-sha-256
#db_user_namespace = off

# GSSAPI using Kerberos
#krb_server_keyfile = ''
#krb_caseins_users = off

# - SSL -

#ssl = on
#ssl_ca_file = ''
ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem'
#ssl_crl_file = ''
ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key'
#ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers
#ssl_prefer_server_ciphers = on
#ssl_ecdh_curve = 'prime256v1'
#ssl_min_protocol_version = 'TLSv1'
#ssl_max_protocol_version = ''
#ssl_dh_params_file = ''
#ssl_passphrase_command = ''
#ssl_passphrase_command_supports_reload = off


#------------------------------------------------------------------------------
# RESOURCE USAGE (except WAL)
#------------------------------------------------------------------------------

# - Memory -

shared_buffers = 128MB                  # min 128kB
                                        # (change requires restart)
#huge_pages = try                       # on, off, or try
                                        # (change requires restart)
#temp_buffers = 8MB                     # min 800kB
#max_prepared_transactions = 0          # zero disables the feature
                                        # (change requires restart)
# Caution: it is not advisable to set max_prepared_transactions nonzero unless
# you actively intend to use prepared transactions.
#work_mem = 4MB                         # min 64kB
#maintenance_work_mem = 64MB            # min 1MB
#autovacuum_work_mem = -1               # min 1MB, or -1 to use maintenance_work_mem
#max_stack_depth = 2MB                  # min 100kB
#shared_memory_type = mmap              # the default is the first option
                                        # supported by the operating system:
                                        #   mmap
                                        #   sysv
                                        #   windows
                                        # (change requires restart)
dynamic_shared_memory_type = posix      # the default is the first option
                                        # supported by the operating system:
                                        #   posix
                                        #   sysv
                                        #   windows
                                        #   mmap
                                        # (change requires restart)

# - Disk -

#temp_file_limit = -1                   # limits per-process temp file space
                                        # in kB, or -1 for no limit

# - Kernel Resources -

#max_files_per_process = 1000           # min 25
                                        # (change requires restart)

# - Cost-Based Vacuum Delay -

#vacuum_cost_delay = 0                  # 0-100 milliseconds (0 disables)
#vacuum_cost_page_hit = 1               # 0-10000 credits
#vacuum_cost_page_miss = 10             # 0-10000 credits
#vacuum_cost_page_dirty = 20            # 0-10000 credits
#vacuum_cost_limit = 200                # 1-10000 credits

# - Background Writer -

#bgwriter_delay = 200ms                 # 10-10000ms between rounds
#bgwriter_lru_maxpages = 100            # max buffers written/round, 0 disables
#bgwriter_lru_multiplier = 2.0          # 0-10.0 multiplier on buffers scanned/round
#bgwriter_flush_after = 512kB           # measured in pages, 0 disables

# - Asynchronous Behavior -

#effective_io_concurrency = 1           # 1-1000; 0 disables prefetching
#max_worker_processes = 8               # (change requires restart)
#max_parallel_maintenance_workers = 2   # taken from max_parallel_workers
#max_parallel_workers_per_gather = 2    # taken from max_parallel_workers
#parallel_leader_participation = on
#max_parallel_workers = 8               # maximum number of max_worker_processes that
                                        # can be used in parallel operations
#old_snapshot_threshold = -1            # 1min-60d; -1 disables; 0 is immediate
                                        # (change requires restart)
#backend_flush_after = 0                # measured in pages, 0 disables


#------------------------------------------------------------------------------
# WRITE-AHEAD LOG
#------------------------------------------------------------------------------

# - Settings -

#wal_level = replica                    # minimal, replica, or logical
                                        # (change requires restart)
#fsync = on                             # flush data to disk for crash safety
                                        # (turning this off can cause
                                        # unrecoverable data corruption)
#synchronous_commit = on                # synchronization level;
                                        # off, local, remote_write, remote_apply, or on
#wal_sync_method = fsync                # the default is the first option
                                        # supported by the operating system:
                                        #   open_datasync
                                        #   fdatasync (default on Linux)
                                        #   fsync
                                        #   fsync_writethrough
                                        #   open_sync
#full_page_writes = on                  # recover from partial page writes
#wal_compression = off                  # enable compression of full-page writes
#wal_log_hints = off                    # also do full page writes of non-critical updates
                                        # (change requires restart)
#wal_init_zero = on                     # zero-fill new WAL files
#wal_recycle = on                       # recycle WAL files
#wal_buffers = -1                       # min 32kB, -1 sets based on shared_buffers
                                        # (change requires restart)
#wal_writer_delay = 200ms               # 1-10000 milliseconds
#wal_writer_flush_after = 1MB           # measured in pages, 0 disables

#commit_delay = 0                       # range 0-100000, in microseconds
#commit_siblings = 5                    # range 1-1000

# - Checkpoints -

#checkpoint_timeout = 5min              # range 30s-1d
max_walsize = 1GB
min_walsize = 80MB
#checkpoint_completion_target = 0.5     # checkpoint target duration, 0.0 - 1.0
#checkpoint_flush_after = 256kB         # measured in pages, 0 disables
#checkpoint_warning = 30s               # 0 disables

# - Archiving -

#archive_mode = off             # enables archiving; off, on, or always
                                # (change requires restart)
#archive_command = ''           # command to use to archive a logfile segment
                                # placeholders: %p = path of file to archive
                                #               %f = file name only
                                # e.g. 'test ! -f /mnt/server/archivedir/%f && cp %p /mnt/server/archivedir/%f'
#archive_timeout = 0            # force a logfile segment switch after this
                                # number of seconds; 0 disables

# - Archive Recovery -

# These are only used in recovery mode.

#restore_command = ''           # command to use to restore an archived logfile segment
                                # placeholders: %p = path of file to restore
                                #               %f = file name only
                                # e.g. 'cp /mnt/server/archivedir/%f %p'
                                # (change requires restart)
#archive_cleanup_command = ''   # command to execute at every restartpoint
#recovery_end_command = ''      # command to execute at completion of recovery

# - Recovery Target -

# Set these only when performing a targeted recovery.

#recovery_target = ''           # 'immediate' to end recovery as soon as a
                                # consistent state is reached
                                # (change requires restart)
#recovery_target_name = ''      # the named restore point to which recovery will proceed
                                # (change requires restart)
#recovery_target_time = ''      # the time stamp up to which recovery will proceed
                                # (change requires restart)
#recovery_target_xid = ''       # the transaction ID up to which recovery will proceed
                                # (change requires restart)
#recovery_target_lsn = ''       # the WAL LSN up to which recovery will proceed
                                # (change requires restart)
#recovery_target_inclusive = on # Specifies whether to stop:
                                # just after the specified recovery target (on)
                                # just before the recovery target (off)
                                # (change requires restart)
#recovery_target_timeline = 'latest'    # 'current', 'latest', or timeline ID
                                # (change requires restart)
#recovery_target_action = 'pause'       # 'pause', 'promote', 'shutdown'
                                # (change requires restart)


#------------------------------------------------------------------------------
# REPLICATION
#------------------------------------------------------------------------------

# - Sending Servers -

# Set these on the master and on any standby that will send replication data.

#max_wal_senders = 10           # max number of walsender processes
                                # (change requires restart)
#wal_keep_segments = 0          # in logfile segments; 0 disables
#wal_sender_timeout = 60s       # in milliseconds; 0 disables

#max_replication_slots = 10     # max number of replication slots
                                # (change requires restart)
#track_commit_timestamp = off   # collect timestamp of transaction commit
                                # (change requires restart)

# - Master Server -

# These settings are ignored on a standby server.

#synchronous_standby_names = '' # standby servers that provide sync rep
                                # method to choose sync standbys, number of sync standbys,
                                # and comma-separated list of application_name
                                # from standby(s); '*' = all
#vacuum_defer_cleanup_age = 0   # number of xacts by which cleanup is delayed

# - Standby Servers -

# These settings are ignored on a master server.

#primary_conninfo = ''                  # connection string to sending server
                                        # (change requires restart)
#primary_slot_name = ''                 # replication slot on sending server
                                        # (change requires restart)
#promote_trigger_file = ''              # file name whose presence ends recovery
#hot_standby = on                       # "off" disallows queries during recovery
                                        # (change requires restart)
#max_standby_archive_delay = 30s        # max delay before canceling queries
                                        # when reading WAL from archive;
                                        # -1 allows indefinite delay
#max_standby_streaming_delay = 30s      # max delay before canceling queries
                                        # when reading streaming WAL;
                                        # -1 allows indefinite delay
#wal_receiver_status_interval = 10s     # send replies at least this often
                                        # 0 disables
#hot_standby_feedback = off             # send info from standby to prevent
                                        # query conflicts
#wal_receiver_timeout = 60s             # time that receiver waits for
                                        # communication from master
                                        # in milliseconds; 0 disables
#wal_retrieve_retry_interval = 5s       # time to wait before retrying to
                                        # retrieve WAL after a failed attempt
#recovery_min_apply_delay = 0           # minimum delay for applying changes during recovery

# - Subscribers -

# These settings are ignored on a publisher.

#max_logical_replication_workers = 4    # taken from max_worker_processes
                                        # (change requires restart)
#max_sync_workers_per_subscription = 2  # taken from max_logical_replication_workers


#------------------------------------------------------------------------------
# QUERY TUNING
#------------------------------------------------------------------------------

# - Planner Method Configuration -

#enable_bitmapscan = on
#enable_hashagg = on
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
#enable_parallel_append = on
#enable_seqscan = on
#enable_sort = on
#enable_tidscan = on
#enable_partitionwise_join = off
#enable_partitionwise_aggregate = off
#enable_parallel_hash = on
#enable_partition_pruning = on

# - Planner Cost Constants -

#seq_page_cost = 1.0                    # measured on an arbitrary scale
#random_page_cost = 4.0                 # same scale as above
#cpu_tuple_cost = 0.01                  # same scale as above
#cpu_index_tuple_cost = 0.005           # same scale as above
#cpu_operator_cost = 0.0025             # same scale as above
#parallel_tuple_cost = 0.1              # same scale as above
#parallel_setup_cost = 1000.0   # same scale as above

#jit_above_cost = 100000                # perform JIT compilation if available
                                        # and query more expensive than this;
                                        # -1 disables
#jit_inline_above_cost = 500000         # inline small functions if query is
                                        # more expensive than this; -1 disables
#jit_optimize_above_cost = 500000       # use expensive JIT optimizations if
                                        # query is more expensive than this;
                                        # -1 disables

#min_parallel_table_scansize = 8MB
#min_parallel_index_scansize = 512kB
#effective_cachesize = 4GB

# - Genetic Query Optimizer -

#geqo = on
#geqo_threshold = 12
#geqo_effort = 5                        # range 1-10
#geqo_poolsize = 0                     # selects default based on effort
#geqo_generations = 0                   # selects default based on effort
#geqo_selection_bias = 2.0              # range 1.5-2.0
#geqo_seed = 0.0                        # range 0.0-1.0

# - Other Planner Options -

#default_statistics_target = 100        # range 1-10000
#constraint_exclusion = partition       # on, off, or partition
#cursor_tuple_fraction = 0.1            # range 0.0-1.0
#from_collapse_limit = 8
#join_collapse_limit = 8                # 1 disables collapsing of explicit
                                        # JOIN clauses
#force_parallel_mode = off
#jit = on                               # allow JIT compilation
#plan_cache_mode = auto                 # auto, force_generic_plan or
                                        # force_custom_plan


#------------------------------------------------------------------------------
# REPORTING AND LOGGING
#------------------------------------------------------------------------------

# - Where to Log -

#log_destination = 'stderr'             # Valid values are combinations of
                                        # stderr, csvlog, syslog, and eventlog,
                                        # depending on platform.  csvlog
                                        # requires logging_collector to be on.

# This is used when logging to stderr:
#logging_collector = off                # Enable capturing of stderr and csvlog
                                        # into log files. Required to be on for
                                        # csvlogs.
                                        # (change requires restart)

# These are only used if logging_collector is on:
#log_directory = 'log'                  # directory where log files are written,
                                        # can be absolute or relative to PGDATA
#log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'        # log file name pattern,
                                        # can include strftime() escapes
#log_file_mode = 0600                   # creation mode for log files,
                                        # begin with 0 to use octal notation
#log_truncate_on_rotation = off         # If on, an existing log file with the
                                        # same name as the new log file will be
                                        # truncated rather than appended to.
                                        # But such truncation only occurs on
                                        # time-driven rotation, not on restarts
                                        # or size-driven rotation.  Default is
                                        # off, meaning append to existing files
                                        # in all cases.
#log_rotation_age = 1d                  # Automatic rotation of logfiles will
                                        # happen after that time.  0 disables.
#log_rotationsize = 10MB               # Automatic rotation of logfiles will
                                        # happen after that much log output.
                                        # 0 disables.

# These are relevant when logging to syslog:
#syslog_facility = 'LOCAL0'
#syslog_ident = 'postgres'
#syslog_sequence_numbers = on
#syslog_split_messages = on

# This is only relevant when logging to eventlog (win32):
# (change requires restart)
#event_source = 'PostgreSQL'

# - When to Log -

#log_min_messages = warning             # values in order of decreasing detail:
                                        #   debug5
                                        #   debug4
                                        #   debug3
                                        #   debug2
                                        #   debug1
                                        #   info
                                        #   notice
                                        #   warning
                                        #   error
                                        #   log
                                        #   fatal
                                        #   panic

#log_min_error_statement = error        # values in order of decreasing detail:
                                        #   debug5
                                        #   debug4
                                        #   debug3
                                        #   debug2
                                        #   debug1
                                        #   info
                                        #   notice
                                        #   warning
                                        #   error
                                        #   log
                                        #   fatal
                                        #   panic (effectively off)

#log_min_duration_statement = -1        # -1 is disabled, 0 logs all statements
                                        # and their durations, > 0 logs only
                                        # statements running at least this number
                                        # of milliseconds

#log_transaction_sample_rate = 0.0      # Fraction of transactions whose statements
                                        # are logged regardless of their duration. 1.0 logs all
                                        # statements from all transactions, 0.0 never logs.

# - What to Log -

#debug_print_parse = off
#debug_print_rewritten = off
#debug_print_plan = off
#debug_pretty_print = on
#log_checkpoints = off
#log_connections = off
#log_disconnections = off
#log_duration = off
#log_error_verbosity = default          # terse, default, or verbose messages
#log_hostname = off
log_line_prefix = '%m [%p] %q%u@%d '            # special values:
                                        #   %a = application name
                                        #   %u = username
                                        #   %d = database name
                                        #   %r = remote host and port
                                        #   %h = remote host
                                        #   %p = process ID
                                        #   %t = timestamp without milliseconds
                                        #   %m = timestamp with milliseconds
                                        #   %n = timestamp with milliseconds (as a Unix epoch)
                                        #   %i = command tag
                                        #   %e = SQL state
                                        #   %c = session ID
                                        #   %l = session line number
                                        #   %s = session start timestamp
                                        #   %v = virtual transaction ID
                                        #   %x = transaction ID (0 if none)
                                        #   %q = stop here in non-session
                                        #        processes
                                        #   %% = '%'
                                        # e.g. '<%u%%%d> '
#log_lock_waits = off                   # log lock waits >= deadlock_timeout
#log_statement = 'none'                 # none, ddl, mod, all
#log_replication_commands = off
#log_temp_files = -1                    # log temporary files equal or larger
                                        # than the specified size in kilobytes;
                                        # -1 disables, 0 logs all temp files
log_timezone = 'America/New_York'

#------------------------------------------------------------------------------
# PROCESS TITLE
#------------------------------------------------------------------------------

cluster_name = '12/main'                        # added to process titles if nonempty
                                        # (change requires restart)
#update_process_title = on


#------------------------------------------------------------------------------
# STATISTICS
#------------------------------------------------------------------------------

# - Query and Index Statistics Collector -

#track_activities = on
#track_counts = on
#track_io_timing = off
#track_functions = none                 # none, pl, all
#track_activity_querysize = 1024       # (change requires restart)
stats_temp_directory = '/var/run/postgresql/12-main.pg_stat_tmp'


# - Monitoring -

#log_parser_stats = off
#log_planner_stats = off
#log_executor_stats = off
#log_statement_stats = off


#------------------------------------------------------------------------------
# AUTOVACUUM
#------------------------------------------------------------------------------

#autovacuum = on                        # Enable autovacuum subprocess?  'on'
                                        # requires track_counts to also be on.
#log_autovacuum_min_duration = -1       # -1 disables, 0 logs all actions and
                                        # their durations, > 0 logs only
                                        # actions running at least this number
                                        # of milliseconds.
#autovacuum_max_workers = 3             # max number of autovacuum subprocesses
                                        # (change requires restart)
#autovacuum_naptime = 1min              # time between autovacuum runs
#autovacuum_vacuum_threshold = 50       # min number of row updates before
                                        # vacuum
#autovacuum_analyze_threshold = 50      # min number of row updates before
                                        # analyze
#autovacuum_vacuum_scale_factor = 0.2   # fraction of table size before vacuum
#autovacuum_analyze_scale_factor = 0.1  # fraction of table size before analyze
#autovacuum_freeze_max_age = 200000000  # maximum XID age before forced vacuum
                                        # (change requires restart)
#autovacuum_multixact_freeze_max_age = 400000000        # maximum multixact age
                                        # before forced vacuum
                                        # (change requires restart)
#autovacuum_vacuum_cost_delay = 2ms     # default vacuum cost delay for
                                        # autovacuum, in milliseconds;
                                        # -1 means use vacuum_cost_delay
#autovacuum_vacuum_cost_limit = -1      # default vacuum cost limit for
                                        # autovacuum, -1 means use
                                        # vacuum_cost_limit


#------------------------------------------------------------------------------
# CLIENT CONNECTION DEFAULTS
#------------------------------------------------------------------------------

# - Statement Behavior -

#client_min_messages = notice           # values in order of decreasing detail:
                                        #   debug5
                                        #   debug4
                                        #   debug3
                                        #   debug2
                                        #   debug1
                                        #   log
                                        #   notice
                                        #   warning
                                        #   error
#search_path = '"$user", public'        # schema names
#row_security = on
#default_tablespace = ''                # a tablespace name, '' uses the default
#temp_tablespaces = ''                  # a list of tablespace names, '' uses
                                        # only default tablespace
#default_table_access_method = 'heap'
#check_function_bodies = on
#default_transaction_isolation = 'read committed'
#default_transaction_read_only = off
#default_transaction_deferrable = off
#session_replication_role = 'origin'
#statement_timeout = 0                  # in milliseconds, 0 is disabled
#lock_timeout = 0                       # in milliseconds, 0 is disabled
#idle_in_transaction_session_timeout = 0        # in milliseconds, 0 is disabled
#vacuum_freeze_min_age = 50000000
#vacuum_freeze_table_age = 150000000
#vacuum_multixact_freeze_min_age = 5000000
#vacuum_multixact_freeze_table_age = 150000000
#vacuum_cleanup_index_scale_factor = 0.1        # fraction of total number of tuples
                                                # before index cleanup, 0 always performs
                                                # index cleanup
#bytea_output = 'hex'                   # hex, escape
#xmlbinary = 'base64'
#xmloption = 'content'
#gin_fuzzy_search_limit = 0
#gin_pending_list_limit = 4MB

# - Locale and Formatting -

datestyle = 'iso, mdy'
#intervalstyle = 'postgres'
timezone = 'America/New_York'
#timezone_abbreviations = 'Default'     # Select the set of available time zone
                                        # abbreviations.  Currently, there are
                                        #   Default
                                        #   Australia (historical usage)
                                        #   India
                                        # You can create your own file in
                                        # share/timezonesets/.
#extra_float_digits = 1                 # min -15, max 3; any value >0 actually
                                        # selects precise output mode
#client_encoding = sql_ascii            # actually, defaults to database
                                        # encoding

# These settings are initialized by initdb, but they can be changed.
lc_messages = 'en_US.UTF-8'                     # locale for system error message
                                        # strings
lc_monetary = 'en_US.UTF-8'                     # locale for monetary formatting
lc_numeric = 'en_US.UTF-8'                      # locale for number formatting
lc_time = 'en_US.UTF-8'                         # locale for time formatting

# default configuration for text search
default_text_search_config = 'pg_catalog.english'

# - Shared Library Preloading -

#shared_preload_libraries = ''  # (change requires restart)
#local_preload_libraries = ''
#session_preload_libraries = ''
#jit_provider = 'llvmjit'               # JIT library to use

# - Other Defaults -

#dynamic_library_path = '$libdir'


#------------------------------------------------------------------------------
# LOCK MANAGEMENT
#------------------------------------------------------------------------------

#deadlock_timeout = 1s
#max_locks_per_transaction = 64         # min 10
                                        # (change requires restart)
#max_pred_locks_per_transaction = 64    # min 10
                                        # (change requires restart)
#max_pred_locks_per_relation = -2       # negative values mean
                                        # (max_pred_locks_per_transaction
                                        #  / -max_pred_locks_per_relation) - 1
#max_pred_locks_per_page = 2            # min 0


#------------------------------------------------------------------------------
# VERSION AND PLATFORM COMPATIBILITY
#------------------------------------------------------------------------------

# - Previous PostgreSQL Versions -

#array_nulls = on
#backslash_quote = safe_encoding        # on, off, or safe_encoding
#escape_string_warning = on
#lo_compat_privileges = off
#operator_precedence_warning = off
#quote_all_identifiers = off
#standard_conforming_strings = on
#synchronize_seqscans = on

# - Other Platforms and Clients -

#transform_null_equals = off


#------------------------------------------------------------------------------
# ERROR HANDLING
#------------------------------------------------------------------------------

#exit_on_error = off                    # terminate session on any error?
#restart_after_crash = on               # reinitialize after backend crash?
#data_sync_retry = off                  # retry or panic on failure to fsync
                                        # data?
                                        # (change requires restart)


#------------------------------------------------------------------------------
# CONFIG FILE INCLUDES
#------------------------------------------------------------------------------

# These options allow settings to be loaded from files other than the
# default postgresql.conf.  Note that these are directives, not variable
# assignments, so they can usefully be given more than once.

include_dir = 'conf.d'                  # include files ending in '.conf' from
                                        # a directory, e.g., 'conf.d'
#include_if_exists = '...'              # include file only if it exists
#include = '...'                        # include file


#------------------------------------------------------------------------------
# CUSTOMIZED OPTIONS
#------------------------------------------------------------------------------

All of my changes, highlighted in yellow, are at the bottom of pg_hba.conf. Scroll down to see them.

/etc/postgresql/12/main/pg_hba.conf

# PostgreSQL Client Authentication Configuration File
# ===================================================
#
# Refer to the "Client Authentication" section in the PostgreSQL
# documentation for a complete description of this file.  A short
# synopsis follows.
#
# This file controls: which hosts are allowed to connect, how clients
# are authenticated, which PostgreSQL usernames they can use, which
# databases they can access.  Records take one of these forms:
#
# local         DATABASE  USER  METHOD  [OPTIONS]
# host          DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
# hostssl       DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
# hostnossl     DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
# hostgssenc    DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
# hostnogssenc  DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
#
# (The uppercase items must be replaced by actual values.)
#
# The first field is the connection type: "local" is a Unix-domain
# socket, "host" is either a plain or SSL-encrypted TCP/IP socket,
# "hostssl" is an SSL-encrypted TCP/IP socket, and "hostnossl" is a
# non-SSL TCP/IP socket.  Similarly, "hostgssenc" uses a
# GSSAPI-encrypted TCP/IP socket, while "hostnogssenc" uses a
# non-GSSAPI socket.
#
# DATABASE can be "all", "sameuser", "samerole", "replication", a
# database name, or a comma-separated list thereof. The "all"
# keyword does not match "replication". Access to replication
# must be enabled in a separate record (see example below).
#
# USER can be "all", a username, a group name prefixed with "+", or a
# comma-separated list thereof.  In both the DATABASE and USER fields
# you can also write a file name prefixed with "@" to include names
# from a separate file.
#
# ADDRESS specifies the set of hosts the record matches.  It can be a
# host name, or it is made up of an IP address and a CIDR mask that is
# an integer (between 0 and 32 (IPv4) or 128 (IPv6) inclusive) that
# specifies the number of significant bits in the mask.  A host name
# that starts with a dot (.) matches a suffix of the actual host name.
# Alternatively, you can write an IP address and netmask in separate
# columns to specify the set of hosts.  Instead of a CIDR-address, you
# can write "samehost" to match any of the server's own IP addresses,
# or "samenet" to match any address in any subnet that the server is
# directly connected to.
#
# METHOD can be "trust", "reject", "md5", "password", "scram-sha-256",
# "gss", "sspi", "ident", "peer", "pam", "ldap", "radius" or "cert".
# Note that "password" sends passwords in clear text; "md5" or
# "scram-sha-256" are preferred since they send encrypted passwords.
#
# OPTIONS are a set of options for the authentication in the format
# NAME=VALUE.  The available options depend on the different
# authentication methods -- refer to the "Client Authentication"
# section in the documentation for a list of which options are
# available for which authentication methods.
#
# Database and usernames containing spaces, commas, quotes and other
# special characters must be quoted.  Quoting one of the keywords
# "all", "sameuser", "samerole" or "replication" makes the name lose
# its special character, and just match a database or username with
# that name.
#
# This file is read on server startup and when the server receives a
# SIGHUP signal.  If you edit the file on a running system, you have to
# SIGHUP the server for the changes to take effect, run "pg_ctl reload",
# or execute "SELECT pg_reload_conf()".
#
# Put your actual configuration here
# ----------------------------------
#
# If you want to allow non-local connections, you need to add more
# "host" records.  In that case you will also need to make PostgreSQL
# listen on a non-local interface via the listen_addresses
# configuration parameter, or via the -i or -h command line switches.


# DO NOT DISABLE!
# If you change this first entry you will need to make sure that the
# database superuser can access the database using some other method.
# Noninteractive access to all databases is required during automatic
# maintenance (custom daily cronjobs, replication, and similar tasks).
#
# Database administrative login by Unix domain socket
local   all             postgres                                md5

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     peer
# IPv4 local connections:
host    all             all             0.0.0.0/0               md5
host    all             all             0.0.0.0/32              md5
# IPv6 local connections:
host    all             all             ::1/128                 md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                     peer
host    replication     all             127.0.0.1/32            md5
host    replication     all             ::1/128                 md5

Now that PostgreSQL was configured, I restarted it:

Shell

$ sudo systemctl restart postgresql

Connecting PostgreSQL to PHP

I created a new database called opencart with this command:

Shell

$ psql -U postgres -c "create database opencart;"
CREATE DATABASE
Time: 1057.887 ms (00:01.058)

I entered PHP interactive mode to verify that PHP could connect properly to the new PostgreSQL database. This command sequence just creates a simple table and deletes it.

Shell

$ php -a
Interactive mode enabled

php > pg_connect("host=localhost dbname=opencart user=postgres password=hithere");
php > pg_query("create table test(id integer)");
php > pg_query("drop table test");
php > exit

Configuring OpenCart

The two configuration files that OpenCart provides are empty. They need to be renamed before OpenCart can be installed.

Shell

$ sudo mv /work/ecommerce/opencart/upload/config{-dist,}.php

$ sudo mv /work/ecommerce/opencart/upload/admin/config{-dist,}.php

These files will contain configuration information after the OpenCart admin user configures the system. The files also need to have their owner or group set to the same user that the web server runs as. For nginx, this username and group are both called www-data.

Shell

$ find . -name config.php -exec sudo chown www-data:dev {} \;

I dislike the idea of having a web application modify its configuration data while running. This is inherently insecure. However, many PHP programs from the era in which OpenCart was originally written operated that way. I have always been acutely uncomfortable with this practice.

Equally distasteful to me was the hack that PHP programmers often do to support multi-tenant web applications where users who self-administer their sites have limited storage options (20 years ago, this was an issue; the rest of the world has moved on): storing logs within the program file structure. This is insecure. OpenCart logs belong in /var/log/opencart. I did not modify the code; instead, I rolled my eyes and made the log files in opencart/upload/system/storage/logs/ group writable.

Shell

$ sudo chmod g+w opencart/upload/system/storage/logs/*.log

Installing OpenCart

Web-Based Installer

The web-based OpenCart installer is fragile and not well maintained. It dies near the end of its work when attempting to install using a Postgres database.

Clicking on http://localhost/upload/install/ starts the web-based OpenCart installation process by displaying the GNU license agreement from 2007. The installation fails on page 3. This problem was first reported on July 15, 2019, but it was never addressed.

Completing the installation only requires that the database be set up. system/helper/db_schema.php contains PHP code for defining the database schema using MySQL, and the SQL to populate the database is found in upload/install/opencart.sql.

At this point, I gave up and tried the command-line installer.

Command-Line Installer

OpenCart has a command-line installer, which is not mentioned in the online installation documentation. I always prefer to use a command-line installer, if possible, because any problems encountered are easier to diagnose and fix than with web-based installers.

In contrast to the publicly promoted web-based installer, the command-line installer appears to be well maintained for and by the current authors, who obviously also operate OpenCart Cloud (more on that in a minute). Once again, we see the inherent conflict of interest in traditional open-source software.

Here is a sample command line for installing OpenCart. The script should be run from the opencart/upload/install directory.

Shell

$ php cli_install.php install \
  --db_database opencart \
  --db_driver   postgre \
  --db_hostname localhost \
  --db_password postgres_password \
  --db_port     5432 \
  --db_username postgres \
  --email       email@example.com \
  --http_server http://localhost/opencart/ \
  --password    admin_password \
  --username    admin \
  | lynx -stdin

I needed to provide proper values for the following options:

--db_database: There is no default value for this option. It makes sense to name the database opencart, but one might have reasons to give it another name.
--db_hostname: The database might not run on the same network node as OpenCart's web server.
--db_password: Friends do not let friends use empty passwords, even on personal machines.
--db_port: I usually use the default PostgreSQL port, 5432.
--db_username: It is more secure to not use the postgres default username.
--email: Email address of the OpenCart administrator.
--http_server: More than just the domain name, this option also specifies the protocol, HTTP port and the path to the OpenCart directory on the web server.
--password: OpenCart admin user password.

The --db-prefix Option

The above omits the --db_prefix option, whose default value is oc_. This is because the installer uses upload/install/opencart.sql, which is hard-coded to use the default value.

The --cloud Option

The above also omits the --cloud option. This option has no documentation. After looking at the source code, I think this parameter is exclusively for OpenCart Cloud installations. This means that most people could omit the option because its value defaults to 0, which means the installation is not intended for Open Cloud.

Why do I think that? Looking at the code, I see this parameter suppresses database configuration and the saving of configuration information. Also, cloud installations require that the admin user password be pre-hashed, which suggests to me that this script can be initiated from another installation script used by Open Cloud.

Using Default Values

If you are installing on a development machine, it is likely to run both the Postgres database and the PHP website, and the software is likely to be set up using default values. Assuming that the PostgreSQL username is the default, postgres, and the database is called opencart, you just need to specify the following parameters:

Shell

$ php cli_install.php install \
  --db_driver   postgre \
  --db_database opencart \
  --db_password postgres_password \
  --db_port     5432 \
  --db_username postgres \
  --email       email@example.com \
  --http_server http://localhost/opencart/ \
  --password    admin_password \
  | lynx -stdin

Patching Source Code

I ran cli_install.php, found problems, fixed them, reran cli_install.php, found more problems, fixed them, etc. etc.

Clearly, no one has ever run cli_install.php to completion when --cloud option was set to 0.

Defining Constants

The people who use cli_install.php provide values for two constants before the program runs. I found some code in upload/index.php that defined them. Using those statements as a guide, I added some lines after line 57 of upload/system/startup.php so these constants were defined:

Shell

// mslinn added:
define('DIR_EXTENSION', DIR_OPENCART . 'extension/');
define('HTTP_SERVER', 'file:' . $_SERVER['HTTP_HOST'] . rtrim(dirname($_SERVER['SCRIPT_NAME']), '/.\\') . '/');
// end mslinn

Checking The Postgres Driver

Clearly no-one has ever tried to install using the PostgreSQL driver before. I had to modify line 198 of upload/install/cli_install.php to add a check, highlighted in yellow, for the PostgreSQL driver extension:

Shell

if (!extension_loaded('mysqli') && !extension_loaded('pgsql')) {
  $error .= 'ERROR: MySQLi extension needs to be loaded for OpenCart to work!' . "\n";
}

Running the Command-Line Installer

The command-line installer spewed out miles and miles of HTML (mostly the GNU license) and died with an error message. So, I fixed the problem and reran it. It died somewhere else with a different error. So, I fixed that problem too and ran it again. It died yet again somewhere else with yet another error. And again, and again, and again...

The last problem I found before quitting was an error message resulting from the command-line installer attempting to rewrite an HTML header. Really! A command-line installer does not need to present HTML to the user. This command-line installer program is clearly a cheap hack.

Evaluation Results

At this point, I felt that I now had a good idea of the quality of this open-source project:

OpenCart is very poorly constructed

The business model for the company that stewards OpenCart is also clear: hire cheap programmers of modest ability and charge for their time by the hour. OpenCart is not something I would want to base an e-commerce business on.

Since OpenCart is supposedly the best open-source shopping cart today, I will next look into building something just for me that provides me with a competitive advantage.

Installing a New SSH Key on AWS EC2 with User Data

2020-10-27T00:00:00-04:00

For some reason the ssh certificates that AWS generated for me 3 years ago are no longer recognized by Ubuntu 20.10. This article shows how to create new certificates and push them to an AWS server that was just upgraded to Ubuntu 20.01, and now cannot be logged into. I decided to use OpenSSH to generate the new keypairs instead of AWS to generate the keypairs because the current problem stems from AWS-generated keys gradually becoming incompatible with OpenSSH servers.

This article describes the following:

Tracking down the problem
Create a new ssh certificate keypair.
Even though the system cannot accept logins, the new ssh public key must be copied to the ubuntu user’s ~/.ssh directory on the problem server. This is done by defining a user data script on the server instance prior to booting it.
Log into the problem server using the new certificates.
Complete the upgrade to XUbuntu 20.10.
Remove the user data script from the problem server instance.

Discovery

I was able to log into another of my machines (gojira), so first I wanted to know if the problem machines (va and france) had OpenSSH configured differently. I used the comm Linux utility to perform the comparison.

Shell

$ for x in cipher mac key kex; do
  comm -3 <(ssh -Q $x france|sort) <(ssh -Q $x gojira|sort)
done

$ for x in cipher mac key kex; do
  comm -3 <(ssh -Q $x france|sort) <(ssh -Q $x gojira|sort)
done

So the problem was not OpenSSH configuration per se. Next I wondered if the ssh connections to the problem machines were different somehow from the ssh connection to the working machine.

Shell

$ comm -3 <(ssh -Gva|sort) <(ssh -G gojira|sort)
hostname gojira
hostname va
        identityfile ~/.ssh/id_rsa
identityfile ~/.ssh/sslTest
        user mslinn
user ubuntu

So the only differences were the hostnames and the keys offered.

One of the problem machines, france, resided on scaleway. I used the most recently available bootscript to launch the server and examined /var/log/auth.log. I found this: sshd[27025]: Unable to negotiate with 205.185.123.173 port 40555: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1 [preauth]

This error message is produced by OpenSSH 7.0+. The release notes say “Support for the 1024-bit diffie-hellman-group1-sha1 key exchange is disabled by default at run-time. It may be re-enabled using the instructions at http://www.openssh.com/legacy.html”

So it seems that the version of OpenSSH installed with Ubuntu 20.10 rejects my old keys. The release notes for the new version of OpenSSH also indicate that OpenSSH 7.1 will be even stricter:

“This focus of this release is primarily to deprecate weak, legacy and/or unsafe cryptography”
“Refusing all RSA keys smaller than 1024 bits (the current minimum is 768 bits)”
“Several ciphers will be disabled by default: blowfish-cbc, cast128-cbc, all arcfour variants and the rijndael-cbc aliases for AES.”
“MD5-based HMAC algorithms will be disabled by default.”

Clearly, I need to generate better ssh keys.

The version of OpenSSH installed by Ubuntu 20.10 is 8.3:

Shell

$ sshd -V
unknown option -- V
OpenSSH_8.3p1 Ubuntu-1, OpenSSL 1.1.1f  31 Mar 2020
usage: sshd [-46DdeiqTt] [-C connection_spec] [-c host_cert_file]
            [-E log_file] [-f config_file] [-g login_grace_time]
            [-h host_key_file] [-o option] [-p port] [-u len]

$ ssh -V
OpenSSH_8.3p1 Ubuntu-1, OpenSSL 1.1.1f  31 Mar 2020

Setup

AWS CLI

I prefer to use the AWS CLI instead of the web console. Installation instructions are here. This article uses the AWS CLI exclusively in favor of the AWS web console.

The code in the remainder of this article references an AWS EC2 instance, whose id I stored in AWS_PROBLEM_INSTANCE_ID. The previous article shows how I did that.

jq

I also use jq for parsing JSON in the bash shell. Install it on Debian-style Linux distros such as Ubuntu like this:

Shell

$ yes | sudo apt install jq

Name the New Keypair

I wanted to make new ecdsa keys because this algorithm is the currently accepted best practice for commercial security concerns. ecdsa stands for Elliptic Curve Digital Signature Algorithm.

Unfortunately, AWS EC2 only accepts RSA keys. The name of the new key pair will be of the form ~/.ssh/rsa-YYYY-MM-DD.

Shell

$ AWS_KEY_PAIR_FILE="$HOME/.ssh/rsa-$( date '+%Y-%m-%d-%M-%S' )"

$ echo $AWS_KEY_PAIR_FILE
/home/mslinn/.ssh/rsa-2020-11-04-08-24-22

The new public key will be called ~/.ssh/rsa-2020-11-04-08-24-22.pub and the new private key will be called ~/.ssh/rsa-2020-11-04-08-24-22.

Create a New Keypair

This is how I would have created a new ECDSA keypair, if AWS supported that type of encryption.

Shell

$ ssh-keygen -b 512 -C "mslinn@mslinn.com" -f "$AWS_KEY_PAIR_FILE" -P "" -t ecdsa
Generating public/private ecdsa key pair.
Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-08-24-22
Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-08-24-22.pub
The key fingerprint is:
SHA256:HEKjAA1GZxHbpwqjm85DXQpQEeIWrcjZ6fl84RHQaHE mslinn@mslinn.com
The key’s randomart image is:
+---[RSA 512]----+
|=O*Bo.*E         |
|+.=oo*.o         |
|+o+.+.o..        |
|o= o .o+ .       |
| o+ +.  S        |
|..o=.  o         |
|o  .o . o        |
|.+   o o         |
|+o.   .          |
+----[SHA256]-----+

$ chmod 400 $AWS_KEY_PAIR_FILE

Instead, I created a new RSA keypair like this:

Shell

$ ssh-keygen -b 2048 -C "mslinn@mslinn.com" -f "$AWS_KEY_PAIR_FILE" -m PEM -P "" -t rsa
Generating public/private rsa key pair.
Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-08-24-22
Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-08-24-22.pub
The key fingerprint is:
SHA256:bQScX0UMn0xGDorxSvElMZzwMyyk7hgs2FNbshBNenA mslinn@mslinn.com
The key’s randomart image is:
+---[RSA 2048]----+
|  ooE  .*++o+**  |
|   =.  ooXo=.B.. |
|  o + o +.X.  =  |
| o = * . =.o     |
|. + = . S o      |
|   o +   .       |
|    . .          |
|                 |
|                 |
+----[SHA256]-----+

$ chmod 400 $AWS_KEY_PAIR_FILE

I would have liked to copy the keypair to the problem system using ssh-copy-id, but that only works when login is possible.
Shell
```
$ ssh-copy-id -i $AWS_KEY_PAIR_FILE ubuntu@$AWS_PROBLEM_IP
```

Instead, I decided to paste the public key into an AWS user data script and execute that script on the problem server the next time it booted. The purpose of the script is to copy the new public key that was just made to ~/.ssh/ on the problem server. This is the user data script I wrote to install the new public key, called rescue_ubuntu2010.sh:

rescue_ubuntu2010.sh

#!/bin/bash

KEY_FILE_NAME=/home/ubuntu/.ssh/rsa-2020-11-03.pub

cat > "$KEY_FILE_NAME" <<EOF
ssh-rsa ABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFA== mslinn@mslinn.com
EOF

chown ubuntu: /home/ubuntu/.ssh/*
chmod 400 "$KEY_FILE_NAME"
cat "$KEY_FILE_NAME" >> /home/ubuntu/.ssh/authorized_keys

The script runs on the problem server as root next time the system boots, and it reboots the server on the last line.

The script need to be converted into base 64, in a file called rescue_ubuntu2010.b64.
Shell
```
$ base64 rescue_ubuntu2010.sh > rescue_ubuntu2010.b64
```

The problem EC2 instance can be shut down like this:

Shell

$ aws ec2 stop-instances --instance-id $AWS_PROBLEM_INSTANCE_ID

$ aws ec2 wait instance-stopped --instance-ids $AWS_PROBLEM_INSTANCE_ID

With the problem EC2 instance stopped, its user data was set to the base64-encoded version of the rescue script.

Shell

$ aws ec2 modify-instance-attribute \
  --instance-id $AWS_PROBLEM_INSTANCE_ID \
  --attribute userData \
  --value file://rescue_ubuntu2010.b64

Now the problem EC2 instance can be restarted. The script will add the new key to /home/ubuntu/.ssh/authorized_keys and login should be possible.

Shell

$ aws ec2 start-instances --instance-id $AWS_PROBLEM_INSTANCE_ID
    {
  "StartingInstances": [
      {
          "CurrentState": {
              "Code": 0,
              "Name": "pending"
          },
          "InstanceId": "i-d3b03954",
          "PreviousState": {
              "Code": 80,
              "Name": "stopped"
          }
      }
  ]
}

$ aws ec2 wait instance-running --instance-ids $AWS_PROBLEM_INSTANCE_ID

Reset User Data for Next Time

Next time the problem server is stopped, clear the user data so it is not provided the next time the server restarts.

Shell

$ aws ec2 modify-instance-attribute \
  --instance-id $AWS_PROBLEM_INSTANCE_ID \
  --user-data Value=

Rescuing a Catastrophic Upgrade to Ubuntu 20.10

2020-10-25T00:00:00-04:00

The upgrade from Ubuntu 20.04 to 20.10 has been especially problematic for each of the half-dozen XUbuntu systems that I manage. One important server that I run on Scaleway became unresponsive and would not boot shortly after starting the installation, and another important server on AWS ran fine, but did not allow logins.

This article details what I did to recover the AWS server using a standard *nix procedure that any competent system administrator would be comfortable with: chroot.

Before Linux had cgroups, we used chroot and its close cousin, jail. I used chroot for the technical basis of Zamples back in 2001.

Because the chroot environment will be set up in a way that it shares the rescue system’s /var/run directory, the rescue system should have all upgrades in place and should be rebooted if /var/run/reboot-required exists.

AWS also provides a tool called EC2Rescue, which does a complicated series of actions to accomplish something similar. Here is additional documentation. I find the AWS documentation is frequently obtuse, and the approach taken by most AWS products and tools is extremely general. Consequently I often find myself wasting a lot of time trying to get things to work. I don’t subscribe to AWS support; if I had subscribed to expensive enterprise-level support, complete with an AWS expert to hold my hand while I attempted to resurrect the server, I might have tried using EC2Rescue. On the other hand, when pressed with an emergency, I prefer to lean on tried-and-true methods like chroot.

This article concludes with two Bash scripts to automate the details. I wrote the second script in late January 2022, approximately 2 years after this article was first published. The second script is less user-friendly, but more fine-grained. That is a reasonable tradeoff, and both approaches have merit.

Setup

AWS CLI

I prefer to use the AWS CLI instead of the web console. Installation instructions are here. This article uses the AWS CLI exclusively in favor of the AWS web console.

jq

I also use jq for parsing JSON in the bash shell. Install it on Debian-style Linux distros such as Ubuntu like this:

Shell

$ yes | sudo apt install jq

Discover information about the Problem EC2 instance

Getting the AWS EC2 Instance Information

Because my problem EC2 instance has a tag called Name with Value production, I was able to easily obtain a JSON representation of all the information about it. I stored the JSON in an environment variable called AWS_EC2_PRODUCTION.

The results are shown in unselectable text. This is so you can easily use this sample code yourself. You can copy the code to run into your clipboard. Just click on the little copy icon at the top right hand corner of the scrolling code display area. Because the prompt and the results and are unselectable, your clipboard will only pick up the code you need to paste in order to run the code example yourself.

Shell

$ AWS_EC2_PRODUCTION="$(
  aws ec2 describe-instances | \
  jq '.Reservations[].Instances[] | select((.Tags[]?.Key=="Name") and (.Tags[]?.Value=="production"))'
)"

$ echo "$AWS_EC2_PRODUCTION"
{
  "AmiLaunchIndex": 0,
  "ImageId": "ami-e29b9388",
  "InstanceId": "i-825eb905",
  "InstanceType": "t2.small",
  "KeyName": "sslTest",
  "LaunchTime": "2017-10-12T16:24:14.000Z",
  "Monitoring": {
    "State": "disabled"
  },
  "Placement": {
    "AvailabilityZone": "us-east-1c",
    "GroupName": "",
    "Tenancy": "default"
  },
  "PrivateDnsName": "ip-10-0-0-201.ec2.internal",
  "PrivateIpAddress": "10.0.0.201",
  "ProductCodes": [],
  "PublicDnsName": "",
  "PublicIpAddress": "52.207.225.143",
  "State": {
    "Code": 16,
    "Name": "running"
  },
  "StateTransitionReason": "",
  "SubnetId": "subnet-49de033f",
  "VpcId": "vpc-f16a0895",
  "Architecture": "x86_64",
  "BlockDeviceMappings": [
    {
      "DeviceName": "/dev/sda1",
      "Ebs": {
        "AttachTime": "2016-04-05T19:07:17.000Z",
        "DeleteOnTermination": true,
        "Status": "attached",
        "VolumeId": "vol-1c8903b4"
      }
    }
  ],
  "ClientToken": "GykZz1459883236367",
  "EbsOptimized": false,
  "Hypervisor": "xen",
  "NetworkInterfaces": [
    {
      "Association": {
        "IpOwnerId": "amazon",
        "PublicDnsName": "",
        "PublicIp": "52.207.225.143"
      },
      "Attachment": {
        "AttachTime": "2016-04-05T19:07:16.000Z",
        "AttachmentId": "eni-attach-a58bd15f",
        "DeleteOnTermination": true,
        "DeviceIndex": 0,
        "Status": "attached"
      },
      "Description": "Primary network interface",
      "Groups": [
        {
          "GroupName": "testSG",
          "GroupId": "sg-4cbc6f35"
        }
      ],
      "Ipv6Addresses": [],
      "MacAddress": "0a:a4:be:1b:8e:eb",
      "NetworkInterfaceId": "eni-fa4f65bb",
      "OwnerId": "031372724784",
      "PrivateIpAddress": "10.0.0.201",
      "PrivateIpAddresses": [
        {
          "Association": {
            "IpOwnerId": "amazon",
            "PublicDnsName": "",
            "PublicIp": "52.207.225.143"
          },
          "Primary": true,
          "PrivateIpAddress": "10.0.0.201"
        }
      ],
      "SourceDestCheck": true,
      "Status": "in-use",
      "SubnetId": "subnet-49de033f",
      "VpcId": "vpc-f16a0895",
      "InterfaceType": "interface"
    }
  ],
  "RootDeviceName": "/dev/sda1",
  "RootDeviceType": "ebs",
  "SecurityGroups": [
    {
      "GroupName": "testSG",
      "GroupId": "sg-4cbc6f35"
    }
  ],
  "SourceDestCheck": true,
  "Tags": [
    {
      "Key": "Name",
      "Value": "production"
    }
  ],
  "VirtualizationType": "hvm",
  "CpuOptions": {
    "CoreCount": 1,
    "ThreadsPerCore": 1
  },
  "CapacityReservationSpecification": {
    "CapacityReservationPreference": "open"
  },
  "HibernationOptions": {
    "Configured": false
  },
  "MetadataOptions": {
    "State": "applied",
    "HttpTokens": "optional",
    "HttpPutResponseHopLimit": 1,
    "HttpEndpoint": "enabled"
  }
}

Getting the AWS EC2 Problem Instance Id

The instance ID for the problem EC2 instance can be extracted from the JSON returned by the preceding results easily:

Shell

$ AWS_PROBLEM_INSTANCE_ID="$(
  jq -r .InstanceId <<< "$AWS_EC2_PRODUCTION"
)"

$ echo "$AWS_PROBLEM_INSTANCE_ID"
i-825eb905

Getting the AWS EC2 Problem Instance IP Address

The IP address for the problem EC2 instance can be extracted from the JSON returned by the preceding results easily:

Shell

$ AWS_PROBLEM_IP="$(
  jq -r .PublicIpAddress <<< "$AWS_EC2_PRODUCTION"
)"

$ echo "$AWS_PROBLEM_IP"
52.207.225.143

Getting the AWS EC2 Problem Availability Zone

The AWS availability zone for the problem EC2 instance can be extracted from the JSON returned by the preceding results easily:

Shell

$ AWS_AVAILABILITY_ZONE="$(
  jq -r .Placement.AvailabilityZone <<< "$AWS_EC2_PRODUCTION"
)"

$ echo "$AWS_AVAILABILITY_ZONE"
us-east-1c

Getting the AWS EC2 Problem Volume ID

The following command line extracts the volume id of the problem server’s system drive into an environment variable called $AWS_PROBLEM_VOLUME_ID:

Shell

$ AWS_PROBLEM_VOLUME_ID="$(
  jq -r '.BlockDeviceMappings[].Ebs.VolumeId' <<< "$AWS_EC2_PRODUCTION"
)"

$ echo "$AWS_PROBLEM_VOLUME_ID"
vol-1c8903b4

Make a Snapshot of the Problem Server

One approach, which would be living dangerously, would be to mount the system volume of the problem server on another server, set up chroot, attempt to repair the drive image, remount the repaired drive on the problem server, and reboot the server. I am never that optimistic. Things invariably go wrong. Instead, we will take a snapshot of the problem drive, turn the snapshot into a volume, repair the volume, swap in the repaired volume on the problem system, and reboot that system.

It is better to shut down the EC2 instance before making a snapshot, however a snapshot can be taken whenever the server is idling. We will need to shut down the server anyway, so that could be done now, or at the last minute.

I made a snapshot with a tag called Name with the value like production 2020-10-25 and saved the snapshot id in an environment variable called AWS_PROBLEM_SNAPSHOT_ID:

Shell

$ AWS_PROBLEM_SNAPSHOT_ID="$(
  aws ec2 create-snapshot --volume-id "$AWS_PROBLEM_VOLUME_ID" \
    --description "production `date '+%Y-%m-%d'`" \
    --tag-specifications "ResourceType=snapshot,Tags=[{Key=Created, Value=`date '+%Y-%m-%d'`},{Key=Name, Value=\"Broken do-release-upgrade 20.{04,10\"}]" | \
  jq -r .SnapshotId
)"

$ echo "$AWS_PROBLEM_SNAPSHOT_ID"
snap-0a856be1f58b8a856 

$ aws ec2 wait snapshot-completed --snapshot-ids "$AWS_PROBLEM_SNAPSHOT_ID"

Snapshots only take a few minutes to complete. The aws ec2 wait command blocks until the specified operation finishes.

Create Rescue Volume From Snapshot

Once the snapshot process has completed, create a new volume from the snapshot. The default volume type is gp2. We’ll refer to this volume as $AWS_RESCUE_VOLUME_ID. It is important to create the volume in the same availability zone as the problem EC2 instance so that it can easily be attached. This command applies a tag called Name, with the value rescue, for easy identification.

Shell

$ AWS_RESCUE_VOLUME_ID="$(
  aws ec2 create-volume \
    --availability-zone $AWS_AVAILABILITY_ZONE \
    --snapshot-id $AWS_PROBLEM_SNAPSHOT_ID \
    --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=rescue}]' | \
  jq -r .VolumeId
)"

$ echo "$AWS_RESCUE_VOLUME_ID"
vol-0e20fd22d2dc5a933 

$ aws ec2 wait volume-available --volume-id "$AWS_RESCUE_VOLUME_ID"

Use an EC2 Spot Instance For the Rescue Server

Now that the rescue volume is available, we need to mount it on a server, which I’ll call the rescue server. We’ll refer to the server where the rescue volume is prepared via its instance id, saved as AWS_EC2_RESCUE_ID. You can either create a new EC2 instance for this purpose, or use an existing EC2 instance.

The rescue server does not need to be anything special; a tiny virtual machine of any description will do fine. However, some rescue operations will be much easier if the type of operating system is the same as that on the problem drive. Yesterday I blogged about how to find a suitable AMI, and determine its image-id.

Shell

$ AWS_AMI="$(
  aws ec2 describe-images \
    --owners 099720109477 \
    --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-━━━━━???-━━━━━???-amd64-server-━━━━━???" \
              "Name=state,Values=available" \
    --query "reverse(sort_by(Images, &CreationDate))[:1]" | \
  jq -r '.[0]'
)"

$ echo "$AWS_AMI"
{
  "Architecture": "x86_64",
  "CreationDate": "2020-10-30T14:07:42.000Z",
  "ImageId": "ami-0c71ec98278087e60",
  "ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030",
  "ImageType": "machine",
  "Public": true,
  "OwnerId": "099720109477",
  "PlatformDetails": "Linux/UNIX",
  "UsageOperation": "RunInstances",
  "State": "available",
  "BlockDeviceMappings": [
    {
      "DeviceName": "/dev/sda1",
      "Ebs": {
        "DeleteOnTermination": true,
        "SnapshotId": "snap-00bf581086dd686e5",
        "VolumeSize": 8,
        "VolumeType": "gp2",
        "Encrypted": false
      }
    },
    {
      "DeviceName": "/dev/sdb",
      "VirtualName": "ephemeral0"
    },
    {
      "DeviceName": "/dev/sdc",
      "VirtualName": "ephemeral1"
    }
  ],
  "Description": "Canonical, Ubuntu, 20.10, amd64 groovy image build on 2020-10-30",
  "EnaSupport": true,
  "Hypervisor": "xen",
  "Name": "ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030",
  "RootDeviceName": "/dev/sda1",
  "RootDeviceType": "ebs",
  "SriovNetSupport": "simple",
  "VirtualizationType": "hvm"
}

Now let's extract the ID of the AMI image and save it as AWS_AMI_ID.

Shell

$ AWS_AMI_ID="$( jq -r '.[0].ImageId' <<< "$AWS_AMI" )"

$ echo "$AWS_AMI_ID"
ami-0c71ec98278087e60

Volumes can be attached to running and stopped server instances. The load on the rescue server will likely be light and short-lived. An EC2 spot instance is ideal, and only costs two cents per hour! The spot instance will likely only be needed for 15 minutes. I specified my VPC id as SubnetId, the security group sg-4cbc6f35 and the AvailabilityZone.

Shell

$ AWS_EC2_RESCUE="$(
  aws ec2 run-instances \
    --image-id "$AWS_AMI_ID" \
    --instance-market-options '{ "MarketType": "spot" }' \
    --instance-type t2.medium \
    --key-name rsa-2020-11-03.pub \
    --network-interfaces '[ {
        "DeviceIndex": 0,
        "Groups": ["sg-4cbc6f35"],
        "SubnetId": "subnet-49de033f",
        "DeleteOnTermination": true,
        "AssociatePublicIpAddress": true
      } ]' \
    --placement '{ "AvailabilityZone": "us-east-1c" }'
)"

$ echo "$AWS_EC2_RESCUE"
{
    "Groups": [],
    "Instances": [
        {
            "AmiLaunchIndex": 0,
            "ImageId": "ami-0dba2cb6798deb6d8",
            "InstanceId": "i-012a54aefcd333de9",
            "InstanceType": "t2.small",
            "KeyName": "rsa-2020-11-03.pub",
            "LaunchTime": "2020-11-03T23:19:50.000Z",
            "Monitoring": {
                "State": "disabled"
            },
            "Placement": {
                "AvailabilityZone": "us-east-1c",
                "GroupName": "",
                "Tenancy": "default"
            },
            "PrivateDnsName": "ip-10-0-0-210.ec2.internal",
            "PrivateIpAddress": "10.0.0.210",
            "ProductCodes": [],
            "PublicDnsName": "",
            "State": {
                "Code": 0,
                "Name": "pending"
            },
            "StateTransitionReason": "",
            "SubnetId": "subnet-49de033f",
            "VpcId": "vpc-f16a0895",
            "Architecture": "x86_64",
            "BlockDeviceMappings": [],
            "ClientToken": "026583fb-c94e-4bca-bdd2-8dcdcaa3aae9",
            "EbsOptimized": false,
            "EnaSupport": true,
            "Hypervisor": "xen",
            "InstanceLifecycle": "spot",
            "NetworkInterfaces": [
                {
                    "Attachment": {
                        "AttachTime": "2020-11-03T23:19:50.000Z",
                        "AttachmentId": "eni-attach-04feb4d36cf5c6792",
                        "DeleteOnTermination": true,
                        "DeviceIndex": 0,
                        "Status": "attaching"
                    },
                    "Description": "",
                    "Groups": [
                        {
                            "GroupName": "testSG",
                            "GroupId": "sg-4cbc6f35"
                        }
                    ],
                    "Ipv6Addresses": [],
                    "MacAddress": "0a:6d:ba:c5:65:4b",
                    "NetworkInterfaceId": "eni-09ef90920cfb29dd9",
                    "OwnerId": "031372724784",
                    "PrivateIpAddress": "10.0.0.210",
                    "PrivateIpAddresses": [
                        {
                            "Primary": true,
                            "PrivateIpAddress": "10.0.0.210"
                        }
                    ],
                    "SourceDestCheck": true,
                    "Status": "in-use",
                    "SubnetId": "subnet-49de033f",
                    "VpcId": "vpc-f16a0895",
                    "InterfaceType": "interface"
                }
            ],
            "RootDeviceName": "/dev/sda1",
            "RootDeviceType": "ebs",
            "SecurityGroups": [
                {
                    "GroupName": "testSG",
                    "GroupId": "sg-4cbc6f35"
                }
            ],
            "SourceDestCheck": true,
            "SpotInstanceRequestId": "sir-rrs9gm3j",
            "StateReason": {
                "Code": "pending",
                "Message": "pending"
            },
            "VirtualizationType": "hvm",
            "CpuOptions": {
                "CoreCount": 1,
                "ThreadsPerCore": 1
            },
            "CapacityReservationSpecification": {
                "CapacityReservationPreference": "open"
            },
            "MetadataOptions": {
                "State": "pending",
                "HttpTokens": "optional",
                "HttpPutResponseHopLimit": 1,
                "HttpEndpoint": "enabled"
            }
        }
    ],
    "OwnerId": "031372724784",
    "ReservationId": "r-0d45e1919e7bad5c9"
}

We can use jq to extract the EC2 InstanceId of the spot instance:

Shell

$ AWS_SPOT_INSTANCE_ID="$(
  jq -r '.Instances[].InstanceId' <<< "$AWS_EC2_RESCUE"
)"

$ echo "$AWS_SPOT_INSTANCE_ID"
ami-0dba2cb6798deb6d8

We need to retrieve the IP address of the newly created EC2 spot instance. This instance will disappear (terminate) once it shuts down, so do not reboot it.

Shell

$ aws ec2 describe-instances --instance-ids "$AWS_SPOT_INSTANCE_ID"
{
    "Reservations": [
        {
            "Groups": [],
            "Instances": [
                {
                    "AmiLaunchIndex": 0,
                    "ImageId": "ami-0dba2cb6798deb6d8",
                    "InstanceId": "i-012a54aefcd333de9",
                    "InstanceType": "t2.small",
                    "KeyName": "rsa-2020-11-03.pub",
                    "LaunchTime": "2020-11-03T23:19:50.000Z",
                    "Monitoring": {
                        "State": "disabled"
                    },
                    "Placement": {
                        "AvailabilityZone": "us-east-1c",
                        "GroupName": "",
                        "Tenancy": "default"
                    },
                    "PrivateDnsName": "ip-10-0-0-210.ec2.internal",
                    "PrivateIpAddress": "10.0.0.210",
                    "ProductCodes": [],
                    "PublicDnsName": "",
                    "PublicIpAddress": "54.242.88.254",
                    "State": {
                        "Code": 16,
                        "Name": "running"
                    },
                    "StateTransitionReason": "",
                    "SubnetId": "subnet-49de033f",
                    "VpcId": "vpc-f16a0895",
                    "Architecture": "x86_64",
                    "BlockDeviceMappings": [
                        {
                            "DeviceName": "/dev/sda1",
                            "Ebs": {
                                "AttachTime": "2020-11-03T23:19:51.000Z",
                                "DeleteOnTermination": true,
                                "Status": "attached",
                                "VolumeId": "vol-0c44c8c009d1fafda"
                            }
                        }
                    ],
                    "ClientToken": "026583fb-c94e-4bca-bdd2-8dcdcaa3aae9",
                    "EbsOptimized": false,
                    "EnaSupport": true,
                    "Hypervisor": "xen",
                    "InstanceLifecycle": "spot",
                    "NetworkInterfaces": [
                        {
                            "Association": {
                                "IpOwnerId": "amazon",
                                "PublicDnsName": "",
                                "PublicIp": "54.242.88.254"
                            },
                            "Attachment": {
                                "AttachTime": "2020-11-03T23:19:50.000Z",
                                "AttachmentId": "eni-attach-04feb4d36cf5c6792",
                                "DeleteOnTermination": true,
                                "DeviceIndex": 0,
                                "Status": "attached"
                            },
                            "Description": "",
                            "Groups": [
                                {
                                    "GroupName": "testSG",
                                    "GroupId": "sg-4cbc6f35"
                                }
                            ],
                            "Ipv6Addresses": [],
                            "MacAddress": "0a:6d:ba:c5:65:4b",
                            "NetworkInterfaceId": "eni-09ef90920cfb29dd9",
                            "OwnerId": "031372724784",
                            "PrivateIpAddress": "10.0.0.210",
                            "PrivateIpAddresses": [
                                {
                                    "Association": {
                                        "IpOwnerId": "amazon",
                                        "PublicDnsName": "",
                                        "PublicIp": "54.242.88.254"
                                    },
                                    "Primary": true,
                                    "PrivateIpAddress": "10.0.0.210"
                                }
                            ],
                            "SourceDestCheck": true,
                            "Status": "in-use",
                            "SubnetId": "subnet-49de033f",
                            "VpcId": "vpc-f16a0895",
                            "InterfaceType": "interface"
                        }
                    ],
                    "RootDeviceName": "/dev/sda1",
                    "RootDeviceType": "ebs",
                    "SecurityGroups": [
                        {
                            "GroupName": "testSG",
                            "GroupId": "sg-4cbc6f35"
                        }
                    ],
                    "SourceDestCheck": true,
                    "SpotInstanceRequestId": "sir-rrs9gm3j",
                    "VirtualizationType": "hvm",
                    "CpuOptions": {
                        "CoreCount": 1,
                        "ThreadsPerCore": 1
                    },
                    "CapacityReservationSpecification": {
                        "CapacityReservationPreference": "open"
                    },
                    "HibernationOptions": {
                        "Configured": false
                    },
                    "MetadataOptions": {
                        "State": "applied",
                        "HttpTokens": "optional",
                        "HttpPutResponseHopLimit": 1,
                        "HttpEndpoint": "enabled"
                    }
                }
            ],
            "OwnerId": "031372724784",
            "ReservationId": "r-0d45e1919e7bad5c9"
        }
    ]
}

Mount the Rescue Volume On the Rescue Server

We need to select a device name to be assigned to the rescue disk once it is attached to an EC2 instance. The available names depend on what names are already in use on the rescue server. After logging into the rescue server, I ran the lsblk Linux command to see the available disk devices and their mount points.

Shell

$ lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop1     7:1    0 53.1M  1 loop /snap/lxd/10984
loop2     7:2    0 88.4M  1 loop /snap/core/7169
loop3     7:3    0 97.8M  1 loop /snap/core/10185
loop4     7:4    0 53.1M  1 loop /snap/lxd/11348
xvda    202:0    0    8G  0 disk
└─xvda1 202:1    0    8G  0 part /

The lsblk output does not show full device paths, instead, the /dev/ prefix is omitted. With that in mind we can see that the only disk device on the rescue server is /dev/xvda, and its only partition called /dev/xvda1 is mounted on the root directory. Because Linux drives are normally named sequentially, we should name the rescue disk /dev/xvdb. Let’s define an environment variable called AWS_RESCUE_DRIVE to memorialize that decision.

Shell

$ AWS_RESCUE_DRIVE=/dev/xvdb

The aws ec2 attach-volume command will attach the rescue volume to the rescue server. It automatically selects an appropriate device name for the rescue volume, which in the following example is /dev/xvdb:

Shell

$ AWS_ATTACH_VOLUME="$(
  aws ec2 attach-volume \
    --device $AWS_RESCUE_DRIVE \
    --instance-id $AWS_EC2_RESCUE_ID \
    --volume-id $AWS_RESCUE_VOLUME_ID
)"

$ echo "$AWS_ATTACH_VOLUME"
{
    "AttachTime": "2020-10-26T14:34:55.222Z",
    "InstanceId": "i-d3b03954",
    "VolumeId": "vol-0e20fd22d2dc5a933",
    "State": "attaching",
    "Device": "/dev/xvdb"
}

$ aws ec2 wait volume-in-use --volume-id "$AWS_RESCUE_VOLUME_ID"

The details of the mounted rescue drive are provided by fdisk -l:

Shell

$ sudo fdisk -l | sed -n -e '/xvdb/,$p'
Disk /dev/xvdb: 12 GiB, 12884901888 bytes, 25165824 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start      End  Sectors Size Id Type
/dev/xvdb1 *    16065 25165790 25149726  12G 83 Linux

Now it is time to mount the rescue drive on the rescue server. Ubuntu has a directory called /mnt whose purpose is to act as a mount point:

Shell

$ sudo mount /dev/xvdb1 /mnt

Let’s confirm that the drive is mounted:

Shell

$ df -h | grep '^/dev/' | grep -v '^/dev/loop'
/dev/xvda1      7.8G  6.3G  1.1G  86% /
/dev/xvdb1       12G  9.0G  2.2G  82% /mnt

The last line shows that this drive is mounted on /mnt and it is 82% full.

Set Up a chroot to Establish an Environment for Making Repairs

We need to mount some more file systems before we perform the chroot. The following mounts the rescue server’s /dev, /dev/shm, /sys, and /run to the same paths within the rescue volume. Because programs like do-release-upgrade need a tty, I also mount devtps and proc. These mounts only last until the next server reboot. After all the mounts the chroot is issued.

Warning - mounting /run and then updating the system on the rescue disk from within a chroot may change the host system’s /run contents; if the package managers (apt and dpkg) get out of sync with the actual state on the host system you won’t be able to update the host system until you restore the host system’s image from the snapshot that we made earlier.

Shell

$ sudo mount -o bind /dev /mnt/dev

$ sudo mount -o bind /dev/shm /mnt/dev/shm

$ sudo mount -o bind /sys /mnt/sys

$ sudo mount -o bind /run /mnt/run

$ sudo mount -t proc proc /mnt/proc

$ sudo mount -t devpts devpts /mnt/dev/pts

$ sudo chroot /mnt

root@ip-10-0-0-189:/#

Notice how the prompt changed after the chroot. That is your clue that it is active.

Correct the Problem

This step depends on whatever is wrong. I won’t bore you with the problem I had.

Unmount the New Volume

Exit the chroot and unmount the rescue volume from the rescue server.

Shell

# exit

$ sudo umount /mnt/dev

$ sudo umount /mnt/dev/shm

$ sudo umount /mnt/sys

$ sudo umount /mnt/run

$ sudo umount /mnt/proc

$ sudo umount /mnt/dev/pts

$ sudo umount /mnt

Detach the rescue volume from the rescue server. This can be done from any machine that is configured with aws cli for use with your account credentials.

Shell

$ aws ec2 detach-volume --volume-id $AWS_RESCUE_VOLUME_ID

$ aws ec2 wait volume-available --volume-id $AWS_RESCUE_VOLUME_ID

Unmount the Problem Volume

The problem server must be shut down for this to work. Detach the problem volume from the problem server. This can be done from any machine that is configured with aws cli for use with your account credentials.

Shell

$ aws ec2 stop-instances --instance-id $AWS_PROBLEM_INSTANCE_ID

$ aws ec2 wait instance-stopped --instance-ids $AWS_PROBLEM_INSTANCE_ID

$ aws ec2 detach-volume --volume-id $AWS_PROBLEM_VOLUME_ID

$ aws ec2 wait volume-available --volume-id $AWS_PROBLEM_VOLUME_ID

Replace the Problem Volume

Now it is time to replace the problem volume containing the problem boot drive on the problem system with the newly created volume. BTW, AWS EC2 always refers to boot drives as /dev/sda1, even when the device has a different name, such as /dev/xvdb1.

replaceSystemVolume Bash function

This Bash function detaches the volume containing the current boot drive of an EC2 instance and replaces it with another volume. If the EC2 instance is running then it is first stopped.

replaceSystemVolume

#!/bin/bash

function replaceSystemVolume {
  # $1 - EC2 instance id
  # $2 - new volume to mount as system boot drive

  export EC2_INSTANCE="$(
    aws ec2 describe-instances --instance-ids "$1" | \
    jq -r ".Reservations[].Instances[0]"
  )"

  export EC2_NAME="$(
    jq -r ".Tags[] | select(.Key==\"Name\") | .Value" <<< "$EC2_INSTANCE"
  )"

  export ATTACHED_VOLUME_ID="$(
    jq -r ".BlockDeviceMappings[].Ebs.VolumeId" <<< "$EC2_INSTANCE"
  )"

  if [[ "$ATTACHED_VOLUME_ID" == "$2" ]]; then
    >&2 echo "VolumeId $2 is already attached to EC2 instance $1"
    exit 1
  fi

  export EC2_STATE="$(
    jq -r ".State.Name" <<< "$EC2_INSTANCE"
  )"

  if [ "$EC2_STATE" == running ]; then
    echo "Stopping EC2 instance $1"
    aws ec2 stop-instances --instance-ids "$1"
    aws ec2 wait instance-stopped --instance-ids "$1"
  fi

  aws ec2 detach-volume --volume-id "$ATTACHED_VOLUME_ID"
  aws ec2 wait volume-available --volume-id "$ATTACHED_VOLUME_ID"

  aws ec2 attach-volume \
    --device /dev/sda1 \
    --instance-id "$1" \
    --volume-id "$2"
  aws ec2 wait volume-in-use --volume-id "$2"

  aws ec2 start-instances --instance-ids "$1"
  aws ec2 wait instance-started --instance-ids "$1"
}


set -e

replaceSystemVolume "$@"

Here is how to use it:

Shell

$ replaceSystemVolume "$AWS_PROBLEM_INSTANCE_ID" "$AWS_RESCUE_VOLUME_ID"

Preview 2 instance id is AWS_EC2_RESCUE_ID. Replace rescue volume on preview with preview's original volume:

Shell

$ replaceSystemVolume "$AWS_EC2_RESCUE_ID" "$AWS_PREVIEW_VOLUME_ID"

Boot the problem system

Boot the problem system and verify the problem is solved.

Shell

$ aws ec2 start-instances --instance-ids $AWS_PROBLEM_INSTANCE_ID

$ aws ec2 wait instance-started --instance-ids $AWS_PROBLEM_INSTANCE_ID

Acknowledgements

This article was inspired by this excellent article, which uses the AWS web console to achieve similar results.

Working With EC2 Spot Instances From AWS CLI

2020-10-24T00:00:00-04:00

AWS EC2 T2.medium spot instances cost less than 2 cents per hour for Linux and can be created very easily from the command line. They self-destruct once shut down. These powerful virtual machines can do an incredible amount of work in an hour for less than 2 cents!

This article shows how all this can be done via the command line. I also provide an interactive bash script for automating the process of obtaining and releasing an EC2 spot instance.

Once again this article uses AWS CLI and jq.

Create and Import a New Keypair

I want to create a new temporary ssh keypair that will just be used for this spot instance. The name of the new key pair will be of the form ~/.ssh/rsa-YYYY-MM-DD-mm-ss.

Shell

$ AWS_KEY_PAIR_NAME="rsa-$( date '+%Y-%m-%d-%H-%M-%S' )"

$ echo "$AWS_KEY_PAIR_NAME"
rsa-2020-11-04-10-43-54 

$ AWS_KEY_PAIR_FILE="~/.ssh/$AWS_KEY_PAIR_NAME"

$ echo "$AWS_KEY_PAIR_FILE"
~/.ssh/rsa-2020-11-04-10-43-54

Now we can make the keypair. AWS EC2 does not accept keys longer than 2048 bits.

Shell

$ ssh-keygen -b 2048 -f "$AWS_KEY_PAIR_FILE" -P "" -N "" -t rsa
Generating public/private rsa key pair.
  Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-10-43-54
  Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-10-43-54.pub
  The key fingerprint is:
  SHA256:bQScX0UMn0xGDorxSvElMZzwMyyk7hgs2FNbshBNenA mslinn@mslinn.com
  The key’s randomart image is:
  +---[RSA 2048]----+
  |  ooE  .*++o+**  |
  |   =.  ooXo=.B.. |
  |  o + o +.X.  =  |
  | o = * . =.o     |
  |. + = . S o      |
  |   o +   .       |
  |    . .          |
  |                 |
  |                 |
  +----[SHA256]-----+

The new public key will be stored in ~/.ssh/2020-11-04-10-43-54.pub and the new private key will be stored in ~/.ssh/2020-11-04-10-43-54.

Now set the permissions for the key.

Shell

$ chmod 400 "$AWS_KEY_PAIR_FILE"

Now we can import the key pair into AWS:

Shell

$ aws ec2 import-key-pair \
  --key-name "$AWS_KEY_PAIR_NAME" \
  --public-key-material "fileb://${AWS_KEY_PAIR_FILE}.pub"
{
  "KeyFingerprint": "c7:76:90:53:17:d0:fc:ba:45:dd:93:d2:93:03:c2:19",
  "KeyName": "2020-11-04-10-43-54",
  "KeyPairId": "key-092a2306ec3f4aff6"
}

Select an AMI

New AMIs become available every day. You probably want your EC2 spot instance to be created from the most recent AMI that matches your needs. For most of my work I want an Ubuntu 64-bit Intel/AMD server distribution. AWS documentation is helpful and gives us a head start in automating the AMI selection.

The following incantation sets an environment variable called AWS_AMI to the details in JSON syntax of the AMI for the most recent 64-bit Ubuntu server release for Intel/AMD architecture. The OwnerId of Canonical, the publisher of Ubuntu, is 099720109477.

Shell

$ AWS_AMI="$(
  aws ec2 describe-images \
    --owners 099720109477 \
    --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-━━━━━???-━━━━━???-amd64-server-━━━━━???" \
              "Name=state,Values=available" \
    --query "reverse(sort_by(Images, &CreationDate))[:1]" | \
  jq -r '.[0]'
)"

$ echo "$AWS_AMI"
{
  "Architecture": "x86_64",
  "CreationDate": "2020-10-30T14:07:42.000Z",
  "ImageId": "ami-0c71ec98278087e60",
  "ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030",
  "ImageType": "machine",
  "Public": true,
  "OwnerId": "099720109477",
  "PlatformDetails": "Linux/UNIX",
  "UsageOperation": "RunInstances",
  "State": "available",
  "BlockDeviceMappings": [
    {
      "DeviceName": "/dev/sda1",
      "Ebs": {
        "DeleteOnTermination": true,
        "SnapshotId": "snap-00bf581086dd686e5",
        "VolumeSize": 8,
        "VolumeType": "gp2",
        "Encrypted": false
      }
    },
    {
      "DeviceName": "/dev/sdb",
      "VirtualName": "ephemeral0"
    },
    {
      "DeviceName": "/dev/sdc",
      "VirtualName": "ephemeral1"
    }
  ],
  "Description": "Canonical, Ubuntu, 20.10, amd64 groovy image build on 2020-10-30",
  "EnaSupport": true,
  "Hypervisor": "xen",
  "Name": "ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030",
  "RootDeviceName": "/dev/sda1",
  "RootDeviceType": "ebs",
  "SriovNetSupport": "simple",
  "VirtualizationType": "hvm"
}

Now let's extract the ID of the AMI image and save it as AWS_AMI_ID.

Shell

$ AWS_AMI_ID="$( jq -r '.[0].ImageId' <<< "$AWS_AMI" )"

$ echo "$AWS_AMI_ID"
ami-0c71ec98278087e60

Create an EC2 Spot Instance

For my work I often want my spot instance to be created in the same VPC subnet as my other resources, with the same security group. That is why the following environment variables are defined for the Groups and SubnetId values within the network-interfaces option, as well as the AWS region. The script at the end of this article offers an easier way of obtaining all these values.

Shell

$ AWS_GROUP=sg-4cbc6f35

$ AWS_SUBNET=subnet-49de033f

$ AWS_ZONE=us-east-1c

$ AWS_EC2_TYPE=t2.medium

The following creates an AWS EC2 spot instance with a public IP address and runs it. Details about the newly created spot instance are stored as JSON in AWS_SPOT_INSTANCE.

Shell

$ AWS_SPOT_INSTANCE="$(
  aws ec2 run-instances \
    --image-id "$AWS_AMI_ID" \
    --instance-market-options '{ "MarketType": "spot" }' \
    --instance-type "$AWS_EC2_TYPE" \
    --key-name "$AWS_KEY_PAIR_NAME" \
    --network-interfaces "[ {
        \"DeviceIndex\": 0,
        \"Groups\": [\"$AWS_GROUP\"],
        \"SubnetId\": \"$AWS_SUBNET\",
        \"DeleteOnTermination\": true,
        \"AssociatePublicIpAddress\": true
      } ]" \
    --placement "{ \"AvailabilityZone\": \"$AWS_ZONE\" }" | \
    jq -r .Instances[0]
)"

$ echo "$AWS_SPOT_INSTANCE"
{
  "AmiLaunchIndex": 0,
  "ImageId": "ami-0dba2cb6798deb6d8",
  "InstanceId": "i-012a54aefcd333de9",
  "InstanceType": "t2.small",
  "KeyName": "rsa-2020-11-03.pub",
  "LaunchTime": "2020-11-03T23:19:50.000Z",
  "Monitoring": {
      "State": "disabled"
  },
  "Placement": {
      "AvailabilityZone": "us-east-1c",
      "GroupName": "",
      "Tenancy": "default"
  },
  "PrivateDnsName": "ip-10-0-0-210.ec2.internal",
  "PrivateIpAddress": "10.0.0.210",
  "ProductCodes": [],
  "PublicDnsName": "",
  "State": {
      "Code": 0,
      "Name": "pending"
  },
  "StateTransitionReason": "",
  "SubnetId": "subnet-49de033f",
  "VpcId": "vpc-f16a0895",
  "Architecture": "x86_64",
  "BlockDeviceMappings": [],
  "ClientToken": "026583fb-c94e-4bca-bdd2-8dcdcaa3aae9",
  "EbsOptimized": false,
  "EnaSupport": true,
  "Hypervisor": "xen",
  "InstanceLifecycle": "spot",
  "NetworkInterfaces": [
      {
          "Attachment": {
              "AttachTime": "2020-11-03T23:19:50.000Z",
              "AttachmentId": "eni-attach-04feb4d36cf5c6792",
              "DeleteOnTermination": true,
              "DeviceIndex": 0,
              "Status": "attaching"
          },
          "Description": "",
          "Groups": [
              {
                  "GroupName": "testSG",
                  "GroupId": "sg-4cbc6f35"
              }
          ],
          "Ipv6Addresses": [],
          "MacAddress": "0a:6d:ba:c5:65:4b",
          "NetworkInterfaceId": "eni-09ef90920cfb29dd9",
          "OwnerId": "031372724784",
          "PrivateIpAddress": "10.0.0.210",
          "PrivateIpAddresses": [
              {
                  "Primary": true,
                  "PrivateIpAddress": "10.0.0.210"
              }
          ],
          "SourceDestCheck": true,
          "Status": "in-use",
          "SubnetId": "subnet-49de033f",
          "VpcId": "vpc-f16a0895",
          "InterfaceType": "interface"
      }
  ],
  "RootDeviceName": "/dev/sda1",
  "RootDeviceType": "ebs",
  "SecurityGroups": [
      {
          "GroupName": "testSG",
          "GroupId": "sg-4cbc6f35"
      }
  ],
  "SourceDestCheck": true,
  "SpotInstanceRequestId": "sir-rrs9gm3j",
  "StateReason": {
      "Code": "pending",
      "Message": "pending"
  },
  "VirtualizationType": "hvm",
  "CpuOptions": {
      "CoreCount": 1,
      "ThreadsPerCore": 1
  },
  "CapacityReservationSpecification": {
      "CapacityReservationPreference": "open"
  },
  "MetadataOptions": {
      "State": "pending",
      "HttpTokens": "optional",
      "HttpPutResponseHopLimit": 1,
      "HttpEndpoint": "enabled"
  }
}

Now extract the EC2 spot instance id and save it in AWS_SPOT_ID.

Shell

$ AWS_SPOT_ID="$(
  jq -r .InstanceId <<< "$AWS_SPOT_INSTANCE"
)"

$ echo "$AWS_SPOT_ID"
i-012a54aefcd333de9

Wait for the instance to start.

Shell

$ aws ec2 wait instance-running --instance-ids "$AWS_SPOT_ID"

Connect to the Spot Instance

In order to ssh into the spot instance we first need to discover its IP address, which is saved in AWS_SPOT_IP.

Shell

$ AWS_SPOT_IP="$(
  aws ec2 describe-instances \
    --instance-ids $AWS_SPOT_ID | \
  jq -r .Reservations[0].Instances[0].PublicIpAddress
)"

$ echo "$AWS_SPOT_IP"
54.242.88.254

Now we can connect to the spot instance via ssh. The default userid for Ubuntu is ubuntu.

Shell

$ ssh -i "$AWS_KEY_PAIR_FILE" "ubuntu@$AWS_SPOT_IP"
Warning: No xauth data; using fake authentication data for X11 forwarding.
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.11.0-1027-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Jan 27 20:28:22 UTC 2022

  System load:  0.06              Processes:             113
  Usage of /:   18.3% of 7.69GB   Users logged in:       0
  Memory usage: 5%                IPv4 address for eth0: 10.0.0.29
  Swap usage:   0%

1 update can be applied immediately.
To see these additional updates run: apt list --upgradable


The list of available updates is more than a week old.
To check for new updates run: sudo apt update


The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

/usr/bin/xauth:  file /home/ubuntu/.Xauthority does not exist
To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.

ubuntu@ip-10-0-0-29:~$

Do your work on the spot instance now. We'll disconnect and clean up next.

Disconnect from the Spot Instance and Clean Up

Using the Command Line

Once the spot instance stops it is automatically terminated. The instance will survive a reboot, but not a halt. From a prompt on the spot instance, type:

shell (Spot Instance)

$ sudo halt

Back in the shell that launched the spot instance, wait for the spot instance to stop before cleaning up.

Shell

$ aws ec2 wait instance-stopped --instance-ids $AWS_SPOT_ID

Delete the temporary ssh keypair we created. Copies exist on AWS and the local machine; we need to remove all of them, like this:

Shell

$ aws ec2 delete-key-pair --key-name $AWS_KEY_PAIR_NAME

$ rm $AWS_KEY_PAIR_FILE $AWS_KEY_PAIR_FILE.pub

Checking With Web Console

You can use the web console to verify that all the spot instances were shut down, and the key pairs were deleted.

Bash Script createEc2Spot

Source Code

This script does everything discussed above, plus it prompts the user with default values for parameters unique to each invocation. Click on the name of the script and save it this script to a directory on your PATH.

createEc2Spot

#!/bin/bash

# Author: Mike Slinn mslinn@mslinn.com
# Initial version 2020-1-25
# Last modified 2022-01-27

set -e

function readWithDefault {
  >&2 printf "\n$1: "
  read -e -i "$2" VALUE
  echo "$VALUE"
}

echo "The AWS EC2 spot instance needs to share settings with the existing EC2 instance you want to affect."
echo "The easiest way to do this is to reference an EC2 instance with a Name tag, so it can be identified."
echo "If there is no such EC2 instance, delete the default value in the next prompt, and you will be able to specify the details manually."
AWS_EC2_NAME="$( readWithDefault "AWS EC2 Name tag value" production )"
if [ "$AWS_EC2_NAME" ]; then
  AWS_EC2_PRODUCTION="$(
    aws ec2 describe-instances | \
    jq ".Reservations[].Instances[] | select((.Tags[]?.Key==\"Name\") and (.Tags[]?.Value==\"$AWS_EC2_NAME\"))"
  )"
  AWS_GROUP="$( jq -r ".NetworkInterfaces[].Groups[].GroupId" <<< "$AWS_EC2_PRODUCTION" )"
  AWS_SUBNET="$( jq -r ".SubnetId" <<< "$AWS_EC2_PRODUCTION" )"
  AWS_ZONE="$( jq -r ".Placement.AvailabilityZone" <<< "$AWS_EC2_PRODUCTION" )"
else
  echo "Please answer a few questions so the AWS EC2 spot instance can be created."
  AWS_GROUP="$( readWithDefault "AWS security group" sg-4cbc6f35 )"
  AWS_SUBNET="$( readWithDefault "EC2 subnet" subnet-49de033f )"
  AWS_ZONE="$( readWithDefault "AWS availability zone" us-east-1c )"
fi

echo "EC2 spot instances are really inexpensive, so be generous with the size of the machine type for this spot instance."
AWS_EC2_TYPE="$( readWithDefault "EC2 machine type" t2.medium )"

AWS_KEY_PAIR_NAME="rsa-$( date '+%Y-%m-%d-%H-%M-%S' )"

AWS_KEY_PAIR_FILE="$HOME/.ssh/$AWS_KEY_PAIR_NAME"

ssh-keygen -b 2048 -f "$AWS_KEY_PAIR_FILE" -P "" -N "" -t rsa # -q
chmod 400 "$AWS_KEY_PAIR_FILE"

aws ec2 import-key-pair \
  --key-name "$AWS_KEY_PAIR_NAME" \
  --public-key-material "fileb://$AWS_KEY_PAIR_FILE.pub"

echo "Searching for the latest 64-bit Intel/AMD Ubuntu AMI by Canonical."
AWS_AMI="$(
  aws ec2 describe-images \
    --owners 099720109477 \
    --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-????????-????????-amd64-server-????????" \
              "Name=state,Values=available" \
    --query "reverse(sort_by(Images, &CreationDate))[:1]" | \
  jq -r '.[0]'
)"

echo "Obtaining the AMI image ID."
AWS_AMI_ID="$( jq -r '.ImageId' <<< "$AWS_AMI" )"

echo "Creating the EC2 spot instance."
AWS_SPOT_INSTANCE="$(
  aws ec2 run-instances \
    --image-id $AWS_AMI_ID \
    --instance-market-options '{ "MarketType": "spot" }' \
    --instance-type $AWS_EC2_TYPE \
    --key-name $AWS_KEY_PAIR_NAME \
    --network-interfaces "[ {
        \"DeviceIndex\": 0,
        \"Groups\": [\"$AWS_GROUP\"],
        \"SubnetId\": \"$AWS_SUBNET\",
        \"DeleteOnTermination\": true,
        \"AssociatePublicIpAddress\": true
      } ]" \
    --placement "{ \"AvailabilityZone\": \"$AWS_ZONE\" }" | \
    jq -r .Instances[0]
)"

echo "Obtaining the EC2 spot instance ID."
AWS_SPOT_ID="$(
  jq -r .InstanceId <<< "$AWS_SPOT_INSTANCE"
)"

echo "Awaiting for the EC2 spot instance $AWS_SPOT_ID to enter the running state."
aws ec2 wait instance-running --instance-ids $AWS_SPOT_ID

echo "Obtaining the IP address of the new EC2 spot instance $AWS_SPOT_ID."
AWS_SPOT_IP="$(
  aws ec2 describe-instances \
    --instance-ids $AWS_SPOT_ID | \
  jq -r .Reservations[0].Instances[0].PublicIpAddress
)"

echo "About to ssh to the EC2 spot instance as ubuntu@$AWS_SPOT_IP using $AWS_KEY_PAIR_FILE."
echo "When you are done, type: sudo halt."
echo "The spot instance will then terminate and be gone forever."
echo "Any predefined resources, such as volumes that you attach will be freed."
ssh -i "$AWS_KEY_PAIR_FILE" "ubuntu@$AWS_SPOT_IP"

echo "Waiting for the EC2 spot instance $AWS_SPOT_ID to enter the stopped state."
aws ec2 wait instance-stopped --instance-ids "$AWS_SPOT_ID"

echo "The spot instance is no longer available. Deleting its keypair."
aws ec2 delete-key-pair --key-name AWS_KEY_PAIR_NAME
rm $AWS_KEY_PAIR_FILE $AWS_KEY_PAIR_FILE.pub

Make the script executable.

Shell

$ chmod a+x createEc2Spot

Sample Usage

The script is easy to use:

Shell

$ createEc2Spot
The AWS EC2 spot instance needs to share settings with the existing EC2 instance you want to affect.
The easiest way to do this is to reference an EC2 instance with a Name tag, so it can be identified.
If there is no such EC2 instance, delete the default value in the next prompt, and you will be able to specify the details manually.

AWS EC2 Name tag value: production

EC2 spot instances are really inexpensive, so be generous with the size of the machine type for this spot instance.
EC2 machine type: t2.medium

Generating public/private rsa key pair.
  Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-10-43-54
  Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-10-43-54.pub
  The key fingerprint is:
  SHA256:bQScX0UMn0xGDorxSvElMZzwMyyk7hgs2FNbshBNenA mslinn@mslinn.com
  The key’s randomart image is:
  +---[RSA 2048]----+
  |  ooE  .*++o+**  |
  |   =.  ooXo=.B.. |
  |  o + o +.X.  =  |
  | o = * . =.o     |
  |. + = . S o      |
  |   o +   .       |
  |    . .          |
  |                 |
  |                 |
  +----[SHA256]-----+
    "KeyFingerprint": "be:19:50:59:a1:83:ea:c1:91:1e:2f:d6:31:64:9a:c0",
    "KeyName": "rsa-2022-01-27-50-02",
    "KeyPairId": "key-072a3f0545864526a"
}
Searching for the latest 64-bit Intel/AMD Ubuntu AMI by Canonical.
Obtaining the AMI image ID.
Creating the EC2 spot instance.
Obtaining the EC2 spot instance ID.
Awaiting for the EC2 spot instance i-03d09e364ed15a448 to enter the running state.
Obtaining the IP address of the new EC2 spot instance i-03d09e364ed15a448.
54.242.88.254
When you are done, type: sudo halt.
The spot instance will then terminate and be gone forever.
Any predefined resources, such as volumes that you attach will be freed.

Waiting for the EC2 spot instance $AWS_SPOT_ID to enter the stopped state.

Now do your work on the spot instance, and then run sudo halt. Once the spot instance shuts down, it is destroyed and the script cleans up. Do not try to reboot the spot instance, because that shuts it down and it goes away instead of coming back up. Now do your work on the spot instance. From a prompt on the spot instance, type:

shell (Spot Instance)

$ sudo halt

Back in the shell on your computer, you should see:

Shell

$ sudo halt
The spot instance is no longer available. Deleting its keypair.

Bash Script aws_ec2_functions

Source Code

This is another script does the same thing as the previous script, but in steps. Click on the name of the script and save it this script to a directory on your PATH.

aws_ec2_functions

#!/bin/bash

# Author: Mike Slinn mslinn@mslinn.com
# Initial version 2022-01-28
# Last modified 2022-01-28


function readWithDefault {
  # prompt user for a value, with a default
  >&2 printf "\n$1: "
  read -e -i "$2" VALUE
  echo "$VALUE"
}


function requires {
  # Halts if any of the supplied arguments is not the name of a defined environment variable
  for ENV_VAR in "$@"; do
    if [ -z "${!ENV_VAR}" ]; then
    echo "Error: ${ENV_VAR} is undefined."
    return 1 2> /dev/null || exit 1
  #else
  #  echo "${ENV_VAR} has value ${!ENV_VAR}"
  fi
  done
}


function attachVolumeToSpot {
  requires AWS_SPOT_ID AWS_NEW_VOLUME_ID || return

  export AWS_ATTACH_VOLUME="$(
    aws ec2 attach-volume \
      --device /dev/xvdh \
      --instance-id $AWS_SPOT_ID \
      --volume-id $AWS_NEW_VOLUME_ID
  )"

  aws ec2 wait volume-in-use --volume-id "$AWS_NEW_VOLUME_ID"

  export AWS_ATTACH_VOLUME_DEVICE="$(
    aws ec2 describe-volumes \
      --volume-id "$AWS_NEW_VOLUME_ID" | \
    jq -r .Volumes[0].Attachments[0].Device
  )"
}


function chroot {
  requires AWS_NEW_VOLUME_ID || return

  # Use the /tmp/mounter script built by attachVolumeToSpot

  aws ec2 detach-volume --volume-id $AWS_NEW_VOLUME_ID
  aws ec2 wait volume-available --volume-id $AWS_NEW_VOLUME_ID
}


function copyScriptToSpot {
  requires AWS_ATTACH_VOLUME_DEVICE

  cat >/tmp/mounter <<EOF
sudo mount "${AWS_ATTACH_VOLUME_DEVICE}1" /mnt

sudo mount -o bind /dev /mnt/dev
sudo mount -o bind /dev/shm /mnt/dev/shm
sudo mount -o bind /sys /mnt/sys
sudo mount -o bind /run /mnt/run
sudo mount -t proc proc /mnt/proc
sudo mount -t devpts devpts /mnt/dev/pts
sudo chroot /mnt

# If the user types 'exit' then this script continues.
# Otherwise, if the user types 'sudo halt' this script is not needed, AWS does the cleanup.

sudo umount /mnt/dev
sudo umount /mnt/dev/shm
sudo umount /mnt/sys
sudo umount /mnt/run
sudo umount /mnt/proc
sudo umount /mnt/dev/pts
sudo umount /mnt
EOF
  set -xv
  chmod a+x /tmp/mounter

  scpToSpot /tmp/mounter mounter

  echo "About to ssh into the spot instance. Run ./mounter to enter chroot using the new volume"
  sshToSpot
  # Proves the drive is mounted:
  #df -h | grep '^/dev/' | grep -v '^/dev/loop'

  rm /tmp/mounter
}


function deleteEc2SpotInstance {
  requires AWS_KEY_PAIR_NAME AWS_SPOT_ID || return

  # Hope that this does not bomb out if the user types 'sudo halt'
  aws ec2 cancel-spot-instance-requests --spot-instance-request-ids "$AWS_SPOT_ID"

  echo "Waiting for the EC2 spot instance $AWS_SPOT_ID to enter the stopped state."
  aws ec2 wait instance-stopped --instance-ids "$AWS_SPOT_ID"

  echo "The spot instance is no longer available. Deleting its keypair."
  aws ec2 delete-key-pair --key-name AWS_KEY_PAIR_NAME
  rm $AWS_KEY_PAIR_FILE $AWS_KEY_PAIR_FILE.pub
}


function findEc2 {
  echo "The existing AWS EC2 instance needs a snapshot to be made, which will then be turned into a volume and then mounted on a new AWS EC2 spot instance."
  echo "This script looks for an EC2 instance with a Name tag, so it can be identified."
  export AWS_EC2_ORIGINAL_NAME="$( readWithDefault "AWS EC2 Name tag value" production )"

  export AWS_EC2_ORIGINAL="$(
    aws ec2 describe-instances | \
    jq ".Reservations[].Instances[] | select((.Tags[]?.Key==\"Name\") and (.Tags[]?.Value==\"$AWS_EC2_ORIGINAL_NAME\"))"
  )"

  export AWS_ORIGINAL_EC2_INSTANCE_ID="$(
    jq -r .InstanceId <<< "$AWS_EC2_ORIGINAL"
  )"

  export AWS_ORIGINAL_EC2_IP="$(
    jq -r .PublicIpAddress <<< "$AWS_EC2_ORIGINAL"
  )"

  AWS_ORIGINAL_VOLUME_ID="$(
    jq -r '.BlockDeviceMappings[].Ebs.VolumeId' <<< "$AWS_EC2_ORIGINAL"
  )"

  export AWS_GROUP="$(
    jq -r ".NetworkInterfaces[].Groups[].GroupId" <<< "$AWS_EC2_ORIGINAL"
  )"

  export AWS_SUBNET="$( jq -r ".SubnetId" <<< "$AWS_EC2_ORIGINAL" )"
  export AWS_ZONE="$( jq -r .Placement.AvailabilityZone <<< "$AWS_EC2_ORIGINAL" )"

  echo "Original EC2 instance $AWS_ORIGINAL_EC2_INSTANCE_ID
  is at IP address $AWS_ORIGINAL_EC2_IP,
  has EBS volume $AWS_ORIGINAL_VOLUME_ID,
  with security group $AWS_GROUP,
  in $AWS_SUBNET,
  in the $AWS_ZONE zone."
}


function findSnapshot {
  # Look for a snapshot previously created by this script
  export AWS_SNAPSHOT_ID="$(
    aws ec2 describe-snapshots \
      --owner-ids self \
      --filters Name=tag:Name,Values=TestScript | \
    jq -r .Snapshots[].SnapshotId
  )"
}


function findVolume {
  # Look for a snapshot previously created by this script

  requires AWS_ZONE AWS_SNAPSHOT_ID || return

  export AWS_ATTACH_VOLUME_DEVICE="$(
    aws ec2 describe-volumes \
      --filters Name=tag:Name,Values=TestScript | \
    jq -r .Volumes[0].Attachments[0].Device
  )"
}


function fn_help {
  echo "Bash functions to make working with AWS EC2 easier via the command line.
Source this file then call the functions:
  attachVolumeToSpot
  chroot
  copyScriptToSpot
  deleteEc2SpotInstance
  findEc2
  findSnapshot
  findVolume
  latestUbuntuAmi
  makeEc2SpotInstance
  makeSnapshot
  makeVolumeFromSnapshot
  mountVolumeOnSpot
  scpToSpot
  sshToSpot

Typical usage requires ' || true' to keep the terminal open if a problem occurs.
This is unnecessary if you invoke from another bash script.
  source aws_ec2_functions
  findEc2 || true
  makeSnapshot || true
  makeVolumeFromSnapshot || true
  latestUbuntuAmi || true
  makeEc2SpotInstance || true
  attachVolumeToSpot || true
  copyScriptToSpot || true

  ... or, to perform all of the above:
  source aws_ec2_functions
  prepare_spot || true

  ... another way to perform all of the above:
  aws_ec2_functions run

  ... to pick up from a failed attempt, which created a snapshot and a volume, but did not make a spot instance, or the spot instance has been cancelled:
  findSnapshot || true
  findVolume || true
  latestUbuntuAmi || true
  makeEc2SpotInstance || true
  attachVolumeToSpot || true
  copyScriptToSpot || true
"
}


function latestUbuntuAmi {
  export AWS_AMI="$(
    aws ec2 describe-images \
      --owners 099720109477 \
      --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-????????-????????-amd64-server-????????" \
                "Name=state,Values=available" \
      --query "reverse(sort_by(Images, &CreationDate))[:1]" | \
    jq -r '.[0]'
  )"

  export AWS_AMI_ID="$( jq -r '.ImageId' <<< "$AWS_AMI" )"

  echo "The most recent Ubuntu AMI ID is $AWS_AMI_ID"
}


function makeEc2SpotInstance {
  requires AWS_AMI_ID AWS_GROUP AWS_SUBNET AWS_ZONE || return

  export AWS_KEY_PAIR_NAME="rsa-$( date '+%Y-%m-%d-%H-%M-%S' )"

  export AWS_KEY_PAIR_FILE="$HOME/.ssh/$AWS_KEY_PAIR_NAME"

  ssh-keygen -b 2048 -f "$AWS_KEY_PAIR_FILE" -P "" -N "" -t rsa # -q
  chmod 400 "$AWS_KEY_PAIR_FILE"

  aws ec2 import-key-pair \
    --key-name "$AWS_KEY_PAIR_NAME" \
    --public-key-material "fileb://$AWS_KEY_PAIR_FILE.pub"

  echo "EC2 spot instances are really inexpensive, so be generous with the size of the machine type for this spot instance."
  export AWS_EC2_TYPE="$( readWithDefault "EC2 machine type" t2.medium )"

  export AWS_SPOT_INSTANCE="$(
    aws ec2 run-instances \
      --image-id $AWS_AMI_ID \
      --instance-market-options '{ "MarketType": "spot" }' \
      --instance-type $AWS_EC2_TYPE \
      --key-name $AWS_KEY_PAIR_NAME \
      --network-interfaces "[ {
          \"DeviceIndex\": 0,
          \"Groups\": [\"$AWS_GROUP\"],
          \"SubnetId\": \"$AWS_SUBNET\",
          \"DeleteOnTermination\": true,
          \"AssociatePublicIpAddress\": true
        } ]" \
      --placement "{ \"AvailabilityZone\": \"$AWS_ZONE\" }" | \
      jq -r .Instances[0]
  )"

  export AWS_SPOT_ID="$(
    jq -r .InstanceId <<< "$AWS_SPOT_INSTANCE"
  )"

  echo "Waiting for the EC2 spot instance $AWS_SPOT_ID to enter the running state. This usually takes about 1 minute."
  aws ec2 wait instance-running --instance-ids "$AWS_SPOT_ID"

  export AWS_SPOT_IP="$(
    aws ec2 describe-instances \
      --instance-ids $AWS_SPOT_ID | \
    jq -r .Reservations[0].Instances[0].PublicIpAddress
  )"
  echo "The IP address of the new EC2 spot instance $AWS_SPOT_ID is $AWS_SPOT_IP."
}


function makeSnapshot {
  # Makes an AWS EC2 snapshot with name TestScript from AWS_ORIGINAL_VOLUME_ID

  requires AWS_ORIGINAL_VOLUME_ID || return

  COMPLETION_TIME="$(date --date="@$(($(date +%s)+120))" +"%H":"%M":"%S")"
  echo "Snapshots take about 2 minutes. This one should complete by $COMPLETION_TIME."

  export AWS_SNAPSHOT_ID="$(
    aws ec2 create-snapshot --volume-id "$AWS_ORIGINAL_VOLUME_ID" \
    --description "production $( date '+%Y-%m-%d' )" \
    --tag-specifications "ResourceType=snapshot,Tags=[{Key=Created, Value=`date '+%Y-%m-%d'`},{Key=Name, Value=\"TestScript\"}]" | \
    jq -r .SnapshotId
  )"

  aws ec2 wait snapshot-completed --snapshot-ids "$AWS_SNAPSHOT_ID"

  echo "Snapshot $AWS_SNAPSHOT_ID is complete."
}


function makeVolumeFromSnapshot {
  # Makes an AWS EC2 volume with name TestScript

  requires AWS_ZONE AWS_SNAPSHOT_ID || return

  export AWS_NEW_VOLUME_ID="$(
    aws ec2 create-volume \
    --availability-zone $AWS_ZONE \
    --snapshot-id $AWS_SNAPSHOT_ID \
    --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=TestScript}]' | \
    jq -r .VolumeId
  )"

  echo "Waiting for AWS volume $AWS_NEW_VOLUME_ID to become available. This usually takes about 20 seconds."

  aws ec2 wait volume-available --volume-id "$AWS_NEW_VOLUME_ID"

  echo "$AWS_NEW_VOLUME_ID"
}


function prepare_spot {
  # Run everything
  findEc2
  makeSnapshot
  makeVolumeFromSnapshot
  latestUbuntuAmi
  makeEc2SpotInstance
  attachVolumeToSpot
  copyScriptToSpot
}


function scpToSpot {
  requires AWS_KEY_PAIR_FILE AWS_SPOT_IP || return

  scp -pi "$AWS_KEY_PAIR_FILE" "$1" "ubuntu@$AWS_SPOT_IP:$2"
}


function sshToSpot {
  requires AWS_KEY_PAIR_FILE AWS_SPOT_IP || return

  echo "When you are done, type: sudo halt."
  echo "The AWS EC2 spot instance will then terminate and be gone forever."
  echo "Any predefined resources, such as volumes that you attach will be freed."

  ssh -i "$AWS_KEY_PAIR_FILE" "ubuntu@$AWS_SPOT_IP" "$*"
}


if [ "$1" == prepare_spot ]; then
  echo "Starting..."
  prepare_spot || true
elif [ "$1" ]; then
  fn_help || true
else
  echo "Type fn_help to obtain help information"
fi

Make the script executable.

Shell

$ chmod a+x createEc2Spot

Sample Usage

View the help like this:

Shell

$ aws_ec2_functions -h
Bash functions to make working with AWS EC2 easier via the command line.
Source this file then call the functions, listed alphabetically:
  attachVolumeToSpot
  chroot
  copyScriptToSpot
  deleteEc2SpotInstance
  findEc2
  findSnapshot
  findVolume
  latestUbuntuAmi
  makeEc2SpotInstance
  makeSnapshot
  makeVolumeFromSnapshot
  mountVolumeOnSpot
  scpToSpot
  sshToSpot

Typical usage requires ' || true' to keep the terminal open if a problem occurs.
This is unnecessary if you invoke from another bash script.
  source aws_ec2_functions
  findEc2 || true
  makeSnapshot || true
  makeVolumeFromSnapshot || true
  latestUbuntuAmi || true
  makeEc2SpotInstance || true
  attachVolumeToSpot || true
  copyScriptToSpot || true

  ... or, to perform all of the above:
  source aws_ec2_functions
  prepare_spot || true

  ... another way to perform all of the above:
  aws_ec2_functions run

  ... to pick up from a failed attempt, which created a snapshot and a volume, but did not make a spot instance, or the spot instance has been cancelled:
  findSnapshot || true
  findVolume || true
  latestUbuntuAmi || true
  makeEc2SpotInstance || true
  attachVolumeToSpot || true
  copyScriptToSpot || true

Using the chroot

Using either of the preceding two ways to set up the chroot, enter it using the mounter script:

shell on Spot Instance

ubuntu@ip-10-0-0-193:~$ ls
mounter

ubuntu@ip-10-0-0-193:~$ ./mounter

ubuntu@ip-10-0-0-193:~$ echo "127.0.1.1 $(hostname)" >> /etc/hosts

root@ip-10-0-0-193:/# su ubuntu

ubuntu@ip-10-0-0-193:/$ # Do whatever you need to do

ubuntu@ip-10-0-0-193:/$ sudo halt  # Shut everything down

Scala-Style Lambda Function Placeholder Syntax in Python 3

2020-10-22T00:00:00-04:00

Pipes are the ultimate in functional programming. It is clear when reading code that uses pipes that data is not mutated. Piping data into and out of lambda functions (and regular functions) is a succinct way of elegantly expressing a computation. This article discusses how to do this using similar syntax in Python 3, Scala 2 and Scala 3.

After programming in Scala for more than ten years I have grown to appreciate Scala's implementation of lambda functions, including the ability to use the underscore character as a placeholder for variables. Scala 2.13 introduced pipelining between functions, which is rather like *nix pipes between processes.

Python 3 can also do something similar. This article demonstrates how to use sspipe and JulienPalard’s pipe with Scala's underscore placeholder for Python 3 lambda functions.

Python 3 Setup

The key concept is to use this specific Python import:

Shell

from sspipe import p, px, px as _

All the Python code examples that follow require this import. The Python code examples are modified versions of the sspipe examples to illustrate how to use underscores as placeholders.

This import is unusual because it imports px twice: once as a normal import, and once aliased to _. I use the _ alias to support Scala-like syntax, and I use px when I need to reference a parameter twice. Python is unlike Scala in that the Python compiler does not treat variables called _ specially; those variables are merely called _. I could use _ in Python code many times to refer to the same value, but a Scala programmer reading that code would expect that each reference to _ would be another input parameter, not a regular variable reference. The examples that follow should make this clear.

Python 3 Installation

Install sspipe using pip:

Shell

$ pip install --upgrade sspipe

Scala 2.13+ Setup

Scala 2.13 introduced ChainingOps, which adds chaining methods tap and pipe to every type. The key concept is to import the following prior to attempting the code examples below:

Shell

implicit class Smokin[A](val a: A) {
    import scala.util.chaining._
    import scala.language.implicitConversions

    implicit def |>[B](f: (A) => B): B = a.pipe(f)
}

Python and Scala Usage Examples

One Lambda Function and 1 Pipe

This Python example employs one lambda function and 1 pipe to add 2 to the number 5:

Shell

>>> 5 | _ + 2
7

The Scala equivalent of the above is:

Shell

scala> 5 |> ((_: Int) + 2)
val res0: Int = 7

Two Lambda Functions and 2 Pipes

This Python example employs two lambda functions and 2 pipes to multiply the previous result by 5 and then add the previous result. Recall that I said that in Python, an underscore when used this way is the name of a normal variable and that the compiler does not treat underscores as placeholders for lambda parameters. A Scala programmer would complain about the following code, because they would expect that the second lambda function would require 2 inputs:

Shell

>>> 5 | _ + 2 | _ * 5 + _
42

A better way to write the above would be to use the special variable px, which was imported above. Now everyone either knows that px holds the piped value, or they complain about px being a magic variable. A possible solution to this complaint would be to alias px to a more descriptive name, such as pipedValue … which is still magical, but at least it is more descriptive.

Shell

>>> 5 | _ + 2 | px * 5 + px
42

The Scala equivalent of the above is:

Shell

scala> 5 |> ((_: Int) + 2) |> ((x: Int) => x * 5 + x)
val res1: Int = 42

Two Lambda Functions and 3 Pipes

This Python example employs 2 lambda functions and 3 pipes to add 10 to the even numbers from 0 to 5, exclusive.

Shell

>>> (
    range(5)
      | p(filter, _ % 2 == 0)
      | p(map, _ + 10)
      | p(list)
)
[10, 12, 14]

Scala has a better way of performing this type of computation that does not require pipes or computation. It is better because it is simpler to understand.

Shell

scala> for {
     |   x <- (0 until 5).toList if x % 2 == 0
     |   y = x + 10
     | } yield y
val res12: List[Int] = List(10, 12, 14)

Other examples of placeholder syntax

NumPy expressions (NumPy is Python-specific):

Shell

range(10) | np.sin(_)+1 | p(plt.plot)

Pandas expressions (Pandas is Python-specific):

Shell

people_df | _.loc[_.age > 10, 'name']

Solution for the 2nd Project Euler exercise:

Shell

>>> def fib():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

>>> euler2 = (
      fib()
        | p.where(_ % 2 == 0)
        | p.take_while(_ < 4000000)
        | p.add()
   )

>>> euler2
4613732

Looking Ahead to Scala 3 (Dotty)

The next major version of Scala, due out in a few months, will probably allow a Scala 3 extension method to define the vertical bar as a method for more readabile code:

Shell

def [A,B](a: A) |(f: (A) => B): B = a.pipe(f)

# Sample usage:
val x = 5 | doSomething | doSomethingElse | doSomethingMore

To Learn More

My Introduction to Scala course on ScalaCourses.com teaches Scala lambda functions.

Converting All Images in a Website to webp Format

2020-08-15T00:00:00-04:00

I first launched this website in 1996. Since then, it has been re-incarnated using many different technologies. Presently I use Jekyll to assemble the site, then push the image to a web-enabled AWS S3 bucket that is edge-cached by an AWS CloudFront distribution.

Until yesterday, the site contained images with a mixture of image formats. I decided to convert them all to the new webp format. Because there are hundreds of images in over 120 web pages, I wrote a bash script called toWebP to do the work. This posting provides the toWebP script plus instructions on how you could use it for your website.

The script converts image types gif, jpg, jpeg, png, tif, and tiff. It also modifies the HTML pages, CSS and SCSS that reference those images.

The conversions are set for maximum fidelity (lossless where possible), and maximum compression. This means the images look great and load quickly.

I also wrote toPng, which works in a similar manner as toWebp.

Caveat

The script assumes that all images are local to your website, which makes sense because the converted images need to be stored, and local storage is the only sensible option. It renames all references to images in HTML, CSS and SCSS files to webp format. If the images are remote (for example, on a CDN), they are not converted, but the image file types in the HTML, CSS and SCSS are adjusted anyway. I suppose I could fix the script, but I don't need to do that for myself. If someone needs that feature, go ahead and enhance the script... and please provide me the enhanced script, so I can update this articleing.

Prerequisites

You need to install the WebP package.

Mac

Use Homebrew or Macports.

Ubuntu (this is the default Linux distribution for Windows Subsystem for Linux)

At a shell prompt type:

Shell

$ yes | sudo apt install webp

Running toWebp

The program may emit warnings when it runs. Those warnings can be safely ignored.

Hopefully, your website is managed by git. I suggest that you commit your work before running the script. That way if something goes wrong you just have to type git stash to return your website to its previous state.

Usage

The general form of the command to convert all images and modify the HTML pages that they are referenced from is:

Shell

$ toWebp <directoryName>

Examples

To convert the website (images, html, scss & css) rooted at the current directory, type:

Shell

$ toWebp .

To convert the website called mySite rooted under your home directory, type:

Shell

$ toWebp ~/mySite

To just convert 1 specific image to webp, type:

Shell

$ toWebp images/blah.jpg

The toWebP Bash Script.

Put this file in one of the directories on your PATH, for example /usr/local/bin, or for Ubuntu users ~/.local/bin/:

toWebp

#!/bin/bash

# SPDX-License-Identifier: Apache-2.0

shopt -s extglob

export CMD="cwebp alpha_q 10 -exact -lossless -m 6 -short -q 100 -z 9"

function help {
  [[ "$1" ]] && >&2 printf "Error: $1\n\n";
  >&2 printf "$( basename $0 ) - Convert all images to webp files in place and update HTML, CSS and SCSS to suit

Usage: toWebp directory|imageFileName

Example: toWebp .  # convert images, html, scss & css in current directory tree

Example: toWebp images/blah.jpg   # just convert 1 specific image
"
}

function checkDependencies {
  if [ -z "$( which cwebp )" ]; then
    echo "Installing webp"
    yes | sudo apt install webp
  fi
}

function convertGifs {
  for F in $( find "$1" -iname '*.gif' ); do
    TOF="${F%.*}.webp"
    gif2webp -loop_compatibility -m 6 -mixed "$F" -q 100 -o "$TOF"
  done
}

function convertMost {
  # See "Extended pattern" extglob for Bash
  # https://wiki.bash-hackers.org/syntax/pattern#extended_pattern_language
  cd "$1" || exit 3
  for F in $( listImages ); do
    TOF="${F%.*}.webp"
    echo "Converting '$F' to '$TOF'"
    # Warning messages might be emitted, but don't worry
    $CMD "$F" -o "$TOF"
    if [ "$DELETE_OLD" ]; then rm "$F"; fi
  done
  cd - || exit
}

function listImages {
  # Does not return gifs because cwebp cannot handle them
  find . -iregex '.*\.\(jpg\|png\|jpeg\|tif\|tiff\)' -printf '%f\n'
}

function renameImage {
  sed -i "s,\b$1\b,.webp,g" "$2"
}

function swapImages {
  for F in $( find "$1" -iname '*.html' -o -iname '*.css' -o -iname '*.scss' ); do
    for X in .gif .jpg .jpeg .tif .tiff .png; do
      echo "Swapping $X images for .webp in '$F'"
      renameImage "$X" "$F"
    done
  done
}


unset DELETE_OLD   # TODO make this a command line option

if [[ -f "$1" ]]; then   # just convert a single image to webp
  F="$1"
  $CMD "$F" -o "${F%.*}.webp"
  if [ "$DELETE_OLD" ]; then rm "$F"; fi
elif [[ -d "$1" ]]; then # process all images, css and scss in directory
  shopt -s globstar
  convertMost "$1"
  convertGifs "$1"
  swapImages "$1"
else
  help "You must either specify a valid file or a directory"
fi

Make it Executable

Remember to make the toWebp script executable before trying to use it:

Shell

$ chmod a+x /usr/local/bin/toWebp

Help Message

Shell

$ toWebp
Error: You must either specify a valid file or a directory

toWebp - Convert all images to webp files in place and update HTML, CSS and SCSS to suit

Usage: toWebp directory|imageFileName

Example: toWebp .  # convert images, html, scss & css in current directory tree

Example: toWebp images/blah.jpg   # just convert 1 specific image

The toPng Bash Script.

toPng

#!/bin/bash

# Convert all webp images to png in place; does not update HTML, CSS or SCSS
# Usage: toPng directory|imageFileName
# Example: toPng .  # convert images in current directory tree
# Example: toPng images/blah.webp   # just convert 1 specific image
#
# SPDX-License-Identifier: Apache-2.0

shopt -s extglob

export CMD="dwebp"

function help {
  [[ "$1" ]] && >&2 printf "Error: $1\n\n";
  >&2 printf "$( basename $0 ) - Convert all images to webp files in place and update HTML, CSS and SCSS to suit

Usage: $( basename $0 ) directory|imageFileName

Example: $( basename $0 ) .  # convert images, html, scss & css in current directory tree

Example: $( basename $0 ) images/blah.jpg   # just convert 1 specific image
"
}

function checkDependencies {
  if [ -z "$(which dwebp)" ]; then
    echo "Installing webp"
    yes | sudo apt install webp
  fi
}

function listImages {
  find . -type f -iname '*.webp'
}

function convert {
  # See "Extended pattern" extglob for Bash
  # https://wiki.bash-hackers.org/syntax/pattern#extended_pattern_language
  cd "$1" || exit
  for F in $( listImages ); do
    TOF="${F%.*}.png"
    echo "Converting '$F' to '$TOF'"
    # Warning messages might be emitted, but don't worry
    $CMD "$F" -o "$TOF"
    if [ "$DELETE_OLD" ]; then rm "$F"; fi
  done
  cd - || exit
}


# Set cwd to project root
GIT_ROOT="$( git rev-parse --show-toplevel )"
cd "${GIT_ROOT}" || exit
[[ -f _bin/loadConfigEnvVars ]] && source _bin/loadConfigEnvVars;

unset DELETE_OLD   # TODO make this a command line option

checkDependencies

if [ -f "$1" ]; then   # just convert a single image to png
  F="$1"
  $CMD "$F" -o "${F%.*}.png"
  if [ "$DELETE_OLD" ]; then rm "$F"; fi
elif [ -d "$1" ]; then # process all images, css and scss in directory
  convert "$1"
else
  help "You must either specify a valid file or a directory."
fi

Make it Executable

Remember to make the toWebp script executable before trying to use it:

Shell

$ chmod a+x /usr/local/bin/toPng

Help Message

Shell

$ toPng
Error: You must either specify a valid file or a directory.

toPng - Convert all images to webp files in place and update HTML,
CSS and SCSS to suit.

Usage: toWebp directory|imageFileName

Example: toWebp .  # convert images, html, scss & css in current directory tree

Example: toWebp images/blah.jpg   # just convert 1 specific image

Dotty (Scala 3 Preview) Presentation at Hopper, Montreal

2019-11-28T00:00:00-05:00

Yesterday I presented Dotty (Scala 3 Preview) to Lambda Montreal.

The slides are here.

The code is here.

The video recording is here.

A Hybrid Machine Learning / Personality Simulation Platform

2019-10-24T00:00:00-04:00

Yesterday I presented “EmpathyWorks: A Hybrid Machine Learning / Personality Simulation Platform” to the Fall 2019 Montreal Machine Learning Mini-Conference.

The slides are here. The video recording is here.

Decentralized Ponytails

2018-09-13T00:00:00-04:00

I’d like to point out the similarity of the early days of the open-source movement with today’s decentralized blockchain movement.

Open-source software was brought to mainstream attention during the last technology bubble at the end of the last millennium. The open-source software movement had a loyal cadre of zealots who believed that their cause would overcome any need for a business case. Sun Microsystems was the hardware company whose servers powered the Internet, and their software included the Java programming language, plus many other important networking-related products. Sun's slogan was “The network is the computer”.

Jonathan Schwartz, the CEO of Sun Microsystems was one of the open-source zealots. He was famous for his ponytail. Unfortunately, zealotry and dogma is bad for business, and as a result Sun Microsystems is no longer with us.

Jonathan Schwartz and his ponytail

Eventually, companies like Red Hat developed solid business models for open-source software, but that took years to develop. Today we see many companies attempting using decentralized blockchain technology to create cryptocurrencies, other token-based economies, and evangelizing decentralized dogma without a solid business case. Most of these ventures will die a horrible death, and the investors will get nothing. It will take years for solid business models based on decentralization to be proven.

Mr. Schwartz's ponytail was the fashion statement that fueled the YouTube parody below. For background, Scott McNealy was the previous CEO at Sun Microsystems.

Full disclosure: I also had a ponytail in 2008, and for a few years I had my Sun Spark 2 workstation at home.

Mike Slinn and his ponytail back in 2008

Here is a transcription of the video, which I paraphrased for clarity:

Steve Gilmore: Hi, this is Steve Gilmore and this is a video special edition of the Gilmore gang. I'm here with Jonathan Schwartz. It's a great pleasure – it's been a long, long time coming – I haven't seen Jonathan for quite a while. Jonathan Schwartz, who is the president and CEO of Sun Microsystems, agreed to sit down for the first time in three or four years and talk about what's going on with Sun. I wanna start, Jonathan, by thanking you for joining us.

Jonathan Schwartz: Thank you for having me, Steve. It has been a long time, nice to see you.

Steve Gilmore: So, you know there's been a lot of turmoil on Wall Street as I know you know.

Jonathan Schwartz: Yes.

Steve Gilmore: What's your take on that?

Jonathan Schwartz: Well, I think that it's a cyclical thing as you know Sun was been very prepared for this because we took three quarters of a billion dollars off of KKR a few years ago so that's in our war chest and I think that we're in a very good position moving forward, Steve.

Steve Gilmore: Ahh, and specifically what are you doing?

Jonathan Schwartz: Well, what we're doing is as you know Sun has always been very proactive in the open-source movement. I know you're very familiar with that. What we want to do to help our customers during this very difficult time is keep up with that trend of open-source. So I'm actually very pleased to announce, Steve, that Sun is starting a new open-source initiative. We're going to be releasing the source code to my ponytail as open-source, Steve.

Steve Gilmore: How's that going to help the situation?

Jonathan Schwartz: It's open-source, Steve.

Steve Gilmore: Yeah.

Jonathan Schwartz: It's my ponytail, Steve.

Steve Gilmore: You know, we've had this conversation in the past.

Jonathan Schwartz: Yes.

Steve Gilmore: Something is open-source, fine, and you get a lot of adoption, you get a lot of exposure in the in the marketing arena...

Jonathan Schwartz: Yes.

Steve Gilmore: ... but how do you make money on your ponytail?

Jonathan Schwartz: Well, what we're going to be doing is releasing my ponytail as open-source. So what we're hoping is that our developers take my ponytail and develop some kind of revenue stream with my ponytail. Did I mention this is open-source, Steve?

Steve Gilmore: Yeah, so how do you open-source a ponytail? What does that mean?

Jonathan Schwartz: Well, basically what we do is, we will have some of our best and brightest engineers here at Sun go through my ponytail and find out the unique attributes about what makes my ponytail so successful in the valley. And what we've done Steve, we've open-sourced my ponytail, Steve.

Steve Gilmore: Jonathan, you're just repeating this over and over again; it doesn't necessarily arrive at a business model.

Jonathan Schwartz: Umm, Steve?

Steve Gilmore: Yeah.

Jonathan Schwartz: We're gonna take my ponytail right and make it open-source. Now I know that this is a big concept for you but I really think it's a game changer, Steve.

Steve Gilmore: So, who do you see as your competition in the open-source ponytail arena?

Jonathan Schwartz: I think that we pretty much have it locked up. I don't see anybody who can compete with Sun Microsystems, when it comes to the open-source ponytail market. I think that we're in very good shape, Steve. How much do you miss Scott McNealy?

Steve Gilmore: Right now, a lot.

Jonathan Schwartz: Not nearly as much as I do, Steve.

Steve Gilmore: Okay, so what are you gonna do about, uh, you've laid off a lot of people in the last few months.

Jonathan Schwartz: Yes, it's going well in fact we have another round of layoffs coming. Once the ponytail is released into the wild, we'll be releasing the team that open-sourced my ponytail, Steve.

Steve Gilmore: Well you know I have to say that I was hoping for something a little bit more visionary from you Jonathan.

Jonathan Schwartz: Well I think that this is quite visionary. I don't think that IBM will be releasing a ponytail, and if they did it certainly wouldn't be open-source, Steve. We are very, very excited about our open-source ponytail program if you want you can go to sunmicrosystems.com/ponytail. Any other questions, Steve?

Steve Gilmore: Yeah I got one that I hope will be a stumper for you, which is a do you see the relationship between your open-source ponytail strategy and micro-messaging as popularized by Twitter?

Jonathan Schwartz: I'd like to open the pipe to the ponytail. Except as you know, Twitter is not exactly handling XMPP correctly at this time. I don't see the correlation between an open-source ponytail, and a closed-off micro-blogging system. Steve, I think that the ponytail is much bigger than Twitter.

Steve Gilmore: And the business model again?

Jonathan Schwartz: Let me see this real slowly and clearly: OPEN. SOURCE. PONYTAIL.

Steve Gilmore: This has been Jonathan Schwartz along with me, Steve Gilmore. Good luck, Jon.

Jonathan Schwartz: Thank you Steve. God, I miss McNealy. Think it's going to work?

Steve Gilmore: No.

Evaluating Technology Companies

2018-09-12T00:00:00-04:00

I have performed evaluations for investors since 1985, and for legal cases since 1998. I typically assess the people, the processes they follow, and the technology they work with. Although the goals of assessments performed for investors are different from the goals of the legal team that retain me as an expert, the work that I do for both types of engagements is in many ways quite similar.

Aspects Common to Investment and Ligigation Engagements

The major steps that I follow for both types of engagements are:

Examine evidence and interview the people involved
Perform analysis
Write a report
Present the report, and defend or discuss it

Strategic risks typically involve technical and business considerations. I assess the strengths and weaknesses of the technical team, the technology they use, and the processes that they follow. I am also called upon to opine on product/market fit; is this a solution that addresses a burning unmet need in a potentially lucrative market in a given time period?

Some engagements require me to develop an opinion on whether significant intellectual property was developed and applied effectively. I develop an opinion on whether the company in question is or was able to generate and profitably maintain and evolve a product or service that meets compelling unmet customer needs in various market segments.

I generally inspect the following primary sources of data for assessments:

My notes from personally using the product or service, stress-testing it, and performing failure mode analysis
One-on-one interviews of employees and customers (when possible)
Examining correspondence and working documents (Jira logs, git logs, internal product requirements and customer-facing documentation)

While assessments for legal cases have many similarities with assessments for investors, the approach is usually quite different. In addition, investors are often interested in sensitivity analysis, while legal cases generally do not require that.

Investments

Investors are primarily interested in minimizing risk and maximizing reward. Investors often request assessments of technology companies to determine if a company might be a good investment, to optimize the performance of an investment, to identify risk and opportunity.

Legal Cases

Legal cases often involve assigning blame, quantifying loss or missed opportunity.

Some cases engagements require an opinion on whether significant intellectual property was developed and applied effectively. Legal cases often require an assessment of how the suitability of a company’s product or service existed at a point in time, or how it changed over time. As an expert, I am often called upon to discuss the relative strengths of the arguments of the parties in a dispute from a technology perspective.

1:1 Interviews

I try to individually interviewing everyone in the company. To the extent possible, I take each person’s photograph, gather job descriptions, resumes and their work history. The information obtained from the lowest, most recently hired person is valuable and important. The most influential people cannot help but apply spin to their story. I look for evidence to support every piece of information that is presented to me.

Open-ended questions yield much more interesting and insightful responses than specific questions. Multiple-choice questions have limited value. When interviewing someone, I recognize that all experiences are individual; I never attempt to equate my experience with those that I am interviewing. These conversations are not self-promotional opportunities.

Each interview takes 45 minutes, followed by 5 minutes of note taking. I prepare for each interview by reading the resume, job description, and any other information before meeting with them. I can process about 40 people a week this way.

If there are more people than I can interview in the time available, I interview everyone from the front-line managers on up, plus a random selection of the others. I do not allow the company to select the lower-level people because I want to know of problems. Generally I interview the top level twice: a half-hour interview at the beginning, then a 90-minute second interview at the end.

Overspecialization vs. Resilience

The more highly specialized and optimized an animal or an organization is, the more fragile it is in an evolutionary sense. Attaining product/market fit is a continuous process, not a one-time event. Markets evolve, products mature, and organizations are social constructs subject to politics and an ever-changing historical interpretation.

I try to evaluate the interdependence, resilience and resourcefulness of the individuals in an organization. Cross training compensates for highly specialized jobs; highly trained people like PhDs generally know more and more about less and less. Extremely specialized people are generally unaware of the big picture; instead, they view events through their personal microscope.

Resilience Superposition

Resilience can only be measured by testing it, and testing an organization’s resilience changes it. This is similar to the story in the field quantum mechanics, which asks if Schrodinger’s cat is alive or dead within a box.

Change Is Traumatic

I generally try to anticipate the results of two sources of change:

A large investment or a change in ownership
Market changes

A large investment or a change in ownership can cause a fundamental behavioral change in the existing company's leadership. Replacing the leadership or changing their behavior is usually a traumatic event for the organization.

Because markets change, an organization that lacks resilience may crumble in the face of dramatic market shifts. I try to evaluate an organization’s ability to continually anticipate and respond to changes in product/market fit. It is all mental, a social construct, stories that the people who work at a company have internalized.

Anarchists Are The Best Assessors

One of the primary tasks of an assessor is to question the boundaries of the target market and the product or service. Rarely do those boundaries remain unchanged through the lifespan of the product or service, or the company that produces them.

Anarchists perform the most useful assessments. An assessor cannot be effective if they restrict themselves to coloring within the given outlines; by this I mean respecting the status quo cannot yield an objective or useful assessment.

IBM Personality Insights

2018-08-29T00:00:00-04:00

EmpathyWorks™ is my name for the original research I've done on modeling personality and behavior of individuals and groups since 2007. For me, the work has been both therapeutic and insightful, and I now have a better basis for understanding myself and others as a result.

Recently, an IBM employee pointed me to IBM Personality Insights, and I eagerly visited the site. The scientific basis for the results is interesting. I am greatly interested in the analysis that IBM Personality Insights provided of the articleing I wrote on my birthday last year. I also submitted a short humorous posting I wrote ten years ago for analysis.

A Modern Horoscope?

I think the results from IBM Personality Insights are about as accurate as a horoscope. Abhishek Srivastava’s Quora posting of November 18, 2016, expresses this well.

I think most answers here have missed the point of IBM Watson’s personality insight service. It is a NLP based approach to find scores of individuals on some well-established scales in psychology like Big 5, Basic Humans Values and Needs. So, when those numbers are arrived using those standard questionnaires or through text analysis done by Watson, they are pretty close statistically. In that respect Watson is definitely very accurate. As far as interpretation of those results are concerned, that’s beyond the purview of current scope of Watson and rather is a question for psychologist. If I was a team member at IBM Watson, I would have actually not given the interpretation and would rather just give the scores and let people interpret them.

Furthermore, until a peer review of IBM Personality Insights concludes that the results are well-founded, I would not be comfortable using the technology for decision-making.

To be fair, a recent examination of nearly 350 published psychological experiments found that 42% failed to show that they were based on a valid foundation of empirical evidence, suggesting that a wide swath of psychological science is based on an untested foundation. With that in mind, let's see what this modern horoscope serves up!

High Level Results

Results were fairly consistent between the two articleings. When I concatenated the two posts, the longer article dominated. Differences between the two personality assessments are shown this way. My comments are shown this way.

Birthday Posting Summary	Humorous Posting Summary
The results in JSON format are here.	The results in JSON format are here.
You are shrewd and skeptical.	You are shrewd, inner-directed and guarded.
You are philosophical: you are open to and intrigued by new ideas and love to explore them. You are independent: you have a strong desire to have time to yourself. And you are authority-challenging: you prefer to challenge authority and traditional values to help bring about positive changes.	You are philosophical: you are open to and intrigued by new ideas and love to explore them. You are independent: you have a strong desire to have time to yourself. And you are solemn: you are generally serious and do not joke much. I guess IBM did not like my humor!
Your choices are driven by a desire for discovery often true.	Your choices are driven by a desire for organization often true.
You are relatively unconcerned with both tradition and taking pleasure in life. You care more about making your own path than following what others have done. And you prefer activities with a purpose greater than just personal enjoyment.	You are relatively unconcerned with both achieving success and taking pleasure in life. You prefer activities with a purpose greater than just personal enjoyment. And you make decisions with little regard for how they show off your talents.

An old friend, who I've known since university, is a clinical psychologist with a PhD and works as a doctor in a mental hospital. His comments on the results were:

I think that some aspects of the profile are pretty accurate; however, other aspects such as calling you shrewd and skeptical are a bit evaluate: I would instead say intelligent and not naïve. The openness results would seem pretty accurate.

I think you're probably a bit more extraverted than this analysis suggests.

The ‘emotional range’ factor of the Big 5 is typically labeled Neuroticism. I kind of think of Agreeableness as a compliance/ conformity dimension, and I if I am not mistaken, it is not unusual for (male) entrepreneurs to score low on that dimension.

But Wait, There's More!

IBM's Personality Insights provides additional information that explains the above in more detail:

Birthday Posting Summary	Humorous Posting Summary
You are likely to: like musical movies Oops, that was way off! be sensitive to ownership cost when buying automobiles have experience playing music True: I am a multi-instrumentalist	You are likely to: like historical movies True! be sensitive to ownership cost when buying automobiles have experience playing music
You are unlikely to: be influenced by social media during product purchases The truth is more complex prefer style when buying clothes True, and to compensate I try to shop for clothes with carefully selected friends like country music Spot on!	You are unlikely to: be influenced by social media during product purchases prefer style when buying clothes be influenced by brand name when making product purchases

Birthday Posting Summary

Humorous Posting Summary

You are likely to:

like musical movies
Oops, that was way off!
be sensitive to ownership cost when buying automobiles
have experience playing music
True: I am a multi-instrumentalist

You are likely to:

like historical movies
True!
be sensitive to ownership cost when buying automobiles
have experience playing music

You are unlikely to:

be influenced by social media during product purchases The truth is more complex
prefer style when buying clothes True, and to compensate I try to shop for clothes with carefully selected friends
like country music Spot on!

You are unlikely to:

be influenced by social media during product purchases
prefer style when buying clothes
be influenced by brand name when making product purchases

I think that the above was a pretty good assessment of me, and the detailed breakdowns which follow are interesting. Before you look at that, however, you should know that the Big 5 Personality Model defines the 5 traits using words with meanings that might seem different from the meanings you might expect. The 5 traits are: openness, conscientiousness, extroversion, agreeableness, and emotional range.

Personality, Needs and Values

Each of the 5 traits are broken down into various aspects. For example, openness consists of adventurousness, artistic interests, emotionality, imagination, intellect, and authority-challenging. Again, these terms are defined in specific ways that might be different from the definitions that you might expect.

Birthday Posting Summary	Humorous Posting Summary
1724 words; decent analysis.	516 words; we need a minimum of 600, preferably 1,200 or more, to compute statistically significant estimates.

Detailed Breakdown

Here is my detailed breakdown, shown as sunburst charts:

Birthday Posting Summary	Humorous Posting Summary

The sunburst charts paint me as an unusual person. If this personality assessment is accurate, I am rather complex. However, further reading suggests that people with very high openness scores defy most structured evaluations, and I score in the 99^th percentile for openness.

Birthday Posting Summary	Humorous Posting Summary
Openness to experience: high (99^th percentile). This means I have an unusually high fluid intelligence (I am able to learn complex concepts and tasks very quickly), and I am likely to be eccentric. I believe this to be true.	Openness to experience: high (99^th percentile) – Same
Extraversion: Low (15th percentile) – Cold, withdrawn, unfriendly. That feels harsh.	Extraversion: Even lower (6^th percentile). Yikes!
Agreeableness: Low (5^th percentile) – Independent, tough, dominant, possibly manipulative. Yes, I make up my mind and I follow what I believe to be the appropriate course, regardless of what others might say or do. Additionally, I have very strong sympathy (97^th percentile) and I am rather uncompromising (82nd percentile), strongly cooperative (82^nd percentile) with strong altruism (87^th percentile). Perhaps that means that I am a crusty individual with a warm heart. According to the breakdown, I am quite trusting (82^nd percentile), yet I am also very cautious (90^th percentile), so for me I go with “trust but verify”.	Agreeableness: Even lower (0^th percentile). Yikes!
Emotional range: average (> 59^th percentile). IBM defines this as “the extent to which a person's emotions are sensitive to the individual's environment”	Emotional range: very high (> 91^st percentile). When coupled with low agreeableness, IBM predicts: temperamental, irritable, quarrelsome, impatient, grumpy. When coupled with low conscientiousness, IBM predicts: compulsive, nosy, self-indulgent, forgetful, impulsive. When coupled with low extroversion, IBM predicts: guarded, fretful, insecure, pessimistic, secretive. When coupled with high openness, IBM predicts: excitable, passionate, sensual. Hmm, quarrelsome, impatient, compulsive, self-indulgent, forgetful, insecure, pessimistic and yet passionate and sensual. What a combination!
Low Conservation (1^st percentile) – This is called Hedonism in the other graph.	Hedonism: very low (1^st percentile) – This is called Conservation in the other graph. Within this category I scored low self-enhancement, low Hedonism, low Openness to change and low conservation.
Low Harmony (5^th percentile) – This is called Closeness in the other graph.	Closeness: low (1^th percentile) – This is called Harmony in the other graph. Within this category I scored low Excitement, Harmony, Ideal, Liberty, Love, Self-expression and Stability. I also scored high Curiosity (78_th percentile) and high Structure (88^th percentile). Sounds like I'm pretty much a curious robot.

These traits, if true, would make me a good expert witness, and a good evaluator for technical due diligence. This might also explain my propensity for constantly inventing new things.

According to these results, I also registered some extreme needs and values. Because the results of analyzing both documents were almost the same, I show them together.

First, let's look at the results that I identify with:

Both articles
An extremely liberal mindset (conservation: 1st percentile)
Virtually no interest in comfort (hedonism: 2nd percentile)
Almost no interest in social power, authority, wealth, success, capability, etc. (self-enhancement: 4th percentile)
Almost no interest in harmony (5th percentile)
Strong curiosity (87th percentile)
Low connection to family and setting up a home (closeness: 9%)
Very little respect, commitment, and acceptance of the customs and ideas that culture and/or religion provides.

I think the following needs assessments are way off the mark. Perhaps I protest too much?

Very little interest in just having fun for its own sake (excitement: 7th percentile).

Low desire for perfection or a sense of community The truth is complicated: I have spent decades building professional and musical communities, yet I am isolated in many ways

Low interest in discovering and asserting their identities (self-expression: 9%) This seems way off the mark, for example consider the motivation for my spending thousands of hours on EmpathyWorks.

Another Assessment

The more clocks you have, the less certain you are of the correct time. Some of the following is true, but my personality expresses itself differently according to circumstance, so no one single description could be realistic.

We3app.com rated my personality type as:

The Good-Timer
Calm, messy, sociable, curious, egoistic
3.3% of women | 5.3% of men

You’re independent, imaginative, and intelligent, and you want to make fun where you can. Life is great, and you’ve got a good idea of how to live it—it’d take a lot to make you change your mind.

Your positive vibe and strong opinions mean that people tend to give you what you need, and are happy to follow your lead. You’ll listen to what they have to say, but you usually end up being right anyway. Besides, you’re the only one who knows how to get away with cutting corners: nobody can beat the system like you.

You’re never sure exactly how much is in your bank account, you regularly have to improvise when you run out of underwear, and you are disgusted at the concept of a cleaning schedule. You might agree to washing the car, but only because it will inevitably turn into a water fight.

You’re not one to say no to temptation, and there are certain habits you know it would be better to kick, but self-discipline is for monks and bodybuilders.

Evaluating Blockchain Companies

2018-08-29T00:00:00-04:00

I have performed technical due diligence for investors since the mid-1980s. In this 40-minute presentation I explain how I evaluate blockchain-related technology companies.

This talk was presented at the 6th Annual Global Big Data Conference in Santa Clara, California on August 29, 2018. The promotional video recording is here.

InfoQ produced the presentation in their unique format.

The slides are here.

The video recording is here.

Bob Summerwill

2018-08-28T00:00:00-04:00

This is a great photo of Bob Summerwill and me in San Francisco August 2018!

Keynote Panel Discussion - The Future of Blockchain

2018-08-23T00:00:00-04:00

I will participate in a keynote panel discussion on the Future of Blockchain on August 23 from 4:50 PM to 6:00 PM at the 6th Annual Global Big Data Conference in Santa Clara, California.

Global Big Data Conference featuring me as a speaker

Following are some ideas I hope to discuss with the other members of the panel. First, I would like to remind the reader that blockchain data is passive; without a program to access it blockchain data is inert. Now I'd like to give the definition that I will use for this discussion.

The word blockchain is defined by Wikipedia as: “a growing list of records, called blocks, which are linked using cryptography.” The main theme I’d like to personally bring forward during this discussion is that Blockchain does not imply or require decentralization, or even distributed systems. Instead, blockchain is a useful technology in its own right, even if it is not distributed / decentralized. I have nothing against decentralization, when used appropriately; the point I want to make is that decentralization is not necessary for blockchain to provide value.

Here are the use cases I want to talk about:

#1: Secure Mobile and Embedded Databases

For me, blockchain is a file format that yields immutable data, highly resistant to modification. It is not a database, instead, it is a generic type of storage technology. Yes, an API could be provided that provides a structured interface to blockchain data, but there is no universally accepted standard in use for this in any blockchain implementation that I am aware of. Such a standard should exist, and I would be interested in learning about any standards work in this area.

A database built using blockchain would not support the full CRUD API (create, read, update, and delete) functionality traditionally supported by SQL. Instead, only create and read functionality would be supported.

This seems especially useful for mobile and embedded data recorders, for example flight recorders used in airplanes. I have not yet attempted to compute how feasible it might be to use blockchain to store video or audio streams from devices such as police body cameras.

#2: Secure Data Distribution

Blockchain is a data storage technology that is known to be highly resistant to corruption or modification. Because data can only be created or read, but not modified or deleted, if you have a file of blockchain data that passes inspection you can be confident that you have all the data that was available at the time the file was last updated.

Blockchain data could be stored in RFID tags. This is possible because RFID tags exist which can store up to 66 KB of data, and read/write RFID tags are available. Read-write RFID tags usually have a serial number that can’t be written over. Additional blocks of data can be used to store additional information about the items the tag is attached to (these can usually be locked to prevent overwriting of data).

This would allow blockchain to be used for smart passports and smart ID cards. Secure, smart passports would provide extra security for passport/ID cardholders and would make counterfeiting extremely difficult, and would provide a digital record of countries visited, instead of relying on passport stamps, which are easy to counterfeit.

Similarly, smart ID cards could hold mutable yet secure data.

#3: Secure Software Build Systems

There have been many instances of corporate spies infiltrating software companies or their repositories. The perpetrators altered the source code, accompanying data or modified other build dependencies. When the next version of the software is published, all the customers who install updates will potentially be compromised.

git uses similar cryptography techniques as does blockchain. A blockchain-based build system would use a git tag to kick off build, and would need to also store and compare hashes for all dependencies to ensure that they had not changed. Because the software build toolchain is also a dependency, hashes for all the software used in the toolchain would need to be validated at build time.

#4: Secure Software Distribution

Computer viruses have been a problem for decades. There have been many instances where programs have viruses embedded after they are published. This is generally true for any software downloaded from a torrent site. If software was packaged using blockchain technology, it could be retrieved for installation with confidence. Software installers that used blockchain in this way could be trusted.

The Panelists

After the panel, we panelists were photographed together. From left to right: Hayden Kirkpatrick, VP of Strategy at Esurance (moderator); Karen Hsu, CRO at BlockXypher; Steve Beauregard, CRO at Bloq; Mike Slinn, CTO at Micronautics Research (holding the microphone); Mark Javier, Account Executive at Ambisafe; and Camille Sanandaji, CEO at Foodstems.

Hayden Kirkpatrick (moderator), Karen Hsu, Steve Beauregard, Mike Slinn, and Camille Sanandaji

Mark Javier, Mike Slinn, Hayden Kirkpatrick (moderator), Steve Beauregard, Karen Hsu, and Camille Sanandaji

Windows Subsystem for Linux Revisited

2018-08-20T00:00:00-04:00

I’ve been using Windows Subsystem for Linux (WSL) headless since it was first released with Windows 10 version 1607 in August 2016. The April 2018 release of Windows 10 (version 1803) significantly improved WSL.

It is possible to work with Ubuntu graphically on a vanilla Windows machine. No special drivers are required. No special Linux or Ubuntu support is required from the computer vendor.

This is my setup for running an X client like xeyes or IntelliJ IDEA from WSL or WSL2, accessed via a Windows X server like VcXsrv. These scripts assume Ubuntu 18.04 was installed under WSL.

Essential Scripts

Here are bash scripts to install everything:

Optional Scripts

The remaining scripts are all optional.

VNC

These bash scripts allow VNC to connect to a remote machine or to WSL on the local machine.

Packages and Programming Environments

Installation of various packages and programming environments follow.

Git

Python

NodeJS

Java Virtual Machine

Miscellaneous

Wine

The advent of WSL means that Wine is no longer required!

Configuration

This is my WSL configuration file, in my Windows home directory (%systemdrive%%homepath%\.wslconfig). See Advanced settings configuration in WSL for more information.

%systemdrive%%homepath%\.wslconfig

# See https://docs.microsoft.com/en-us/windows/wsl/wsl-config#configure-global-options-with-wslconfig
# See https://github.com/microsoft/WSL/issues/4166
[wsl2]
memory=8GB
swap=60GB
#processor=4
localhostForwarding=true

[experimental]
autoMemoryReclaim=drop
sparseVhd=true

2024-02-20 Update

The WSL v2.1.3 prerelease sets the the default memory reclamation mode to dropcache. The following is now a default setting:

%systemdrive%%homepath%\.wslconfig

wsl2.autoMemoryReclaim=drop

The memory will be freed using dropcache after 10 minutes of inactivity. To upgrade WSL using bash, cmd or powershell:

bash, cmd or powershell

wsl.exe --update --pre-release

Wait a few minutes before restarting a terminal session. It is not necessary to reboot after upgrading.

Upgrading from Ubuntu 19.11 to Ubuntu 20.04

Update Aug 17, 2020: Here are my notes on upgrading WSL and WSL2 from Ubuntu 19.11 to Ubuntu 20.04.

Run Associated Windows Program

To run the Windows program associated with a file, preface the file with cmd.exe /C start.

For example, to run the default Windows video viewer for .mkv files, type:

Shell

$ cmd.exe /C start my_file.mkv

I defined a bash alias to make this easier:

~/.bash_aliases

alias run='cmd.exe /C start'

Reload the bash aliases after editing ~/.bash_aliases:

Shell

$ source ~/.bash_aliases

Now use the run alias to run the default Windows video viewer for .mkv files like this:

Shell

$ run my_file.mkv

Ethereum Source Code Walkthrough

2018-06-13T00:00:00-04:00

April 27, 2020 Update

This was a sample work in progress as part of a proposal to the Ethereum Foundation Grants committee. They declined to fund this activity, but gave no reason and no feedback as to what might be acceptable. The preparation of my proposal took weeks, and the preliminary feedback that I had received from many knowledgeable people was that it had great value.

I felt that the evaluation process was broken, in fact the entire organization was broken, and there was little hope that Ethereum would ever become a professional organization. Two years later, I believe history has proved me right. This was the last blockchain-related initiative I participated in.

I no longer maintain web3j-scala, an Ethereum-related project for Scala programmers. I created that open source project and worked on it for free for 3 years. It has been forked and I’ve been told it is or was used in production. However, I see no reason to continue working for free on it. Others can carry the project forward if they want.

This is a brief walkthrough of some of the core source files for smart contracts in the official Go language Ethereum implementation, which includes the geth command-line Ethereum client program, along with many other programs. Ethereum clients include an implementation of the Ethereum Virtual Machine (EVM), which are able to parse and verify the Ethereum blockchain, including smart contracts, and provides interfaces to create transactions and mine blocks.

I’ve added some suggestions for how the source code might be improved. If there is general agreement that these suggestions make sense (tell me in the comments!) then I’ll create a pull request.

License

This Ethereum client project was released under the GNU Lesser General Public License, version 3 or later, which permits use of the code as a library in proprietary programs.

Source Files

The gocloc program counted the following source files and lines:

Language	Files	Blank Lines	Comment Lines	Code Lines
Go	1824	58,134	81,861	639,435
C	55	17,257	30,909	84,719
C Header	97	2,559	6,318	15,083
Markdown	88	3,152	0	9,175
JavaScript	13	1,845	4,495	7,986
Assembly	39	557	957	3,783
JSON	17	0	0	2,065
Protocol Buffers	2	113	40	1,030
Plain Text	11	217	0	954
C++	4	132	102	937
BASH	10	178	315	931
Perl	10	268	1,289	879
JSX	11	119	245	722
XML	9	0	0	651
M4	4	79	99	649
YAML	20	77	42	581
NSIS	5	86	154	446
Java	4	143	187	438
Makefile	11	101	84	381
Python	6	154	250	339
HTML	3	15	9	245
Solidity	7	56	171	213
Bourne Shell	6	23	25	119
CMake	1	9	0	35
Awk	1	4	4	17
TOML	1	0	0	3
Total	2,260	85,278	127,556	771,825

Packages

I used the following incantation to discover that geth defines 244 packages:

Shell

$ grep -rh "^package" | grep -v "not installed" | \
  tr -d ';' | sed 's^//.*^^' | awk '{$1=$1};1' | \
  sort | uniq | wc -l

I won't list them all. The godoc for the project contains much of the following documentation for the top-level packages. I provided the rest of the information from disparate sources, including reading the source code:

accounts implements high-level Ethereum account management.

trie provides a binary Merkle tree implementation.

cmd

Contains the following command-line tools. Most tools support the --help option.

abigen	source code generator to convert Ethereum contract definitions into easy to use, compile-time type-safe Go packages. It operates on plain Ethereum contract ABIs with expanded functionality if the contract bytecode is also available. However it also accepts Solidity source files, making development much more streamlined. Please see the Native DApps wiki page for details.
bootnode	runs a bootstrap node for the Ethereum Discovery Protocol. This is a stripped-down version of `geth` that only takes part in the network node discovery protocol, and does not run any of the higher level application protocols. It can be used as a lightweight bootstrap node to aid in finding peers in private networks.
clef	a standalone signer that manages keys across multiple Ethereum-aware apps such as Geth, Metamask, and cpp-ethereum. Alpha quality, not released yet.
ethkey	a key/wallet management tool for Ethereum keys. Allows user to add, remove and change their keys, and supports cold wallet device-friendly transaction inspection and signing. This documentation was written for the C++ Ethereum client implementation, but it is probably suitable for the Go implementation as well.
evm	a version of the EVM (Ethereum Virtual Machine) for running bytecode snippets within a configurable environment and execution mode. Allows isolated, fine-grained debugging of EVM opcodes. Example usage: Shell evm --code 60ff60ff --debug
faucet	an Ether faucet backed by a light client.
geth	official command-line client for Ethereum. It provides the entry point into the Ethereum network (main-, test- or private net), capable of running as a full node (default) archive node (retaining all historical state) or a light node (retrieving data live). It can be used by other processes as a gateway into the Ethereum network via JSON RPC endpoints exposed on top of HTTP, WebSocket and/or IPC transports. For more information see the CLI Wiki page.
p2psim	a simulation HTTP API. Docs are here.
puppeth	assembles and maintains private networks.
rlpdump	a pretty-printer for RLP data. RLP (Recursive Length Prefix) is the data encoding used by the Ethereum protocol. Sample usage: Shell rlpdump --hex CE0183FFFFFFC4C304050583616263
swarm	provides the `bzzhash` command, which computes a swarm tree hash, and implements the swarm daemon and tools. See the swarm documentation for more information.
wnode	simple Whisper node. It could be used as a stand-alone bootstrap node. Also could be used for different test and diagnostics purposes.

common contains various helper functions worth checking out

consensus implements different Ethereum consensus engines (which must conform to the Engine interface): clique implements proof-of-authority consensus, and ethash implements proof-of-work consensus.

console Ethereum implements a JavaScript runtime environment (JSRE) that can be used in either interactive (console) or non-interactive (script) mode. Ethereum's JavaScript console exposes the full web3 JavaScript Dapp API and the admin API. More documentation is here. This package implements JSRE for the geth console and geth console subcommands.

containers

contracts

core implements the Ethereum consensus protocol, implements the Ethereum Virtual Machine, and other miscellaneous important bits

crypto cryptographic implementations

dashboard

eth implements the Ethereum protocol

ethclient provides a client for the Ethereum RPC API

ethdb

ethstats implements the network stats reporting service

event deals with subscriptions to real-time events

internal Debugging support, JavaScript dependencies, testing support

les implements the Light Ethereum Subprotocol

light implements on-demand retrieval capable state and chain objects for the Ethereum Light Client

log provides an opinionated, simple toolkit for best-practice logging that is both human and machine readable

metrics port of Coda Hale's Metrics library. Unclear why this was not implemented as a separate library, like this one.

miner implements Ethereum block creation and mining

mobile contains the simplified mobile APIs to go-ethereum

node sets up multi-protocol Ethereum nodes

p2p implements the Ethereum p2p network protocols: Node Discovery Protocol, RLPx v5 Topic Discovery Protocol, Ethereum Node Records as defined in EIP-778, common network port mapping protocols, and p2p network simulation.

params

rlp implements the RLP serialization format

rpc provides access to the exported methods of an object across a network or other I/O connection

signer

swarm

tests implements execution of Ethereum JSON tests

trie implements Merkle Patricia Tries

vendor contains a minimal framework for creating and organizing command line Go applications, and a rich testing extension for Go's testing package

whisper implements the Whisper protocol

I used the following incantation to list the package names:

Shell

find . -maxdepth 1 -type d | sed 's^\./^^' | sed '/\..*/d'

The build/ directory does not contain a Go source package; instead, it contains scripts and configurations for building the package in various environments.

Smart Contract Source Code

The core/vm directory contains the files that implement the EVM. These files are part of the vm package. Let's look at two of them:

contract.go, which defines smart contract behavior.
contracts.go, responsible for executing smart contracts on the EVM.

Referenced Types

Two of the types used in the source files that we would like to understand are defined in common/types.go. Let's look at them first.

Address is defined as an array of 20 bytes:

Shell

const (
    HashLength    = 32
    AddressLength = 20
)

// Address represents the 20 byte address of an Ethereum account.
type Address [AddressLength]byte

Hash is defined as an array of 32 bytes:

Shell

// Hash represents the 32 byte Keccak256 hash of arbitrary data.
type Hash [HashLength]byte

The opcodes for each version of the EVM are defined in jump_table.go. The operation struct defines the properties:

Shell

type operation struct {
    // execute is the operation function
    execute executionFunc
    // gasCost is the gas function and returns the gas required for execution
    gasCost gasFunc
    // validateStack validates the stack (size) for the operation
    validateStack stackValidationFunc
    // memorySize returns the memory size required for the operation
    memorySize memorySizeFunc

    halts   bool // indicates whether the operation should halt further execution
    jumps   bool // indicates whether the program counter should not increment
    writes  bool // determines whether this a state modifying operation
    valid   bool // indication whether the retrieved operation is valid and known
    reverts bool // determines whether the operation reverts state (implicitly halts)
    returns bool // determines whether the operations sets the return data content
}

Notice the jumps property, a Boolean, which if set indicates that the program counter should not increment after executing any form of jump opcode.

The destinations type maps the hash of a smart contract to a bit vector for each the smart contract's entry points. If a bit is set, that indicates the EMV's program counter should increment after executing the entry point. analysis.go defines the destinations type like this:

Shell

// destinations stores one map per contract (keyed by hash of code).
// The maps contain an entry for each location of a JUMPDEST instruction.
type destinations map[common.Hash]bitvec

contract.go

This file defines smart contract behavior.

Imports

This comment applies to all of the Go source files in the entire project. I think the following absolute import would have been better specified as a relative import:

Shell

"github.com/ethereum/go-ethereum/common"

The relative import would look like this instead:

Shell

"../../common"

If relative imports were used instead of absolute imports that point to the github repo, local changes to the project made by a developer would automatically be picked up. As currently written, absolute imports cause local changes to be ignored, in favor of the version on github. It might take a software developer a while to realize that the reason why their changes are ignored by most of the code base is because absoluate imports were used. It would then be painful to for the developer to modify the affected source files throughout the project such that they used relative imports.

Types

The publicly visible AccountRef type is defined as:

Shell

// Account references are used during EVM initialisation and
// it's primary use is to fetch addresses. Removing this object
// proves difficult because of the cached jump destinations which
// are fetched from the parent contract (i.e. the caller), which
// is a ContractRef.
type AccountRef common.Address

The same file defines a type cast from AccountRef to Address:

Shell

// Address casts AccountRef to a Address
func (ar AccountRef) Address() common.Address { return (common.Address)(ar) }

The ContractRef interface is used by the Contract struct, which we'll see in a moment. This ContractRef interface just consists of an Address.

Shell

// ContractRef is a reference to the contract's backing object
type ContractRef interface {
    Address() common.Address
}

The Contract struct defines the behavior of Ethereum smart contracts, and is central to the topic, so here it is in all its glory:

Shell

type Contract struct {
    CallerAddress common.Address
    caller    ContractRef
    self      ContractRef

    jumpdests destinations // result of JUMPDEST analysis.

    Code     []byte
    CodeHash common.Hash
    CodeAddr *common.Address
    Input    []byte

    Gas   uint64
    value *big.Int

    Args []byte

    DelegateCall bool
}

CallerAddress is a publicly visible Address of the caller. caller and self are private ContractRefs, which as we know are really just Addresses.

jumpdests, a private field, has type destinations, which as we've already discussed defines if the entry point in the smart contract that need the program counter to be incremented after executing.

Code is a a publicly visible byte slice. We don't yet know if this is the smart contract source code, compiled code, or something else.

CodeHash is the publicly visible hash of the Code, while CodeAddr is a publicly visible pointer to the Address (of the code, presumably).

Gas is the publicly visible amount of Ethereum gas allocated by the user for executing this smart contract, stored as an unsigned 64-bit integer.

Value is a private pointer to a big integer. Possibly this might be the result of executing the contract?

Args is a publicly visible byte slice, not sure what it is for.

DelegateCall is a publicly visible Boolean value, unclear if this means the smart contract was invoked using delegatecall. From the documentation: "This means that a contract can dynamically load code from a different address at runtime. Storage, current address and balance still refer to the calling contract, only the code is taken from the called address. This makes it possible to implement the “library” feature in Solidity: Reusable library code that can be applied to a contract’s storage, e.g. in order to implement a complex data structure."

contracts.go

This file is responsible for executing smart contracts on the EVM.

Imports

The following imports are used:

Package sha256 from the crypto project implements the SHA224 and SHA256 hash algorithms as defined in FIPS 180-4.
errors, the Go language simple error handling primitives, such as error.
math/big implements arbitrary-precision arithmetic (big numbers).

Other packages in this project (go-ethereum):

Shell

"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/common/math"
"github.com/ethereum/go-ethereum/crypto"
"github.com/ethereum/go-ethereum/crypto/bn256"
"github.com/ethereum/go-ethereum/params"

Again, I think the above imports would have been better specified as relative imports:

Shell

"../../common"
"../../common/math"
"../../crypto"
"../../crypto/bn256"
"../../params"

ripemd160 implements the RIPEMD-160 hash algorithm, a secure replacement for the MD4 and MD5 hash functions. These hashes are also termed RIPE message digests.

Type PrecompiledContract

PrecompiledContract is the interface for native Go smart contracts. This interface is used by precompiled contracts, as we will see next. Contract is a struct defined in contract.go.

Pre-Compiled Contract Maps

These maps specify various types of cryptographic hashes and utility functions, accessed via their address.

PrecompiledContractsHomestead contains the default set of pre-compiled contract addresses used in the Frontier and Homestead releases of Ethereum: ecrecover, sha256hash, ripemd160hash and dataCopy.

PrecompiledContractsByzantium contains the default set of pre-compiled contract addresses used in the Byzantium Ethereum release. All of the previously defined pre-compiled contract addresses are provided in Byzantium, plus: bigModExp, bn256Add, bn256ScalarMul and bn256Pairing.

I’m not happy about the code duplication, whereby the contents of PrecompiledContractsHomestead are incorporated into PrecompiledContractsByzantium by listing the values again; this would be better expressed by referencing the values of PrecompiledContractsHomestead instead of duplicating them.

Contract Evaluator Function

The RunPrecompiledContract function runs and evaluates the output of a precompiled contract. It accepts three parameters:

A PrecompiledContract instance.
A byte array of input data.
A reference to a Contract, defined in contract.go, discussed above.

The function returns:

A byte array containing the output of the contract.
An error value, which could be nil.

Other Functions

RunPrecompiledContract – runs and evaluates the output of a precompiled contract; returns the output as a byte array and an error.
RequiredGas (overloaded) – Computes the gas required for input data, specified as a byte array and returns a uint64.
Run (overloaded) – Computes the smart contract for input data, specified as a byte array and returns the result as a left-padded byte array and an error.
newCurvePoint – Unmarshals a binary blob into a bn256 elliptic curve point. BN-curves are an elliptic curves suitable for cryptographic pairings that provide high security and efficiency cryptographic schemes. See the IETF paper on Barreto-Naehrig Curves for more information.

Smart Contracts That Learn

2018-04-03T00:00:00-04:00

It seems natural that machine learning and smart contracts will commonly be integrated over the next few years.

I presented Smart Contracts That Learn at the Global Blockchain Conference on April 3, 2018, in Santa Clara. This was a 40-minute technical presentation. Slides are here.

InfoQ.com published the presentation in their unique format, featuring synchronized video and transcripts.

A full-length (70 minute) version of this presentation was presented at IBM Ottawa on May 28, 2018. Here are the slides for the full-length version.

SVIEF Blockchain Conference at Stanford University

2018-03-02T00:00:00-05:00

Images from the SVIEF Blockchain Conference at Stanford University March 24, 2018. My presentation was entitled “Smart Contracts that Learn”.

Smart Contracts for Enterprises

2018-01-18T00:00:00-05:00

‘Polyglot’ means “of many languages”, and I believe that enterprises that integrate smart contracts into their infrastructure will require solutions that incorporate many simultaneous software languages, libraries and runtime environments, all working as one in a distributed environment.

I anticipate significant need for system integration. I am interested in distributed consensus / blockchain / cryptocurrency architecture that support enterprise needs by providing useful features that are easy to use effectively.

I presented “Polyglot Ethereum Smart Contracts for the Enterprise” at the World Crypto Economic Forum in San Francisco, January 16, 2018. Here is my slide deck.

Tweet Stream Manager

2018-01-03T00:00:00-05:00

I publish multiple tweet streams, so I wrote a Node.js web application using Express and Pug to help me manage them. This is what the user interface looks like. This is a silent video. Hopefully, it makes sense to everyone.

For example, ScalaCourses.com currently has two tweet streams, each with their schedule (times shown are HH:MM for a 24-hour clock):

ScalaCourses – tweets are published at 6:45 every day
SaleMore33 – This is a promotional campaign, starting on a certain day/time and ending on another day/time. This stream tweets at 3:24, 9:24, 13:24, 18:24, and 23:24.

crontab, running on a local XUbuntu machine, runs a Node.js command-line app that publishes the tweet streams:

NODE=/usr/bin/node
TWEETER=/var/work/training/projects/tweeter

45 6 * * *              $NODE $TWEETER/index.js $TWEETER/ScalaCourses
24 3,9,13,18,23 * * *   $NODE $TWEETER/index.js $TWEETER/SaleMore33 2018-01-03 2018-01-15

I have been thinking about re-implementing this command-line program as an AWS Lambda function one day. Instead of using CSV files for persistence, I'll probably go with Dynamo.

The web3j-scala Ethereum Library

2017-11-29T00:00:00-05:00

This video shows web3j-scala, an idiomatic Scala wrapper I wrote around web3j, which is a Java 8 version of web3.js. These 3 libraries all leverage the json-rpc protocol that all Ethereum clients support. Web3j is a lightweight, reactive, somewhat type-safe library for Java and Android that integrates with nodes on Ethereum blockchains. Web3j-scala provides type safety and enhanced scalability over its Java and JavaScript cousins, as well as the pleasure of writing solutions in Scala.

Transcript

Much of the material for this presentation was taken from the web3j-scala GitHub page.

Kafka Streams vs. Akka

2017-07-28T00:00:00-04:00

Context switches are expensive for CPUs to perform, and each incoming message in an Akka system requires a context switch when fairness (minimal latency deviation) is important ^[1]. In contrast, Apache Kafka Streams offers consistent processing times without requiring context switches, so Kafka Streams produces significantly higher throughput than Akka can when contending with a high volume of small computations that must be applied fairly.

Fairness

“Fairness” in a streaming system is a measure of how evenly computing resources are applied to each incoming message. A fair system is characterized by consistent and predictable latency for the processing of each message. An emergent effect of fair systems is that messages are journaled, processed, and transformed for downstream computation in approximately the order received. The output of a fair system roughly matches input in real time; a perfectly fair system would provide a perfect correspondence between input messages and output messages.

“Fair enough” systems have a some acceptable amount of jitter in the ordering of the input vs. output messages.

You might expect that streaming systems are generally fair, but systems based on Akka rarely are because of the implementation details of Akka's multithreaded architecture. Instead, Akka-based systems enqueue incoming messages for each actor, and the Akka scheduler periodically initiates the processing of each actor's input message queue. This is actually a type of batch processing. The reason Akka does this is so the high cost of CPU context switches can be amortized over several messages, to keep system throughput at an acceptable rate.

Context Switching

It is desirable for the primary type of context switching in Akka systems to be between tasks on a thread because other types of context switching are more expensive. However, switching tasks on a thread costs a surprisingly large amount of CPU cycles. To put this into context, the accompanying table shows various latencies compared to the amount of time necessary for a context switch on an Intel Xeon 5150 CPU .

As you can see, a lot of computing can be done during the time that a CPU performs a context switch. My Intermediate Scala course has several lectures that explore multithreading in great detail, with lots of code examples.

Building Distributed OSes is Expensive and Hard

Akka is rather low-level, and the actor model it is built around was originally conceived 45 years ago. Applications built using Akka require a lot of custom plumbing. Akka applications are rather like custom-built distributed operating systems spanning multiple layers of the OSI model, where application-layer logic is intertwined with transport-layer, session-layer and presentation-layer logic.

Debugging and tuning a distributed system is inherently difficult. Building an operating system that spans multiple network nodes and continues to operate properly in the face of network partitions is even harder. Because each Akka application is unique, customers find distributed systems built with Akka to be expensive to maintain.

Kafka vs. Akka

For most software engineering projects, it is better to use vendor-supplied or open-source libraries for building distributed systems because the library maintainers can amortize the maintenance cost over many systems. This allows users to focus on their problem, instead of having to develop and maintain the complex and difficult-to-understand plumbing for their distributed application.

Akka is a poor choice for apps that need to do small amounts of computation on high volumes of messages where latency must be minimal and roughly equal for all messages. Kafka Streams is a better choice for those types of applications because programmers can focus on the problem at hand, without having to build and maintain a custom distributed operating system. KTable is also a nice abstraction to work with.

Motivation

EmpathyWorks™ is a platform for modeling human emotions and network effects amongst groups of people as events impinge upon them. Each incoming event has the potential to cause a cascade of second-order events, which are spread throughout the network via people's relationships with one another. The actual processing required for each event is rather small, but given that the goal is to model everyone on earth (all 7.5 billion people), this is a huge computation to continuously maintain. For EmpathyWorks to function properly, incoming messages must be processed somewhat fairly.

KTables and Compacted Topics

Kafka Streams makes it easy to transform one or more immutable streams into another immutable stream. A KTable is an abstraction of a changelog stream, where each record represents an update. Instead of a using actors to track the state of multiple entities, a KTable provides an analogy to database tables, where the rows in a table contains the current state for each of the entities. Kafka is responsible for dealing with network partitions and data collisions, so the application layer does not become polluted with lower-layer concerns.

Kafka’s compacted topics are a unique type of stream that only maintains the most recent message for each key. This produces something like a materialized or table-like view of a stream, with up-to-date values (subject to eventual consistency) for all key/value pairs in the log.

[1] From the Akka blog: “... pay close attention to your ‘throughput’ setting on your dispatcher. This defines thread distribution ‘fairness’ in your dispatcher – telling the actors how many messages to handle in their mailboxes before relinquishing the thread so that other actors do not starve. However, a context switch in CPU caches is likely each time actors are assigned threads, and warmed caches are one of your biggest friends for high performance. It may behoove you to be less fair so that you can handle quite a few messages consecutively before releasing it.”

[2] Taken from Latency numbers every programmer should know. The Intel Xeon 5150 was released in 2006; for a more up-to-date information about CPU context switch time, please see this Quora answer. Although CPUs have gotten slightly faster since 2006, networks and SSDs have gotten much faster, so the relative latency penalty has actually increased over time.

Resume-Driven Development

2017-05-30T00:00:00-04:00

I had occasion to speak with a technical manager at a Very Large Corporation at length recently. He wanted advice on where to find Spark programmers. After my usual 20 questions, I realized that his project did not have any requirement for the usual Spark capabilities:

No need of machine learning or pattern recognition.
He emphatically did not want to process streaming data.
He had a rather small volume of data.
He had no need of interactive analysis.
There was no need for ETL.
“Data enrichment” in his case was no more difficult than joining two SQL tables.
There was no requirement for trigger event detection.
... and there was no requirement for session analysis.

This man just wanted to put “I managed the development of an Apache Spark application” on his resume.

Shine Up That Turd

Buffing up a resume with a nonsensical misapplication of the latest software fashion is nothing new. Few people involved with such a project have a positive experience, however. When complex technologies are misapplied they often fail to deliver value, or the project just fails, which contributes to the trough of disillusionment in the Gartner Hype Cycle.

Consider the cynical and selfish world view that such a person must have to engage in this sort of activity. Software professionals are rewarded for being fashionable, and part of the mystique is to say and do things that others don’t fully comprehend. If one keeps moving fast enough, and learns some key phrases to use as put downs that sound impressive but have no meaning when taken out of context (“your approach would likely cause a data race condition”), others are kept off balance long enough for the perpetrator to achieve advantage.

As W.C. Fields said 100 years ago: “If you can’t dazzle them with brilliance, baffle them with bullshit.” He also made a movie entitled Never Give a Sucker an Even Break... but I digress.

Fashion is the Enemy of Delivering Valuable Results on Time

I often remark that the software business is just like the fashion business. The cycle goes like this:

The latest thing is preannounced. No documentation is available. Big claims are made.
Early code is provided, but it is slow, very buggy, there is no documentation, and the wild claims continue.
Seminars pump up the hype. Arm-waving abounds. ‘Experts’ associated with vendors are interviewed.
A whole new vocabulary is invented to describe the features. However, since those terms are never defined, few people realize that the message is actually “same old stuff with a new coat of paint.” The cool kids learn the words and can parrot them back they way they heard them, but they don't actually know what they are saying. Even if you knew what the phrase “a monad is a monoid in the category of endofunctors” means, it would not help you add value to a business process... and for most programmers who work in the business world, that is what actually pays their bills.
Gartner, ThoughtWorks and O’Reilly Radar say nice things about the new tech.
Marketing and sales activity shifts into high gear. Everyone’s laptop has stickers on it that show the owner is hip.
Some incomplete and mostly useless docs trickle out. Release candidates are eventually made available but it is not obvious how they provide value without significant magic being applied.
Version 1 finally arrives, completely useless to everyone except those few customers who paid huge dollars to the vendor, so they could get the bragging rights for what is essentially a custom build.
No useful benchmarks are available. Bugs crawl out from every crevasse. Documentation reads like it was badly translated from Chinese into Bantu, then into Navajo and then English.
Version 1.1 comes out, to address stop-ship problems that somehow did not prevent version 1 from shipping.
Version 2 is announced, completely incompatible with Version 1.x, and it is touted as revolutionary.
The digital lemmings decide that this is the cliff they hold dearest in their hearts, and proceed to jump from it en masse.
Gartner, ThoughtWorks and O’Reilly Radar downgrade the new tech to “meh”
... and the churn continues.

Documentation is not written to be understood, they are written to encourage payments to the digital elite – who have no interest in providing value; they just want to make money from foolish lemmings. This encourages others to aspire to be elitists. The hype cycle only works because our society accepts bullshit, and software has become a cultural phenomenon. We have the software quality and utility that we deserve as a society. If we want better software, we as a society will have to modify our purchasing behavior accordingly.

The software business has often demonstrated an insatiable need to be cool, at the expense of providing little or no value because evidencing value is either too hard for the many posers in the software business, or too boring for those who are capable but jaded.

Better Syntactic Sugar for Scala Futures

2017-05-25T00:00:00-04:00

Ever since Scala’s Futures were initially provided as part of Akka 2.0, programmers have been confused by the non-intuitive syntax. That is why ScalaCourses.com dedicates an entire lecture to For-comprehensions With Futures, as part of an 8 lecture series on Scala Futures within the Intermediate Scala course.

As Viktor Klang points out in his blog, the following code does not run 3 Futures in parallel:

Shell

def doSomething(someParameter: SomeType)
               (implicit ec: ExecutionContext): Future[Something] =
  for {
    v1 <- Future(someCalculation())
    v2 <- Future(someOtherCalculation())
    v3 <- Future(someDifferentCalculation())
  } yield doSomethingWith(v1, v2, v3)

The compiler has no way of ascertaining the programmer’s intent – perhaps it is desirable for some reason to run the 3 Futures one after the other.

Viktor suggests this syntax to make futures run in parallel:

Shell

def doSomething(someParameter: SomeType)
               (implicit ec: ExecutionContext): Future[Something] =
  for {
    f1 = Future(someCalculation())
    f2 = Future(someOtherCalculation())
    f3 = Future(someDifferentCalculation())
    v1 <- f1
    v2 <- f2
    v3 <- f3
  } yield doSomethingWith(v1, v2, v3)

While Viktor’s solution works, it is verbose. Worse, it silently fails to run the futures in parallel if the programmer accidentally writes even one of the expressions out of order:

Shell

def doSomething(someParameter: SomeType)
               (implicit ec: ExecutionContext): Future[Something] =
  for {
    f1 = Future(someCalculation())
    v1 <- f1
    f2 = Future(someOtherCalculation())
    v2 <- f2
    f3 = Future(someDifferentCalculation())
    v3 <- f3
  } yield doSomethingWith(v1, v2, v3)

Again, the compiler has no way of ascertaining the programmer's intent, so it should not generate an error or warning message.

We Need A Macro

A new right-associative operator, implemented as a Scala macro, would make the programmer's intent clear. Let’s call this operator <=: (parallel generator). The above code could be rewritten using the parallel generation operator like this:

Shell

def doSomething(someParameter: SomeType)
               (implicit ec: ExecutionContext): Future[Something] =
  for {
    v1 <=: Future(someCalculation())
    v2 <=: Future(someOtherCalculation())
    v3 <=: Future(someDifferentCalculation())
  } yield doSomethingWith(v1, v2, v3)

There is no longer any doubt that the programmer intended for all 3 futures to run in parallel.

The macro would examine all the for-expression’s generators and expand consecutive expressions that use the <=: operator to a series of variable declarations using the = operator followed by a series of assignments using the <- operator, exactly as we saw earlier:

Shell

def doSomething(someParameter: SomeType)
              (implicit ec: ExecutionContext): Future[Something] =
  for {
    f1 = Future(someCalculation())
    f2 = Future(someOtherCalculation())
    f3 = Future(someDifferentCalculation())
    v1 <- f1
    v2 <- f2
    v3 <- f3
  } yield doSomethingWith(v1, v2, v3)

Because the compiler ‘knows’ that the programmer’s intention was to run the Futures in parallel, this sort of error could cause an error or warning message to be generated:

Shell

def doSomething(someParameter: SomeType)
              (implicit ec: ExecutionContext): Future[Something] =
  for {
    v1 <=: Future(someCalculation())
    x <- List(1, 2, 3)
    v2 <=: Future(someOtherCalculation())
    y <- List("a", "b")
    v3 <=: Future(someDifferentCalculation())
  } yield doSomethingWith(v1, v2, v3)

... or the macro might reorder the generators and issue a warning that it did so.

Include the Macro in Scala 2.12.x

This macro should become part of the Scala language so long as Futures are part of the standard runtime. If and when Futures are hived out of the standard runtime, the macro should be packaged with Futures.

PS: @flaviusbraz tweeted on May 25, 2017: “Easily doable with a macro transformation. In fact, I’ve implemented this transformation at Twitter, but it’s not open source.”

Exploratory Conversation With AIs

2017-04-29T00:00:00-04:00

This is a high-level proposal, not documentation for any specific system that exists today. I would be happy to discuss possible implementation details with interested parties, including the creation of a prototype.

Intents and Subgoals

Some of today’s interactive voice response systems, such as AWS’s Lex, Amazon’s Alexa, and Google’s Dialogflow, use intents as a means of gathering the required parameters to fulfill a voice command. The next big step in voice interfaces to computation would be the ability to have an exploratory conversation, where one or more subgoals may or may not emerge.

Subgoals might be fulfilled by intents. Continuous tracking of viable subgoals could be accomplished by a constraint-based solver that uses rules to participate in an exploratory conversation by recognizing and prioritizing potential subgoals, thereby activating and deactivating intents during the conversation. Once a user confirms that a potential subgoal is desirable, the system might consider that goal to be the only goal worth pursuing, or it might continuously elicit more information from the user regarding other potential subgoals. User-supplied information might be applied to a model associated with a subgoal/intent, and/or it might be retained as a user preference.

Anthropomorphism

If an AI system is a good conversationalist, even if it has no physical presence, people will anthropomorphically attribute the system with human traits, emotions, and intentions. This should not be discouraged. Rather, AI systems should aspirationally model our higher selves. The movie Her explored this in a charming way.

Breaking It Down

An interactive voice response system recognizes an intent by a key word or phrase uttered by the user. For example, if the user says: “Computer, order dinner”, and the system was previously ‘taught’ to understand the word “order” as a key word for launching the OrderDinner intent, the system would then elicit the kind of food the user wanted, how many people would be eating, etc, and then order the desired dinner.

Anthropomorphically: if intents are the only programming technique employed, they present interactive voice systems as slaves.

Intents are useful for recognizing commands and gathering enough associated data to carry out the command. Intents are not useful for open-ended conversations where there is no explicit goal, or potential subgoals emerge as the result of a dialog. Once a subgoal is identified and confirmed, however, processing an intent is an efficient mechanism for fulfilling the emergent subgoal.

Designers of chatbots and video games are familiar with goal-seeking behavior. Exploratory conversation is how people interact with each other IRL (in real life). Each individual uses a variety of strategies for initiating exploratory conversation, and participating in it. For example, small talk at a gathering is a useful skill for getting to know other people, and a socially adept practitioner of small talk can innocuously gather a lot of information in this way. Another strategy is to make a controversial statement, and use the resulting banter to learn about the conversationalists and the possible subgoals that they might request. A wide variety of such strategies exist, and could be utilized by conversational systems.

Anthropomorphically: with exploratory conversation capability, interactive voice systems can be presented socially as co-operative entities.

Agenda and Strategy

Unlike people, computers can only do a finite number of tasks, and voice recognition systems are programmed with a finite number of intents. I define the potential agenda for a chatbot or video game to be the entire scope of its pre-programmed intents. Agendas may be fully disclosed, or they might be obvious, or they might be unveiled over time. Ethical considerations might apply to the design and implementation of conversational AI systems.

Users should be apprised of the agenda of every autonomous computer entity they encounter. To mitigate potential problems, an industrial code of conduct should be established. The Europe Union will likely be the first government to require published standards and possibly a certification process.

I define strategy to mean the autonomous modification to goal-seeking behavior when a criterion is met. For example, an insurance chatbot might begin to solicit sensitive information only after it has reason to believe that a modicum of trust has been established with the prospect. How a strategy is executed is quite significant; by this I mean the degree of diplomacy. Some circumstances might sacrifice diplomacy to save lives, while under normal circumstances the AI entity should treat everyone with respect and kindness. Again, the Europe Union is likely to be the first government to require published standards and possibly a certification process.

Update May 5, 2017

Today, a week after I first published this articleing, I received the following email from api.ai. Two points worth mentioning:

I have no apps running on api.ai that use their Small Talk feature.
The Small Talk feature, even after the changes mentioned below, does not approach the capability I proposed above.

Nonetheless, it was great to get the information.

Hi Michael,

We noticed you are using the Small Talk customization feature in your agent, and that's why we are getting in touch.

The Small Talk customization has been updated.

The new version will be even faster and more accurate. It also includes some additional intents that you can customize. The update also includes changes in actions and parameters returned with the responses.

If you use them in your business logic, please review the changes here. If you’re ready, you can apply change right now here. Otherwise, the changes will be applied automatically on May 29, 2017.

If you’d like a more flexible way to implement small talk, then get your source agent and implement the use cases you care about in your timeline.

Regards,

API.AI Team

About the Author

Mike Slinn has been pursuing EmpathyWorks™ original AI research (augmenting machine learning with simulations and event-driven architecture) since 2007.

UI Considerations for the Visually Impaired

2017-04-04T00:00:00-04:00

If you look at a computer screen all day, and the screen has a lot of glare, the resulting eyestrain can temporarily degrade your vision to the same degree as a visually impaired person.

2022-05-18 Update

An excellent article was just published by Andrew Somers about WCAG 2.0 (Web Content Accessibility Guidelines). The article discusses contrast, dark mode, minimum font size, line spacing, and the limitations of current scientific support for readability. Mr. Somers is a W3 AGWG invited expert and a readability and color science researcher.

Background

According to the Farlex Free Medical Dictionary, visual impairment, or low vision, is a severe reduction in vision that cannot be corrected with standard glasses or contact lenses and reduces a person's ability to function at certain or all tasks. The World Health Organization estimates that 285 million people are visually impaired worldwide, of which 246 million have low vision and 39 million are blind. People's vision generally deteriorates with age.

In the US, cataracts are the most common form of visual impairment, according to the National Eye Institute. The NEI estimates that more than 24 million Americans over the age of 40 have cataracts. “The risk of cataracts increases with each decade of life starting around age 40. By age 75, half of white Americans have cataracts. By age 80, 70 percent of whites have cataracts, compared with 53 percent of blacks and 61 percent of Hispanic Americans.” Males are twice as likely as females to get cataracts.

Cataracts start slowly. If you wear glasses, at first you think you can‘t quite clean them properly. Then you notice your sunglasses are also always dirty. Then you notice that white screens with black text are hard to read because of the glare from the white areas.

The medical treatment is to replace the cloudy and .webp lens in the eye that has the cataract with a plastic lens. In many European countries, cataract surgery is the most common operation performed in that country. Usually, people delay the surgery until their quality of life and independence his diminished markedly.

Coping With Visual Impairment

Saturated Color and White Backgrounds

Dark screens with a minimum of saturated color helps. Increasing font size helps, and sanserif fonts, which have fewer spindly features, are easier to read.

Some OSes offer a dark theme. You can enable the Windows 10 dark theme from Settings / Personalization / Colors. Scroll down and select ‘Dark’ under ‘Choose your app mode’. The Universal Windows Platform applications will immediately darken. However, if a developer did not make the effort to support the dark theme, their program will continue with their original color theme. Unfortunately, Windows File Explorer is one of those applications.

Enabling the Windows 10 dark theme

Some software offers a nice dark theme that does not require any fussing, for example IntelliJ IDEA's Darkula, Adobe Premiere Pro and Adobe Audition.

The Firefox web browser’s Dark Background and Light Text plugin is very helpful. By default, the plugin overrides the CSS styles for all web pages, so the background is black, and text is white. The plugin can be trained to leave certain pages untouched, for those pages that become unusable with the modified default stylesheet, for example, Google Calendar. Other sites already offer a nice dark theme, so they do not need the modified theme thrust upon them, for example tweetdeck.twitter.com.

Some software offers a dark theme which requires further modification before a good result is a achieved. For example, the Thunderbird email client with the TT DeepDark addon gets you partway there, then Tools / Options / Display / Fonts & Colors / Colors shows the following dialog, where you can set the colors as shown:

Display Fonts & Colors

As another example, Adobe Reader has a dark theme, which can be enabled via Edit / Preferences / Accessibility; enable Replace Document Colors and click on Custom Color; change the Page Background to black and Document Text to light gray. Disable Only change the color of black text or line art, and ensure that Change the color of line art as well as text is enabled.

Enabling the Adobe Reader dark theme

Small Text and Low Contrast Colors

Users who know CSS can enforce a custom stylesheet on a per-site or a per-page basis for hard-to-read web pages. The My Style plugin for the Chrome browser activates with Ctrl-M, and users can enter CSS to suit. As any web developer knows, CTRL-Shift-C starts HTML inspection, so the HTML in question can be viewed, and the CSS experimented with before immortalizing the changes with My Style.

Design Considerations

If your web page has images in it, try to use transparency instead of a white background. It would help visually impaired readers to only inflict strong primary colors, or a white background, when necessary. For example, this entry in a stylesheet would only show a light gray background when the user mouses over the image, otherwise, the background would remain transparent:

Shell

img:hover { background-color: #666; }

Are My Hands-Free Devices Always Listening?

2017-01-15T00:00:00-05:00

2022-04-27 Update

Amazon is monetizing your conversations, according to a study published by 10 researchers at the University of Washington, the University of California-Davis, the University of California-Irvine, and Northeastern University. The study is titled “Your Echos are Heard: Tracking, Profiling, and Ad Targeting in the Amazon Smart Speaker Ecosystem”.

Abstract

Smart speakers collect voice input that can be used to infer sensitive information about users. Given a number of egregious privacy breaches, there is a clear unmet need for greater transparency and control over data collection, sharing, and use by smart speaker platforms as well as third party skills supported on them. To bridge the gap, we build an auditing framework that leverages online advertising to measure data collection, its usage, and its sharing by the smart speaker platforms.

We evaluate our framework on the Amazon smart speaker ecosystem. Our results show that Amazon and third parties (including advertising and tracking services) collect smart speaker interaction data; where Amazon shares it with as many as 41 advertising partners. We find that Amazon processes voice data to infer user interests and uses it to serve targeted ads on-platform (Echo devices) as well as off-platform (web). Smart speaker interaction leads to as much as 30x higher ad bids from advertisers. Finally, we find that Amazon’s and skills’ (apps’) operational practices are often inconsistent with their privacy policies.

Some of the references cited by the paper are disturbing. This is the reference that concerned me the most:

Alexa has been eavesdropping on you this whole time.
– Washington Post, 2019-05-06.

The short answer: probably not.

A longer answer: Depends on what you mean by ‘listening’. If you mean “is Alexa storing or sending every sound or voice utterance to a server”, the answer is again “probably not”.

A more complete answer: unless all the source code for a device is made available for scrutiny, and the build process is similarly examined, the only way to be sure that device does not have operating modes that are contrary to expectations would be to connect the device to a network monitor that only allows communication with specific servers at designated times. Normal computer security concerns also pose risks; for example, viruses and other malware could be injected into a device could cause it to behave differently.

At Least Echo’s Mute Button Works

Update 2021-01-12 – Amazon Echo’s mute button was reverse engineered.

The mute button appears to be very real and functional. When the button glows red the power is removed from the microphones.

Some Paranoia is Healthy

Mattel’s Aristotle is targeted at children. It uses Microsoft Bing for searching, Microsoft Cortana for voice processing and streams video to the cloud. It can read bedtime stories and play soothing sounds if your child wakes up at the night. It can recognize your children’s imperfect speech, and apparently adapts as they get older and become curious about the world. Aristotle is an AI to help raise your child. Did they do a terrific job or a terrible job? It depends on the factors you take into consideration.

Aristotle can respond to adults differently than to children. Its logging capability is profound. It tracks things like wet diapers and feedings, and can order supplies when asked by an adult.

Aristotle can use object recognition to identify flashcards, or co-opt a toy without electronics and thereby enhance it with sound effects or even another personality.

Mattel has about 500 partners, and they have been invited to build connected toys and apps. The hardware uses 256-bit encryption for all transmissions to Aristotle’s servers, and data is handled internally in compliance with COPAA and HIPAA.

It is not difficult to write code that detects when a child is alone

How hard would it be to write code that detects when the child is alone? If a malicious entity wanted to, they could embed their program in the device, use IP geolocation and other characteristics to identify specific families, and cause the device to behave differently and/or tell the child things when no adults were paying attention. Who knows what malicious software might do?

Open Source To The Rescue

Closed source systems cannot be adequately vetted for public usage. It is conceivable that selected children might become secretly radicalized by their toy. In 1913 Justice Louis Brandeis said “Sunlight is the best disinfectant”. Open-source applications, with updates that can be vetted by any interested party, are the only way to ensure these devices are truly safe.

Hands-Free Voice as a User Interface

2017-01-10T00:00:00-05:00

Amazon’s Alexa is a runaway success. Already, 5% of US households have a hands-free voice operated device. 50% of Echo owners keep one in the kitchen. Gartner predicts that by 2018, 30% of our interactions with technology will be via voice conversations. The near future will have many hands-free, voice-driven, Internet-aware applications, and some will also provide a web interface in order to satisfy use cases that require a text or graphic display.

The Near Future

The near future will have many hands-free, voice-driven, Internet-aware applications, and some will also provide a web interface to satisfy use cases that require a text or graphic display. Potential usages include hands-free voice control of applications running on computers, tablets and phones, hands-free voice control of home devices and vehicles, dictaphones, and games that provide anthropomorphic characters with the ability to generate and understand speech.

Services that developers and integrators can use for hands-free, voice-operated applications include Amazon’s Alexa, Google Assistant, Google Voice, Apple’s Siri, and Microsoft Cortana.

The Present

Only recently has it become feasible to use hands-free voice as a user interface. The best hands-free, voice applications are innately distributed – that is, they perform some computation on a local device, and they also require a connection to a server to do the heavy computation necessary for a high-quality experience. Most companies developing applications that use hands-free voice as a user interface require and will continue to require services provided by third parties. However, from the point of view of the application developer's company, the data shared with these third parties can represent a significant security risk. From the user's point of view, this data can represent a potentially significant privacy breach. I'm going to discuss why this is so in this article, and what can be done about it.

Privacy and security concerns for synthesizing speech are minimal because quality speech can be synthesized without context specific to an individual. This means that user data need not be associated with the words or phrases being synthesized, so anonymous phrases can be and should be sent to the remote service that generates the audio files produced by the speech synthesis.

The best value for high-quality voice generation is achieved with a distributed solution. In this scenario, voice generation is simple to initiate: a text string is sent to a remote service, and an audio clip containing synthesized speech is returned. Most voice generators support embedded markup to control inflection. For example, Amazon Polly and Google Assistant (formerly api.ai) both use SSML; Apple uses a variety of techniques across its products, and Microsoft uses SAPI.

In contrast, the heavy lifting necessary to recognize unconstrained vocabulary requires lots of compute resource, and raises privacy and ethical issues. The main issues are:

Recognizing a trigger word or phrase
Training a voice recognition engine
Determining the appropriate privacy/effectiveness trade-off for your application
Integrating with third-party or proprietary services

What Are These Things Made From?

Hands-free voice operated devices have most of the same components as a tablet, but often do not have a screen. An ARM A8 to A11 processor is typically embedded in a chip die that also contains a powerful digital signal processor. In other words, they can do a lot of computing; they are more powerful than any mobile phone, and they are more powerful than most tablets.

Amazon Echo parts

Amazon Echo PC board

TI DM37x processors

The open source ecosystem that has grown up around these DSP/CPU chips includes a variety of operating systems, including real-time and various Linux distributions, and many applications. The OSes are: Android, DSP/BIOS, Neutrino, Integrity, Windows Embedded CE, Linux, and VxWorks. It is quick and easy to design a complete device similar to Alexa and bring it to market.

Recognizing a trigger word or phrase

One of the first major challenges one encounters when designing a device that responds to voice, is how to ignore silence, or recognizing noise that should be ignored. This requires some level of signal processing. In contrast, requiring the user to push a button makes for a much simpler user interface, but this requirement would greatly restrict the types of applications possible.

mobile transceiver' class="imgImg rounded shadow" src="/blog/images/voiceActivated/Icom-IC-706MKIIG_300x180.png" style='width: 100%; padding: 1em;' title='Icom 706 Mark II g
mobile transceiver' />

Icom 706 Mark II g
mobile transceiver

I am a ham radio operator, and I use the well-known protocol for addressing a specific individual when using a broadcast medium. If I want to talk to another ham operator, I address them by their call sign three times and await their acknowledgement before giving my message. “KG6LBG, KG6LBG, KG6LBG, this is KG6LDE, do you read me?” If that person hears their call sign, they reply “KG6LDE, this is KG6LBG, go ahead.” “KG6LBG, I just called to say hello, over.” “KG6LDE, Hello yourself, over and out.”

Hands-free voice applications also need to recognize a trigger word or phrase (also known as a hotword) that prefaces an audio stream which will be processed for voice recognition. Without such a trigger, either the user would need to press a button to start recording their speech, or a continuous audio stream would have to be processed. I'm interested in hands-free voice recognition. Since the voice recognition processing must be done on a remote service, a lot of bandwidth and CPU power would be wasted processing silence or irrelevant sound. Because it is undesirable to a continuous audio stream to a server to accurately recognize trigger words, the methods used to recognize them are less accurate.

Proprietary Alternatives

Both of these products can be configured to perform the trigger word recognition without requiring any bandwidth between the CPU running the recognition program and a server. Neither of them provide a JavaScript implementation, which means that unless the trigger word recognition program in installed on the local machine, it must be installed on a connected server and bandwidth will be used at all times.

Sensory, Inc's TrulyHandsfree library (license), based in Santa Clara, CA. “We do not have any low cost or free license or library. We are unable to support any student or individual.”
kitt.ai's Snowboy, based in Seattle, WA and partially funded by Amazon.

CMU Sphinx

Designed for low-resource platforms, implementations of CMU’s Sphinx exist for C (which supports Python) and Java. Hotword spotting is supported. Several versions of Sphinx exist, with varying free and commercial licenses. Reports suggest that Sphinx works reasonably well but I have not tested yet. Sphinx powers Jasper.

JavaScript alternatives

Annyang works well, but requires Google's web browsers and servers, so it is probably not reasonable to use it just for hotword detection.
Voice-commands.js also requires Google's web browsers and servers.
Sonus is a Node framework which will be able to be configured to use a variety of back ends one day. It does hotword detection by using Snowboy; the authors obviously ignored Snowboy's license terms.
JuliusJS is a JavaScript port of the “Large Vocabulary Continuous Speech Recognition Engine Julius”. It does not call any servers and runs in most browsers. Recognition is weak and it requires a lot of CPU.
PocketSphinx.js is a free browser-based alternative, unfortunately it is horrible.

Training a voice recognition engine

Voice recognition engines need to be trained on a large dataset for the desired languages. Recognition effectiveness is less than linearly proportional to the size of the dataset, and high-quality datasets are important. Truly large amounts of data are required. Amazon, Apple, Google and Microsoft have commercial products that were trained using enormous proprietary datasets. This is a substantial investment, so only well-capitalized organizations will be able to offer their own voice recognition engines for unconstrained vocabularies.

Determining Your Application's Privacy / Effectiveness Tradeoff

A voice recognition's effectiveness increases for specific users if their voice streams are recorded and stored, then used for further training. However, this means that privacy and security are traded off for effectiveness. This service provider's tradeoff might not be optimal for your use case. Apple's Siri only associates the stored voice recordings with you for 6 months. Google's Assist and Voice, and Amazon Alexa store all your voice recordings forever, unless you explicitly delete them. I could not discover how long Microsoft's Cortana and Skype store voice recordings, or how to delete them.

Integrating With Third-party or Proprietary Services

Because voice recognition returns a structured document like JSON or XML, and voice generation is simple to initiate, integration is well understood and many options exist.

I Updated the Apache Spark Reference Applications

2016-11-15T00:00:00-05:00

Overview

The Apache Spark committers just accepted my pull request that updated the official Twitter Classifier Reference Application from Spark 1.4 / Scala 2.10 to Spark 2 / Scala 2.11. This article discusses some things I did in the pull request from the point of view of a Scala programmer. A primary goal was to rewrite the reference application using idiomatic and functional-style Scala. This article briefly discusses two unique aspects that I addressed: command-line parsing and DRYing up the code by importing scopes. I did several other things to improve the reference application, such as modularizing the code and providing run scripts, but this article does not address them because those techniques are generally well understood.

I did not upgrade the reference application to Scala 2.12, which was released a couple of weeks ago because Spark does not yet support Scala 2.12. Josh Rosen of Databricks wrote me and said:

“Some of Spark’s dependencies and by Scala-version-specific code changes necessary to work around method overloads became ambiguous in 2.12. The umbrella ticket tracking 2.12 support can be found at issues.apache.org/jira/browse/SPARK-14220. One of the hardest pieces will be issues.apache.org/jira/browse/SPARK-14643 (see the linked design document on that issue). Lack of 2.12 support for Breeze and its dependencies is likely to be another serious blocker, but that might be avoidable by only publishing a subset of the projects with 2.12 to begin with (e.g. only Spark core / SQL at first).”

Command Line Parsing

I modified the reference applications’ command line parsing to use a Scala library that supported idiomatic Scala (Commander Scala), instead of Apache Commons CLI, which is the Java library that was previously used. The result was simple, clean and very terse code that is intuitive to understand and easy to maintain. Commander Scala automatically generates the help message. Take a look at the collect command’s parsing. You’ll notice that it uses some common code for parsing Twitter authentication parameters. This code is much shorter than the previous code, easier to understand and modify, and is more flexible.

Shell

import com.github.acrisci.commander.Program
import java.io.File

abstract sealed case class CollectOptions(
  twitterOptions: TwitterOptions,
  overWrite: Boolean = false,
  tweetDirectory: File = new File(System.getProperty("user.home"), "/sparkTwitter/tweets/"),
  numTweetsToCollect: Int = 100,
  intervalInSecs: Int = 1,
  partitionsEachInterval: Int = 1
)

object CollectOptions extends TwitterOptionParser {
  override val _program = super._program
    .option(flags="-w, --overWrite", description="Overwrite all data files from a previous run")
    .usage("Collect [options] <tweetDirectory> <numTweetsToCollect> <intervalInSeconds> <partitionsEachInterval>")

  def parse(args: Array[String]): CollectOptions = {
    val program: Program = _program.parse(args)
    if (program.args.length!=program.usage.split(" ").length-2) program.help

    new CollectOptions(
      twitterOptions = super.apply(args),
      overWrite = program.overWrite,
      tweetDirectory = new File(program.args.head.replaceAll("^~", System.getProperty("user.home"))),
      numTweetsToCollect = program.args(1).toInt,
      intervalInSecs = program.args(2).toInt,
      partitionsEachInterval = program.args(3).toInt
    ){}
  }
}

Here is how to sidestep the Spark help message and display the help message for the collect entry point:

Shell

$ spark-shell \
-class com.databricks.apps.twitterClassifier.Collect \
-jars target/scala-2.11/spark-twitter-lang-classifier-assembly-2.0.0.jar \
-- -help
Usage: Collect [options] <tweetDirectory> <numTweetsToCollect> <intervalInSeconds> <partitionsEachInterval>
Options:
-h, — help output usage information
 -V, — version output the version number
 -w, — overWrite Overwrite all data files from a previous run
 -v, — accessTokenSecret [type] Twitter OAuth Access Token Secret
 -t, — accessToken [type] Twitter OAuth Access Token
 -s, — consumerSecret [type] Twitter OAuth Consumer Secret
 -c, — consumerKey [type] Twitter OAuth Consumer Key

Importing Inner Scope Into Another Object

Apache Spark is unusual in that you cannot encapsulate a Spark streaming context in a type instance. A memory overflow occurs when you try to instantiate a Scala trait or class that creates a Spark context. The solution is to use a unique Scala feature: the ability to import inner scope from an object into another scope. This meant that the code was made DRY (common code was not repeated), without using classes or traits.

Here is how I took advantage of this little-known Scala technique: first I defined the SparkObject object within a package object so it was easily found:

Shell

object SparkSetup {
  val spark = SparkSession
    .builder
    .appName(getClass.getSimpleName.replace("$", ""))
    .getOrCreate()
  val sqlContext = spark.sqlContext
  val sc: SparkContext = spark.sparkContext
  sc.setLogLevel("ERROR")
}

Next I imported all the variables defined in SparkSetup into the Collect object’s scope, including sc, which was used twice, like this:

Shell

object Collect extends App {
  val options = CollectOptions.parse(args)
  import SparkSetup._
  val ssc = new StreamingContext(sc, Seconds(options.intervalInSecs))
  Collector.doIt(options, sc, ssc)
}

Want to learn more practical Scala techniques? Head over to ScalaCourses.com and enroll! The combination of the Introduction to Scala and Intermediate Scala courses will teach you everything you need to know to start your journey with Apache Spark.

Mike Slinn is the lead Scala instructor at ScalaCourses.com.

Publishing Maven Artifacts to AWS S3

2013-07-07T00:00:00-04:00

I wanted a simple, flexible and cheap way of publishing proprietary Maven artifacts created by sbt projects such that they could be securely retrieved by authorized individuals. I liked the idea of versioned artifacts, but did not want to use GitHub or BitBucket to host the artifacts because of the hassle of maintaining ever-larger git repos. Instead, I opted for S3's optional versioning mechanism.

The technique described here relies on the fact that publishing to a local file (using sbt publish) generates all the necessary artifacts, which merely need to be copied to the right directory on the Artifactory server. The server need not be anything special: a normal web server works fine, as does webdav. A variety of other protocols are also supported by sbt.

I created two test projects on GitHub: testPublishLib and testPublishApp. You could clone them if you would like to try this. Each of them has a README that explains what they do.

I used s3cmd to manage the AWS S3 buckets that hold the repository. You can obtain s3cmd for most OSes. I found I needed to install python-magic:

Shell

$ sudo pip install python-magic

Let s3cmd know your s3 keys:
Shell
```
$ s3cmd --configure
```
Create the S3 bucket, which must be unique. If you want to repository to be publicly visible, be sure that the bucket name starts with www. If you already have the S3 bucket then just omit this step.
Shell
```
$ s3cmd mb s3://www.mymavenrepo
```
If you want to repository to be publicly visible, you need to enable the S3 bucket web site option:
Shell
```
$ s3cmd ws-create s3://www.mymavenrepo
```
Publish your library to a local repository. testPublishLib is an example of how to set up a project properly.
Shell
```
$ sbt publish
```
Copy your locally published artifacts to S3. You can either type it out longhand:
Shell
```
$ s3cmd -P sync ~/.ivy2/local/com/micronautics/test_publish_lib \
s3://www.mymavenrepo/snapshots/com/micronautics/test_publish_lib
```
... (note that the -P option makes the files publicly visible), or you can use s3publish:
Shell
```
$ s3publish com/micronautics/test_publish_lib
```
If your repository is not public, you will need to provide authentication in a file which I called ~/.sbt/awsCreds.sbt for convenience:
~/.sbt/awsCreds.sbt
```
credentials +=
  Credentials("AWS Realm", "www.mavenrepo.s3.amazonaws.com",
    "myUserId", "myPassword")
```
Use the published artifact from your sbt project by including a resolver of the form:
Shell
```
"AWS Snapshots" at "https://www.mavenrepo.s3.amazonaws.com/snapshots"
```
The TestPublishApp project is a working example of how to do that.

Following is the script I wrote to upload to S3, which I called s3publish. It assumes you are always publishing a snapshot; and that the files should be public. I leave it to you to extend this script to handle releases and private content if you have the need.

#!/bin/bash if [ $# -eq 0 ]; then echo "Usage: `basename $0` pubpath/name [repo]" echo " Where pubpath might be something like com/micronautics" echo " name is name of artifact to publish" echo " repo is the optional name of the bucket to publish to" echo " Example: `basename $0` com/micronautics/test_publish_lib" exit 1 fi REPO=www.mymavenrepo OPTIONS=-P if [ $# -eq 2 ]; REPO=$2; fi s3cmd $OPTIONS sync ~/.ivy2/local/$1 s3://$REPO/snapshots/$1

Load Testing ScalaCourses.com

2013-06-01T00:00:00-04:00

ScalaCourses.com, which will be announced next week, is built using the entire Typesafe stack: Scala 2.10, Play 2.1, Slick 1.0 and Akka 2.1. It runs on Heroku.

I ran a load test on the app running on only one Heroku dyno. I configured JMeter to use 300 threads, hammering at full speed (no pauses between hits). Testing was done from my desktop. The test generated 16.5Mb/s inbound and 4.1Mb/s outbound according to iptraf. The test did not download page assets because they are served directly from AWS S3.

50% of all responses were received in under 110ms, and 95% were received in under 165ms. The distance from my workstation in Half Moon Bay, CA, USA to the Heroku app server, running from an AWS server in Ashburn, Virginia, USA is about 3000 miles or 4,828 km away. Considering that average ping time is ~100ms, that is amazingly good! Ping measures round-trim time for a test packet, and mtr showed the average ping time as ~95ms with a standard deviation of 18.

Yes, I did put a lot of care into the design of the app so that it would scale well, but I had not expected such fantastic results. Kudos to each of the Typesafe product teams, and to Heroku!

Cleaning the Heroku Cache

2013-03-18T00:00:00-04:00

In an earlier article, I talked about using a bash console to experiment with a Heroku app. I mentioned that you should not run sbt. Of course, that is exactly what I did, and I discovered the hard way that the cache can't be cleared from the Heroku bash shell. The build cache is held outside the dyno and cannot be accessed from inside a running dyno.

Cleaning the cache is accomplished with a special cache cleaner buildpack. Set the buildpack in your app like this:

Shell

heroku config:add BUILDPACK_URL=https://github.com/heroku/heroku-buildpack-scala.git#cleancache --app myapp

Push your code:

Shell

$ git push heroku master

Remove the cache cleaner buildpack:

Shell

$ heroku config:remove BUILDPACK_URL --app myapp

Change a file, commit and push again:

Shell

$ git push heroku master

All better!

Using Scala’s String Interpolation to Access a Map

2013-03-15T00:00:00-04:00

Given a map, would it not be nice to have a shorthand way of looking up a value from a key, and to provide a default value? I’ve wanted to be able to do this for a long time. Scala 2.10 makes this really easy!

The MapLookup class below contains a Map[String, Int] that can be looked up by using a dollar sign ($) from code that contains a reference to the implicit class. If the lookup key is not defined by the map, a zero is returned.

Shell

implicit class MapLookup(val sc: StringContext) {
  val map = Map(("a", 1), ("b", 2), ("c", 3)).withDefaultValue(0)

  def $(args: Any*): Int = {
    val orig = sc.s (args : _*)
    map.get(orig)
  }
}

Assuming that the above is stored in a file called strInterp.scala, here are some examples of usage:

Shell

$ scala -i strInterp.scala
Loading strInterp.scala...
defined class MapLookup

Welcome to Scala version 2.10.1 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_17).
Type in expressions to have them evaluated.
Type :help for more information.

scala> $"a"
res2: Int = 1

scala> $"z"
res3: Int = 0

Short and sweet!

Listing of all AWS Elastic Transcoder Presets

2013-03-15T00:00:00-04:00

Here is a listing of all AWS Elastic Transcoder presets, provided ‘out of the box’ as system-wide presets.

Id: 1351620000001-000001
Name: System preset: Generic 1080p
Description: System preset generic 1080p
Container: mp4
Audio: {Codec: AAC,SampleRate: 44100,BitRate: 160,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=4},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  5400,FrameRate: 29.97,AspectRatio: 16:9,MaxWidth: 1920,MaxHeight: 1080,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-000010
Name: System preset: Generic 720p
Description: System preset generic 720p
Container: mp4
Audio: {Codec: AAC,SampleRate: 44100,BitRate: 160,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  2400,FrameRate: 29.97,MaxWidth: 1280,MaxHeight: 720,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-000020
Name: System preset: Generic 480p 16:9
Description: System preset generic 480p 16:9
Container: mp4
Audio: {Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  1200,FrameRate: 29.97,MaxWidth: 854,MaxHeight: 480,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-000030
Name: System preset: Generic 480p 4:3
Description: System preset generic 480p 4:3
Container: mp4
Audio: {Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  900,FrameRate: 29.97,MaxWidth: 640,MaxHeight: 480,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-000040
Name: System preset: Generic 360p 16:9
Description: System preset generic 360p 16:9
Container: mp4
Audio: {Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  720,FrameRate: 29.97,MaxWidth: 640,MaxHeight: 360,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-000050
Name: System preset: Generic 360p 4:3
Description: System preset generic 360p 4:3
Container: mp4
Audio: {Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  600,FrameRate: 29.97,MaxWidth: 480,MaxHeight: 360,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-000060
Name: System preset: Generic 320x240
Description: System preset generic 320x240
Container: mp4
Audio: {Codec: AAC,SampleRate: 22050,BitRate: 64,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=1.3},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  300,FrameRate: 15,MaxWidth: 320,MaxHeight: 240,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-100010
Name: System preset: iPhone4
Description: System preset: iPod touch 5G, 4G, iPad 1G, 2G
Container: mp4
Audio: {Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=3.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  2200,FrameRate: 30,MaxWidth: 1280,MaxHeight: 720,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-100020
Name: System preset: iPhone4S
Description: System preset: iPhone 5, iPad 3G, 4G, iPad mini, Samsung Galaxy S2/S3/Tab 2
Container: mp4
Audio: {Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=high, Level=4.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  5000,FrameRate: 30,MaxWidth: 1920,MaxHeight: 1080,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-100030
Name: System preset: iPhone3GS
Description: System preset: iPhone 3GS
Container: mp4
Audio: {Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  600,FrameRate: 30,MaxWidth: 640,MaxHeight: 480,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-100040
Name: System preset: iPod Touch
Description: System preset: iPhone 1, 3, iPod classic
Container: mp4
Audio: {Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  1500,FrameRate: 30,MaxWidth: 640,MaxHeight: 480,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-100050
Name: System preset: Apple TV 2G
Description: System preset: Apple TV 2G
Container: mp4
Audio: {Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=3.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  5000,FrameRate: 30,MaxWidth: 1280,MaxHeight: 720,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-100060
Name: System preset: Apple TV 3G
Description: System preset: Apple TV 3G, Roku HD/2 XD
Container: mp4
Audio: {Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=high, Level=4},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  5000,FrameRate: 30,MaxWidth: 1920,MaxHeight: 1080,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-100070
Name: System preset: Web
Description: System preset: Facebook, SmugMug, Vimeo, YouTube
Container: mp4
Audio: {Codec: AAC,SampleRate: 44100,BitRate: 160,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=3.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  2200,FrameRate: 30,MaxWidth: 1280,MaxHeight: 720,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-100080
Name: System preset: KindleFireHD
Description: System preset: Kindle Fire HD
Container: mp4
Audio: {Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=4},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  2200,FrameRate: 30,MaxWidth: 1280,MaxHeight: 720,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-100090
Name: System preset: KindleFireHD8.9
Description: System preset: Kindle Fire HD 8.9
Container: mp4
Audio: {Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=4},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  5400,FrameRate: 30,MaxWidth: 1920,MaxHeight: 1080,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-100100
Name: System preset: KindleFire
Description: System preset: Kindle Fire
Container: mp4
Audio: {Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=3.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate:
  1600,FrameRate: 30,MaxWidth: 1024,MaxHeight: 576,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-200010
Name: System preset: HLS 2M
Description: System preset: HLS 2M
Container: ts
Audio: {Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=3.1},KeyframesMaxDist: 90,FixedGOP: true,BitRate:
  1872,FrameRate: auto,MaxWidth: 1024,MaxHeight: 768,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-200020
Name: System preset: HLS 1.5M
Description: System preset: HLS 1.5M
Container: ts
Audio: {Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=3.1},KeyframesMaxDist: 90,FixedGOP: true,BitRate:
  1372,FrameRate: auto,MaxWidth: 960,MaxHeight: 640,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-200030
Name: System preset: HLS 1M
Description: System preset: HLS 1M
Container: ts
Audio: {Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=3.1},KeyframesMaxDist: 90,FixedGOP: true,BitRate:
  872,FrameRate: auto,MaxWidth: 640,MaxHeight: 432,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-200040
Name: System preset: HLS 600k
Description: System preset: HLS 600k
Container: ts
Audio: {Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3.0},KeyframesMaxDist: 90,FixedGOP: true,BitRate:
  472,FrameRate: auto,MaxWidth: 480,MaxHeight: 320,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Id: 1351620000001-200050
Name: System preset: HLS 400k
Description: System preset: HLS 400k
Container: ts
Audio: {Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}
Video:

{Codec: H.264,CodecOptions: {MaxReferenceFrames=1, Profile=baseline, Level=3.0},KeyframesMaxDist: 90,FixedGOP: true,BitRate:
  272,FrameRate: auto,MaxWidth: 400,MaxHeight: 288,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}

Bash shell on a Heroku Dyno

2013-02-27T00:00:00-05:00

Up to now most of my work with Heroku has been via the heroku command line client, and pushing builds to app instances to have them compiled and run. Recently, I've been messing around with the remote bash shell of my Heroku apps. To enter a remote shell, just type the following from the root directory of a git project that declares your Heroku app as a remote repository. This command uses the heroku client provided by the Heroku toolbelt:

Shell

$ heroku run bash

Upon login, you are placed into the root directory of your deployed app. Unfortunately, you do not have access to the table of mounted file systems and you cannot run sudo.

Shell

~ $ pwd
/app 

~ $ ls -alF
total 64
drwx------ 10 u37570 37570 4096 Dec  1 01:07 ./
drwxr-xr-x 15 root   root  4096 Oct 31  2011 ../
-rwx------  1 u37570 37570  183 Dec  1 01:05 .gitignore*
drwx------  3 u37570 37570 4096 Dec  1 01:05 .ivy2/
drwxrwxr-x  6 u37570 37570 4096 Dec  1 01:05 .jdk/
drwx------  4 u37570 37570 4096 Dec  1 01:08 .sbt_home/
-rw-------  1 u37570 37570   98 Dec  1 01:05 Procfile
-rw-------  1 u37570 37570 3833 Dec  1 01:05 README.md
drwx------  5 u37570 37570 4096 Dec  1 01:05 app/
drwx------  2 u37570 37570 4096 Dec  1 01:05 conf/
drwx------  5 u37570 37570 4096 Dec  1 01:06 project/
drwx------  5 u37570 37570 4096 Dec  1 01:05 public/
-rwx------  1 u37570 37570   27 Dec  1 01:05 system.properties*
drwx------  6 u37570 37570 4096 Dec  1 01:08 target/

Discover total used file space (my slug uses 1.7 GB):

Shell

~ $ du -sh --exclude='/proc/*' /
du: cannot read directory `/lost+found': Permission denied
du: cannot read directory `/etc/ssl/private': Permission denied
1.7G /

All your environment variables are available:

Shell

~ $ set
... miles of output ...

ifconfig is not available, but the IP address of your dyno can be discovered this way:

Shell

~ $ tail -n 1 /etc/hosts
10.92.81.76 e3d315e6-4392-4f3a-b7ae-75438c469697

You can run sbt, but this is a bad idea because the Ivy cache is held outside the dyno, and there is no way for you to clean it without using some voodoo.

Shell

~ $ sbt update
[info] Loading global plugins from /app/.sbt_home/.sbt/plugins
[info] Loading project definition from /app/project
[info] Set current project to blahblah (in build file:/app/)
[info] Updating {file:/app/}blahblah...
... miles of output ...
[info] Done updating.
[success] Total time: 6 s, completed Feb 28, 2013 4:40:21 AM

AWS S3 websites and Naked HTTP Redirects

2012-11-14T00:00:00-05:00

Sites hosted directly off AWS S3 only respond to the www subdomain. In other words, if you tried to navigate to https://mysite.com for a site that was hosted on AWS S3, a 404 status would result, however https://www.mysite.com would work. Some registrars, like Namecheap and GoDaddy have URL Redirect, while others, like GKG.net, do not.

Here is how to set up Namecheap to redirect requests like https://mysite.com/blah to https://www.mysite.com/blah so AWS S3 will respond. I also show how to set up the DNS for email with Rackspace.

Originally AWS buckets could only be used for serving websites if they started with www, for example: www.artforhealingenvironments. This restriction no longer exists.
Go to the Namecheap FreeDNS service. This will set up your domain for a smooth transfer to Namecheap, for uninterrupted web presence during and after the transfer.
Enter your domain name, for example artforhealingenvironments.com, and click on Get DNS
On the next page, click on Add DNS Service For the Selected Domains
Fill in the Namecheap Hosted Domains page. It will look something like this when you are done, for the artforhealingenvironments.com domain.

Namecheap Hosted Domains page

In the first IP ADDRESS/URL, put the fully resolved HTTP URL, with a trailing slash; set the record type to URL Redirect: https://www.artforhealingenvironments.com/
In the second IP ADDRESS/URL, put the AWS bucket name, followed by the S3 site's domain; Namecheap will automatically add a period after, so you do not have to. Set the record type to CNAME:www.artforhealingenvironments.com.s3-website-us-east-1.amazonaws.com
Namecheap's minimum TTL is 60 minutes.
Namecheap's website automatically adds a period after each domain name.

The MX records for email are shown at the bottom; they won't appear until you press the Save Changes button once (not shown). The MAILSERVER HOST NAME values are mx1.emailsrvr.com and mx2.emailsrvr.com.
After the transfer completes, the FreeDNS entry will automatically be removed, and you will manage the domain on the Namecheap Manage Domains page.

Debugging JVM Programs on Heroku

2012-09-28T00:00:00-04:00

Debugging Java Applications

Sometimes there is no substitute for debugging a remote application. The Java virtual machine provides the JPDA facility for this. JPDA is flexible, and can be configured in a variety of ways. Two attachment mechanisms are supported for debugging remote applications: inbound connections, whereby a debugging process on your machine attaches to a remote process via designated port at a specific IP address, and outbound connections, whereby a debugging process on your local machine listens to a designated port for a remote process to attach to it. JPDA can be used by all JVM-based languages, such as Java, Scala, Groovy and Clojure.

Heroku only allows one incoming port, so because an incoming port is used by Heroku to connect to the hosted app, debug connections originating from your IDE will not succeed. Instead, you must set up your Heroku application to initiate an outbound connection for debugging. This cannot be done if your app uses more than one dyno, so you must scale your Heroku back to one dyno before you can remotely debug it, like this:

Shell

$ heroku scale web=1

A Heroku app can initiate an outbound connection for debugging with or without a proxy server. The examples below use the Java debugger (jdb) and IntelliJ IDEA, but you could equally well set up a debug configuration for Eclipse in a similar manner. The settings below were used with a Play 2 application with Java and Scala controller classes.

Tip: heroku restart will restart your Heroku app using your existing slug. This accomplishes the same thing as:

Shell

$ heroku scale web=0

$ heroku scale web=1

The other way you can restart your app is by building a new slug, which then automatically starts. You can do this by checking in a bogus file:

Shell

$ date > ignoreme.txt; git add ignoreme.txt; git push heroku

Warning: Your Heroku app will crash if there is no listening process. Use the heroku logs command to check for a crash.

Shell

$ heroku logs
2012-09-29T18:20:53+00:00 heroku[web.1]: Starting process with command `target/start -Dhttp.port=${PORT} ${JAVA_OPTS}`
2012-09-29T18:20:55+00:00 app[web.1]: ERROR: transport error 202: connect failed: Connection refused
2012-09-29T18:20:55+00:00 app[web.1]: ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)
2012-09-29T18:20:55+00:00 app[web.1]: JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized [../../../src/share/back/debugInit.c:741]
2012-09-29T18:20:55+00:00 app[web.1]: FATAL ERROR in native method: JDWP No transports initialized, jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)
2012-09-29T18:20:56+00:00 heroku[web.1]: Process exited with status 134
2012-09-29T18:20:56+00:00 heroku[web.1]: State changed from starting to crashed

Remote Debugging Without a Proxy Server

If your local machine is accessible from the Internet at an IP address (a domain such as blah.no-ip.info would work equally well):

Shell

$ jdb -listen 9999& # Do not type this line if you are using an IDE
# If using an IDE, start debugging now

$ # Assumes that eth0 accesses the Internet; only works if you are not behind NAT
$ IPADDR=`ifconfig eth0|grep "inet addr"|awk -F: '{print $2}'|awk '{print $1}'`

$ # If you are behind NAT, you will need to use a dynamic DNS and set IPADDR to the machine name instead:
$ IPADDR=mycomputer.no-ip.info # modify to suit

$ # This is a really long line. I wrapped it, but you should not:
$ heroku config:add JAVA_OPTS='-Xdebug
  -Xrunjdwp:transport=dt_socket,address=$IPADDR:9999
  -Xms512M -Xmx1024M -Xss1M -XX:+CMSClassUnloadingEnabled
  -XX:MaxPermSize=256M -XX:+UseCompressedOops'

$ heroku restart

Here is what the run configuration for IntelliJ IDEA looks like. Note that debugger mode is set to listen, which implies server="n". You need to launch this run configuration before restarting your Heroku app.

IntelliJ IDEA listening as a client

Bash Script Implementation

I put the bash scripts in a gist:

#!/bin/bash set -e export JAVA_OPTS="" export JAVA_OPTS="$JAVA_OPTS -Xmx2048m -Xss512m -Xverify:none" # Java 8 no longer supports MaxPermSize: # export JAVA_OPTS="$JAVA_OPTS -XX:MaxPermSize=384m"

Remote Debugging With a Proxy Server

If your local machine is hidden from the Internet by a proxy server at blah.domain.com, open a tunnel to it from your local machine, and listen to it:

Shell

$ # Assumes that eth0 accesses the Internet; only works if you are not behind NAT
$ IPADDR=`ifconfig eth0|grep "inet addr"|awk -F: '{print $2}'|awk '{print $1}'`

$ # If you are behind NAT, you will need to use a dynamic DNS and set IPADDR to the machine name instead:
$ IPADDR=mycomputer.no-ip.info # modify to suit

$ ssh -NR *:9999:localhost:9999 $IPADDR&

$ jdb -listen 9999& # Do not type this line if you are using an IDE

If using an IDE, start debugging now.

Shell

$ # This is a really long line. I wrapped it, but you should not:

$ heroku config:add JAVA_OPTS='-Xdebug
  -Xrunjdwp:transport=dt_socket,address=$IPADDR:9999
  -Xms512M -Xmx1024M -Xss1M -XX:+CMSClassUnloadingEnabled
  -XX:MaxPermSize=256M -XX:+UseCompressedOops'

$ heroku restart

Disabling Remote Debugging

The JAVA_OPTS value set earlier will remain in effect across multiple restarts of your Heroku app until you change it. To disable remote debugging, redefine the environment variable without the -Xdebug and -Xrunjdwp options:

Shell

$ # This is a really long line. I wrapped it, but you should not:

$ heroku config:add JAVA_OPTS='-Xms512M -Xmx1024M -Xss1M
  -XX:+CMSClassUnloadingEnabled
  -XX:MaxPermSize=256M -XX:+UseCompressedOops'

$ heroku restart

Saving the Run Configuration

If you enable the Share checkbox in the IntelliJ IDEA run configuration dialog box, the definition will be written to .idea/runConfiguration/. Normally you would not check in the contents of .idea/ to your source code repository, but this subdirectory is an exception, and because this run configuration has no local dependencies it can be shared without modification amongst all of your developer team.

Shell

$ cat .idea/runConfigurations/Heroku_remote.xml
<component name="ProjectRunConfigurationManager">
  <configuration default="false" name="Heroku remote" type="Remote" factoryName="Remote">
    <option name="USE_SOCKET_TRANSPORT" value="true" />
    <option name="SERVER_MODE" value="true" />
    <option name="SHMEM_ADDRESS" value="javadebug" />
    <option name="HOST" value="localhost" />
    <option name="PORT" value="9999" />
    <method />
  </configuration>
</component>

Composable Futures with Akka 2.0

2012-08-09T00:00:00-04:00

My latest book is finally complete! Composable Futures with Akka 2.0 features Java and Scala code examples. The book is available in PDF format.

Akka is hot, and Akka futures are important for creating responsive applications that can scale. This book is intended for ‘the rest of us’ – Java and Scala programmers who would like to quickly learn how design and integrate applications using composable futures.

I presented a preview of the book at Googleplex April 18, 2012 to the Silicon Valley Java User Group.

Proposal – Mandatory Countervailing Tip

2012-08-07T00:00:00-04:00

Inequity between countries and ethnicities exploits individuals who are the primary producers and harms the individuals and societies that contain the primary consumers; it only benefits international traders. Over 99% of the world is exploited to serve less than the top 1%. This imbalance threatens world stability, as well as national, regional and local stability.

This proposal is designed to create jobs in every country where the trade balance is negative, and is intended to reverse the exploitation of individuals in producing countries. I propose a Mandatory Countervailing Tip (MCT). It’s like a tip that you might offer someone who provides service, and whose wages are not sufficient to justify their employment. Waiters, waitresses, bellhops and musicians are paid tips for this reason. This proposal is unlike a countervailing tax because the consuming country does not keep the surcharge; instead, it would provide money directly to individuals in producing regions. The exact mechanism(s) for disbursing funds locally is not addressed in this proposal.

The MCT would be levied at the point of purchase, like a sales tax, and would be disbursed directly to residents at the point(s) of origin of the goods who created the product. This would serve to improve the lives of all parties involved in the transaction, except the international trader.

The MCT would be calculated such that exploited individuals themselves in the producing nation would directly receive enough money to raise their standard of living to within 25% of the standard of living for the middle 80% of the consuming region in the producing country. The money would be collected by the same government departments that collect sales tax and would be remitted every 30 days to a new agency in the UN. The entire staff of the UN agency would be required to be replaced every 2 years, and working committees and task forces in the agency would be comprised of residents of primary producers and consumers, as well as enough individuals from neutral countries to ensure impartiality and honesty. The states collecting the sales tax would be entitled to 1% of the funds collected for administrative overhead. The UN agency would be entitled to an additional 2%. The UN would be responsible for computing the MCT for every category of product sold, for every producing / consuming country.

The sums involved would be enormous, so the potential for fraud is also enormous. To counteract this, unprecedented transparency must be implemented; this is possible because of modern computer and communications technology. Access to the data should be made public to the world at large. No sign-in should be required to download data concerning anonymized transactions and aggregated statistics. No data about individuals in producers or consumers must be available without security clearance.

Transactions and queries requiring authorization must be audited, and auditors must be selected from presumably hostile countries and regions. For example, given that India and China are fierce competitors, they should provide the personnel to audit each other’s transactions. Those auditors would be paid from 0.002% of the total receipts. Auditors must also be reassigned every year, and may not ever be reassigned to the same region.

Implementation need not require world-wide agreement; only one country need to decide that it wants to go ahead, and rough estimates of economic disparity would suffice to start. The first country to implement will be the first country to enjoy the benefits of a local economic resurgence. This can be done before the UN is ready, by setting up a temporary agency.

This proposal is intended to greatly reduce the disparity between rich and poor throughout the world. It is fair because it is based on consumption, and the degree of equalization applied is proportional to the disparity between the standards of living between producer and consumer. This should greatly decrease the suicide rate of the indentured slaves who build iPads in China, while providing an opportunity for manufacturing to resurge in the US. Think of it as a regulated fair trade for world stability, prosperity and peace.

Colonialism is based on exploitation, but capitalism does not require it. One might argue that a certain amount of exploitation is healthy because without some measure of economic disparity, undeveloped regions would not have a cost advantage and therefore never develop. This is probably true, so the question then becomes: how much economic disparity is healthy? I don’t know the answer, but a legislated policy on whatever that figure needs to be is preferable to uncontrolled exploitation. I doubt that the figure would need to be exceed 50%, and probably should be less. Current figures are in the range of 1% to 5%.

The MCT is an example of how local, regional, national and world government can provide for the greater good. The only parties who have reason to oppose this type of proposal are those who benefit in some way from exploitation. Think of the goodwill that the recipients will feel towards their end users, and the countries that they live in.

Scala Existential Types and Salat

2012-08-06T00:00:00-04:00

For Scala programmers, the term 'existential types' does not refer to philosophers, or even authentic people. Ironically, most of the discussions of Scala's existential types that I found are too abstract to be useful to me. I learn by doing; rarely do I learn from reading an abstract treatise. I guess this means that I could be labelled an existentialist.

Section 31.3 of Programming in Scala has some good information on Scala's existential types. Existential types is an abstraction of Java types, and abstraction is the antithesis of existentialism... aaaanyway, here is a sentence I stole from the book:

Iterator[_] means the same thing as Iterator[T] forSome { type T }

forSome is the keyword which tells the compiler that an existential type is being defined. This becomes useful if upper and/or lower bounds are used to define the type. For example, you can use a lower bound to specify that the type must be a subclass of PubSubAction with the following existential type:

Shell

T forSome { type T <: PubSubAction }

That is a lot of characters, so let's give this type a name:

Shell

type PubSubActionSubclass = T forSome { type T <: PubSubAction }

We can also define a type to help us with SalatDAO:

Shell

type SalatObject = T forSome { type T <: AnyRef }

Now lets use SalatObject as the parametric type for an invocation of Manifest.classType():

Shell

val mot = Manifest.classType[SalatObject](msg.getClass)

This is useful because the manifest can be passed to a SalatDAO constructor, which will create a DAO object for whatever type is supplied. In the following code, PubSubActionClass is just used to guarantee that the msg parameter is of the correct type; SalatDAO's constructor is defined to accept all subclasses of AnyRef.

Shell

import com.novus.salat.global.ctx

abstract class PubSubAction

object PubSubAction {
  def makeDAO[PubSubActionSubclass](msg: PubSubActionSubclass)
                              (implicit coll: MongoCollection) = {
    val mot = Manifest.classType[SalatObject](msg.getClass)
    val mid = Manifest.classType[Int](classOf[Int])
    new SalatDAO(coll)(mot, mid, ctx){}
  }
}

Now you know that you do not have to hard-code the creation of a DAO for every PubSubAction subclass. You can examine the source code for SalatDAO if you would like to learn more.

Assuming that psaMsg is an instance of a PubSubActionSubclass, you could call insert() as follows to do a generic insert. Because Salat has multiple methods called insert(), there is not enough type information for Scala to disambiguate the method reference unless the returned value from the single insert is stored in a variable, and that variable's type is provided:

Shell

val ignoredIndex: Option[Int] = makeDAO.insert(psaMsg)

There is only one variant of insert() which accepts a collection, so the reference is not ambiguous and the return value can be ignored:

Shell

dao.insert(Seq(psaMsg, psaMsg2, psaMsg3), WriteConcern.Normal)

Pushing Notifications to Nagios from Java and Scala

2012-08-04T00:00:00-04:00

I was investigating how to push notifications from JVM-based programs when I found the NagiosAppender project. Push notifications are termed ‘passive checks’ because Nagios does not poll for results. For the curious, see the Nagios Plugins – Passive Service Check section of Application Monitoring Made Easy for Java Applications Using Nagios.

NagiosAppender integrates Log4j or Logback with Nagios’ optional NSCA server. The only ‘programming’ required is setting up configuration files for Nagios client and Nagios server, adding a new dependency, and writing appropriate log messages for forwarding to Nagios by the plugin. Unfortunately, NagiosAppender is not compatible with Akka because it uses MDC, which uses ThreadLocal variables, which should not be used with Akka. I took the NagiosAppender project, slimmed it down, removed the Log4j interface and the MDC code, and created the PushToNagios project.

NB: The document mentions MDC without defining it. From the Apache log4j docs: “A Mapped Diagnostic Context, or MDC, is an instrument for distinguishing interleaved log output from different sources. Log output is typically interleaved when a server handles multiple clients near-simultaneously. The MDC is managed on a per-thread basis. A child thread automatically inherits a copy of the mapped diagnostic context of its parent.” The Logback documentation has a whole chapter on MDC.

NSCA

NSCA is a Nagios add-on that allows you to send passive check results from remote hosts to the Nagios daemon running on the monitoring server. This is very useful in distributed and redundant/failover monitoring setups. The NSCA addon can be found on Nagios Exchange. For more information, see Addon – Nagios Passive Checks with NSCA.

Installation

For Ubuntu:

Shell

$ sudo apt-get install nagios3 nsca

Installs Nagios Core 3.2.3, which is outdated but compatible, and nsca 2.7.2+nmu2, which is current. The current version of Nagios Core is 3.4.1, released on 2012-05-14. Nagios starts automatically after installation, but NSCA needs to be started manually (don't do that yet, keep reading). Navigate your web browser to http://localhost/nagios3 and specify userid nagiosadmin.

FYI, /etc/init.d/nagios3 contains:

/etc/init.d/nagios3

DAEMON=/usr/sbin/nagios3
NAGIOSCFG="/etc/nagios3/nagios.cfg"
CGICFG="/etc/nagios3/cgi.cfg"

/etc/nagios3/nagios.cfg contains:

/etc/nagios3/nagios.cfg

log_file=/var/log/nagios3/nagios.log
cfg_file=/etc/nagios3/commands.cfg
cfg_dir=/etc/nagios-plugins/config

/etc/nagios3/resource.cfg contains:

/etc/nagios3/resource.cfg

# Sets $USER1$ to be the path to the plugins
$USER1$=/usr/lib/nagios/plugins

/etc/init.d/nsca contains:

/etc/init.d/nsca

DAEMON=/usr/sbin/nsca
CONF=/etc/nsca.cfg
OPTS="--daemon -c $CONF"
PIDFILE="/var/run/nsca.pid"

We saw above that plugins are in /usr/lib/nagios/plugins/. I added one called check_domain_bus with permissions set to 755, and owned by nagios:nagios:

check_domain_bus

#!/bin/sh
echo "All OK: $1"
exit 0
Configuration

Edit /etc/nagios3/nagios.cfg and enable external commands on line 145 so the entry looks like this:

/etc/nagios3/nagios.cfg

check_external_commands=1

Edit the last line of /etc/nsca.cfg to disable encryption:

/etc/nsca.cfg

decryption_method=0

Define a new Nagios command called check_domain_bus in /etc/nagios3/commands.cfg by adding the following anywhere in that file:

/etc/nagios3/commands.cfg

define command {
  command_name check_domain_bus
  command_line $USER1$/check_domain_bus $ARG1$
}

Define a template for passive services, and an instance of a passive service called domainBus that responds to the check_domain_bus command by adding the following to /etc/nagios3/conf.d/services_nagios2.cfg:

/etc/nagios3/conf.d/services_nagios2.cfg

define service {
       name                    passive-service
       use                     generic-service
       check_freshness         1
       passive_checks_enabled  1
       active_checks_enabled   0
       is_volatile             0
       flap_detection_enabled  0
       notification_options    w,u,c,s
       freshness_threshold     57600     ;12hr
}

define service {
    use                     passive-service 
    host_name               localhost
    service_description     domainBus
    check_command           check_domain_bus!0
}

Usage

Start NSCA server, then restart Nagios:

Shell

$ sudo service nsca start
$ sudo service nagios3 restart

The custom service, called domainBus, should be viewable as a Nagios service, shown in the red rectangle below:

Nagios domainBus service

Nagios will need to be restarted each time a service definition is modified. New services are shown as PENDING until they receive their first result. Passive services have no scheduled updates.

Testing with the PushToNagios Java Client

See the PushToNagios documentation.

Testing With the Compiled C NSCA Client

Let’s send a message and have the result displayed on the web interface. send_nsca is a compiled C nsca client that can be used to send a test message. Unpack nsca-2.7.2.tar.gz into a directory, and compile it:

Shell

$ ./configure

$ make install

Again, edit the last line of sample-config/send_nsca.cfg and change it to read:

sample-config/send_nsca.cfg

decryption_method=0

Create a test message in the root of the unpacked NSCA project. The format for a service check packet using NSCA contains tab characters and ends in a newline, like this:

Shell

<hostname>[tab]<svc_description>[tab]<return_code>[tab]<plugin_output>

I am unsure if <hostname> refers to the Nagios host or the sending host. The allowable values for <return_code> are:

0 - OK state

1 - Warning state

2 - Error state

3 - Unknown state

<plugin_output> can be up to 512 bytes long.

Create a text message called testCritical, with embedded tabs, that NSCA uses as a field delimiter.

Shell

localhost   domainBus    2   This is a Test Error

Watch the Nagios log and syslog in one console:

Shell

$ tail -f /var/log/nagios3/nagios.log /var/log/syslog

Send the test message like this in another console; let's call this the command console:

Shell

$ src/send_nsca localhost -c sample-config/send_nsca.cfg < testCritical

Notice the log output in the console with the log output:

Shell

==> /var/log/nagios3/nagios.log <==
[1343251589] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;localhost;domainBus;2;This is a Test Error

==> /var/log/syslog <==
Jul 25 14:26:29 natty nagios3: EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;localhost;domainBus;2;This is a Test Error

==> /var/log/nagios3/nagios.log <==
[1343251590] PASSIVE SERVICE CHECK: localhost;domainBus;2;This is a Test Error

==> /var/log/syslog <==
Jul 25 14:26:30 natty nagios3: PASSIVE SERVICE CHECK: localhost;domainBus;2;This is a Test Error

==> /var/log/nagios3/nagios.log <==
[1343251590] SERVICE ALERT: localhost;domainBus;CRITICAL;SOFT;1;This is a Test Error

==> /var/log/syslog <==
Jul 25 14:26:30 natty nagios3: SERVICE ALERT: localhost;domainBus;CRITICAL;SOFT;1;This is a Test Error

In the web browser, click on Services again and notice that the status of the domainBus service is now CRITICAL, and Status Information now reads:

Shell

This is a Test Error

Status of the domainBus service is now 'CRITICAL'

Create a text message called testClear, and do not forget the embedded tabs:

testClear

localhost   domainBus    0   Mischief Managed

Send this new test message in the command console:

Shell

$ src/send_nsca localhost -c sample-config/send_nsca.cfg < testClear

The log output console should show something like this:

Shell

==> /var/log/nagios3/nagios.log <==
[1343252049] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;localhost;domainBus;0;Mischief Managed

==> /var/log/syslog <==
Jul 25 14:34:09 natty nagios3: EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;localhost;domainBus;0;Mischief Managed

==> /var/log/nagios3/nagios.log <==
[1343252050] PASSIVE SERVICE CHECK: localhost;domainBus;0;Mischief Managed
[1343252050] SERVICE ALERT: localhost;domainBus;OK;SOFT;2;Mischief Managed

==> /var/log/syslog <==
Jul 25 14:34:10 natty nagios3: PASSIVE SERVICE CHECK: localhost;domainBus;0;Mischief Managed

In the web browser, the Services information should automatically update after a pause of up to 90 seconds (by default), or you can click on Services to immediately see the new status. Notice that the status of the domainBus service is now OK, and Status Information now reads Mischief Managed.

Status Information now reads 'Mischief Managed'

The third possible message status is warning. Create a text message called testWarning and do not forget the embedded tabs:

testWarning

localhost   domainBus    1   Do you know where your chocolate is?

Send the test message like this in the command console:

Shell

$ src/send_nsca localhost -c sample-config/send_nsca.cfg < testWarning

In the web browser, click on Services to immediately see the new status. Notice that the status of the domainBus service is now OK, and Status Information now reads Do you know where your chocolate is?

Status Information now reads ‘Do you know where your chocolate is?’

Scala Type Parameters, Implicit Manifests and Salat

2012-08-02T00:00:00-04:00

Type parameters provide types for constructing any arguments that are implicit manifests.

This helps when working with Salat. In the following code, dbo is a com.mongodb.DBObject, perhaps retrieved as the result of a find(). Salat uses _typeHint to store the fully qualified name of the persisted class; this property is returned as one of the properties of dbo. The call to Salat's grater() method converts dbo into the desired type. The type is constrained using a lower bound to be a subclass of PubSubAction.

Shell

def findWithSeqNum[T <: PubSubAction](seqNum: Long)
    (implicit manifest: Manifest[T]): Option[T] = {
  val klass = manifest.erasure.asInstanceOf[Class[T]]
  val item = myCollection.findOne(
    MongoDBObject("seqNum" -> seqNum, "_typeHint" -> klass.getName))
  item match {
    case Some(dbo: DBObject) =>
      Some(grater(ctx, Manifest.classType(klass)).asObject(dbo))

    case None =>
      None
  }
}

When called as follows, the type parameter need not be provided because there is an explicit manifest argument:

Shell

findWithSeqNum(11)(Manifest.classType(classOf("com.micronautics.Blah"))

When called as follows, the type parameter provides the value for constructing the implicit manifest argument:

Shell

findWithSeqNum[Blah](11)

Of course, you can also explicitly define an implicit manifest at any scope that you desire:

Shell

implicit val manifest: ClassManifest[Blah]
findWithSeqNum(11)

If the type of the manifest is dynamic (only known at runtime), you can define manifest this way:

Shell

val mtype = "com.micronautics.Blah"
implicit val manifest = Manifest.classType(Class.forName(mtype))
findWithSeqNum(11)

Pigs Can Fly

2011-09-30T00:00:00-04:00

Details matter. If all you read are headlines and subtitles, then it is likely that you are horribly misinformed. Let’s imagine that the text of an article was something like this:

Pigs Can Fly

Outfitted with 3D goggles and haptic feedback mechanisms customized for a pig’s extremities, porcine subjects showed that they quickly adapted to visual stimulus that suggested to them that their movements caused them to fly. With training, pigs demonstrated that they could control the flight of simulated cargo planes, stunt triplanes, jet fighters and paper planes.

So, does the passage say that pigs can actually fly unassisted? Does it say that pigs might think they can fly? Does it discuss the possibility that pigs might be able to be trained as pilots? You would have to read the passage carefully to be able to answer those questions. If you just skim headlines, then you would have no idea what the passage was about, or even if it was just humor.

Details matter a lot when you are working with technologists. If you are easily put off by even the slightest techno-speak, then you have no business managing technologists. Specialized vocabulary summarizes complex thoughts succinctly; learn it, don’t ask for the baby-talk version.

You can’t
Summarize complex interactions
Into a few bullet points
Without getting it wrong

This is true for reading and writing. If the best that a manager can grunt via a keyboard is “you know what I mean” then it is probable that the recipient of the email has at best only a vague idea of what the lazy, inarticulate writer meant. In truth, the writer has probably not thought through their ideas properly, and they are not clear what they meant either.

Pick your information sources and sinks with discretion, and strive to communicate in complete thoughts.

Here is some related science.

AMF over HTTP in another Multiverse

2011-09-21T00:00:00-04:00

This silly story is taken from my book Flex Data Services, Hibernate and Eclipse. No other part of that book is deliberately silly. I thought it would be fun to post this here. In the story, produce and food are allegories for data and a cellar is an allegory for a database. As for Klay and Adobe, well, you get the idea.

Once upon a time, in an alternate multiverse, there might or might not have been a world called Klay. Klay was a world very like Earth, except that the peaceful inhabitants were obsessed with food and food technology. This world had perfected the art and science of growing and cooking food. Each region had its specialty, however because transportation was not mechanized. Only seasoned travelers had ever experienced the wondrous tastes and textures of the exquisite culinary creations in far-off lands. Because growing conditions varied widely, food could only be made from local produce, and thus could not be replicated outside where the food was grown. Instead, the pudgy townsfolk would gather around wandering minstrels as they sang of the nutritional marvels that they had experienced in far-off lands.

Food was grown, prepared, cooked and eaten nearby the same fields that it was grown. The Klaylings valued freshness and texture very highly, but they found that some produce actually improved when it was aged. This produce was stored in cellars next to the fields where the produce was grown. When a chef prepared a meal, he/she would gather the ingredients and cook it to order.

A group of rich merchants in a prosperous region realized that there was profit to be made by selling prepared food to neighboring regions. Their first attempt was to have a chef prepare a dish that resembled a soufflé, and have it delivered by horseback. The rider, holding the dish in front of him, ended up with egg all over himself. Clearly, a means of packaging the food was required.

The merchants also had another problem. Each country had different customs regarding the shape of the containers from which they ate. It was inconceivable that a person accustomed to eating from a square bowl might eat instead from a round or oval bowl. This complication was considered serious because as any chef will tell you, presentation is very important. The food had to be prepared for the specific containers that it was served in, and once prepared could not be transferred to another container without ruining the presentation, thereby severely reducing the selling price.

The merchants called upon the local wise woman, who meditated and went into a trance. An acolyte wrote down the muttered sentences that she uttered while communing with the Fat God. Without knowing what the words meant, she wrote down the words and presented them to the merchants after the wise woman fell asleep. The words were:

Awesome Mouthwatering Food over Highspeed Trusty Transport Provider.

Another acolyte, skilled in numerology, tried turning the words into sacred acronyms: AMF over HTTP. Try as he might, he could not understand any significance in those words.

When the wise woman awoke, she drew this diagram:

The merchants gathered around the wise woman as she explained the diagram. “The food from the fields and the cellars are brought into the kitchen as required. The meal is cooked to order, then deconstituted and packaged. A magic carpet uses AMF over HTTP to deliver the deconstituted food to the client’s kitchen, where it is reconstituted by one of the local chefs. The food is served into the appropriate containers for the patrons, and the carpet returns with the chef and payment.

“The key is AMF over HTTP. This magical process, yet to be invented, would consist of a tesseract, larger on the inside than it is on the outside. Within the tesseract, AMF would absorb the raw essence of the prepared food. The magic carpet would transport the tesseract almost instantaneously to the far-off land where the client lives. The magical AMF process would then reconstitute the food into the client’s desired containers.”

The merchants were incredulous. “But we have never heard of AMF over HTTP. Where can we find this magical process, and the magic carpet?”

The old woman replied that she did not know. “It was only a dream,” she replied despondently.

Fortunately, in this multiverse, AMF over HTTP does exist, and it transports data between client and servers just as the wise old woman described – and a lot more. Under the covers, AMF is used internally by Flex for many purposes. The next few chapters discuss how that can be done, and more. The Internet is our magic carpet.

Tracking Down a Mismatched JAR

2011-08-04T00:00:00-04:00

For debugging mismatched jars, first build a list of the jars in question, then run jwhich to discover the jar that provides a specific Java class. Put the jar and the script provided in the download in a directory on your PATH such as /usr/local/bin.

For *nix and Mac, build the list of jars under the current directory like this:

Shell

export CLASSPATH=$(find . -name \*.jar -printf %h/%p:)

For Cygwin, build the list of jars under the current directory like this:

Shell

export CLASSPATH=$(find . -name \*.jar -printf '%h/%p;')

Run the script like this:

Shell

$ jwhich classNameToFindHere

Debugging Spring's SEVERE Error ListenerStart

2011-08-04T00:00:00-04:00

This Spring startup error can be mystifying if you don”t know how to tackle it. listenerStart() configures and invokes application event listeners for a Context. Most Spring applications have several listeners, and they should be invoked in the following order. Each of them causes debug output, so if you do not see any, the first listener died:

org.springframework.web.context.ContextLoaderListener
org.springframework.web.context.request.RequestContextListener
flex.messaging.HttpFlexSession (for Spring/Flex integration)

If the error message appeared, one of the above did not initialize correctly. Put breakpoints in each of the listener classes’ contextInitialized() or requestInitialized() methods.

To view debug log messages for StandardContext, add:

Shell

catalina.org.apache.juli.FileHandler.bufferSize = -1

SEVERE: Error listenerStart messages can be debugged by setting a breakpoint at org.springframework.web.context.ContextLoaderListener, line 47.

Shell

contextLoader.initWebApplicationContext(event.getServletContext());

Step return once and wait for the container to load everything. If there is an error you will now be at standardcontext.listenerstart and you will see the error in the variable window under the t variable.

For debug output, add this to web.xml:

Shell

<listener>
  <listener-class>
    org.springframework.web.util.Log4jConfigListener
  </listener-class>
</listener>

Assessing Sample Code from Job Applicants

2011-05-24T00:00:00-04:00

Once again I am reviewing programmer resumes for a client. This client needs to hire an entire team, including Flex and Java programmers. Because programmers write software, I always request sample code from candidates. I do not ask stupid hypothetical questions or play mind games – I just want to assess the quality of their work.

I request the code along with a resume. Code is truth. I do not interview candidates who do not provide sample code that passes inspection. The requirement for sample code is published in the job description. Surprisingly, most applicants do not provide sample code. I do not respond to those applications because clearly those people do not follow simple instructions.

This is worth repeating: If a job applicant wants to be considered as a candidate, they need to provide sample code along with their resume.

I assume that the programs that candidates show me are examples of their best work, or at least work that they are proud of. Most of my clients need programmers to develop software products or in-house applications, so efficiency and maintainability are important.

I ask for sample code in at least two languages. It is up to the candidate to decide how much to show me, and what they show me. The more sample code they show me, the more sure I can be about my assessment.

Here are some issues that I look for in submitted code:

Is it difficult or expensive to maintain?
Was the code written with the next programmer on this project in mind?
Is logic used instead of data structures?
Is the code neat, and does it follow a consistent pattern? I don’t care if particular formatting conventions are followed, I am looking for an ordered mind and a conscientious worker.
Is the code testable? Were unit tests and/or integration tests provided?
Does the code contain magic values instead of declared constants?
Is the code inefficient (are there lots of conversions between string and int, for example)?
Does the programmer use idioms specific to the language that it is written in?
Are there appropriate comments, especially Javadoc / asdoc style comments?
Are default actions or values present in the code? Defaults are tacitly understood by competent programmers, and providing them just adds noise. Code and documentation should be written with the assumption that readers will be technically competent.
Is encapsulation violated because most methods are public?
Are there syntax errors?

For Adobe Flex programmers I also look for:

Is there proper usage of the Flex component life cycle?
Is there a heavy dependence on binding?
Are events used properly?
Are heavyweight Halo containers used, when Spark Groups would suffice?
Is all logic in a controller, or is it shoved into views (or worse, item renderers)?
Are style sheets used, or is style information explicitly coded?

I also provide a reading and comprehension test, and ask them to critique the code I provide. Some code I provide is good, some bad.

Many enticing resumes are simply not supported by good sample code. Would you hire a chef without tasting a sample of their food first?

Mobile and Desktop Technology Trends and Issues

2011-05-13T00:00:00-04:00

As principal of Micronautics Research I develop or oversee development of desktop and mobile client/server applications. We use whatever technology is appropriate for a client project.

I make a distinction between applications (apps) and web pages (HTML and HTML5). Over time this distinction will blur. This article deals with current reality for apps that need to be developed and deployed using today’s technology.

Mobile and desktop apps can be described in terms of usability, performance, development cost, and maintainability; the latter is an indicator of the time, cost and risk of modifying an existing application. Whereas desktop environments are quite stable, new mobile devices and operating system versions now appear daily. Mobile apps are therefore have a much more difficult time staying relevant than do desktop apps. Mobile app environments also vary a lot more than desktop app environments.

There is no perfect one-size-fits-all technology for every app, and this is especially true for mobile apps. Some mobile apps must integrate tightly with the operating system; for mobile apps this currently means writing in Java for Android or Objective C for Apple iOS. The majority of mobile apps, however, must be deployed to any device that the user might happen to own; the time and cost required to target every possible device explodes unless a cross-platform technology is used.

I’d like to share a lesson learned from migrating several desktop apps to mobile apps. To reuse the original logic and data structures from the desktop app in various mobile apps, the logic and data structures must be cleanly separated from presentation to the greatest extent possible. We have yet to see a project where rework was not required in this regard before we could begin the task of making mobile versions.

Presentation logic for adaptable views is not much of an issue for desktop apps, but it is important when targeting devices whose resolutions and pixel densities vary greatly. Implementing the logic for adaptable views is not difficult, but designers today are generally unfamiliar with the issues of how to create adaptable designs for such a wide variety of platforms. We find that our interaction with designers has deepened when working on mobile projects.

Processing power varies greatly between mobile devices. Tablets have quite a bit more power than phones, and desktops have yet more power. However, mobile devices have specialized hardware not present in most desktops, such as GPS and various forms of telephony communications; many Android devices also have a built-in bar code scanner. When adapting a desktop app to run on mobile apps, we strongly suggest using built-in hardware to the greatest extent possible. Mobile apps should obtain data from hardware or remote services instead of asking the user to type, click or swipe.

We also enhance data structures as much as possible so less processing is required; although this is generally a good practice, today’s desktops are so powerful that careless programming often goes unnoticed. When an app is ported from a desktop environment to a mobile device, the reduced CPU and memory resources cause the app to run unacceptably slowly.

For applications that must continue to work on desktops after they are ported to mobile devices, we recommend identifying and optimizing the application core such that it is present without changes for all environments. Sometimes clients are surprised by the results of this partitioning process because their assumptions of what is core do not align with actual commonalities between target environments.

Mounting compressed folders / Looping back zip files

2010-02-19T00:00:00-05:00

Big file systems make computers run more slowly. This problem is most noticeable for laptops. On a project that I am currently working on, I am using multiple versions of three frameworks: The Flex SDK (source and ASDoc), the MicroStrategy Flex SDK and the Blaze DS source tree. The result is thousands of files that takes hours to copy and slows down disk access even when I’m not programming.

Read-only Files

Because these files are only read and never written, the solution is to not unzip them. Instead, the files should only be accessed from within the zip files that contains them. This capability is called a “local loopback filesystem” on Linux, and “accessing compressed folders” under Windows.

Results are dramatic. Usage is simple. What’s not to love?

Windows

On Windows, I use the free Pismo File Mount Audit Package. It provides the ability for zip files to be mounted as virtual drives.

Linux

On Linux, use fuse-zip. It provides the ability for zip files to be mounted on any directory mount point.

Shutting down Zamples.com

2009-11-23T00:00:00-05:00

I’ve decided that it is long past the time that zamples.com should be shut off. The site got steady traffic, but it generated almost no revenue and demanded a lot of my time. Now that the server needs to be replaced, it is time to say goodbye.

I first created the predecessor to Zamples in 1994 for The Internet Factory, manufacturers of the first programmable web server. Eleven years later, it is abundantly clear that no-one really cares about live online code examples, at least not enough to spend money for the privilege of using the facility.

w3schools built their own client-side only version recently.

It’s rather sad to turn the switch off, but I won’t miss spending any more time and money on Zamples.

2009 Open Source Community Leadership Summit

2009-07-20T00:00:00-04:00

The first Open Source Community Leadership Summit happened this weekend, and I was lucky enough to be able to attend. Jono Bacon, Ubuntu Community Manager, and the other volunteers did a terrific job of putting this together.

I had a chance to get to know and renew acquaintances with some fascinating people from Canonical, Cisco, Collab.net, the Linux Foundation, the Linux Fund, Novell, the Open Voting Consortium, O’Reilly Media, Red Hat, the Software Freedom Law Center, Sun and two of the the ‘old men’ of the open source world, Bruce Perens and Larry Rosen.

Larry Rosen

Bruce Perens

Commonwealth Club Performance

2009-06-27T00:00:00-04:00

Cindie Steinmetz and I performed at the San Francisco Commonwealth Club two evenings ago. We had been invited to preview a jingle I wrote entitled “I Love Chocolate” for the “Women and Chocolate” event. I played the rest of the evening, while enjoying excellent wine and chocolate.

Presenting EmpathyWorks at a Private Equity Roundtable

2009-02-22T00:00:00-05:00

The private equity round table vSIG is focused on applying virtual worlds to solve pressing business issues. The group meets once a month on Sandhill Road in Palo Alto, CA. Members include software entrepreneurs, VCs, academia and researchers.

2020 Update: the vSIG has morphed into Venture Capital Roundtable Silicon Valley.

This will be part two of two. During the last meeting I introduced the topics of modeling relationships and behavior and presented background concepts.

The main points I made were:

Relationships are far more than just directory entries, they are the foundation of expression and provide context for interpreting behavior
The ability to recognize another individual is necessary in order to form a relationship with them
Personality is perceived, not an absolute or fixed quality
Behavior is defined by responses and is contextually dependent
Today’s artificial beings suffer from Alzheimer’s
It is more useful to model an entire community than a single individual
Personas represent an individual in various contexts

I then introduced EmpathyWorks™, a generalized relationship and behavior modeler that I have been working on for a few years.

Good Code is Beautiful

2008-10-15T00:00:00-04:00

First, I’d like to point out a few practical reasons why orderliness is important. Can you spot the bug in the following Flex code? This custom component is not as wide as expected:

Shell

<mx:TextArea  width="100" height="100%"
contextMenu="{cm}" xmlns:components="components.*"
creationComplete="init()" width="100%"
xmlns:mx="https://www.adobe.com/2006/mxml">
...
</mx:TextArea>

The bug occurs because two width attributes were provided, one with the value 100 pixels and one with the value 100%. Jamming a long list of attributes into an MXML tag in no particular order is an invitation to chaos. No error message is generated if an attribute is specified twice.

Long ago I adopted the convention of ordering all attributes in alphabetical order, one per line. I would write the above like this:

Shell

<mx:TextArea
    contextMenu="{cm}"
    creationComplete="init()"
    height="100%"
    width="100%"
    xmlns:components="components.*"
    xmlns:mx="https://www.adobe.com/2006/mxml">
...
</mx:TextArea>

Other people write certain attributes first, like id, width and height. Whatever convention you adopt, stick to it.

Here are the official ActionScript coding conventions, as published by Adobe back in 2011, prior to canceling the Flex product. I take some of it the way the characters of “Pirates of the Caribbean” consider the Pirate’s Code: “Argh, it be more like a set of guidelines.”

ANSI/Unix vs. Vertically Compressed Styles

I realize that this topic might make me flame-bait, but my experience has shown that the ANSI/Unix style of vertically spacing parenthesis increases the maintenance costs of a program.

Functions/methods should fit on a single screen; so should data structures. Because programs written in bygone days had to be editable on a green screen, 80 columns was the widest that a program could be written. In the ‘bad old days’, it was better to let a program run to more lines in order to avoid folding code. Here is an example:

Shell

private function GridContextEventHandler(event:GridContextEvent):void
{
   if (event.menuItem=='Edit')
   {
      EditGridItem(event.selectedIndex);
   }
   else if (event.menuItem=='Remove')
   {
      RemoveGridItem(event.selectedIndex);
   }
}

A vertically compressed style is much more easily read. When the Python programming language was first introduced it was initially controversial in part because it used indentation to define control structure. Python programs do not need braces for scoping as a result. Why not write ActionScript in the same manner, so parenthesis are virtually redundant? The Sun/Java convention is a good example of this formatting convention. The above program fragment would be written like this using a vertically compressed style:

Shell

private function GridContextEventHandler(event:GridContextEvent):void {
   if (event.menuItem=='Edit') {
      EditGridItem(event.selectedIndex);
   } else if (event.menuItem=='Remove') {
      RemoveGridItem(event.selectedIndex);
   }
}

Enforcing Style

Coding style conventions can be automatically applied by installing the Flex Pretty Printer plugin for Eclipse / Flash Builder. If you like, you can importing my formatting rules, which enforce the vertically compressed style above.

Breathless Delirium

2008-09-24T00:00:00-04:00

In his keynote presentation to the Agile 2008 conference in Toronto entitled “The Wisdom of Experience”, Alan Cooper had a great sentence that described the ‘first to market’ goal behind some products:

There is no large group of people out there waiting in a breathless delirium to purchase your lousy product sooner rather than later.

Sure, it is terrific to be first. But the product still has to be good.

Alan also makes some good points about design, engineering and construction. He took the long way around, because he doesn’t get into his main topic, interactive design, until slide 75. Requirements are discussed starting at slide 87. In the slide entitled “Requirements are not design”, Alan says:

Giving people what they say they desire does not result in success.
Your customers are not the same as your users.
Neither your customers nor your users know what they want or even what they do.
What people tell you has little bearing on the truth.
Good user experience is not dependent on features
Radically different products can have identical features.
A list of features is not the same as the design of behavior.
Expertise in a subject does not correlate to expertise in designing software behavior.

Philosophy starts on slide 92. I like philosophy, it is one of the foundations of architecture.

Cult of the Software God

2008-04-28T00:00:00-04:00

I invented a ‘religion’ in the mid-80s. Admittedly, it was done in jest, but I find that I make frequent reference to it and so it deserves a mention lest people I converse with find me just babbling. Have no fear, other whacked-out religions exist with high-profile devotees (you know some, I’m sure) so I’m in good company.

The religion is called “The Cult of the Software God”, and this god oversees technical and financial success for software entrepreneurs and IT workers. Like many ancient religions of days gone by, the Software God requires sacrifices, which amounts to the destruction of software media. At one time he/she/it was content when I would hold a 5.25" floppy disk over a garbage can and cut it up, while intoning praises to “the great Software God” and praying that the bits stored on the media be returned to their maker, but this is a God of the Geeks, and he/she/it has kept up with advances in storage media.

When 3.5" diskettes became available, the Software God soon lost interest in clunky old floppy disks and demanded sacrifices of the smaller rigid diskettes instead. I, as High Priest of the Cult, interpreted the wishes of the Software God and used a Sacred Hammer stored on a bench near our diskette duplicating station to smash up diskettes that had write errors during the duplication process. “Oh great Software God”, I would say. “We humbly beseech you to accept this offering and pray that you bring abundance and a measurable quality improvement to our next release.”

Time moved on, I moved to other ventures, and yet I never abandoned my role as High Priest of the Cult of the Software God. Occasionally I would snap CDs in half in the name of the Software God when a write error occurred, and moved on to DVDs. I even used a blowtorch on a RLL-encoded 5.25” hard drive and ground my heel on a flash drive, while singing praises to the Software God.

You might think I had too much time on my hands. Certainly I am at least eccentric. Yet there is more!

As High Priest of the Cult of the Software God, I became aware that he/she/it is also fed by /dev/null. Where do you think bits piped to /dev/null go, anyway? Why, they nourish the Software God, of course! You can turn this otherwise wasted gift to the Software God into a sacrificial offering simply by making a prayer as the bits are piped. Of course, since the Software God is a god for the geeks and by the geeks, automation counts. That is why I wrote a program in the late 90s (since lost) that would print out any prayer you desired on the screen, over and over... rather like how prayer wheels work.

Next time you really need your computer to not crash while running a semi-stable program, or have a serious virus infestation, remember the Cult of the Software God. He/she/it is virtually there for you. 😉

AAAI Symposium

2008-03-26T00:00:00-04:00

This week I am attending the AAAI Symposium for Emotion, Personality and Social Behavior at Stanford. The symposium provides a forum for an interdisciplinary discussion of modeling affect and personality in social behavior. Attendees include researchers studying social computing, virtual reality, game design, robotics, believable agents, affective computing, psychotherapy and the arts. Panel discussions discussed successful interactions between artificial beings and humans.

Some of the discussion topics that I am interested in include:

How can compelling artificial characters be designed?
How to facilitate social interaction between humans, and / or artificial beings?
What are the components in the software toolbox?
What are the emerging standards in affective artificial characters, robots and systems?
What architectural designs work best?
What knowledge base designs work best?

Believability of Robot Affect

Matthias Scheutz, Associate Professor of Informatics at Indiana University, presented a paper entitled “Empirical Investigations into the Believability of Robot Affect”. During the discussion afterwards he suggested that people are more critical of animatronic (physical) entities, compared to characters that are merely imaged. Seems that the bar for believable behavior for robots is higher than for characters which are merely rendered on a computer screen because people don't expect a rendered character to be real anyway.

Socially Assistive Robots

Adriana Tapus’ presentation, “Socially Assistive Robots: The Link between Personality, Empathy, Physiological Signals and Task Performance” provided me two pieces of interesting common-sense information:

A robot that challenges the user during therapy rather than offering praise will be preferred by extroverts;
A robot that offers nurturing praise rather than challenging-based motivation by introverts.

In the discussion that followed Adriana’s talk, mention was made of (uncited) research which found that extroverts are more sensitive to robot personalities because they seek outside stimulus; introverts are self-stimulated. It seems logical that when a robot first encounters a person about which nothing is known regarding their personality, the robot should display mildly extroverted behavior, and then adapt according to social cues.

Emotions for Strategic Real-time Systems

Megan Olsen discussed how adding an emotion framework to game engines improved performance in her talk entitled “Emotions for Strategic Real-time Systems”. The emotional framework models fear and frustration in the form of emotional maps, which are overlaid on a physical map of terrain. Emotions diffuse out and are picked up by co-operating agents nearby. Emotions have an intensity value which linearly decays over time. She showed an interesting video which illustrated the aggregated emotions of many agents over time. She also showed that emotions can cause groups of synthetic individuals to be more successful in combat. Different game engines responded differently to each emotion; some responded better to the addition of the ability to model fear, which others responded better to the addition of frustration, and NicoWar was able to benefit from the addition of both emotions.

Emotion Modeling 101

Dr. Eva Hudlicka of Psychometric Associates spoke on “Emotion Modeling 101”. Emotion modeling incorporates emotion expression, recognition, generation, and the effect on agent behavior. One could model feelings, moods, emotions, affective states and personality traits; some people mean all of the above when discussing modeling emotion. As well, attitudes and preferences can be modeled. Interpersonal roles include: social coordination, rapid communication of intent; intrapsychic roles include: motivation, homeostatis, and adaptive behavior. Emotions are manifested across multiple interacting modalities: somatic / physiological, cognitive / interpretive, behavioral / motivational and experiential / subjective. Unfortunately, the literature on emotional models is inconsistent and terms are unclear. Eva suggested we view emotion models in terms of emotion generation and effects; that emotion modeling building blocks be identified. Emotion is generated from stimuli and appraisal by the individual; Eva briefly discussed both topics and the need to standardize the process of mapping stimulus to emotion.

Coffee With Dr. Antonio Camurri

During a coffee break, I chatted with Dr. Antonio Camurri, Associate Professor at the University of Genoa, Italy. Seems we both share a passion for boats. Dr. Camurri mentioned EyesWeb, an open software platform project that he leads, that enables the development of real-time multimodal distributed interactive applications.

Wrapup

In the wrap-up session, I drew attention to one of the issues mentioned in the preface to the Technical Report that the Symposium was to intended to address: “What are the emerging standards in affective artificial characters, robots and systems?” One member of the audience (Rosamaria Barone?) mentioned some standards work for characters (unfortunately, I didn't write it down), then an awkward silence followed. I believe that standards will be key to artificial personality and emotion moving from the lab to mainstream society.

IKVM.NET – Java applications on .NET

2008-01-16T00:00:00-05:00

I am building the first implementation of EmpathyWorks™ as a Java library packaged as a JAR. After looking around for a while to understand how EmpathyWorks might run on .NET, I found IKVM.NET.

IKVM Works Well

IKVM is an open source project that converts Java JARs into a .NET assembly. I was quite skeptical, but after downloading IKVM I was able to convert EmpathyWorks into an EXE that ran on Windows XP in about ten minutes.

IKVM also has a mechanism for converting .NET stubs into JARs, important when developing .NET code in an IDE so that Intellisense tooltips appear for code completion. I am next going to try converting the Microsoft Robotics Studio runtime to a JAR so I can try writing a sample MSR application that co-operates with EmpathyWorks.

There is a brief note in the IKVM documentation that says debugging from Java in this circumstance does not work. Hmm, what about the other way... can one debug the .NET application from Visual Studio, and reach right into the converted Java code? I suppose no source code would be available.

Update Jan 18/08

I built a DLL from Java using IKVM's -target:library option. Visual C# 2008's object browser happily showed all the public objects and methods in the DLL. Once I discovered the “Add to References” button in the object browser my test C# program was able to call into the DLL. Most cool!

The End of IKVM.NET

Jeroen Frijters, the principal author of IKVM.NET posted the following Apr 21/17.

After almost fifteen years I have decided to quit working on IKVM.NET. The decision has been a long time coming. Those of you that saw yesterday’s Twitter spat, please don’t assume that was the cause. It rather shared an underlying cause. I’ve slowly been losing faith in .NET. Looking back, I guess this process started with the release of .NET 3.5. On the Java side things don’t look much better. The Java 9 module system reminds me too much of the generics erasure debacle.

I hope someone will fork IKVM.NET and continue working on it. Although, I’d appreciate it if they’d pick another name. I’ve gotten so much criticism for the name over the years, that I’d like to hang on to it 😊

I’d like to thank the following people for helping me make this journey or making the journey so much fun: Brian Goetz, Chris Brumme, Chris Laffra, Dawid Weiss, Erik Meijer, Jb Evain, John Rose, Mads Torgersen, Mark Reinhold, Volker Berlin, Wayne Kovsky, The GNU Classpath Community, The Mono Community.

And I want to especially thank my friend Miguel de Icaza for his guidance, support, inspiration and tireless efforts to promote IKVM.

Thank you all and goodbye.

I'm a Dragon on a Business Plan Panel

2007-11-21T00:00:00-05:00

A few months ago the Canadian Consulate in Silicon Valley asked me if I would be interested in working with Canadian entrepreneurs. I was happy to oblige. Since then the Canadian National Research Council has been using me to provide guidance to young entrepreneurs.

I was asked by Ean Jackson, an instructor at Simon Fraser University in Burnaby, BC to sit on a “Dragon Panel” to evaluate student projects. The 4th year undergraduate business students enrolled in the New Ventures Planning Program share the course with 4th year Engineering students at SFU.

One of the “Suits” is teamed up with three “Geeks” enrolled in the Engineering program. The Geeks come up with a technology idea and the Suits and Geeks work together to create a business plan. That business is entered into the Ken Spencer Business plan competition at SFU where cash prizes are offered.

Today (November 21, 2007) is the final class. The 14 Suits will pitch their business ideas to a panel of “investors”. I am acting in the role of a prospective investor; each of the “investors” will grade the pitch. Sounds like fun! I'm looking forward to hearing the presentations this afternoon.

Digital Mentat

2007-09-23T00:00:00-04:00

It is by caffeine alone I set my mind in motion.

It is by the Beans of Java that thoughts acquire speed, the hands acquire shaking, the shaking becomes a warning.

It is by caffeine alone I set my mind in motion.

– From Brian Herbert, Mentat Mantra of Dune

It’s just possible that I might have been spending too much time programming EmpathyWorks™ lately!

Singularity Summit

2007-09-09T00:00:00-04:00

I’m at the Singularity Summit today, after spending two months on my sailboat in B.C. Quite a change in venue!

As you can tell by reading this blog, I’m quite interested in AI. The theme of ‘friendly intelligence’ strikes a chord in me. I’m currently working on an artificial personality simulator, EmpathyWorks™, a project that I’ve been noodling on for over thirty years. Hopefully I’ll get a chance to chat with various people here at the conference about integrating artificial personality into a whole artificial organism.

2nd Annual Silicon Valley Ruby Conference

2007-04-25T00:00:00-04:00

Whew, it is a wrap! The Second Annual Silicon Valley Ruby Conference is done, and I’m just about finished doing post-production on the videos of the talks. I’ll hand a DVD to SDForum to make arrangements for attendees of the event to view the videos.

Slides for most of the presentations are available at slideshare.net.

2007 seems to be shaping up as the Year of the Ruby/Rails IDE. Later this year commercial-quality IDEs from Sun and CodeGear will join Aptana’s RadRails IDE. All three of these software publishers presented at this year’s event.

I ran a quick survey by a show of hands, and asked 24 questions. Here are a few results:

Which of you use Rails to make web sites but don’t care about Ruby?	25%
Which of you use Dreamweaver, Illustrator or Photoshop?	17%
Which of you use a Mac?	45%
Which of you use Linux?	35%
Which of you use Windows?	50%

It’s been fun leading the organization of these first two years of this event, but this year will be my last.

Announcing Micronautics Research

2006-10-17T00:00:00-04:00

Cool robot mascot, huh? I’m going to use it for EmpathyWorks™, which at this point is still just a twinkle in my eye.

A physical metaphor for IT innovation

2006-06-23T00:00:00-04:00

I got an email today from a friend, an innovative engineer:

Hi Mike,

The other day, I had a reasonably good idea for a startup. And I ran it by a friend, who said "Why would the enterprise buy that?"

I replied "Much better than existing solutions."

He said "Unless it’s a 10x, nope. And, while it is better, it’s not a 10x."

I thought "That’s gotta be wrong."

But then it occurred to me. In spite of having worked as a consultant to enterprises, and in spite of having run professional services for an enterprise software company, I still don’t have a very good model of enterprise software adoption. To wit:

What makes a new piece of software or IT infrastructure compelling for the enterprise?
When do enterprises decide to upgrade or change infrastructure?

Do you know of any references or models?

Here is what I told him:

A 10x return on investment applies to investors (although 7x works too). If the sales pitch is predicated on cost reduction (optimizing operations), then 5-10x would apply and is most interesting when the client’s business is relatively mature.

Increasing client sales (optimizing business development) needs to be presented along the client’s strategic direction, which might be a moving target. 2x or 3x return would suffice if the timing between a vendor’s presentation and a client’s view of their strategic direction aligned. This would be difficult for the vendor to guess without incredible intelligence on specific targets.

CIOs are risk-averse, so solutions that do not require vetting new servers and client-side software are easier. When selling into a datacenter, a 10x return might not be enough if security issues are not mitigated. On the other hand, repurposing existing investments are much easier to sell: a 2x or 3x return might be sufficient.

In other words: inertia is a dominant force of nature that also applies in IT. If you consider trends to be acceleration (positive or negative), then the net force driving the inertia is composed of external and internal factors. Those factors might be political, economic, technical, etc. Newton’s F = m * a can be restated in entrepreneurial terms as:

net force = investment times rate of adoption

or, more to the point:

rate of adoption = net force / investment

The investment (mass) a client has made in developing and deploying their intellectual property; training personnel; capital investment in infrastructure; and intimate knowledge of the current system. ("The devil you know".)

For entrepreneurs that want to convince clients that a new technology that is worth adopting, continuing our physics metaphor may be illuminating. Let’s equate profitability with speed, and technology platform with direction. Thus velocity is the cost-effectiveness of a given technical platform.

It follows that in order to change a company’s direction, and increase their profitability (speed), the client will need to experience a strong enough net force over a long enough period of time to overcome inertia and justify the risk and the investment.

In addition, the prospect will only understand new information in their current context, which might not use the same dictionary that an entrepreneur uses, since they tend to live in the future.

Success! Silicon Valley Ruby Conference

2006-04-24T00:00:00-04:00

After months of planning, the big weekend arrived and a great time was had by all. Thank you, presenters!

I organized the event and chaired the second day of the conference.

Yours truly, addressing the 2006 Silicon Valley Ruby Conference

See you next year...

Silicon Valley Ruby Conference

2006-02-21T00:00:00-05:00

SDForum just announced the Silicon Valley Ruby Conference. I am one of the two co-chairmen.

We expect the event to sell out very quickly. Some great speakers have been lined up, it's quite inexpensive and only 210 people can attend.

See you there!

Update: Andrew Burke wrote up some good notes for day 1 and day 2 of the conference.

Instantiating Java Inner Classes

2005-12-30T00:00:00-05:00

I've been writing Java code for years. Today I learned something new. I have been adding a new feature to Zamples that was best expressed as a doubly nested inner class. The hierarchy looks like this:

Shell

public class CodeRange {
   public CodeRange(String str) { /* ... */ }

   public class RangeSpec {
      public RangeSpec(int i, int j) { /* ... */ }
      public RangeSpec(RangeItem start, RangeItem end) { /* ... */ }

      public class RangeItem {
         public RangeItem(String str, int i) { /* ... */ }
      }
   }
}

When it came time to write JUnit tests, I needed to instantiate the hierarchy. The syntax surprised me:

Shell

CodeRange.RangeSpec.RangeItem range =
   new CodeRange("test").new RangeSpec(1, 1).new  RangeItem("middle", 3);

Not exactly intuitive, eh? It gets even more interesting when trying to write unit tests for the second RangeSpec constructor, the one that accepts two RangeItems. I found I had to create a protected no-args constructor for RangeSpec with an empty body, plus a method within RangeSpec to create RangeItems on demand:

Shell

/** For JUnit only */
   protected RangeSpec() {}

   /** For JUnit only */
   protected RangeItem newRangeItem(String searchString, int offset) {
      return new RangeItem(searchString, offset);
   }

Now I could write my test case setup code:

Shell

codeRange = new CodeRange(code, "");
   CodeRange.RangeSpec.RangeItem rangeItemStart =
      codeRange.new RangeSpec().newRangeItem("middle", 3);
   CodeRange.RangeSpec.RangeItem rangeItemEnd =
      codeRange.new RangeSpec().newRangeItem("back", 0);

   CodeRange.RangeSpec rangeSpec =
      codeRange.new RangeSpec(rangeItemStart, rangeItemEnd);
   /* ... */

Everything I read about doubly nested classes amounted to a warning to the effect that they should be avoided. I never had occasion to need this type of solution before, however the particular problem I am solving yields very nicely to this approach. It is simple, elegant and efficient. Don't believe everything you read (except this blog!) 😉.

Zamplized Ruby User’s Guide

2005-12-16T00:00:00-05:00

Santa's elves have been cheering us on at Zamples! Just in time for the holidays, Zamples v2.6 is now live.

We've “Zamplized” the Ruby User’s Guide and made it available for free. Now you can curl up with a glass of eggnog and sit by the fire with your laptop, while learning Ruby, the hottest new programming language today.

Zamples’ live code examples let you “learn by doing” and give you a hands-on and foolproof experience. There is no faster, easier way to gain experience with a language or programming interface than by interacting with a Zamplized document.

Best of the season!

Programmatic IM Generation

2005-12-16T00:00:00-05:00

I thought it would be fun to create a live code example using Zamples that would generate an IM (instant message).

The following Java code example uses Jive Software’s Smack library to send a message to a Jabber user with a Gaim client. You will need to put real values in for from, password and to before a message will be sent. BTW, if the userid and password are wrong, you’ll get an odd error message. Since the program fragment automatically compiles and runs after you click on the Try It! button, the error message will appear on the right.

Once you provide the to and from user ids and password and rerun the code fragment, a message will be sent. I tested this by sending myself a message and found that from and to were the same.

Shell

XMPPConnection connection = new XMPPConnection("jabber.org");
connection.login("from", "password");
connection.createChat("to@jabber.org/Gaim").sendMessage("Howdy!");

Zamples REST Interface

2005-12-15T00:00:00-05:00

Wouldn’t it be great to be able to build in a facility for live code examples into your favorite software tools? We thought so, and are proud to announce the Zamples REST Interface. Now you can use any programming language you choose to embed Zamples’ functionality as an IDE plugin or online learning environment.

“I Want to Build a Girl”

2005-07-28T00:00:00-04:00

I said that as an undergraduate engineering student in 1975. Now a Japanese researcher has made real progress towards a synthetic companion. The artificial skin doesn't seem to have a high density sensory grid that other researchers are experimenting with (future artificial skins might incorporate sensors not only for pressure and temperature, but also for light, humidity, strain or sound), but I’m sure it won't be long before a similar android with tactile response and expression ability graduates from the lab.

I still want to be involved in bringing that girl... and a boy... and a whole range of socially acceptable companions to market. Some day soon...

Update: Sony’s Qrio, a humanoid robot under development is exactly along the lines I’ve been dreaming of!

Update: from PC Magazine, “Eleksen, a small UK-based firm is introducing electronic fabric, essentially carbon-embedded nylon sandwiched between layers of nylon mesh that, when a milliamps charge is passed through it, can recognize touch, pressure and even the direction and path of a stroke. This thin, flexible, and washable fabric connects to a small 8-bit processor, which then can be connected to a standard electronic device like an iPod. Eleksen company executives said the washable fabric can also withstand extreme pressure; they’ve rolled a car over it without any ill effects.” Seems we are moving towards artificial skin!

Update: from The Economist, Japan’s humanoid robots – Better than people:
“What seems to set Japan apart from other countries is that few Japanese are all that worried about the effects that hordes of robots might have on its citizens. Nobody seems prepared to ask awkward questions about how it might turn out. If this bold social experiment produces lots of isolated people, there will of course be an outlet for their loneliness: they can confide in their robot pets and partners. Only in Japan could this be thought less risky than having a compassionate Filipina drop by for a chat.”

Self-Fullfilling Prophecy

2005-05-15T00:00:00-04:00

Heisenberg’s Uncertainty Principle

The Greek philosopher Democritus believed that the natural world was immutable. In 1927 Werner Heisenberg published his “Uncertainty Principle” which implied, amongst other things, that observing a quantum phenomenon also significantly altered the phenomenon. For many people, such an idea was counter-intuitive.

What is true in the physical realm is even more true in the realms of interpersonal relationships and business. Our beliefs and belief systems affect outcomes more than we perhaps realize. Consider the diagram below.

Describing the Trend

Without knowing what the diagram represents, can you postulate the trend for the next cycle or two? The answer is, of course, no. You need to have an internal model of the behavior of the system and the entity’s behavior within that system represented in the graph before you can make a prediction. Without any additional information, you might anticipate the trend like this:

Your model is based on incomplete information, and is necessarily a simplification of the actual system. If you are told that the graph depicts rainfall, and if you believe in a rain god, then your natural reaction might be to find a virgin to sacrifice. If you instead had the belief that rainfall had diminished due to clear-cutting of a vast forest which no longer acted as a natural storage mechanism for rainwater, your reaction might be to start lobbying for an intensive tree planting effort. Both belief systems have at their core a central belief that action can affect the outcome to some degree, so your prediction as to how much rain will fall in subsequent years will rely on the vigor with which countermeasures are applied, and your belief in their efficacy. Probably your belief would be that rainfall levels will tend to vary around historical norms, like this:

What if you were told instead that the first graph represented sales of a hybrid car model, and that ‘experts’ believed that increasing SUV sales and the economy were believed to be responsible for declining sales? A marketing campaign that addressed the rising cost of gas and promoted decreasing the reliance of the US economy on foreign oil as being truly patriotic might gain traction. One might attribute the initial sales spike as being due to purchases by early adopters. The tapering off of sales might be the result of tax incentives that artificially promoted the sales of vehicles over 6,000 pounds.

It is clear that China’s consumption of gas will soon outpace even the US’s rate of consumption, so gas prices will continue to rise, and remain at higher prices than any we have experienced to date. The US Department of Energy produced the following projection of Chinese gas consumption in June 2004.

In 2003, BP showed China’s new influence on global oil supply: “Global oil demand, meanwhile, was broadly flat, increasing 290,000 bpd to 75.7 mm bpd from 75.5 mm bpd. All the increase is attributable to China where oil consumption increased 5.8 % or 332,000 bpd.” As demand for a commodity increases, prices increase. As China continues to dramatically increase its demand for oil, prices can only continue higher. In a feature article on Alexander’s Gas & Oil Connections, John Vidal writes “China's oil consumption, which accounted for a third of extra global demand last year, grew 17% and is expected to double over 15 years to more than 10 mm bpd – half the US's present demand. India's consumption is expected to rise by nearly 30% in the next five years.” Jim Meyer, an expert at London's Oil Depletion Analysis Centre was quoted in an article last month as saying “After 2007, we can’t see enough new supplies to meet almost any reasonable level of demand growth.” Expect gas prices to permanently exceed $50/barrel, and $100+/barrel will be reached with startling speed. With that in mind, a long term sales trend for hybrid cars in the US might be projected to look like this:

This projection assumes that historical sales figures are not a significant factor when gauging future sales of this hybrid car model, and that factors on the market that the product sells into have not yet been fully felt or internalized by consumers.

Self-Fullfilling Prophecy

What if a hard-nosed automotive executive, looking at ROI on a quarter-by-quarter basis decided that the return was not immediate enough, and terminated the hybrid car based on sales history to date? That automotive manufacturer would soon miss the world’s largest market, China, as the demand for fuel-efficient cars will soon be supported by a Chinese middle class newly able to afford more expensive hybrid cars. Unless the manufacturer was somehow able to introduce another technology, such as hydrogen powered vehicles, and successfully put together a national infrastructure network to support hydrogen fuel stations, they will have traded off short-term profitability for long-term viability.

Returning to the original idea behind this articleing, one’s expectations and beliefs do color what one might think to be reasonable. If you believe that most people are kind and generous, your behavior will probably reinforce this belief – with the result that even when you meet Scrooge incarnate, he might crack a smile despite himself. If you think that sales of product XYZ are going down, down, down and that nothing can be done about it then that product is as good as dead. If instead, you come up with an alternate world view, communicate higher expectations and provide sufficient resources over the appropriate time period, that product’s best years may be yet to come.

Marshall Brain is a very smart man

2005-04-03T00:00:00-05:00

Marshall Brain founded How Stuff Works, and wrote many interesting and informative articles for the site.

We share an interest in personal robotics. Like many US citizens, Mr. Brain's focus in Robotic Nation is on the potentially negative aspects of a integrating robots into society, such as mass unemployment. His writing reminds me of the endless fear-mongering that CNN's Lou Dobbs serves up daily on the evils of outsourcing and illegal immigration. I have a more optimistic point of view on personal robotics.

The Japanese attitude for robotics is quite positive, and their enthusiasm for personal robots resembles their enthusiasm for advanced cell phones. iMode mobile phone systems have features beyond what is known in North America, and Japanese society has evolved to incorporate behavioral modes that depend on the advanced features of the iMode phone network. Beyond mobile phones, the Japanese culture celebrates electronic gadgetry, and they already accept robots, including personal robots, into their daily experience.

Change is coming, and we can either embrace the future or bemoan the loss of the past. I think personal robotics will become a plaything of the rich in 2015; Mr. Brain predicts the mass market will explode in 2030. It's not like we didn't have any warning.

I think I need to start learning Japanese.

Top 12 Klingon Programmer Sayings

2005-04-02T00:00:00-05:00

Specifications are for the weak and timid!
This machine is a piece of GAGH! I need dual Opteron processors if I am to do battle with this code!
You cannot really appreciate Dilbert unless you've read it in the original Klingon.
Indentation?! – I will show you how to indent when I indent your skull!
What is this talk of ‘release?’ Klingons do not make software ‘releases.’ Our software ‘escapes,’ leaving a bloody trail of designers and quality assurance people in its wake.
Klingon function calls do not have ‘parameters’ – they have ‘arguments’ – and they ALWAYS WIN THEM.
Debugging? Klingons do not debug. Our software does not coddle the weak.
I have challenged the entire quality assurance team to a Bat-Leth contest. They will not concern us again.
A TRUE Klingon Warrior does not comment his code!
By filing this PTR you have challenged the honor of my family. Prepare to die!
You question the worthiness of my code? I should kill you where you stand!
Our users will know fear and cower before our software! Ship it! Ship it and let them flee like the dogs they are!

I lifted this from Monsters Island, now defunct.

Zample in a article

2005-01-05T00:00:00-05:00

Erik Dasque, Product Manager for Mono at Ximian (now part of Novell) was the first person to include a Zample in a article. Way to go, Erik! May there be many more bloggers using Zamples to display live code in their blogs.

I have had some trouble doing this – when I used SnipSnap it simply wouldn't accept raw HTML. I wrote a SnipSnap plugin to allow raw HTML, but then my SnipSnap installation b0rked and I gave up on it. WordPress really mangles the HTML for a Zamples button; here is my best effort (can't get the 'Try It!' button to remain vertically centered.)

Shell

public class Test {
    public static void Main() {
        System.Console.WriteLine("I love Zamples!");
    }
}

If you would like to include a live code example in your blog, first create it using the Zamplizer (easiest if you click on one of the selections under the Create Sample left-hand menu on the Zamples Solutions page – that will set up the Zamplizer with a short working program in the language you want, which you can then modify.)

Another Software Expert Assignment

2004-12-24T00:00:00-05:00

I‘ve just picked up another software litigation case, this time doing research for a software vendor prior to going to court. My client is concerned that two former employees used their code in a product sold by a new company that the former employees started.

I've worked on this type of case before. They are often settled out of court. It is interesting to see how human interactions affect technology and business decisions.

I know what I'll be doing over the Christmas/New Year's holidays!

Nabu cast a long shadow

2004-12-18T00:00:00-05:00

Twenty years ago an Ottawa-based startup was formed by the merger of nine Canadian companies. Nabu sold a cable set-top box with a computer that booted off the cable for under $1000. I was one of the early employees.

Nabu was first in many ways, but it never became the commercial success that the founders had hoped. Many years have since passed, and I forgot about the experience. In the last couple of years I have been contacted by various people regarding Nabu’s products and services, mostly former customers.

Nabu was the first computer that I had when I was a kid. I remember playing games like Mummy’s Tomb, Kiddy Park etc. Talk about being ahead of it's time! ...

I think maybe people that were influenced by NABU when they were young are probably at the point in their life where they can stop and take a look around...

Anyway, I thought I would drop you a message and say that this Ottawa boy (who now lives in Seattle, WA) still remembers the old Nabu days.

I’m now a professional video game developer, which I’m sure was influenced by playing the Nabu so long ago. Of course today’s machines have a little more power! What you guys did was a pretty amazing accomplishment IMHO.

As someone who was fortunate enough to have access to the NABU network in my after school program at Regina Public School in Ottawa... I really enjoyed it as a kid.

Recently I was contacted by a legal firm that has retained me to assist them in a patent litigation case. They are going to cite some of Nabu’s products as prior art. I’m having fun going back and reconnecting with people that I knew from that bygone era, who I lost track of. So far no-one that I knew has died of old age, which pleases me.

Live Code Examples for JDK 6

2004-12-09T00:00:00-05:00

No sooner did Sun push J2SE 5 (also known as JDK 1.5) out the door, then talk of the next version began. Java releases are generally 18 months apart, but for the first time Sun is letting the world see development builds as they are created.

We at Zamples took advantage of this new openness and proudly bring you Live Code Examples for J2SE 6. Let us know if you discover any cool new features!

All Global 2000 Businesses Are Software Developers

2004-11-04T00:00:00-05:00

With the advent of web services, any company that wishes to expose a programmable interface to their business processes also inherits the issues faced by software vendors. Connecting up a supply chain isn't quite as simple as one might want it to be.

Software vendors have an urgent need for their channel partners and customers to adopt their technology quickly. The faster their products are understood, the more money they make. Shorter adoption times drive top line revenue.

Similar dynamics affect companies that use web services to transact business. Businesses with custom interfaces need to somehow show system integrators how to program to their web services. Executives need to be able to wire their enterprises together quickly and easily in order to take advantage of business opportunities. Web services are supposed to make this easy... but nothing is ever quite as easy as it is supposed to be.

The answer for both software vendors and businesses with custom web services interfaces is a comprehensive developer relations program. Not surprisingly, live online code examples are an important aspect of such a program. I have begun writing a follow-on to my article entitled Top Ten Issues for Developer Relations Managers. I would be happy to hear from you if you would like to review a draft before it is published.

Reading Java Properties Files from Bash

2004-11-03T00:00:00-05:00

Ever notice that Java properties files are fairly similar to bash scripts? Here is a short bash script that takes a valid Java properties file called language.properties, massages it into a format that is compatible with bash syntax, writes it to a temporary file and sources it.

Shell

#!/bin/bash
TEMPFILE=$(mktemp)
cat language.properties | \
  sed -re 's/"/"/'g | \
  sed -re 's/=(.*)/="\1"/g' > $TEMPFILE
source $TEMPFILE
rm $TEMPFILE

After running this script, all of the properties in the Java properties file become available as environment variables.