Mike Slinn's Blog 2022-09-14T21:28:59-04:00 https://mslinn.github.io/blog Mike Slinn mslinn@gmail.com JiraCLI, a Feature-rich Interactive Jira Command Line 2022-08-12T00:00:00-04:00 https://mslinn.github.io/blog/2022/08/12/jiracli <p> When first released in 2002, <a href='https://www.atlassian.com/software/jira' target='_blank' rel='nofollow'>Jira</a> was merely a bug tracker. Since then, it has gained features and is now used as a project management tool. Jira only provides a web user interface. </p> <p> <a href='https://github.com/ankitpokhrel/jira-cli' target='_blank' rel='nofollow'>JiraCLI</a>, an independent F/OSS project, provides command-line explorers for Jira issues, epics, and sprints. </p> <h2 id="install">Installation</h2> <p> JiraCLI is a Go program, so it is quick and easy to install on Ubuntu once Go is installed. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id817a5aae28d4'><button class='copyBtn' data-clipboard-target='#id817a5aae28d4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install golang-go <span class='unselectable'>$ </span>go install github.com/ankitpokhrel/jira-cli/cmd/jira@latest <span class='unselectable'>go: downloading github.com/ankitpokhrel/jira-cli v1.0.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/spf13/cobra v1.5.0 go: downloading github.com/spf13/viper v1.12.0 go: downloading github.com/zalando/go-keyring v0.2.1 go: downloading github.com/briandowns/spinner v1.18.1 go: downloading github.com/fatih/color v1.13.0 go: downloading github.com/mitchellh/go-homedir v1.1.0 go: downloading github.com/AlecAivazis/survey/v2 v2.3.5 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 go: downloading github.com/cpuguy83/go-md2man/v2 v2.0.2 go: downloading github.com/spf13/pflag v1.0.5 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/atotto/clipboard v0.1.4 go: downloading github.com/charmbracelet/glamour v0.5.0 go: downloading github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d go: downloading github.com/cli/safeexec v1.0.0 go: downloading github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 go: downloading github.com/kentaro-m/blackfriday-confluence v0.0.0-20220126124413-8e85477b49b3 go: downloading github.com/russross/blackfriday/v2 v2.1.0 go: downloading github.com/mattn/go-colorable v0.1.12 go: downloading github.com/mattn/go-isatty v0.0.14 go: downloading github.com/gdamore/tcell/v2 v2.5.1 go: downloading github.com/rivo/tview v0.0.0-20220610163003-691f46d6f500 go: downloading github.com/fsnotify/fsnotify v1.5.4 go: downloading github.com/mitchellh/mapstructure v1.5.0 go: downloading github.com/spf13/afero v1.8.2 go: downloading github.com/spf13/cast v1.5.0 go: downloading github.com/spf13/jwalterweatherman v1.1.0 go: downloading github.com/godbus/dbus/v5 v5.1.0 go: downloading golang.org/x/text v0.3.7 go: downloading golang.org/x/term v0.0.0-20220526004731-065cf7ba2467 go: downloading golang.org/x/sys v0.0.0-20220615213510-4f61da869c0c go: downloading github.com/subosito/gotenv v1.4.0 go: downloading github.com/hashicorp/hcl v1.0.0 go: downloading gopkg.in/ini.v1 v1.66.6 go: downloading github.com/magiconair/properties v1.8.6 go: downloading github.com/pelletier/go-toml/v2 v2.0.2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/gdamore/encoding v1.0.0 go: downloading github.com/pelletier/go-toml v1.9.5 go: downloading github.com/lucasb-eyer/go-colorful v1.2.0 go: downloading github.com/mattn/go-runewidth v0.0.13 go: downloading github.com/muesli/termenv v0.12.0 go: downloading github.com/yuin/goldmark v1.4.12 go: downloading github.com/yuin/goldmark-emoji v1.0.1 go: downloading github.com/rivo/uniseg v0.2.0 go: downloading github.com/alecthomas/chroma v0.10.0 go: downloading github.com/microcosm-cc/bluemonday v1.0.18 go: downloading github.com/muesli/reflow v0.3.0 go: downloading github.com/olekukonko/tablewriter v0.0.5 go: downloading github.com/aymerick/douceur v0.2.0 go: downloading golang.org/x/net v0.0.0-20220621193019-9d032be2e588 go: downloading github.com/dlclark/regexp2 v1.4.0 go: downloading github.com/gorilla/css v1.0.0 </span></pre> <p> Add <code>$HOME/go/bin</code> to the <code>PATH</code> so <code>jira</code> can be found: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb396f0b28937'><button class='copyBtn' data-clipboard-target='#idb396f0b28937' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo 'PATH=$HOME/go/bin:$PATH' >> ~/.bashrc <span class='unselectable'>$ </span>source ~/.bashrc <span class='unselectable'>$ </span>which jira <span class='unselectable'>/home/mslinn/go/bin/jira </span></pre> <h2 id="usage">Usage</h2> <p>This is the <code>jira </code> help message:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id69900e1d2e2d'><button class='copyBtn' data-clipboard-target='#id69900e1d2e2d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira <span class='unselectable'>Interactive Jira CLI.<br/> USAGE jira [flags]<br/> MAIN COMMANDS board Board manages Jira boards in a project epic Epic manage epics in a project issue Issue manage issues in a project open Open issue in a browser project Project manages Jira projects sprint Sprint manage sprints in a project board<br/> OTHER COMMANDS completion Output shell completion code for the specified shell (bash or zsh) help Help about any command init Init initializes jira config man Help generate man(7) pages for Jira CLI. me Displays configured jira user version Print the app version information<br/> FLAGS -c, --config string Config file (default is /home/mslinn/.config/.jira/.config.yml) --debug Turn on debug output -h, --help help for jira -p, --project string Jira project to look into (defaults to /home/mslinn/.config/.jira/.config.yml)<br/> LEARN MORE Use &#39;jira &lt;command&gt; &lt;subcommand&gt; --help&#39; for more information about a command. </span></pre> <p> You need an Jira API token before you can use JiraCLI. Get it from <a href='https://id.atlassian.com/manage-profile/security/api-tokens' target='_blank' rel='nofollow'>here</a>. </p> <div style=""> <picture> <source srcset="/blog/images/atlassian_api_token.webp" type="image/webp"> <source srcset="/blog/images/atlassian_api_token.png" type="image/png"> <img src="/blog/images/atlassian_api_token.png" class=" liImg2 rounded shadow" /> </picture> </div> <p> You need to configure the program before you use it. Set the <code>JIRA_API_TOKEN</code> environment variable before running <code>jira init</code>. If you do not, then you will get an error like <code>Received unexpected response '401 Unauthorized' from jira.</code> in the next step. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9519d0cc9395'><button class='copyBtn' data-clipboard-target='#id9519d0cc9395' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>export JIRA_API_TOKEN=asdfasdfasdf <span class='unselectable'>$ </span>jira init <span class='unselectable'>? Installation type: Cloud ? Link to Jira server: https://xxx.atlassian.net/ ? Login email: mslinn@mslinn.com ⠼ Verifying login details... ? Default project: My ? Default board: [Use arrows to move, type to filter, ? for more help] > [Search...] ---------- My Scrum Board None ✓ Configuration generated: /home/mslinn/.config/.jira/.config.yml </span></pre> <p> This is the configuration file that was generated: </p> <div class='codeLabel unselectable' data-lt-active='false'>/home/mslinn/.config/.jira/.config.yml</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddaa8d928fe05'><button class='copyBtn' data-clipboard-target='#iddaa8d928fe05' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>board: id: 350 name: My Scrum Board type: scrum epic: name: customfield_10009 link: customfield_10008 installation: Cloud issue: fields: custom: - name: Epic Link key: customfield_10008 schema: datatype: any - name: Epic Name key: customfield_10009 schema: datatype: string types: - id: "3" name: Task handle: Task subtask: false - id: "5" name: Sub-task handle: Sub-task subtask: true - id: "7" name: Story handle: Story subtask: false - id: "1" name: Bug handle: Bug subtask: false - id: "6" name: Epic handle: Epic subtask: false login: mslinn@mslinn.com project: key: My type: classic server: https://xxx.atlassian.net</pre> <h2 id="ex">Example Commands</h2> <p> Help for the <code>jira issue</code> subcommand: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id93aa8125b426'><button class='copyBtn' data-clipboard-target='#id93aa8125b426' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue -h <span class='unselectable'>Issue manage issues in a given project. See available commands below.<br/> USAGE jira issue [flags]<br/> MAIN COMMANDS assign Assign issue to a user clone Clone duplicates an issue comment Manage issue comments create Create an issue in a project delete Delete an issue edit Edit an issue in a project link Link connects two issues list List lists issues in a project move Transition an issue to a given state unlink Unlink disconnects two issues from each other view View displays contents of an issue worklog Manage issue worklog<br/> FLAGS -h, --help help for issue<br/> INHERITED FLAGS -c, --config string Config file (default is /home/mslinn/.config/.jira/.config.yml) --debug Turn on debug output -p, --project string Jira project to look into (defaults to /home/mslinn/.config/.jira/.config.yml)<br/> ALIASES issues<br/> LEARN MORE Use &#39;jira &lt;command&gt; &lt;subcommand&gt; --help&#39; for more information about a command. </span></pre> <p> Help for the <code>jira issue list</code> sub-subcommand: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddbf739b0b6d3'><button class='copyBtn' data-clipboard-target='#iddbf739b0b6d3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue list -h <span class='unselectable'>List lists issues in a given project.<br/> You can combine different flags to create a unique query. For instance,<br/> # Issues that are of high priority, is in progress, was created this month, and has given labels jira issue list -yHigh -s&quot;In Progress&quot; --created month -lbackend -l&quot;high prio&quot;<br/> Issues are displayed in an interactive list view by default. You can use a --plain flag to display output in a plain text mode. A --no-headers flag will hide the table headers in plain view. A --no-truncate flag will display all available fields in plain mode.<br/> USAGE jira issue list [flags]<br/> FLAGS -t, --type string Filter issues by type -R, --resolution string Filter issues by resolution type -s, --status string Filter issues by status -y, --priority string Filter issues by priority -r, --reporter string Filter issues by reporter (email or display name) -a, --assignee string Filter issues by assignee (email or display name) -C, --component string Filter issues by component -l, --label stringArray Filter issues by label -P, --parent string Filter issues by parent --history Issues you accessed recently -w, --watching Issues you are watching --created string Filter issues by created date Accepts: today, week, month, year, or a date in yyyy-mm-dd and yyyy/mm/dd format, or a period format using w = weeks, d = days, h = hours, m = minutes. eg: -10d Created filter will have precedence over created-after and created-before filter --updated string Filter issues by updated date Accepts: today, week, month, year, or a date in yyyy-mm-dd and yyyy/mm/dd format, or a period format using w = weeks, d = days, h = hours, m = minutes. eg: -10d Updated filter will have precedence over updated-after and updated-before filter --created-after string Filter by issues created after certain date --updated-after string Filter by issues updated after certain date --created-before string Filter by issues created before certain date --updated-before string Filter by issues updated before certain date -q, --jql string Run a raw JQL query in a given project context --order-by string Field to order the list with (default &quot;created&quot;) --reverse Reverse the display order (default &quot;DESC&quot;) --paginate string Paginate the result. Max 100 at a time, format: &lt;from&gt;:&lt;limit&gt; where &lt;from&gt; is optional (default &quot;0:100&quot;) --plain Display output in plain mode --no-headers Don&#39;t display table headers in plain mode. Works only with --plain --no-truncate Show all available columns in plain mode. Works only with --plain --columns string Comma separated list of columns to display in the plain mode. Accepts: TYPE, KEY, SUMMARY, STATUS, ASSIGNEE, REPORTER, PRIORITY, RESOLUTION, CREATED, UPDATED -h, --help help for list<br/> INHERITED FLAGS -c, --config string Config file (default is /home/mslinn/.config/.jira/.config.yml) --debug Turn on debug output -p, --project string Jira project to look into (defaults to /home/mslinn/.config/.jira/.config.yml)<br/> EXAMPLES $ jira issue list<br/> # Limit list to 20 items $ jira issue list --paginate 20<br/> # Get 50 items starting from 10 $ jira issue list --paginate 10:50<br/> # List issues in a plain table view without headers $ jira issue list --plain --no-headers<br/> # List some columns of the issue in a plain table view $ jira issue list --plain --columns key,assignee,status<br/> # List issues in a plain table view and show all fields $ jira issue list --plain --no-truncate<br/> # List issues of type &quot;Epic&quot; in status &quot;Done&quot; $ jira issue list -tEpic -sDone<br/> # List issues in status other than &quot;Open&quot; and is assigned to no one $ jira issue list -s~Open -ax<br/> # List issues from all projects $ jira issue list -q&quot;project IS NOT EMPTY&quot;<br/> ALIASES lists ls<br/> LEARN MORE Use &#39;jira &lt;command&gt; &lt;subcommand&gt; --help&#39; for more information about a command. </span></pre> <h3 id="updated">Issues Updated on a Certain Date</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6a5c41e30197'><button class='copyBtn' data-clipboard-target='#id6a5c41e30197' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue list --updated 2021-11-17</pre> <h3 id="updated">Issues Selected by Complex Criteria</h3> <p> The following command will yield a list of high-priority issues, created this month, with status <code>To Do</code>, that are assigned to you, and have the label <code>backend</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id56df7797bc4b'><button class='copyBtn' data-clipboard-target='#id56df7797bc4b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue list -yHigh -s"To Do" --created month -lbackend -a$(jira me)</pre> <p> Output might look something like the following, which was redacted: </p> <div style=""> <picture> <source srcset="/blog/images/jira_issues.webp" type="image/webp"> <source srcset="/blog/images/jira_issues.png" type="image/png"> <img src="/blog/images/jira_issues.png" class=" liImg2 rounded shadow" /> </picture> </div> <h3 id="ids">Issue Ids Only</h3> <p> The <code>--plain</code> and <code>--no-headers</code> options are useful for driving scripts. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc7ead38e91db'><button class='copyBtn' data-clipboard-target='#idc7ead38e91db' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue list --updated 2021-11-17 --plain --no-headers --columns KEY</pre> <h3 id="detail">View Issue</h3> <p> The following returns the available information about an issue, in a plain-text format. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4cd5223bda8e'><button class='copyBtn' data-clipboard-target='#id4cd5223bda8e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue view ISSUE-1 --comments 9999 --plain</pre> <h2 id="verdict">Verdict</h2> <p> JiraCLI is useful project! <span style='font-size: 3em; '>&#x1F601;</span> </p> ImageMagick Slicing on Ubuntu/WSL 2022-07-28T00:00:00-04:00 https://mslinn.github.io/blog/2022/07/28/imagemagick-slicing <p> Lawyers like the Microsoft Office software suite; so when I am working on a court case as an expert, I endeavor to provide my clients with Word documents that contain necessary information. I like working in WSL/WSL2 because I can use Windows programs and Ubuntu programs together effectively. </p> <h2 id="steps">Grab Image, Then Slice</h2> <p> Recently, I used <a href='https://www.techsmith.com/screen-capture.html' target='_blank' rel='nofollow'>SnagIt</a>, a Windows program, to capture large web pages as single images. Some of these images were quite tall. </p> <p> The CSS for the web pages made some content invisible on the printed page. Yes, I could have injected CSS using a Chrome plugin like <a href='https://chrome.google.com/webstore/detail/my-style/ljdhjpmbnkbengahefamnhmegbdifhlb' target='_blank' rel='nofollow'>My Style</a> to ensure that all content will be printed, like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Injected Style</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida3d4a5845b9b'>@media print { * { display: initial; visibility: visible; } }</pre> <p> I decided to use screen grabs, which would guarantee that the contents of my report would exactly match what had been displayed on the screen, without injecting anything into the web pages. </p> <p> <a href='https://imagemagick.org/index.php' target='_blank' rel='nofollow'>ImageMagick</a> is preinstalled on Ubuntu Desktop. I used ImageMagick to slice the image captures into smaller page-sized images, so they could be inserted into a Word document. </p> <h2 id="grunt">The Computer Worked Hard</h2> <p> Grabbing such large web pages was a lot of work for my desktop computer. The only programs active during the screen grab process were the Google Chrome browser and SnagIt. I found that 10GB RAM and 30% of the GPU capability (an NVidia GTX 1660 Super) was used. </p> <p> The screen grab failed if I did not start scrolling from the top of the web page; while it is possible to scrub up and down smaller web pages in order to grab portions of interest, this fails for large pages. </p> <p> I also found that scrolling too fast caused the screen grabbing process to fail. Clicking and holding the bottom scroll arrowhead at the bottom right of the screen seemed to result in a smooth and optimal scrolling speed. This meant that grabbing large web pages took a few minutes as the page slowly scrolled downward. </p> <h2 id="setup">Setting Up the Conversion</h2> <p> The Word documents I usually work with are formatted for North American standards. This means one-inch margins on letter-sized paper (8.5" x 11"), which gives a working area of 6.5" x 9", yielding an aspect ratio of 0.72. </p> <p> The tall captured images needed to be sliced into rectangles that fit efficiently into Word documents. The computations are as follows. </p> <ol> <li> Determine the width of a screen grab and save it into <code>W</code>. The ImageMagick <code>identify</code> command does not provide a newline after its output, however I have inserted one for readability: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8bc9dc3aacad'><button class='copyBtn' data-clipboard-target='#id8bc9dc3aacad' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>identify -ping -format '%w' ../IMG2005.png <span class='unselectable'>1536 </span> <span class='unselectable'>$ </span>export W="$( identify -ping -format '%w' ../IMG2005.png )"</pre> </li> <li> Determine the height of a screen grab and save it into <code>H</code>: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id94ff21af048e'><button class='copyBtn' data-clipboard-target='#id94ff21af048e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>identify -ping -format '%h' ../IMG2005.png <span class='unselectable'>$ </span>export H="$( identify -ping -format '%h' ../IMG2005.png )"</pre> </li> <li> The width can be divided by the aspect ratio to obtain the desired height of each slice so they can be inserted optimally into the Word documents. I used the <a href='https://linux.die.net/man/1/bc' target='_blank' rel='nofollow'><code>bc</code> calculator</a> provided with <code>Bash</code> to divide <code>W / ASPECT_RATIO</code>. The <code>H2</code> integer variable contains the computed height for the images. <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0db1abdbb204'><button class='copyBtn' data-clipboard-target='#id0db1abdbb204' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>export ASPECT_RATIO=0.72 <span class='unselectable'>$ </span>export H2="$( echo "scale=0 ; $W / $ASPECT_RATIO" | bc )"</pre> </li> <li> Now the image called <code>IMG2005.png</code> can be sliced using ImageMagick’s convert command. The slices are stored into a subdirectory called <code>slices</code>, with file names like <code>IMG2005-1.jpg</code>, <code>IMG2005-2.jpg</code>, etc. <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4123dda9ef02'><button class='copyBtn' data-clipboard-target='#id4123dda9ef02' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>convert IMG2005.png -crop ${W}x${H2} \ -quality 100% -scene 0 slices/IMG2005-%d.jpg</pre> </li> </ol> <h2 id="script">Automating the Conversion</h2> <p> I wrote the following bash script, which incorporates the above computations. It slices all the images in a directory and saves the results to a second directory. </p> <div class='codeLabel unselectable' data-lt-active='false'>sliceImages</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id775f4c7a3c76'><button class='copyBtn' data-clipboard-target='#id775f4c7a3c76' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash function help { if [ "$1" ]; then echo "Error: $1"; fi echo " $(basename $0): slice all images in the given directory and place them into a specified directory, which will be created if required. " exit 1 } function setup { export ASPECT_RATIO=0.72 export W="$( identify -ping -format '%w' "$1" )" export H="$( identify -ping -format '%h' "$1" )" export H2="$( echo "scale=0 ; $W / $ASPECT_RATIO" | bc )" } function convert1 { FULLNAME=$(basename -- "$1") FILENAME="${FULLNAME%.*}" FILETYPE="${FULLNAME##*.}" convert "$1" \ -crop "${W}x${H2}" \ -quality 100% \ -scene 0 \ "$DIR_OUTPUT/$FILENAME-%d.png" } if [ -z "$1" ]; then help "No directory path for images to be converted was provided."; fi export DIR_INPUT="$( realpath $1 )" if [ -z "$2" ]; then help "No directory path for the image slices to be saved into was provided."; fi export DIR_OUTPUT="$( realpath $2 )" mkdir -p "$DIR_OUTPUT" find $DIR_INPUT -type f -exec file --mime-type {} \+ | awk -F: '{if ($2 ~/image\//) print $1}' | while read FILE; do setup "$FILE" echo "Slicing $FILE into ${W}x${H2} pixels" convert1 "$FILE" done</pre> <h2 id="limits">Overcoming ImageMagick Processing Limits</h2> <p> Some of the web pages that I needed to grab were quite long, which resulted in those images requiring more computational resources than the default Imagemagick configuration allows. This caused errors such as the following to appear: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id97eafd3ec4c5'>convert-im6.q16: no images defined `/mnt/c/images/slices/IMG1466-%d.png' @ error/convert.c/ConvertImageCommand/3229. convert-im6.q16: cache resources exhausted `/mnt/c/images/IMG1091.png' @ error/cache.c/OpenPixelCache/4095.</pre> <p> Imagemagick defines computational resources limits in <code>/etc/ImageMagick-6/policy.xml</code>. The default maximum memory is 256 KB, the default maximum allowable height is 16,000 pixels (16KP), and the default maximum area is 128M pixels. These values are defined by the following entries: </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/ImageMagick-6/policy.xml</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7a6deab9fe17'>&lt;policy domain="resource" name="memory" value="256MiB"/> &lt;policy domain="resource" name="height" value="16KP"/> &lt;policy domain="resource" name="area" value="128MP"/></pre> <p> I changed the maximum memory limit to 2 GB RAM, the maximum height limit to 10,000,000 pixels (10MP), and the maximum area limit to 2G pixels with these entries: </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/ImageMagick-6/policy.xml</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfee9235c6abf'>&lt;policy domain="resource" name="memory" value="2GiB"/> &lt;policy domain="resource" name="height" value="10MP"/> &lt;policy domain="resource" name="area" value="2GP"/></pre> <p> Alternatively, I could have simply commented out the limits, as shown in highlighted text below. </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/ImageMagick-6/policy.xml</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id55a7cae59976'><span class="bg_yellow">&lt;!--</span> &lt;policy domain="resource" name="memory" value="256MiB"/> &lt;policy domain="resource" name="height" value="16KP"/> &lt;policy domain="resource" name="area" value="128MP"/> <span class="bg_yellow">--></span></pre> <p> The largest web page to be sliced was converted to a very tall image, which was 83,703 pixels high. It was sliced into 40 images. </p> <h2 id="macro">Word Macro</h2> <p> A Word macro is also needed to insert the images into the currently open Word document in alphabetical order. I modified <a href='https://software-solutions-online.com/word-vba-insert-images/' target='_blank' rel='nofollow'>this one</a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft Word Macro</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide8b01e83f0eb'>Sub insertImages() Dim intResult As Integer Dim strPath As String Dim strFolderPath As String Dim objFSO As Object Dim objFolder As Object Dim objFile As Object Dim i As Integer intResult = Application.FileDialog(msoFileDialogFolderPicker).Show 'Check if user canceled the dialog If intResult <> 0 Then 'dispaly message box strFolderPath = Application.FileDialog(msoFileDialogFolderPicker).SelectedItems(1) 'Create an instance of the FileSystemObject Set objFSO = CreateObject("Scripting.FileSystemObject") 'Get the folder object Set objFolder = objFSO.GetFolder(strFolderPath) i = 1 'loops through each file in the directory and prints their names and path For Each objFile In objFolder.Files 'get file path strPath = objFile.Path 'insert the image Selection.InlineShapes.AddPicture FileName:= _ strPath, LinkToFile:=False, _ SaveWithDocument:=True Next objFile End If End Sub</pre> <h2 id="done">Done!</h2> <span style='font-size: 3em; float: right; margin-left: 5px;'>&#x1F601;</span> <p> Thanks to the above automation, I was able to deliver the Word documents containing the sliced web pages to my client soon after they were requested. </p> Ruby on Rails / Solidus for ECommerce 2022-07-22T00:00:00-04:00 https://mslinn.github.io/blog/2022/07/22/solidus <editor-fold intro> <p> The internet is an echo chamber. Influencers quote each other. Statistics are often cited without considering how relevant they might be. </p> <div class="pullQuote"> The business value of ecommerce platforms is defined by their integrations. </div> <h2 id="goal">My Goal</h2> <p> I decided to make a prototype ecommerce website for selling digital goods. The website should be quick to create, easy to maintain, and attractive to users. </p> </editor-fold> <editor-fold contenders> <h2 id="contenders">Contenders</h2> <p> Two of the most popular technologies for ecommerce websites are <a href='https://rubyonrails.org/' target='_blank' rel='nofollow'>Ruby on Rails</a> (Rails) and <a href='https://www.djangoproject.com/' target='_blank' rel='nofollow'>Django</a>, which is based on Python. Many comparisons of these frameworks have been written. I have worked with both of them. </p> <div class='quote'> Python is 7.5x as popular as Ruby (48% vs 7%). <br><br> Ruby on Rails is approximately as popular as Django (58% vs 55%) <br><br> <span style='font-style:normal;'> &nbsp;&ndash; From <a href='https://insights.stackoverflow.com/survey/2021#overview' rel='nofollow' target='_blank'>2021 Stack Overflow Developer Survey</a></span> </div> <div class='quote'> Overall, the net change for the month has the total number of websites using Rails growing by 2,242 and the total number of websites using Django growing by 338. <br><br> If we focus on the tendencies inside the web frameworks ecosystem (websites added vs websites dropped), we see that Rails has a 2.5% growth rate and 98.4% retention rate compared to a growth rate of 2.3% and retention rate of 98.3% for Django. <br><br> Despite the similar growth and retention rates, Rails currently has a small edge in overall popularity, holding 3.3% of the market share, while Django holds only 0.68%. <br><br> <span style='font-style:normal;'> &nbsp;&ndash; From <a href='https://keyua.org/blog/rails-vs-django-comparison/' rel='nofollow' target='_blank'>Django vs Rails in 2022 - Comparison Performance Frameworks</a></span> </div> <h2 id="integrations">Integrations Make the Difference</h2> <p> Most of the comparisons of Rails vs. Django do not consider integrations. That is unfortunate, because the business value of an ecommerce platform is largely dependent on how it is integrated with other services. Examples of desirable integrations include authentication and authorization mechanisms, payment processing, content management, etc. </p> <p> <a href='https://www.dictionary.com/browse/bespoke' target='_blank' rel='nofollow'>Bespoke</a> integrations are time-consuming, expensive to write, and expensive to maintain. Ecommerce platforms that provide quality integrations are more competitive. </p> </editor-fold> <editor-fold solidus> <h2 id="solidus">Solidus</h2> <div style="text-align: right;"> <picture> <source srcset="/blog/images/rails/solidus-io-logo-vector.webp" type="image/webp"> <source srcset="/blog/images/rails/solidus-io-logo-vector.png" type="image/png"> <img src="/blog/images/rails/solidus-io-logo-vector.png" class="right quartersize " /> </picture> </div> <p> <a href='https://solidus.io' target='_blank' rel='nofollow'>Solidus</a> provides a componentized approach to Ruby on Rails. It includes a <a href='https://guides.rubyonrails.org/engines.html' target='_blank' rel='nofollow'>Rails Engine</a>. Solidus is <a href='https://github.com/solidusio/solidus' target='_blank' rel='nofollow'>hosted on GitHub</a>. Solidus originated as a <a href='https://stembolt.com/blog/why-we-forked-spree-and-started-solidus/' target='_blank' rel='nofollow'>fork of Spree</a> (the linked website has an expired SSL certificate). </p> <p> <a href='https://solidus.io/integrations/' target='_blank' rel='nofollow'>Integrations</a> that interest me at this time include Payments (<a href='https://github.com/solidusio-contrib/solidus_paypal_commerce_platform' target='_blank' rel='nofollow'>PayPal</a> and <a href='https://github.com/solidusio/solidus_stripe' target='_blank' rel='nofollow'>Stripe</a>), Authentication (<a href='https://github.com/solidusio/solidus_auth_devise' target='_blank' rel='nofollow'>Devise</a> and <a href='https://github.com/boomerdigital/solidus_user_roles' target='_blank' rel='nofollow'>User Roles</a>), Content Management (<a href='https://github.com/solidusio-contrib/solidus_static_content' target='_blank' rel='nofollow'>Static Content</a>), <a href='https://github.com/karmakatahdin/solidus_seo' target='_blank' rel='nofollow'>SEO</a>, <a href='https://github.com/solidusio-contrib/solidus_sitemap' target='_blank' rel='nofollow'>Sitemap</a>, <a href='https://github.com/solidusio-contrib/solidus_klaviyo' target='_blank' rel='nofollow'>Klaviyo</a>, <a href='https://github.com/solidusio-contrib/solidus_subscriptions' target='_blank' rel='nofollow'>Subscriptions</a>, <a href='https://github.com/solidusio-contrib/solidus_digital' target='_blank' rel='nofollow'>Digital products</a>, <a href='https://github.com/solidusio-contrib/solidus_editor' target='_blank' rel='nofollow'>Editor</a>, and <a href='https://github.com/solidusio-contrib/solidus_social' target='_blank' rel='nofollow'>Social Authentication</a>. </p> <p class="alert rounded shadow"> This blog post is a work in progress. I will update it as I learn more. </p> <h2 id="frontEnd">Front End</h2> <p> It seems that <a href='https://github.com/solidusio/solidus/issues/580' target='_blank' rel='nofollow'>Solidus favors Bootstrap</a> as a frontend, which is fine with me. That said, <a href='https://github.com/solidusio/solidus/issues/3246' target='_blank' rel='nofollow'>work has been done to support React</a>. </p> <div class='quote'> In 2021 React.js surpassed jQuery as the most commonly used web framework (40% vs 34%). <br><br> <span style='font-style:normal;'> &nbsp;&ndash; From <a href='https://insights.stackoverflow.com/survey/2021#overview' rel='nofollow' target='_blank'>2021 Stack Overflow Developer Survey</a></span> </div> <p> I do not agree with calling React or jQuery &ldquo;web frameworks&rdquo;. The React website has a better descriptive phrase: <a href='https://reactjs.org/' target='_blank' rel='nofollow'>a JavaScript library for building user interfaces</a>. </p> <h2 id="poc">Proof of Concept</h2> <p> I decided to use Solidus on Ubuntu to make a <a href='https://www.merriam-webster.com/dictionary/proof%20of%20concept' target='_blank' rel='nofollow'>proof of concept</a> (POC) to evaluate the technology. </p> <p> The <a href='https://guides.solidus.io/developers/' target='_blank' rel='nofollow'>Solidus Developer&rsquo;s Guide</a> is helpful, but it was written for Mac users, using the SQLite database. SQLite is not suitable for production, and it is only somewhat PostgreSQL compatible. I felt that it would be more instructive to evaluate the proof of concept if it used the actual production database. </p> <p> Following are the steps that I followed to create the POC on Ubuntu using PostgreSQL. </p> <p> At the time of this writing, the most recently published Solidus gem only supported Rails v6.1.6.1. This post describes how to build a more recent gem that supports Rails 7. </p> <p> Regarding the front end, I will learn more about what makes sense to use with Solidus as I play with it. This blog post will be updated as I gain experience. </p> </editor-fold> <editor-fold packages> <h2 id="packages">Installing System Packages</h2> <p> I have previously discussed <a href='/jekyll/500-ruby-setup.html#rbenv'>installing Ruby</a>. You also need to install <code>Node.js</code> because Rails&rsquo; asset pipeline compiler requires a JavaScript runtime. The <code>Node.js</code> version does not matter. To install the remaining packages, type: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc4cb7baf847d'><button class='copyBtn' data-clipboard-target='#idc4cb7baf847d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install \ git imagemagick libpq-dev nodejs npm postgresql postgresql-contrib <span class='unselectable'>Reading package lists... Done Building dependency tree... Done Reading state information... Done postgresql is already the newest version (14+238). postgresql-contrib is already the newest version (14+238). Setting up liblqr-1-0:amd64 (0.4.2-2.1) ... Setting up libde265-0:amd64 (1.0.8-1) ... Setting up libjxr-tools (1.2~git20170615.f752187-5) ... Setting up libheif1:amd64 (1.12.0-2build1) ... Setting up libmagickcore-6.q16-6:amd64 (8:6.9.11.60+dfsg-1.3build2) ... Setting up libmagickwand-6.q16-6:amd64 (8:6.9.11.60+dfsg-1.3build2) ... Setting up libmagickcore-6.q16-6-extra:amd64 (8:6.9.11.60+dfsg-1.3build2) ... Setting up imagemagick-6.q16 (8:6.9.11.60+dfsg-1.3build2) ... update-alternatives: using /usr/bin/compare-im6.q16 to provide /usr/bin/compare (compare) in auto mode update-alternatives: using /usr/bin/compare-im6.q16 to provide /usr/bin/compare-im6 (compare-im6) in auto mode update-alternatives: using /usr/bin/animate-im6.q16 to provide /usr/bin/animate (animate) in auto mode update-alternatives: using /usr/bin/animate-im6.q16 to provide /usr/bin/animate-im6 (animate-im6) in auto mode update-alternatives: using /usr/bin/convert-im6.q16 to provide /usr/bin/convert (convert) in auto mode update-alternatives: using /usr/bin/convert-im6.q16 to provide /usr/bin/convert-im6 (convert-im6) in auto mode update-alternatives: using /usr/bin/composite-im6.q16 to provide /usr/bin/composite (composite) in auto mode update-alternatives: using /usr/bin/composite-im6.q16 to provide /usr/bin/composite-im6 (composite-im6) in auto mode update-alternatives: using /usr/bin/conjure-im6.q16 to provide /usr/bin/conjure (conjure) in auto mode update-alternatives: using /usr/bin/conjure-im6.q16 to provide /usr/bin/conjure-im6 (conjure-im6) in auto mode update-alternatives: using /usr/bin/import-im6.q16 to provide /usr/bin/import (import) in auto mode update-alternatives: using /usr/bin/import-im6.q16 to provide /usr/bin/import-im6 (import-im6) in auto mode update-alternatives: using /usr/bin/identify-im6.q16 to provide /usr/bin/identify (identify) in auto mode update-alternatives: using /usr/bin/identify-im6.q16 to provide /usr/bin/identify-im6 (identify-im6) in auto mode update-alternatives: using /usr/bin/stream-im6.q16 to provide /usr/bin/stream (stream) in auto mode update-alternatives: using /usr/bin/stream-im6.q16 to provide /usr/bin/stream-im6 (stream-im6) in auto mode update-alternatives: using /usr/bin/display-im6.q16 to provide /usr/bin/display (display) in auto mode update-alternatives: using /usr/bin/display-im6.q16 to provide /usr/bin/display-im6 (display-im6) in auto mode update-alternatives: using /usr/bin/montage-im6.q16 to provide /usr/bin/montage (montage) in auto mode update-alternatives: using /usr/bin/montage-im6.q16 to provide /usr/bin/montage-im6 (montage-im6) in auto mode update-alternatives: using /usr/bin/mogrify-im6.q16 to provide /usr/bin/mogrify (mogrify) in auto mode update-alternatives: using /usr/bin/mogrify-im6.q16 to provide /usr/bin/mogrify-im6 (mogrify-im6) in auto mode Setting up imagemagick (8:6.9.11.60+dfsg-1.3build2) ... Processing triggers for fontconfig (2.13.1-4.2ubuntu5) ... Processing triggers for desktop-file-utils (0.26-1ubuntu3) ... Processing triggers for hicolor-icon-theme (0.17-2) ... Processing triggers for gnome-menus (3.36.0-1ubuntu3) ... Processing triggers for libc-bin (2.35-0ubuntu3) ... Processing triggers for man-db (2.10.2-1) ... Processing triggers for mailcap (3.70+nmu1ubuntu1) ... </span></pre> <!-- <p> Make <code>nodejs</code> available as <code>node</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6bbebb3de38d'><button class='copyBtn' data-clipboard-target='#id6bbebb3de38d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo ln -sf /usr/bin/nodejs /usr/local/bin/node</pre> --> <p> The most recent version of Ruby at the time of this writing was 3.1.2, however it would not build on my Ubuntu 22.04 system. The most recent version of Ruby that I was able to build was 3.1.0. To install Ruby 3.1.0 and make it default, type: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id09349a27097c'><button class='copyBtn' data-clipboard-target='#id09349a27097c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>rbenv install ruby 3.1.0 <span class='unselectable'>$ </span>rbenv global 3.1.0</pre> </editor-fold> <editor-fold gem_install> <h2 id="geminst">Install Necessary Gems</h2> <p> Some of the required gems are built from a long compile. If you just installed Ruby the compilation will take a long time and use 100% of the CPU. What&rsquo;s more, if you are using <code>ssh</code> to connect to the machine doing the compile, the connection might drop for various reasons. Depending on the hardware, doing this for the first time might take hours. </p> <p> I ran the build under <a href='https://linux.die.net/man/1/nice' target='_blank' rel='nofollow'><code>nice</code></a> so other processes could continue unimpeded. <code>Nice</code> only throttles the process it manages when other processes request a time slice. If the system load is low then <code>nice</code> does not throttle the process that it supervises. </p> <p> I also used <a href='https://linux.die.net/man/1/nohup' target='_blank' rel='nofollow'><code>nohup</code></a> to continue the process if the <code>ssh</code> connection dropped. The <span class="bg_yellow">&</span> at the end of the command line below causes the command-line process to run asynchronously, detached from the console foreground. </p> <p> Just for interest&rsquo;s sake, I also measured how long the entire build took with the <a href='https://linux.die.net/man/1/time' target='_blank' rel='nofollow'><code>time</code></a> command wrapper. </p> <p> These 3 bash command wrappers, <code>nohup</code>, <code>time</code> and <code>nice</code> are often used together like this to start a long, CPU-intensive process that won&rsquo;t get interrupted and won&rsquo;t overload the system, while providing an indication of how much time was required to complete the process. </p> <p> Because <code>nohup</code> swallows the output and saves it to <code>nohup.out</code>, <code>tee nohup.out</code> displays the build as it progresses. The <code>1>&2</code> incantation causes both the <code>gem install</code> command&rsquo;s <code>STDOUT</code> and <code>STDERR</code> output streams to be piped into <a href='https://man7.org/linux/man-pages/man1/tee.1.html' target='_blank' rel='nofollow'><code>tee</code></a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4df5c5da8df4'><button class='copyBtn' data-clipboard-target='#id4df5c5da8df4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span><span class="bg_yellow">nohup time nice</span> gem install rails 1>&2 | tee nohup.out <span class="bg_yellow">&</span> <span class='unselectable'>Fetching method_source-1.0.0.gem Fetching zeitwerk-2.6.0.gem Fetching activesupport-7.0.3.1.gem Fetching crass-1.0.6.gem Fetching loofah-2.18.0.gem Fetching rails-html-sanitizer-1.4.3.gem Fetching rails-dom-testing-2.0.3.gem Fetching rack-2.2.4.gem Fetching erubi-1.10.0.gem Fetching builder-3.2.4.gem Fetching actionview-7.0.3.1.gem Fetching actionpack-7.0.3.1.gem Fetching railties-7.0.3.1.gem Fetching mini_mime-1.1.2.gem Fetching marcel-1.0.2.gem Fetching activemodel-7.0.3.1.gem Fetching activerecord-7.0.3.1.gem Fetching globalid-1.0.0.gem Fetching activejob-7.0.3.1.gem Fetching activestorage-7.0.3.1.gem Fetching actiontext-7.0.3.1.gem Fetching mail-2.7.1.gem Fetching actionmailer-7.0.3.1.gem Fetching actionmailbox-7.0.3.1.gem Fetching rails-7.0.3.1.gem Fetching websocket-extensions-0.1.5.gem Fetching websocket-driver-0.7.5.gem Fetching nio4r-2.5.8.gem Fetching actioncable-7.0.3.1.gem Successfully installed zeitwerk-2.6.0 Successfully installed method_source-1.0.0 Successfully installed activesupport-7.0.3.1 Successfully installed crass-1.0.6 Successfully installed loofah-2.18.0 Successfully installed rails-html-sanitizer-1.4.3 Successfully installed rails-dom-testing-2.0.3 Successfully installed rack-2.2.4 Successfully installed erubi-1.10.0 Successfully installed builder-3.2.4 Successfully installed actionview-7.0.3.1 Successfully installed actionpack-7.0.3.1 Successfully installed railties-7.0.3.1 Successfully installed mini_mime-1.1.2 Successfully installed marcel-1.0.2 Successfully installed activemodel-7.0.3.1 Successfully installed activerecord-7.0.3.1 Successfully installed globalid-1.0.0 Successfully installed activejob-7.0.3.1 Successfully installed activestorage-7.0.3.1 Successfully installed actiontext-7.0.3.1 Successfully installed mail-2.7.1 Successfully installed actionmailer-7.0.3.1 Successfully installed actionmailbox-7.0.3.1 Successfully installed websocket-extensions-0.1.5 Building native extensions. This could take a while... Successfully installed websocket-driver-0.7.5 Building native extensions. This could take a while... Successfully installed nio4r-2.5.8 Successfully installed actioncable-7.0.3.1 Successfully installed rails-7.0.3.1 Parsing documentation for zeitwerk-2.6.0 Installing ri documentation for zeitwerk-2.6.0 Parsing documentation for method_source-1.0.0 Installing ri documentation for method_source-1.0.0 Parsing documentation for activesupport-7.0.3.1 Installing ri documentation for activesupport-7.0.3.1 Parsing documentation for crass-1.0.6 Installing ri documentation for crass-1.0.6 Parsing documentation for loofah-2.18.0 Installing ri documentation for loofah-2.18.0 Parsing documentation for rails-html-sanitizer-1.4.3 Installing ri documentation for rails-html-sanitizer-1.4.3 Parsing documentation for rails-dom-testing-2.0.3 Installing ri documentation for rails-dom-testing-2.0.3 Parsing documentation for rack-2.2.4 Installing ri documentation for rack-2.2.4 Parsing documentation for erubi-1.10.0 Installing ri documentation for erubi-1.10.0 Parsing documentation for builder-3.2.4 Installing ri documentation for builder-3.2.4 Parsing documentation for actionview-7.0.3.1 Installing ri documentation for actionview-7.0.3.1 Parsing documentation for actionpack-7.0.3.1 Installing ri documentation for actionpack-7.0.3.1 Parsing documentation for railties-7.0.3.1 Installing ri documentation for railties-7.0.3.1 Parsing documentation for mini_mime-1.1.2 Installing ri documentation for mini_mime-1.1.2 Parsing documentation for marcel-1.0.2 Installing ri documentation for marcel-1.0.2 Parsing documentation for activemodel-7.0.3.1 Installing ri documentation for activemodel-7.0.3.1 Parsing documentation for activerecord-7.0.3.1 Installing ri documentation for activerecord-7.0.3.1 Parsing documentation for globalid-1.0.0 Installing ri documentation for globalid-1.0.0 Parsing documentation for activejob-7.0.3.1 Installing ri documentation for activejob-7.0.3.1 Parsing documentation for activestorage-7.0.3.1 Installing ri documentation for activestorage-7.0.3.1 Parsing documentation for actiontext-7.0.3.1 Installing ri documentation for actiontext-7.0.3.1 Parsing documentation for mail-2.7.1 Installing ri documentation for mail-2.7.1 Parsing documentation for actionmailer-7.0.3.1 Installing ri documentation for actionmailer-7.0.3.1 Parsing documentation for actionmailbox-7.0.3.1 Installing ri documentation for actionmailbox-7.0.3.1 Parsing documentation for websocket-extensions-0.1.5 Installing ri documentation for websocket-extensions-0.1.5 Parsing documentation for websocket-driver-0.7.5 Installing ri documentation for websocket-driver-0.7.5 Parsing documentation for nio4r-2.5.8 Installing ri documentation for nio4r-2.5.8 Parsing documentation for actioncable-7.0.3.1 Installing ri documentation for actioncable-7.0.3.1 Parsing documentation for rails-7.0.3.1 Installing ri documentation for rails-7.0.3.1 Done installing documentation for zeitwerk, method_source, activesupport, crass, loofah, rails-html-sanitizer, rails-dom-testing, rack, erubi, builder, actionview, actionpack, railties, mini_mime, marcel, activemodel, activerecord, globalid, activejob, activestorage, actiontext, mail, actionmailer, actionmailbox, websocket-extensions, websocket-driver, nio4r, actioncable, rails after 44 seconds 29 gems installed<br> real XmX.XXXs user XmX.XXXs sys XmX.XXX </span></pre> <p> If you&rsquo;re using <code>rbenv</code>, you&rsquo;ll need to run the following command to make the rails executable available: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc4221b9b4e8e'><button class='copyBtn' data-clipboard-target='#idc4221b9b4e8e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>rbenv rehash</pre> </editor-fold> <editor-fold postgres> <h2 id="postgres">Configure and Start PostgreSQL</h2> <p> PostgreSQL v14 is the version that is installed by Ubuntu 22.04. </p> <p> Edit <a href='https://www.postgresql.org/docs/current/config-setting.html#CONFIG-SETTING-CONFIGURATION-FILE' target='_blank' rel='nofollow'><code>/etc/postgresql/14/main/postgresql.conf</code></a> and change the following line: </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/postgresql/14/main/postgresql.conf</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id35a3d3c8ae8b'>#listen_addresses = 'localhost'</pre> <p> To: </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/postgresql/14/main/postgresql.conf</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idafad62f434a2'>listen_addresses = '*'</pre> <p> I edited <a href='https://www.postgresql.org/docs/current/auth-pg-hba-conf.html' target='_blank' rel='nofollow'><code>/etc/postgresql/14/main/pg_hba.conf</code></a> and changed the <code>METHOD</code> to <code>trust</code>. This file is read when the database starts and when it receives a SIGHUP signal. The <code>trust</code> value is suitable when security is not an issue, for example, when developing. I removed most of the comments from the file for clarity. </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/postgresql/14/main/pg_hba.conf</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb70dacf737ce'><button class='copyBtn' data-clipboard-target='#idb70dacf737ce' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># TYPE DATABASE USER ADDRESS METHOD # "local" is for Unix domain socket connections only local all postgres trust local all all trust # IPv4 local connections: host all all 127.0.0.1/32 trust # IPv6 local connections: host all all ::1/128 trust</pre> <p> Now I started PostgreSQL v14 (this is the version that is installed by Ubuntu 22.04). </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb8d99cf37633'><button class='copyBtn' data-clipboard-target='#idb8d99cf37633' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo pg_ctlcluster 14 main start <span class='unselectable'>$ </span>sudo service postgresql restart <span class='unselectable'>* Restarting PostgreSQL 14 database server ...done. </span></pre> <p> Next I created a Postgres user with the same name as the Rails application. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id31645b1cea7d'><button class='copyBtn' data-clipboard-target='#id31645b1cea7d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo -u postgres createuser -s solidus_poc</pre> <p> The new PostgreSQL user needed a password. This example uses the <a href='https://www.postgresql.org/docs/current/app-psql.html' target='_blank' rel='nofollow'><code>\password</code> meta-command</a>, but you could also use the <a href='https://www.postgresql.org/docs/8.0/sql-alteruser.html' target='_blank' rel='nofollow'><code>ALTER USER</code> command</a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idaf935934566a'><button class='copyBtn' data-clipboard-target='#idaf935934566a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo -u postgres psql -c '\password solidus_poc' <span class='unselectable'>Enter new password for user "solidus_poc": Enter it again: </span></pre> </editor-fold> <editor-fold Create> <h2>Create a New Rails Project</h2> <p> Use the <code>rails new</code> command to create a new Rails project. This is the help message: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0430bed6fc4f'><button class='copyBtn' data-clipboard-target='#id0430bed6fc4f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>rails new -h <span class='unselectable'>Usage: rails new APP_PATH [options]<br/> Options: [--skip-namespace], [--no-skip-namespace] # Skip namespace (affects only isolated engines) [--skip-collision-check], [--no-skip-collision-check] # Skip collision check -r, [--ruby=PATH] # Path to the Ruby binary of your choice # Default: /home/mslinn/.rbenv/versions/3.1.0/bin/ruby -m, [--template=TEMPLATE] # Path to some application template (can be a filesystem path or URL) -d, [--database=DATABASE] # Preconfigure for selected database (options: mysql/postgresql/sqlite3/oracle/sqlserver/jdbcmysql/jdbcsqlite3/jdbcpostgresql/jdbc) # Default: sqlite3 -G, [--skip-git], [--no-skip-git] # Skip .gitignore file [--skip-keeps], [--no-skip-keeps] # Skip source control .keep files -M, [--skip-action-mailer], [--no-skip-action-mailer] # Skip Action Mailer files [--skip-action-mailbox], [--no-skip-action-mailbox] # Skip Action Mailbox gem [--skip-action-text], [--no-skip-action-text] # Skip Action Text gem -O, [--skip-active-record], [--no-skip-active-record] # Skip Active Record files [--skip-active-job], [--no-skip-active-job] # Skip Active Job [--skip-active-storage], [--no-skip-active-storage] # Skip Active Storage files -C, [--skip-action-cable], [--no-skip-action-cable] # Skip Action Cable files -A, [--skip-asset-pipeline], [--no-skip-asset-pipeline] # Indicates when to generate skip asset pipeline -a, [--asset-pipeline=ASSET_PIPELINE] # Choose your asset pipeline [options: sprockets (default), propshaft] # Default: sprockets -J, [--skip-javascript], [--no-skip-javascript] # Skip JavaScript files [--skip-hotwire], [--no-skip-hotwire] # Skip Hotwire integration [--skip-jbuilder], [--no-skip-jbuilder] # Skip jbuilder gem -T, [--skip-test], [--no-skip-test] # Skip test files [--skip-system-test], [--no-skip-system-test] # Skip system test files [--skip-bootsnap], [--no-skip-bootsnap] # Skip bootsnap gem [--dev], [--no-dev] # Set up the application with Gemfile pointing to your Rails checkout [--edge], [--no-edge] # Set up the application with Gemfile pointing to Rails repository --master, [--main], [--no-main] # Set up the application with Gemfile pointing to Rails repository main branch [--rc=RC] # Path to file containing extra configuration options for rails command [--no-rc], [--no-no-rc] # Skip loading of extra configuration options from .railsrc file [--api], [--no-api] # Preconfigure smaller stack for API only apps [--minimal], [--no-minimal] # Preconfigure a minimal rails app -j, [--javascript=JAVASCRIPT] # Choose JavaScript approach [options: importmap (default), webpack, esbuild, rollup] # Default: importmap -c, [--css=CSS] # Choose CSS processor [options: tailwind, bootstrap, bulma, postcss, sass... check https://github.com/rails/cssbundling-rails] -B, [--skip-bundle], [--no-skip-bundle] # Don&#39;t run bundle install<br/> Runtime options: -f, [--force] # Overwrite files that already exist -p, [--pretend], [--no-pretend] # Run but do not make any changes -q, [--quiet], [--no-quiet] # Suppress status output -s, [--skip], [--no-skip] # Skip files that already exist<br/> Rails options: -h, [--help], [--no-help] # Show this help message and quit -v, [--version], [--no-version] # Show Rails version number and quit<br/> Description: The &#39;rails new&#39; command creates a new Rails application with a default directory structure and configuration at the path you specify.<br/> You can specify extra command-line arguments to be used every time &#39;rails new&#39; runs in the .railsrc configuration file in your home directory, or in $XDG_CONFIG_HOME/rails/railsrc if XDG_CONFIG_HOME is set.<br/> Note that the arguments specified in the .railsrc file don&#39;t affect the defaults values shown above in this help message.<br/> Example: rails new ~/Code/Ruby/weblog<br/> This generates a skeletal Rails installation in ~/Code/Ruby/weblog. </span></pre> <p> I created a new Rails project called <code>solidus_poc</code>, in a new directory called <code>solidus_poc</code>, under the directory pointed to by the <code>$work</code> environment variable. Note that I specified that the <a href='https://getbootstrap.com/' target='_blank' rel='nofollow'>Bootstrap frontend</a> would be used. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id769fa7870f55'><button class='copyBtn' data-clipboard-target='#id769fa7870f55' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cd $work/rails <span class='unselectable'>$ </span>rails new -d postgresql -c bootstrap solidus_poc <span class='unselectable'>exist create README.md create Rakefile identical .ruby-version create config.ru create .gitignore create .gitattributes create Gemfile run git init from &quot;.&quot; Reinitialized existing Git repository in /var/work/rails/solidus_poc/.git/ create app create app/assets/config/manifest.js create app/assets/stylesheets/application.css create app/channels/application_cable/channel.rb create app/channels/application_cable/connection.rb create app/controllers/application_controller.rb create app/helpers/application_helper.rb create app/jobs/application_job.rb create app/mailers/application_mailer.rb create app/models/application_record.rb create app/views/layouts/application.html.erb create app/views/layouts/mailer.html.erb create app/views/layouts/mailer.text.erb create app/assets/images create app/assets/images/.keep create app/controllers/concerns/.keep create app/models/concerns/.keep create bin create bin/rails create bin/rake create bin/setup create config create config/routes.rb create config/application.rb create config/environment.rb create config/cable.yml create config/puma.rb create config/storage.yml create config/environments create config/environments/development.rb create config/environments/production.rb create config/environments/test.rb create config/initializers create config/initializers/assets.rb create config/initializers/content_security_policy.rb create config/initializers/cors.rb create config/initializers/filter_parameter_logging.rb create config/initializers/inflections.rb create config/initializers/new_framework_defaults_7_0.rb create config/initializers/permissions_policy.rb create config/locales create config/locales/en.yml create config/master.key append .gitignore create config/boot.rb create config/database.yml create db create db/seeds.rb create lib create lib/tasks create lib/tasks/.keep create lib/assets create lib/assets/.keep create log create log/.keep create public create public/404.html create public/422.html create public/500.html create public/apple-touch-icon-precomposed.png create public/apple-touch-icon.png create public/favicon.ico create public/robots.txt create tmp create tmp/.keep create tmp/pids create tmp/pids/.keep create tmp/cache create tmp/cache/assets create vendor create vendor/.keep create test/fixtures/files create test/fixtures/files/.keep create test/controllers create test/controllers/.keep create test/mailers create test/mailers/.keep create test/models create test/models/.keep create test/helpers create test/helpers/.keep create test/integration create test/integration/.keep create test/channels/application_cable/connection_test.rb create test/test_helper.rb create test/system create test/system/.keep create test/application_system_test_case.rb create storage create storage/.keep create tmp/storage create tmp/storage/.keep remove config/initializers/cors.rb remove config/initializers/new_framework_defaults_7_0.rb run bundle install Fetching gem metadata from https://rubygems.org/........... Resolving dependencies.......... Using rake 13.0.6 Using concurrent-ruby 1.1.10 Using mini_mime 1.1.2 Using strscan 3.0.3 Using public_suffix 4.0.7 Using racc 1.6.0 Using crass 1.0.6 Using bundler 2.3.18 Using nio4r 2.5.8 Using matrix 0.4.2 Using marcel 1.0.2 Using digest 3.1.0 Using thor 1.2.1 Using zeitwerk 2.6.0 Using builder 3.2.4 Using erubi 1.10.0 Using bindex 0.8.1 Using msgpack 1.5.3 Using rubyzip 2.3.2 Using websocket-extensions 0.1.5 Using i18n 1.12.0 Using method_source 1.0.0 Using childprocess 4.1.0 Using minitest 5.16.2 Using timeout 0.3.0 Using io-console 0.5.11 Using bootsnap 1.12.0 Using websocket-driver 0.7.5 Using puma 5.6.4 Using websocket 1.2.9 Using regexp_parser 2.5.0 Using tzinfo 2.0.4 Using mail 2.7.1 Using activesupport 7.0.3.1 Using nokogiri 1.13.7 (x86_64-linux) Using globalid 1.0.0 Using activemodel 7.0.3.1 Using pg 1.4.1 Using loofah 2.18.0 Using net-protocol 0.1.3 Using rails-html-sanitizer 1.4.3 Using net-imap 0.2.3 Using net-pop 0.1.1 Using net-smtp 0.3.1 Using activejob 7.0.3.1 Using activerecord 7.0.3.1 Using xpath 3.2.0 Using reline 0.3.1 Using addressable 2.8.0 Using irb 1.4.1 Using rack 2.2.4 Using rexml 3.2.5 Using rack-test 2.0.2 Using selenium-webdriver 4.3.0 Using capybara 3.37.1 Using sprockets 4.1.1 Using debug 1.6.1 Using rails-dom-testing 2.0.3 Using webdrivers 5.0.0 Using actionview 7.0.3.1 Using actionpack 7.0.3.1 Using jbuilder 2.11.5 Using actioncable 7.0.3.1 Using activestorage 7.0.3.1 Using actionmailer 7.0.3.1 Using actionmailbox 7.0.3.1 Using sprockets-rails 3.4.2 Using railties 7.0.3.1 Using actiontext 7.0.3.1 Using jsbundling-rails 1.0.3 Using rails 7.0.3.1 Using web-console 4.2.0 Using cssbundling-rails 1.1.1 Using turbo-rails 1.1.1 Fetching stimulus-rails 1.1.0 Installing stimulus-rails 1.1.0 Bundle complete! 16 Gemfile dependencies, 75 gems now installed. Use `bundle info [gemname]` to see where a bundled gem is installed. run bundle binstubs bundler rails javascript:install:esbuild Compile into app/assets/builds create app/assets/builds create app/assets/builds/.keep append app/assets/config/manifest.js append .gitignore append .gitignore Add JavaScript include tag in application layout insert app/views/layouts/application.html.erb Create default entrypoint in app/javascript/application.js create app/javascript create app/javascript/application.js Add default package.json create package.json Add default Procfile.dev create Procfile.dev Ensure foreman is installed run gem install foreman from &quot;.&quot; Error loading RubyGems plugin &quot;/home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/plugins/gem-release_plugin.rb&quot;: cannot load such file -- gem/release (LoadError) Successfully installed foreman-0.87.2 Parsing documentation for foreman-0.87.2 Done installing documentation for foreman after 1 seconds 1 gem installed Add bin/dev to start foreman create bin/dev Install esbuild run yarn add esbuild from &quot;.&quot; yarn add v1.22.19 info No lockfile found. [1/4] Resolving packages... [2/4] Fetching packages... [3/4] Linking dependencies... [4/4] Building fresh packages... success Saved lockfile. success Saved 2 new dependencies. info Direct dependencies &#9492;&#9472; esbuild@0.14.49 info All dependencies &#9500;&#9472; esbuild-linux-64@0.14.49 &#9492;&#9472; esbuild@0.14.49 Done in 2.81s. Add build script run npm set-script build &quot;esbuild app/javascript/*.* --bundle --sourcemap --outdir=app/assets/builds --public-path=assets&quot; from &quot;.&quot; npm WARN set-script set-script is deprecated, use `npm pkg set scripts.scriptname=&quot;cmd&quot; instead. run yarn build from &quot;.&quot; yarn run v1.22.19 $ esbuild app/javascript/*.* --bundle --sourcemap --outdir=app/assets/builds --public-path=assets<br/> app/assets/builds/application.js 62b app/assets/builds/application.js.map 93b<br/> Done in 0.19s. rails turbo:install stimulus:install Import Turbo append app/javascript/application.js Install Turbo run yarn add @hotwired/turbo-rails from &quot;.&quot; yarn add v1.22.19 [1/4] Resolving packages... [2/4] Fetching packages... [3/4] Linking dependencies... [4/4] Building fresh packages... success Saved lockfile. success Saved 3 new dependencies. info Direct dependencies &#9492;&#9472; @hotwired/turbo-rails@7.1.3 info All dependencies &#9500;&#9472; @hotwired/turbo-rails@7.1.3 &#9500;&#9472; @hotwired/turbo@7.1.0 &#9492;&#9472; @rails/actioncable@7.0.3 Done in 1.92s. Run turbo:install:redis to switch on Redis and use it in development for turbo streams Create controllers directory create app/javascript/controllers create app/javascript/controllers/index.js create app/javascript/controllers/application.js create app/javascript/controllers/hello_controller.js Import Stimulus controllers append app/javascript/application.js Install Stimulus run yarn add @hotwired/stimulus from &quot;.&quot; yarn add v1.22.19 [1/4] Resolving packages... [2/4] Fetching packages... [3/4] Linking dependencies... [4/4] Building fresh packages... success Saved lockfile. success Saved 1 new dependency. info Direct dependencies &#9492;&#9472; @hotwired/stimulus@3.1.0 info All dependencies &#9492;&#9472; @hotwired/stimulus@3.1.0 Done in 1.66s. rails css:install:bootstrap Build into app/assets/builds exist app/assets/builds identical app/assets/builds/.keep File unchanged! The supplied flag value not found! app/assets/config/manifest.js Stop linking stylesheets automatically gsub app/assets/config/manifest.js File unchanged! The supplied flag value not found! .gitignore File unchanged! The supplied flag value not found! .gitignore Remove app/assets/stylesheets/application.css so build output can take over remove app/assets/stylesheets/application.css Add stylesheet link tag in application layout File unchanged! The supplied flag value not found! app/views/layouts/application.html.erb append Procfile.dev Add bin/dev to start foreman identical bin/dev Install Bootstrap with Bootstrap Icons and Popperjs/core create app/assets/stylesheets/application.bootstrap.scss run yarn add sass bootstrap bootstrap-icons @popperjs/core from &quot;.&quot; yarn add v1.22.19 [1/4] Resolving packages... [2/4] Fetching packages... [3/4] Linking dependencies... [4/4] Building fresh packages... success Saved lockfile. success Saved 20 new dependencies. info Direct dependencies &#9500;&#9472; @popperjs/core@2.11.5 &#9500;&#9472; bootstrap-icons@1.9.1 &#9500;&#9472; bootstrap@5.2.0 &#9492;&#9472; sass@1.53.0 info All dependencies &#9500;&#9472; @popperjs/core@2.11.5 &#9500;&#9472; anymatch@3.1.2 &#9500;&#9472; binary-extensions@2.2.0 &#9500;&#9472; bootstrap-icons@1.9.1 &#9500;&#9472; bootstrap@5.2.0 &#9500;&#9472; braces@3.0.2 &#9500;&#9472; chokidar@3.5.3 &#9500;&#9472; fill-range@7.0.1 &#9500;&#9472; glob-parent@5.1.2 &#9500;&#9472; immutable@4.1.0 &#9500;&#9472; is-binary-path@2.1.0 &#9500;&#9472; is-extglob@2.1.1 &#9500;&#9472; is-glob@4.0.3 &#9500;&#9472; is-number@7.0.0 &#9500;&#9472; normalize-path@3.0.0 &#9500;&#9472; picomatch@2.3.1 &#9500;&#9472; readdirp@3.6.0 &#9500;&#9472; sass@1.53.0 &#9500;&#9472; source-map-js@1.0.2 &#9492;&#9472; to-regex-range@5.0.1 Done in 7.92s. insert config/initializers/assets.rb Appending Bootstrap JavaScript import to default entry point append app/javascript/application.js Add build:css script run npm set-script build:css &quot;sass ./app/assets/stylesheets/application.bootstrap.scss:./app/assets/builds/application.css --no-source-map --load-path=node_modules&quot; from &quot;.&quot; npm WARN set-script set-script is deprecated, use `npm pkg set scripts.scriptname=&quot;cmd&quot; instead. run yarn build:css from &quot;.&quot; yarn run v1.22.19 $ sass ./app/assets/stylesheets/application.bootstrap.scss:./app/assets/builds/application.css --no-source-map --load-path=node_modules Done in 3.92s. </span></pre> <p> Make the <code>solidus_poc</code> directory current. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id47d8d0fb9f18'><button class='copyBtn' data-clipboard-target='#id47d8d0fb9f18' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cd solidus_poc</pre> <p> The <a href='https://guides.rubyonrails.org/getting_started.html#creating-the-blog-application' target='_blank' rel='nofollow'>Rails documentation</a> describes the contents of the generated files in detail. </p> <h3 id="commit">Commit to a New Git Repo</h3> <p> This is a good time to commit the work to a git repo so far. I used the <a href='https://github.com/cli/cli' target='_blank' rel='nofollow'>GitHub CLI</a> to create a repo called <code>solidus_poc</code>, and also set the remote origin in <code>.git/config</code> within the current directory. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id567d09e3ded5'><button class='copyBtn' data-clipboard-target='#id567d09e3ded5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>gh repo create solidus_poc --public --source=. --remote=upstream <span class='unselectable'>✓ Created repository mslinn/solidus_poc on GitHub </span> <span class='unselectable'>$ </span>git add -A <span class='unselectable'>$ </span>git commit -m "Initial commit" <span class='unselectable'>$ </span>git push -u origin master</pre> </editor-fold> <editor-fold gem> <h2 id="buidgem">Meet the Solidus Gems</h2> <p> The most recent release of Solidus (v3.1.7) only supports Rails 6.1.x, which was released in January 2021 (18 months ago, as of this writing). However, support for Rails v7.0.3 was merged into Solidus on 2022-04-21. The <a href='https://github.com/solidusio/solidus/pull/4220#issuecomment-1186723094' target='_blank' rel='nofollow'>Solidus developers said</a> the next release might occur in a few weeks. I did not want to wait, so I decided to build the gems from <code>master</code>. </p> <p> Instead of manually running <code>git clone</code> and pulling down a copy of the repository, I used <code>bundle</code>&rsquo;s ability to build a local copy of a gem and all its dependencies directly from a git repository. </p> <p> Several gems are defined in this project. All of them are automatically built: </p> <ul> <li><code>solidus.gemspec</code></li> <li><code>solidus_api.gemspec</code></li> <li><code>solidus_backend.gemspec</code></li> <li><code>solidus_core.gemspec</code></li> <li><code>solidus_frontend.gemspec</code></li> <li><code>solidus_sample.gemspec</code></li> </ul> <h2 id="quickbuild">How I Built the Solidus Gems</h2> <p> Building the gems is quick. The freshly built gems are automatically installed locally, along with all their dependencies. </p> <p> I manually added the following line to the project <code>Gemfile</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Gemfile</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddb56198ac417'><button class='copyBtn' data-clipboard-target='#iddb56198ac417' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>gem 'solidus', git: 'https://github.com/solidusio/solidus'</pre> <p> Then I ran <code>bundle install</code>, taking care not to bog down the local machine doing so by using the <code>nice</code> command. This caused the <code>solidus</code> gems to be built from the <code>master</code> branch of the GitHub project. Those gems, and their dependencies, were automatically installed: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfb6c6803fbcc'><button class='copyBtn' data-clipboard-target='#idfb6c6803fbcc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>nohup nice bundle install 1>&2 | tee nohup.out & <span class='unselectable'>Fetching https://github.com/solidusio/solidus Fetching gem metadata from https://rubygems.org/......... Resolving dependencies.... Using rake 13.0.6 Using minitest 5.16.2 Using rack 2.2.4 Using erubi 1.10.0 Using racc 1.6.0 Using digest 3.1.0 Using nio4r 2.5.8 Using concurrent-ruby 1.1.10 Using builder 3.2.4 Using websocket-extensions 0.1.5 Using marcel 1.0.2 Using mini_mime 1.1.2 Using bundler 2.3.18 Using timeout 0.3.0 Using strscan 3.0.3 Using public_suffix 4.0.7 Using execjs 2.8.1 Using bindex 0.8.1 Using msgpack 1.5.3 Using crass 1.0.6 Using cancancan 3.4.0 Using method_source 1.0.0 Using io-console 0.5.11 Using zeitwerk 2.6.0 Using matrix 0.4.2 Using regexp_parser 2.5.0 Using childprocess 4.1.0 Using climate_control 0.2.0 Using thor 1.2.1 Using ffi 1.15.5 Using tilt 2.0.11 Using mini_magick 4.11.0 Using kaminari-core 1.2.2 Using mime-types-data 3.2022.0105 Using omnes 0.2.2 Using pg 1.4.1 Using stringio 3.0.2 Using rexml 3.2.5 Using rubyzip 2.3.2 Using websocket 1.2.9 Using state_machines 0.5.0 Using truncate_html 0.9.3 Using rack-test 2.0.2 Using nokogiri 1.13.7 (x86_64-linux) Using puma 5.6.4 Using i18n 1.12.0 Using tzinfo 2.0.4 Using sprockets 4.1.1 Using websocket-driver 0.7.5 Using mail 2.7.1 Using net-protocol 0.1.3 Using addressable 2.8.0 Using autoprefixer-rails 10.4.7.0 Using bootsnap 1.12.0 Using reline 0.3.1 Using terrapin 0.6.0 Using ruby-vips 2.1.4 Using sassc 2.4.0 Using mime-types 3.4.1 Using psych 4.0.4 Using selenium-webdriver 4.3.0 Using loofah 2.18.0 Using irb 1.4.1 Using money 6.16.0 Using activesupport 7.0.3.1 Using handlebars_assets 0.23.9 Using net-imap 0.2.3 Using net-pop 0.1.1 Using net-smtp 0.3.1 Using xpath 3.2.0 Using image_processing 1.12.2 Using webdrivers 5.0.0 Using rails-html-sanitizer 1.4.3 Using debug 1.6.1 Using monetize 1.12.0 Using rails-dom-testing 2.0.3 Using globalid 1.0.0 Using activemodel 7.0.3.1 Using activemerchant 1.126.0 Using activerecord 7.0.3.1 Using state_machines-activemodel 0.8.0 Using acts_as_list 1.0.4 Using awesome_nested_set 3.5.0 Using kt-paperclip 7.1.1 Using kaminari-activerecord 1.2.2 Using ransack 2.6.0 Using discard 1.2.1 Using actionview 7.0.3.1 Using activejob 7.0.3.1 Using actionpack 7.0.3.1 Using carmen 1.1.3 Using capybara 3.37.1 Using state_machines-activerecord 0.8.0 Using jbuilder 2.11.5 Using friendly_id 5.4.2 Using kaminari-actionview 1.2.2 Using actioncable 7.0.3.1 Using activestorage 7.0.3.1 Using actionmailer 7.0.3.1 Using railties 7.0.3.1 Using sprockets-rails 3.4.2 Using cssbundling-rails 1.1.1 Using jsbundling-rails 1.0.3 Using actiontext 7.0.3.1 Using font-awesome-rails 4.7.0.8 Using jquery-rails 4.5.0 Using responders 3.0.1 Using kaminari 1.2.2 Using actionmailbox 7.0.3.1 Using stimulus-rails 1.1.0 Using turbo-rails 1.1.1 Using web-console 4.2.0 Using sassc-rails 2.1.2 Using solidus_core 3.2.0.alpha from https://github.com/solidusio/solidus (at master@a227230) Using rails 7.0.3.1 Using canonical-rails 0.2.14 Using solidus_api 3.2.0.alpha from https://github.com/solidusio/solidus (at master@a227230) Using solidus_sample 3.2.0.alpha from https://github.com/solidusio/solidus (at master@a227230) Using solidus_backend 3.2.0.alpha from https://github.com/solidusio/solidus (at master@a227230) Using solidus_frontend 3.2.0.alpha from https://github.com/solidusio/solidus (at master@a227230) Using solidus 3.2.0.alpha from https://github.com/solidusio/solidus (at master@a227230) Bundle complete! 17 Gemfile dependencies, 121 gems now installed. Use `bundle info [gemname]` to see where a bundled gem is installed. </span></pre> <p> <code>Solidus_core</code> displayed the following Post-install message: </p> <div class='codeLabel unselectable' data-lt-active='false'>solidus_core post-install message</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8625fa48b04f'>Post-install message from solidus_core: ------------------------------------------------------------- Thank you for using Solidus ------------------------------------------------------------- If this is a fresh install, don't forget to run the Solidus installer with the following command: $ bin/rails g solidus:install If you are updating Solidus from an older version, please run the following commands to complete the update: $ bin/rails g solidus:update Please, don't forget to look at the CHANGELOG to see what has changed and whether you need to perform other tasks. https://github.com/solidusio/solidus/blob/master/CHANGELOG.md Please report any issues at: - https://github.com/solidusio/solidus/issues - http://slack.solidus.io/ ------------------------------------------------------------- Post-install message from solidus_frontend: ---------------------------------------------------------------------------- DEPRECATION WARNING: SolidusFrontend is deprecated. It will be removed from the solidus meta-package gem in a Solidus v4. Furthermore, its code will be extracted from https://github.com/solidusio/solidus to a new repo. Once extracted, you'll need to explicitly add `solidus_frontend` to your Gemfile in order to continue using it. For fresh Solidus applications, we recommend you use SolidusStarterFrontend instead. ----------------------------------------------------------------------------</pre> <p> Let&rsquo;s take a quick peek at one of the newly built gems: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7fa8819067fa'><button class='copyBtn' data-clipboard-target='#id7fa8819067fa' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>bundle info solidus <span class='unselectable'>&nbsp; * solidus (3.2.0.alpha a227230) Summary: Full-stack e-commerce framework for Ruby on Rails. Homepage: http://solidus.io Path: /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/bundler/gems/solidus-a22723079f88 </span></pre> <p> Following the directions in the post-install message, I typed: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id91005514538c'><button class='copyBtn' data-clipboard-target='#id91005514538c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>nohup time nice \ bin/rails generate solidus:install 1>&2 | tee nohup.out & <span class='unselectable'>create config/initializers/spree.rb Installing Active Storage rake active_storage:install Copied migration 20220730004449_create_active_storage_tables.active_storage.rb from active_storage append public/robots.txt exist app/assets/images create vendor/assets/javascripts/spree/frontend create vendor/assets/javascripts/spree/backend create vendor/assets/stylesheets/spree/frontend create vendor/assets/stylesheets/spree/backend create vendor/assets/images/spree/frontend create vendor/assets/images/spree/backend create vendor/assets/javascripts/spree/frontend/all.js create vendor/assets/stylesheets/spree/frontend/all.css create vendor/assets/javascripts/spree/backend/all.js create vendor/assets/stylesheets/spree/backend/all.css create app/overrides Solidus has a default authentication extension that uses Devise. You can find more info at https://github.com/solidusio/solidus_auth_devise. Would you like to install it? (Y/n) You can select a payment method to be included in the installation process. Please select a payment method name: [paypal, none] (paypal) append db/seeds.rb copying migrations creating database rake db:create We could not find your database: postgres. Which can be found in the database configuration file located at config/database.yml.<br/> To resolve this issue:<br/> - Did you create the database for this app, or delete it? You may need to create your database. - Has the database name changed? Check your database.yml config has the correct database name.<br/> To create your database, run:<br/> bin/rails db:create Couldn&#39;t create &#39;slinnbooks&#39; database. Please check your configuration. rake aborted! ActiveRecord::NoDatabaseError: We could not find your database: postgres. Which can be found in the database configuration file located at config/database.yml.<br/> To resolve this issue:<br/> - Did you create the database for this app, or delete it? You may need to create your database. - Has the database name changed? Check your database.yml config has the correct database name.<br/> To create your database, run:<br/> bin/rails db:create<br/><br/> Caused by: PG::ConnectionBad: connection to server at &quot;127.0.0.1&quot;, port 5432 failed: FATAL: password authentication failed for user &quot;postgres&quot; connection to server at &quot;127.0.0.1&quot;, port 5432 failed: FATAL: password authentication failed for user &quot;postgres&quot;<br/> Tasks: TOP =&gt; db:create (See full trace by running task with --trace) gemfile solidus_auth_devise gemfile solidus_paypal_commerce_platform run bundle install from &quot;.&quot; Fetching gem metadata from https://rubygems.org/......... Resolving dependencies.... Using rake 13.0.6 Using concurrent-ruby 1.1.10 Using rack 2.2.4 Using builder 3.2.4 Using erubi 1.10.0 Using digest 3.1.0 Using crass 1.0.6 Using nio4r 2.5.8 Using minitest 5.16.2 Using public_suffix 4.0.7 Using marcel 1.0.2 Fetching bcrypt 3.1.18 Using racc 1.6.0 Using timeout 0.3.0 Using bundler 2.3.18 Using websocket-extensions 0.1.5 Using execjs 2.8.1 Using mini_mime 1.1.2 Using bindex 0.8.1 Using msgpack 1.5.4 Using strscan 3.0.4 Using cancancan 3.4.0 Using method_source 1.0.0 Using thor 1.2.1 Using zeitwerk 2.6.0 Using matrix 0.4.2 Using regexp_parser 2.5.0 Using childprocess 4.1.0 Using ffi 1.15.5 Using io-console 0.5.11 Using rainbow 3.1.1 Fetching polyglot 0.3.5 Fetching orm_adapter 0.5.0 Using climate_control 0.2.0 Using tilt 2.0.11 Using mini_magick 4.11.0 Using pg 1.4.2 Using mime-types-data 3.2022.0105 Using omnes 0.2.2 Using kaminari-core 1.2.2 Fetching paypalhttp 1.0.1 Using stringio 3.0.2 Using rexml 3.2.5 Using rubyzip 2.3.2 Using websocket 1.2.9 Using i18n 1.12.0 Using truncate_html 0.9.3 Fetching solidus_support 0.9.1 Using state_machines 0.5.0 Using tzinfo 2.0.5 Fetching warden 1.2.9 Using rack-test 2.0.2 Using sprockets 4.1.1 Using puma 5.6.4 Using addressable 2.8.0 Using nokogiri 1.13.8 (x86_64-linux) Using net-protocol 0.1.3 Using websocket-driver 0.7.5 Using autoprefixer-rails 10.4.7.0 Using mail 2.7.1 Using bootsnap 1.13.0 Using ruby-vips 2.1.4 Using sassc 2.4.0 Using terrapin 0.6.0 Using reline 0.3.1 Using mime-types 3.4.1 Using psych 4.0.4 Using selenium-webdriver 4.3.0 Using activesupport 7.0.3.1 Using money 6.16.0 Using handlebars_assets 0.23.9 Installing polyglot 0.3.5 Installing orm_adapter 0.5.0 Using loofah 2.18.0 Using xpath 3.2.0 Installing solidus_support 0.9.1 Installing paypalhttp 1.0.1 Installing warden 1.2.9 Using net-imap 0.2.3 Using net-smtp 0.3.1 Using image_processing 1.12.2 Using irb 1.4.1 Using net-pop 0.1.1 Using rails-dom-testing 2.0.3 Using globalid 1.0.0 Using activemodel 7.0.3.1 Using activemerchant 1.126.0 Using webdrivers 5.0.0 Using monetize 1.12.0 Using carmen 1.1.3 Using rails-html-sanitizer 1.4.3 Using debug 1.6.1 Using activejob 7.0.3.1 Using activerecord 7.0.3.1 Using kt-paperclip 7.1.1 Using state_machines-activemodel 0.8.0 Using actionview 7.0.3.1 Using acts_as_list 1.0.4 Using awesome_nested_set 3.5.0 Using discard 1.2.1 Using friendly_id 5.4.2 Using capybara 3.37.1 Installing bcrypt 3.1.18 with native extensions Using kaminari-activerecord 1.2.2 Using state_machines-activerecord 0.8.0 Using ransack 2.6.0 Using jbuilder 2.11.5 Using kaminari-actionview 1.2.2 Using actionpack 7.0.3.1 Using kaminari 1.2.2 Using actioncable 7.0.3.1 Using railties 7.0.3.1 Using sprockets-rails 3.4.2 Using cssbundling-rails 1.1.1 Using actionmailer 7.0.3.1 Fetching deface 1.9.0 Using activestorage 7.0.3.1 Using font-awesome-rails 4.7.0.8 Using jquery-rails 4.5.0 Using responders 3.0.1 Using stimulus-rails 1.1.0 Using turbo-rails 1.1.1 Using web-console 4.2.0 Using sassc-rails 2.1.2 Using jsbundling-rails 1.0.3 Using actionmailbox 7.0.3.1 Using actiontext 7.0.3.1 Using rails 7.0.3.1 Using canonical-rails 0.2.14 Using solidus_core 3.2.0.alpha from https://github.com/solidusio/solidus (at master@a227230) Using solidus_api 3.2.0.alpha from https://github.com/solidusio/solidus (at master@a227230) Using solidus_sample 3.2.0.alpha from https://github.com/solidusio/solidus (at master@a227230) Using solidus_backend 3.2.0.alpha from https://github.com/solidusio/solidus (at master@a227230) Using solidus_frontend 3.2.0.alpha from https://github.com/solidusio/solidus (at master@a227230) Using solidus 3.2.0.alpha from https://github.com/solidusio/solidus (at master@a227230) Fetching paypal-checkout-sdk 1.0.4 Fetching solidus_webhooks 0.4.0 Installing deface 1.9.0 Installing paypal-checkout-sdk 1.0.4 Installing solidus_webhooks 0.4.0 Fetching solidus_paypal_commerce_platform 0.3.2 Installing solidus_paypal_commerce_platform 0.3.2 Fetching devise 4.8.1 Installing devise 4.8.1 Fetching devise-encryptable 0.2.0 Installing devise-encryptable 0.2.0 Fetching solidus_auth_devise 2.5.4 Installing solidus_auth_devise 2.5.4 Bundle complete! 19 Gemfile dependencies, 135 gems now installed. Use `bundle info [gemname]` to see where a bundled gem is installed. Post-install message from solidus_auth_devise: NOTE: Rails 6 has removed secret_token in favor of secret_key_base, which was deprecated in Rails 5.2. solidus_auth_devise will keep using secret_token, when present, as the pepper. If secret_token is undefined or not available, secret_key_base will be used instead. generate solidus:auth:install --skip_migrations=true rails generate solidus:auth:install --skip_migrations=true /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/solidus_paypal_commerce_platform-0.3.2/lib/solidus_paypal_commerce_platform/engine.rb:18:in `block in &lt;class:Engine&gt;&#39;: uninitialized constant SolidusPaypalCommercePlatform::PaymentMethod (NameError)<br/> app.config.spree.payment_methods &lt;&lt; SolidusPaypalCommercePlatform::PaymentMethod ^^^^^^^^^^^^^^^ from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/railties-7.0.3.1/lib/rails/initializable.rb:32:in `instance_exec&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/railties-7.0.3.1/lib/rails/initializable.rb:32:in `run&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/railties-7.0.3.1/lib/rails/initializable.rb:61:in `block in run_initializers&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/3.1.0/tsort.rb:228:in `block in tsort_each&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/3.1.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/3.1.0/tsort.rb:431:in `each_strongly_connected_component_from&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/3.1.0/tsort.rb:349:in `block in each_strongly_connected_component&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/3.1.0/tsort.rb:347:in `each&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/3.1.0/tsort.rb:347:in `call&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/3.1.0/tsort.rb:347:in `each_strongly_connected_component&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/3.1.0/tsort.rb:226:in `tsort_each&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/3.1.0/tsort.rb:205:in `tsort_each&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/railties-7.0.3.1/lib/rails/initializable.rb:60:in `run_initializers&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/railties-7.0.3.1/lib/rails/application.rb:372:in `initialize!&#39; from /mnt/f/work/rails/solidus_poc/config/environment.rb:5:in `&lt;main&gt;&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bootsnap-1.13.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bootsnap-1.13.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/zeitwerk-2.6.0/lib/zeitwerk/kernel.rb:35:in `require&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/railties-7.0.3.1/lib/rails/application.rb:348:in `require_environment!&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/railties-7.0.3.1/lib/rails/command/actions.rb:28:in `require_environment!&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/railties-7.0.3.1/lib/rails/command/actions.rb:15:in `require_application_and_environment!&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/railties-7.0.3.1/lib/rails/commands/generate/generate_command.rb:21:in `perform&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/railties-7.0.3.1/lib/rails/command/base.rb:87:in `perform&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/railties-7.0.3.1/lib/rails/command.rb:48:in `invoke&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/railties-7.0.3.1/lib/rails/commands.rb:18:in `&lt;main&gt;&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bootsnap-1.13.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require&#39; from /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bootsnap-1.13.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require&#39; from bin/rails:4:in `&lt;main&gt;&#39; </span></pre> <p class="alert rounded shadow"> I going to fix this and start <i>again</i>. Sigh. </p> <h2 id="dbdef">Define and Create the Database</h2> <p> Set the usernames and passwords for the development, test and production databases in <code>config/database.yml</code>. I removed most of the comments for clarity. Read the actual file's comments, there is a lot of good information in them. </p> <div class='codeLabel unselectable' data-lt-active='false'>config/database.yml</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1b12525ddadc'><button class='copyBtn' data-clipboard-target='#id1b12525ddadc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>default: &amp;default adapter: postgresql encoding: unicode pool: &lt;%= ENV.fetch(&quot;RAILS_MAX_THREADS&quot;) { 5 } %&gt;<br/> development: &lt;&lt;: *default database: scalacourses_development #host: localhost password: &lt;%= ENV[&quot;SCALACOURSES_DATABASE_PASSWORD_DEV&quot;] %&gt; #port: 5432 #schema_search_path: myapp,sharedapp,public username: scalacourses<br/> # Log levels, in increasing verbosity: # debug5, debug4, debug3, debug2, debug1, # log, notice, warning, error, fatal, and panic # Default is warning. #min_messages: notice<br/> test: &lt;&lt;: *default database: scalacourses_test #host: localhost #port: 5432<br/> production: &lt;&lt;: *default database: scalacourses_production #host: localhost #port: 5432 password: &lt;%= ENV[&quot;SCALACOURSES_DATABASE_PASSWORD_PROD&quot;] %&gt; username: scalacourses</pre> <p> Use environment variable to set values for <code>SCALACOURSES_DATABASE_PASSWORD_DEV</code>, and perhaps <code>SCALACOURSES_DATABASE_PASSWORD_PROD</code> </p> <div class='codeLabel unselectable' data-lt-active='false'>~/.bashrc</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idadf816d59930'><button class='copyBtn' data-clipboard-target='#idadf816d59930' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>export SCALACOURSES_DATABASE_PASSWORD_DEV=my_secret</pre> <p> Define the environment variable in the current shell, then launch Rails. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id02dbd3e9a990'><button class='copyBtn' data-clipboard-target='#id02dbd3e9a990' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>source ~/.bashrc</pre> <p> Here is the help message for creating the datatabase using <a href='https://ruby.github.io/rake/' target='_blank' rel='nofollow'><code>rake</code></a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id93bf5444e53c'><button class='copyBtn' data-clipboard-target='#id93bf5444e53c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>rake db:create -h <span class='unselectable'>rake [-f rakefile] {options} targets...<br/> Options are ... --backtrace=[OUT] Enable full backtrace. OUT can be stderr (default) or stdout. --comments Show commented tasks only --job-stats [LEVEL] Display job statistics. LEVEL=history displays a complete job list --rules Trace the rules resolution. --suppress-backtrace PATTERN Suppress backtrace lines matching regexp PATTERN. Ignored if --trace is on. -A, --all Show all tasks, even uncommented ones (in combination with -T or -D) -B, --build-all Build all prerequisites, including those which are up-to-date. -C, --directory [DIRECTORY] Change to DIRECTORY before doing anything. -D, --describe [PATTERN] Describe the tasks (matching optional PATTERN), then exit. -e, --execute CODE Execute some Ruby code and exit. -E, --execute-continue CODE Execute some Ruby code, then continue with normal task processing. -f, --rakefile [FILENAME] Use FILENAME as the rakefile to search for. -G, --no-system, --nosystem Use standard project Rakefile search paths, ignore system wide rakefiles. -g, --system Using system wide (global) rakefiles (usually &#39;~/.rake/*.rake&#39;). -I, --libdir LIBDIR Include LIBDIR in the search path for required modules. -j, --jobs [NUMBER] Specifies the maximum number of tasks to execute in parallel. (default is number of CPU cores + 4) -m, --multitask Treat all tasks as multitasks. -n, --dry-run Do a dry run without executing actions. -N, --no-search, --nosearch Do not search parent directories for the Rakefile. -P, --prereqs Display the tasks and dependencies, then exit. -p, --execute-print CODE Execute some Ruby code, print the result, then exit. -q, --quiet Do not log messages to standard output. -r, --require MODULE Require MODULE before executing rakefile. -R, --rakelibdir RAKELIBDIR, Auto-import any .rake files in RAKELIBDIR. (default is &#39;rakelib&#39;) --rakelib -s, --silent Like --quiet, but also suppresses the &#39;in directory&#39; announcement. -t, --trace=[OUT] Turn on invoke/execute tracing, enable full backtrace. OUT can be stderr (default) or stdout. -T, --tasks [PATTERN] Display the tasks (matching optional PATTERN) with descriptions, then exit. -AT combination displays all of tasks contained no description. -v, --verbose Log message to standard output. -V, --version Display the program version. -W, --where [PATTERN] Describe the tasks (matching optional PATTERN), then exit. -X, --no-deprecation-warnings Disable the deprecation warnings. -h, -H, --help Display this help message. </span></pre> <p> Create the development database (<code>scalacourses_development</code>) mentioned in the above configuration file. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id95f92425c873'><button class='copyBtn' data-clipboard-target='#id95f92425c873' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>rake db:create <span class='unselectable'>Created database 'scalacourses' </span></pre> <h2 id="runit">Run Rails Application</h2> <p> Start the Ruby on Rails application. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id83e6da652108'><button class='copyBtn' data-clipboard-target='#id83e6da652108' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>bin/rails server <span class='unselectable'>=> Booting Puma => Rails 7.0.3.1 application starting in development => Run `bin/rails server --help` for more startup options Puma starting in single mode... * Puma version: 5.6.4 (ruby 3.1.0-p0) ("Birdie's Version") * Min threads: 5 * Max threads: 5 * Environment: development * PID: 23159 * Listening on http://127.0.0.1:3000 * Listening on http://[::1]:3000 Use Ctrl-C to stop </span></pre> <p> View the running Rails app in a web browser, such as Google Chrome, Firefox or <a href='https://lynx.invisible-island.net/' target='_blank' rel='nofollow'>lynx</a>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0f0b9e13b5a9'><button class='copyBtn' data-clipboard-target='#id0f0b9e13b5a9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>firefox localhost:3000 &</pre> <p> The page should look like this: </p> <div style=""> <picture> <source srcset="/blog/images/rails/rails_app_front_page.webp" type="image/webp"> <source srcset="/blog/images/rails/rails_app_front_page.png" type="image/png"> <img src="/blog/images/rails/rails_app_front_page.png" class=" liImg2 rounded shadow" /> </picture> </div> <p> As the web browser loads the front page of the new Rails app, notice that the output of the Rails server now includes: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbaee894beb69'><button class='copyBtn' data-clipboard-target='#idbaee894beb69' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>Started GET "/" for ::1 at 2022-07-16 21:04:46 -0400 Processing by Rails::WelcomeController#index as HTML Rendering /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/railties-7.0.3.1/lib/rails/templates/rails/welcome/index.html.erb Rendered /home/mslinn/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/railties-7.0.3.1/lib/rails/templates/rails/welcome/index.html.erb (Duration: 10.8ms | Allocations: 545) Completed 200 OK in 65ms (Views: 33.0ms | ActiveRecord: 0.0ms | Allocations: 4124) </span></pre> </editor-fold> <editor-fold references> <h2 id="refs">References</h2> <ol> <li><a href='https://appinventiv.com/blog/django-vs-ruby-on-rails' target='_blank' rel='nofollow'>Django Vs Ruby on Rails – Which Framework is Best for 2022?</a></li> <li><a href='https://lp.jetbrains.com/django-developer-survey-2021-486/' target='_blank' rel='nofollow'>JetBrains Django Developer Survey 2021</a></li> <li><a href='https://www.monocubed.com/blog/ruby-on-rails-vs-django/' target='_blank' rel='nofollow'>Ruby on Rails vs Django Comparison: Choosing Best Web Framework</a></li> <li><a href='https://keyua.org/blog/rails-vs-django-comparison/' target='_blank' rel='nofollow'>Django vs Rails in 2022: Comparison Performance Frameworks</a></li> </ol> </editor-fold> Uncomplicated Firewall on Ubuntu 2022-07-16T00:00:00-04:00 https://mslinn.github.io/blog/2022/07/16/ufw <p> All websites, especially ecommerce sites, need to be secure. A properly set up firewall is an essential component for a secure server. </p> <p> Ubuntu 22.04 uses the <a href='https://wiki.ubuntu.com/UncomplicatedFirewall' target='_blank' rel='nofollow'>Uncomplicated Firewall <code>ufw</code> firewall</a> frontend by default. <code>Ufw</code> has been provided for Ubuntu since v8.04 (Hardy Heron). </p> <h2 id="tldr">Quick Setup</h2> <p> Enable <code>ufw</code> as follows: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2284a7ac02ea'><button class='copyBtn' data-clipboard-target='#id2284a7ac02ea' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo ufw enable <span class='unselectable'>Firewall is active and enabled on system startup </span></pre> <p> Enable the <a href='https://github.com/ageis/ufw-application-profiles' target='_blank' rel='nofollow'><code>ufw</code> application profiles</a> for <code>ssh</code> and nginx (HTTP/HTTPS) like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id411ad1b54180'><button class='copyBtn' data-clipboard-target='#id411ad1b54180' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo ufw allow OpenSSH <span class='unselectable'>Output Rule added Rule added (v6) </span> <span class='unselectable'>$ </span>sudo ufw allow 'Nginx Full' <span class='unselectable'>Output Rule added Rule added (v6) </span></pre> <h3 id="ufwDetails">Diving Deeper</h3> <p> The following is mostly true: </p> <div class='quote'> The default firewall on Ubuntu 22.04 Jammy Jellyfish is <code>ufw</code>, which is short for “uncomplicated firewall.” <code>Ufw</code> is a frontend for the typical Linux <code>iptables</code> commands, but it is developed in such a way that basic firewall tasks can be performed without the knowledge of <code>iptables</code>. <br><br> Additionally, <code>ufw</code> can be managed from a graphical interface. In this tutorial, you will learn how to enable and disable the <code>ufw</code> firewall on Ubuntu 22.04 Jammy Jellyfish from both command line and GUI. <br><br> <span style='font-style:normal;'> &nbsp;&ndash; From <a href='https://linuxconfig.org/how-to-enable-disable-firewall-on-ubuntu-22-04-lts-jammy-jellyfish-linux' rel='nofollow' target='_blank'>How to enable/disable firewall on Ubuntu 22.04 LTS Jammy Jellyfish Linux</a></span> </div> <p> The above makes no mention of how Ubuntu 22.04 replaced <code>iptables</code> with <code>nftables</code>, as described below. </p> <div class='quote'> <h2><span class="code">nftables</span> as the default firewall backend</h2> Firewalling on Linux consists of two components &ndash; the firewall mechanism within the Linux kernel, and the tools used to configure this from userspace. The Linux kernel has traditionally supported two different subsystems for firewall policies – <code>iptables</code> / <code>xtables</code> and the newer <code>nftables</code>. <br><br> <code>Nftables</code> brings significant benefits both in terms of performance and flexibility when creating and deploying firewall rules, particularly for dual stack IPv4/IPv6 systems. <br><br> The traditional <code>iptables</code> userspace management tool now configures the <code>nftables</code> kernel backend, whilst the new <code>nft</code> userspace tool is also present to allow the creation of more flexible rules not supported by the traditional iptables paradigm. <br><br> <span style='font-style:normal;'> &nbsp;&ndash; From <a href='https://ubuntu.com/blog/whats-new-in-security-for-ubuntu-22-04-lts/docs-internal-guid-117e0493-7fff-ea13-3537-bde70965f89d' rel='nofollow' target='_blank'>What’s new in Security for Ubuntu 22.04 LTS?</a></span> </div> <p> Digital Ocean has a <a href='https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-18-04' target='_blank' rel='nofollow'>good <code>ufw</code> tutorial</a>. </p> Using Nginx as a reverse proxy with SSL 2022-07-08T00:00:00-04:00 https://mslinn.github.io/blog/2022/07/08/reverse-proxy <div class="right"> <div style=""> <a href="https://www.ScalaCourses.com" target="_blank" ><picture> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/webp"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/png"> <img src="/assets/images/ScalaCoursesLogo207x207.png" title="ScalaCourses.com" class=" liImg2 rounded shadow" alt="ScalaCourses.com" /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://www.ScalaCourses.com" target="_blank" > ScalaCourses.com </a> </figcaption> </figure> </div> </div> <p> Recently I <a href='https://mslinn.com/blog/2022/05/26/aws-hijacking.html' target='_blank' rel='nofollow'>migrated</a> ScalaCourses.com from AWS EC2/S3/CloudFront to a server in my apartment, which has fiber optic internet service. The server&rsquo;s motherboard is an ASUS Sabertooth x79 with an Intel i7 4820, 32 GB DDR3 RAM, and a 4TB SATA SSD. At the time of this writing, the server runs Ubuntu 22.04. </p> <p> The backup server is currently being set up. I am repurposing an old Hackintosh as a fallback to the Ubuntu server. That motherboard is a <a href='https://www.newegg.com/p/N82E16813128423?Item=N82E16813128423&Tpk=ud3r' target='_blank' rel='nofollow'>Gigabyte GA-X58A-UD3R (Rev 2)</a>. It also has 32 GB DDR3 RAM and a 2 TB SATA drive. </p> <h2 id="why">Why Am I Doing This?</h2> <p> Once again I control my hardware, my software, my network, and all ancillary services. After several years of using PaaS vendor servers, I am now reverting to running ScalaCourses.com on my own hardware and software, using my own network. </p> <div class="pullQuote"> No longer will I subject myself to the unlimited financial liability that current PaaS vendors expose their customers to. </div> <h2 id="details">Dealing With Details</h2> <p> The old <a href='https://en.wikipedia.org/wiki/Pound_(networking)' target='_blank' rel='nofollow'>Pound</a> v2.8-2 <a href='https://www.cloudflare.com/en-ca/learning/cdn/glossary/reverse-proxy/' target='_blank' rel='nofollow'>reverse proxy</a> that was the front end for the old Play Framework app that runs <a href='https://scalacourses.com' target='_blank' rel='nofollow'>ScalaCourses.com</a> is no longer viable, and the new version 3 of Pound is incomplete. Depending on configuration, reverse proxies can provide extra security from external attacks on a website, decrypt https requests to http, and act as stream editors for the content. </p> <p> This site still uses AWS CloudFront, for the moment. I am researching alternatives, with the goal of closing my AWS account... unless they introduce a way to cap financial liability before I fully migrate off AWS. </p> <h2 id="vs">Apache httpd vs. Nginx</h2> <p> Two popular reverse proxies are <a href='https://httpd.apache.org/docs/trunk/mod/mod_proxy_http2.html' target='_blank' rel='nofollow'>Apache <code>mod_proxy_http2</code></a> and <a href='https://www.nginx.com/' target='_blank' rel='nofollow'>nginx</a>. Anything Pound can do, both nginx and Apache httpd/2 can do. While they both work well, Apache httpd has an older code base, in fact, many of my websites ran on Apache httpd at the turn of the century. </p> <p> Nginx is relatively newer than Apache httpd. It is performant, well supported, well documented, and widely available. I decided to use nginx as the reverse proxy because I had found that nginx worked well on previous projects. </p> <p> Installing nginx and the Letsencrypt software is easy on Debian distros such as Ubuntu: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9ca9f3e5e32a'><button class='copyBtn' data-clipboard-target='#id9ca9f3e5e32a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install nginx certbot python3-certbot-nginx</pre> <h2 id="vs">Verifying the Nginx Build</h2> <p> When acting as a proxy server, nginx requires the <code>http_sub_module</code> to translate HTTP content. This allows the website content to be stream edited, so various links and paths that are not translated properly can be fixed up. </p> <p> The nginx package provided by Ubuntu includes the <code>&#8209;&#8209;with&#8209;http_sub_module</code> option. You can verify that your instance of nginx was built with the <code>&#8209;&#8209;with&#8209;http_sub_module</code> option as follows: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf667fd7adfb2'><button class='copyBtn' data-clipboard-target='#idf667fd7adfb2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>nginx -V <span class='unselectable'>nginx version: nginx/1.18.0 (Ubuntu) built with OpenSSL 3.0.2 15 Mar 2022 TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -ffile-prefix-map=/build/nginx-9P0wNJ/nginx-1.18.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --add-dynamic-module=/build/nginx-9P0wNJ/nginx-1.18.0/debian/modules/http-geoip2 --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module <span class="bg_yellow">--with-http_sub_module</span> </span></pre> <h2 id="vs">Nginx SSL Configuration</h2> <p> Two files are needed for Letsencrypt to be able to create SSL certificates for nginx: </p> <ol> <li> The contents of <a href='https://github.com/certbot/certbot/blob/master/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf' target='_blank' rel='nofollow'><code>/etc/letsencrypt/options-ssl-nginx.conf</code></a> need to be stored into <code>/etc/letsencrypt/options-ssl-nginx.conf</code>. I later discovered this file was also available locally at <code>/usr/lib/python3/dist-packages/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf</code>. </li> <li> The contents of <a href='https://github.com/certbot/certbot/blob/master/certbot/certbot/ssl-dhparams.pem' target='_blank' rel='nofollow'><code>/etc/letsencrypt/ssl-dhparams.pem</code></a> need to be stored into <code>/etc/letsencrypt/ssl-dhparams.pem</code>. I later discovered that this file was also available locally at <code>/usr/lib/python3/dist-packages/certbot/ssl-dhparams.pem</code>. </li> </ol> <h2 id="ssl">Making a Wildcard SSL Certificate</h2> <p> I knew an SSL wildcard certificate would be needed, so I made one using Letsencrypt. Please read <a href='/blog/2022/06/15/certbot.html'>Creating and Renewing Letsencrypt Wildcard SSL Certificates</a> for details. </p> <h2 id="vs">Defining the Nginx Website Reverse Proxy</h2> <p> I saved the following configuration in a new file called <code>/etc/nginx/sites-available/scalacourses.com</code>. Note that a single <code>server</code> block answers on ports 80 and 443, using IPv4 and IPv6, for SSL and non-SSL requests, for the apex domain (<code>scalacourses.com</code>) and the <code>www.scalacourses.com</code> subdomain. No redirects are used. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbe5e9a768acd'><button class='copyBtn' data-clipboard-target='#idbe5e9a768acd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>server { listen 80; listen [::]:80; listen 443 ssl; listen [::]:443 ssl; server_name scalacourses.com www.scalacourses.com; ssl_certificate /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/fullchain.pem; ssl_certificate_key /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/privkey.pem; include /etc/letsencrypt/options-ssl-nginx.conf; ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; root /var/www/html; index index.html; # This gets served if the proxied website is down location / { proxy_pass http://localhost:9000/; proxy_set_header Accept-Encoding ""; # sub_filter requires this sub_filter 'https://localhost:9000/authenticate/' 'https://www.scalacourses.com/authenticate/' ; sub_filter_once off; sub_filter_types text/html; } }</pre> <p> I disabled the <code>default</code> site: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id18998d612df8'><button class='copyBtn' data-clipboard-target='#id18998d612df8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo rm /etc/nginx/sites-enabled/default</pre> <p> I enabled the new <code>scalacourses.com</code> site: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7f7b39354c4a'><button class='copyBtn' data-clipboard-target='#id7f7b39354c4a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo ln /etc/nginx/sites-{available,enabled}/scalacourses.com</pre> <p> The <code>nginx</code> configuration was tested for syntax: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1e5aa04258f0'><button class='copyBtn' data-clipboard-target='#id1e5aa04258f0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo nginx -t <span class='unselectable'>nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful </span></pre> <p> The nginx configuration was reloaded: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0a71faaf5b62'><button class='copyBtn' data-clipboard-target='#id0a71faaf5b62' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl reload nginx</pre> <h2 id="flush">Flush DNS Cache</h2> <p> Ensure that DNS requests receive up-to-date values by flushing the DNS cache. The following works on Ubuntu, but not when running on WSL/WSL2. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida533da26a575'><button class='copyBtn' data-clipboard-target='#ida533da26a575' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo resolvectl flush-caches</pre> <p> The following works on Windows 10: </p> <div class='codeLabel unselectable' data-lt-active='false'>Command and PowerShell consoles</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5dd967739f8f'><button class='copyBtn' data-clipboard-target='#id5dd967739f8f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>ipconfig /flushdns</pre> <h2 id="verify">Verifying nginx Works</h2> <p> I verified that <code>nginx</code> was listening on ports 80 and 443. I highlighted the <code>nginx</code> process number &ndash; you might need to scroll the output below to the right to see it. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida3d85e5b43a3'><button class='copyBtn' data-clipboard-target='#ida3d85e5b43a3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo netstat -tulpn | grep ':\(443\|80\)' <span class='unselectable'>tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN - tcp6 0 0 :::80 :::* LISTEN - tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN <span class="bg_yellow">87487</span>/nginx: master </span></pre> <span> <p> The executable for process 87487 can be found by: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id16a122881292'><button class='copyBtn' data-clipboard-target='#id16a122881292' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls -l /proc/87487/exe <span class='unselectable'>lrwxrwxrwx 1 mslinn mslinn 0 Jul 7 09:13 /proc/5166/exe -> <span class="bg_yellow">/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java*</span> </span></pre> <p> If more detail is desired, get it from the process list. Enclosing a character of the process id within square brackets is an old trick for only showing the desired process. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfe4e6bb31067'><button class='copyBtn' data-clipboard-target='#idfe4e6bb31067' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ps aux | grep [8]7487 <span class='unselectable'>mslinn 87487 2.4 1.3 6596476 439456 ? Sl 09:13 0:20 java -Xms1024m -Xmx1024m -Dhttp.port=9000 /path/to/jar </span></pre> <p> If there is any problem getting things to work, it is often helpful to monitor the logs as you click on the web pages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id21f14b8151ba'><button class='copyBtn' data-clipboard-target='#id21f14b8151ba' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo tail -f /var/log/nginx/*.log</pre> <h2 id="persist">Finishing Up</h2> <p> Make nginx start each time the system starts as follows. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbc42bc0c61e0'><button class='copyBtn' data-clipboard-target='#idbc42bc0c61e0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl enable nginx <span class='unselectable'>Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable nginx </span> <span class='unselectable'>$ </span>sudo update-rc.d nginx defaults</pre> <h3 id="performance">Performance Test</h3> <p> <a href='https://tools.keycdn.com/performance?url=https://www.scalacourses.com' target='_blank' rel='nofollow'>KeyCDN</a> offfers free website performance statistics. The result column labeled <b>TTFB</b> means &ldquo;time to first byte&rdquo;, which is the length of time required for the website content to begin being received by a user's web browser. </p> <p> I am in Montreal, Canada. Response time for TTFB reported by KeyCDN varies, from a minimum of 68ms in New York, to a maximum of 815ms in Bangalore. This range of response times is typical for websites that have a centralized process. The speed of light is finite, after all. </p> <h3 id="monitor">Free Availability Monitoring</h3> <p> <a href='https://hetrixtools.com/' target='_blank' rel='nofollow'>HetrixTools</a> offers free availability monitoring for up to 10 websites. It was quick and easy to set up. </p> <div style=""> <picture> <source srcset="/blog/images/hetrix.webp" type="image/webp"> <source srcset="/blog/images/hetrix.png" type="image/png"> <img src="/blog/images/hetrix.png" class=" liImg2 rounded shadow" /> </picture> </div> <h3 id="live">We Are Live!</h3> <p> <a href='https://www.scalacourses.com' target='_blank' rel='nofollow'><code>ScalaCourses.com</code></a> now serves Scala students from its newly refurbished server! <span style='font-size: 3em; '>&#x1F601;</span> </p> Trialing mslinn.com on Linode Storage 2022-07-01T00:00:00-04:00 https://mslinn.github.io/blog/2022/07/01/trialing-linode-storage <edit-fold intro> <p> This is another post in my ongoing saga of moving off AWS, which <a href='/blog/2022/05/26/aws-hijacking.html'>does not integrate security with real-time billing</a>. I recently learned this the hard way: when my AWS account was hijacked, in less than 15 minutes, a huge bill was incurred. </p> <div class="pullQuote"> &ldquo;Pay-as-you-go&rdquo; is shorthand for &ldquo;there is nothing you can do to limit your financial liability&rdquo; </div> <p> The world of pain that I experienced after the breach was inflicted by broken and wasteful AWS remedial processes, and an ineffective AWS management structure. This type of issue only is enabled because of deficiencies in the AWS architecture. Those AWS architectural deficiencies feel like the result of an exploitive mindset: </p> <div class='quote'> Unlimited financial liability is our customer&rsquo;s problem, not ours &ndash; as a result, exploits are quite profitable for us. <br><br> <span style='font-style:normal;'> &nbsp;&ndash; From <a href='' rel='nofollow' target='_blank'>A mythical retrospective discussion at an AWS offsite.</a></span> </div> <div class='quote'> At this moment that feature of setting limits does not exist, Azure is not able to safeguard customers from unlimited financial liability. <br><br> <span style='font-style:normal;'> &nbsp;&ndash; From <a href='' rel='nofollow' target='_blank'>From an email sent to me from Microsoft Azure support staff on 2022-06-22.</a></span> </div> </edit-fold> <div class="alert rounded shadow"> <h2>Demand Limits to Financial Liability From PaaS Vendors</h2> <p> PaaS vendors currently provide accounts with all services ready to go, without limit. That is good for the vendor's bottom line, but highly dangerous for their customers. </p> <p> All PaaS customers should demand the ability for themselves to be able to set firm limits on budgeted expenses, along with the ability to deny all services not explicitly authorized. </p> <p> You can buy insurance against losses resulting from various calamities, but you cannot limit your financial liability with PaaS vendors. </p> <p> <span style="font-size:larger; font-weight: bold;">Yet.</span> </p> </div> <edit-fold market> <h2 id="market">Website Hosting Market</h2> <p> Recently, I have spent a lot of time looking at options for hosting websites. I found the following types of products: </p> <style> .row th { text-align: right; } </style> <table class="table"> <tr> <td></td> <th>WordPress</th> <th>General Web Server</th> <th>VM</th> <th>S3 Compatible</th> </tr> <tr class="row"> <th>CDN</th> <td>No</td> <td>No</td> <td>No</td> <td>Yes</td> </tr> <tr class="row"> <th>Speed</th> <td>Slow to Medium</td> <td>Slow to Medium</td> <td>Slow to Fast</td> <td>Fast</td> </tr> <tr class="row"> <th>Reliability</th> <td>Fair</td> <td>Fair</td> <td>Depends on you</td> <td>Good</td> </tr> <tr class="row"> <th>Financial Liability</th> <td>Fixed monthly cost</td> <td>Fixed monthly cost</td> <td class="bg_yellow">Depends</td> <td class="bg_yellow">Unlimited</td> </tr> <tr class="row"> <th>Storage</th> <td>Small</td> <td>Small</td> <td>Depends</td> <td>Unlimited</td> </tr> <tr class="row"> <th>CLI & API</th> <td>No</td> <td>No</td> <td>Depends</td> <td>Yes</td> </tr> </table> <p> This website requires about 220 GB of storage. It has numerous images, in 2 versions: <a href='/blog/2020/08/15/converting-all-images-to-webp-format.html'><code>webp</code></a> and <code>png</code>. So for this website, &lsquo;small&rsquo; means less than 250 GB storage. </p> <p> I decided to give the S3-compatible product <a href='https://www.linode.com/docs/products/storage/object-storage/' target='_blank' rel='nofollow'>Linode Storage</a> a try. </p> </edit-fold> <edit-fold linode> <h2 id="why">Why Linode?</h2> <p> First, the positive: <a href='https://www.linode.com/' target='_blank' rel='nofollow'>Linode</a> is a pioneer in virtual computing. It has been on my short list of places for hosting my pet projects for many years. Prices have always been quite competitive, products solid, with good support ... and smart people have answered the phone when I have called. Being able to easily speak with a capable human provides huge value. </p> <p> On the other hand, as I shall demonstrate in this blog post, Linode&rsquo;s documentation presents as a significant barrier to customers considering adopting their services. I had to work a lot harder than I should have to get my evaluation website up and running. Hopefully, this document is complete enough, so others can follow along and host their static websites on Linode Storage. </p> <h2 id="akamai">Acquired by Akamai</h2> <p> <a href='https://www.akamai.com/newsroom/press-release/akamai-completes-acquisition-of-linode' target='_blank' rel='nofollow'>Linode was acquired by Akamai</a> 3 months ago. Akamai is the original CDN, and their network is gigantic. </p> <p> Linode does not yet offer a CDN product, but the person I spoke to at Linode when I started writing this post suggested that a CDN product from Linode based on Akamai&rsquo;s network might be available soon. He also said that they were planning to working on a mechanism for limiting financial liability to their customers in 2022. </p> <p> This was music to my financially risk-averse ears! </p> </edit-fold> <div class="notepaper shadow" style="width: 80%;"> <p> The remainder of this blog post will take you through all the steps necessary to: </p> <ol> <li>Install and configure software tools on a WSL / WSL2 / Ubuntu computer</li> <li>Make an S3-compatible bucket and set it up to hold a website</li> <li>Upload the website, and easily handle mimetype issues</li> <li>Generate and install a free 4096-bit SSL wildcard certificate</li> <li>Get a security rating from Qualys / SSL Labs</li> </ol> </div> <h2 id="setup">Trialing Linode Storage</h2> <edit-fold s3cmd> <h3 id="s3cmd">Installing <span class="code">s3cmd</span></h3> <p> I installed the recommended S3-compatible command-line program <code>s3cmd</code> on WSL2 / Ubuntu like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4202917cc7e0'><button class='copyBtn' data-clipboard-target='#id4202917cc7e0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install s3cmd <span class='unselectable'>Reading package lists... Done Building dependency tree... Done Reading state information... Done The following additional packages will be installed: python3-magic The following NEW packages will be installed: python3-magic s3cmd 0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 133 kB of archives. After this operation, 584 kB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-magic all 2:0.4.24-2 [12.6 kB] Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 s3cmd all 2.2.0-1 [120 kB] Fetched 133 kB in 0s (278 kB/s) Selecting previously unselected package python3-magic. (Reading database ... 169821 files and directories currently installed.) Preparing to unpack .../python3-magic_2%3a0.4.24-2_all.deb ... Unpacking python3-magic (2:0.4.24-2) ... Selecting previously unselected package s3cmd. Preparing to unpack .../archives/s3cmd_2.2.0-1_all.deb ... Unpacking s3cmd (2.2.0-1) ... Setting up python3-magic (2:0.4.24-2) ... Setting up s3cmd (2.2.0-1) ... Processing triggers for man-db (2.10.2-1) ... Scanning processes... Scanning processor microcode... Scanning linux images... Failed to retrieve available kernel versions. Failed to check for processor microcode upgrades. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. </span></pre> <p> Here is the <code>s3cmd</code> help message: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide1a348499b39'><button class='copyBtn' data-clipboard-target='#ide1a348499b39' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd <span class='unselectable'>Usage: s3cmd [options] COMMAND [parameters]<br/> S3cmd is a tool for managing objects in Amazon S3 storage. It allows for making and removing &quot;buckets&quot; and uploading, downloading and removing &quot;objects&quot; from these buckets.<br/> Options: -h, --help show this help message and exit --configure Invoke interactive (re)configuration tool. Optionally use as &#39;--configure s3://some-bucket&#39; to test access to a specific bucket instead of attempting to list them all. -c FILE, --config=FILE Config file name. Defaults to $HOME/.s3cfg --dump-config Dump current configuration after parsing config files and command line options and exit. --access_key=ACCESS_KEY AWS Access Key --secret_key=SECRET_KEY AWS Secret Key --access_token=ACCESS_TOKEN AWS Access Token -n, --dry-run Only show what should be uploaded or downloaded but don&#39;t actually do it. May still perform S3 requests to get bucket listings and other information though (only for file transfer commands) -s, --ssl Use HTTPS connection when communicating with S3. (default) --no-ssl Don&#39;t use HTTPS. -e, --encrypt Encrypt files before uploading to S3. --no-encrypt Don&#39;t encrypt files. -f, --force Force overwrite and other dangerous operations. --continue Continue getting a partially downloaded file (only for [get] command). --continue-put Continue uploading partially uploaded files or multipart upload parts. Restarts parts/files that don&#39;t have matching size and md5. Skips files/parts that do. Note: md5sum checks are not always sufficient to check (part) file equality. Enable this at your own risk. --upload-id=UPLOAD_ID UploadId for Multipart Upload, in case you want continue an existing upload (equivalent to --continue- put) and there are multiple partial uploads. Use s3cmd multipart [URI] to see what UploadIds are associated with the given URI. --skip-existing Skip over files that exist at the destination (only for [get] and [sync] commands). -r, --recursive Recursive upload, download or removal. --check-md5 Check MD5 sums when comparing files for [sync]. (default) --no-check-md5 Do not check MD5 sums when comparing files for [sync]. Only size will be compared. May significantly speed up transfer but may also miss some changed files. -P, --acl-public Store objects with ACL allowing read for anyone. --acl-private Store objects with default ACL allowing access for you only. --acl-grant=PERMISSION:EMAIL or USER_CANONICAL_ID Grant stated permission to a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all --acl-revoke=PERMISSION:USER_CANONICAL_ID Revoke stated permission for a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all -D NUM, --restore-days=NUM Number of days to keep restored file available (only for &#39;restore&#39; command). Default is 1 day. --restore-priority=RESTORE_PRIORITY Priority for restoring files from S3 Glacier (only for &#39;restore&#39; command). Choices available: bulk, standard, expedited --delete-removed Delete destination objects with no corresponding source file [sync] --no-delete-removed Don&#39;t delete destination objects [sync] --delete-after Perform deletes AFTER new uploads when delete-removed is enabled [sync] --delay-updates *OBSOLETE* Put all updated files into place at end [sync] --max-delete=NUM Do not delete more than NUM files. [del] and [sync] --limit=NUM Limit number of objects returned in the response body (only for [ls] and [la] commands) --add-destination=ADDITIONAL_DESTINATIONS Additional destination for parallel uploads, in addition to last arg. May be repeated. --delete-after-fetch Delete remote objects after fetching to local file (only for [get] and [sync] commands). -p, --preserve Preserve filesystem attributes (mode, ownership, timestamps). Default for [sync] command. --no-preserve Don&#39;t store FS attributes --exclude=GLOB Filenames and paths matching GLOB will be excluded from sync --exclude-from=FILE Read --exclude GLOBs from FILE --rexclude=REGEXP Filenames and paths matching REGEXP (regular expression) will be excluded from sync --rexclude-from=FILE Read --rexclude REGEXPs from FILE --include=GLOB Filenames and paths matching GLOB will be included even if previously excluded by one of --(r)exclude(-from) patterns --include-from=FILE Read --include GLOBs from FILE --rinclude=REGEXP Same as --include but uses REGEXP (regular expression) instead of GLOB --rinclude-from=FILE Read --rinclude REGEXPs from FILE --files-from=FILE Read list of source-file names from FILE. Use - to read from stdin. --region=REGION, --bucket-location=REGION Region to create bucket in. As of now the regions are: us-east-1, us-west-1, us-west-2, eu-west-1, eu- central-1, ap-northeast-1, ap-southeast-1, ap- southeast-2, sa-east-1 --host=HOSTNAME HOSTNAME:PORT for S3 endpoint (default: s3.amazonaws.com, alternatives such as s3-eu- west-1.amazonaws.com). You should also set --host- bucket. --host-bucket=HOST_BUCKET DNS-style bucket+hostname:port template for accessing a bucket (default: %(bucket)s.s3.amazonaws.com) --reduced-redundancy, --rr Store object with &#39;Reduced redundancy&#39;. Lower per-GB price. [put, cp, mv] --no-reduced-redundancy, --no-rr Store object without &#39;Reduced redundancy&#39;. Higher per- GB price. [put, cp, mv] --storage-class=CLASS Store object with specified CLASS (STANDARD, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER or DEEP_ARCHIVE). [put, cp, mv] --access-logging-target-prefix=LOG_TARGET_PREFIX Target prefix for access logs (S3 URI) (for [cfmodify] and [accesslog] commands) --no-access-logging Disable access logging (for [cfmodify] and [accesslog] commands) --default-mime-type=DEFAULT_MIME_TYPE Default MIME-type for stored objects. Application default is binary/octet-stream. -M, --guess-mime-type Guess MIME-type of files by their extension or mime magic. Fall back to default MIME-Type as specified by --default-mime-type option --no-guess-mime-type Don&#39;t guess MIME-type and use the default type instead. --no-mime-magic Don&#39;t use mime magic when guessing MIME-type. -m MIME/TYPE, --mime-type=MIME/TYPE Force MIME-type. Override both --default-mime-type and --guess-mime-type. --add-header=NAME:VALUE Add a given HTTP header to the upload request. Can be used multiple times. For instance set &#39;Expires&#39; or &#39;Cache-Control&#39; headers (or both) using this option. --remove-header=NAME Remove a given HTTP header. Can be used multiple times. For instance, remove &#39;Expires&#39; or &#39;Cache- Control&#39; headers (or both) using this option. [modify] --server-side-encryption Specifies that server-side encryption will be used when putting objects. [put, sync, cp, modify] --server-side-encryption-kms-id=KMS_KEY Specifies the key id used for server-side encryption with AWS KMS-Managed Keys (SSE-KMS) when putting objects. [put, sync, cp, modify] --encoding=ENCODING Override autodetected terminal and filesystem encoding (character set). Autodetected: UTF-8 --add-encoding-exts=EXTENSIONs Add encoding to these comma delimited extensions i.e. (css,js,html) when uploading to S3 ) --verbatim Use the S3 name as given on the command line. No pre- processing, encoding, etc. Use with caution! --disable-multipart Disable multipart upload on files bigger than --multipart-chunk-size-mb --multipart-chunk-size-mb=SIZE Size of each chunk of a multipart upload. Files bigger than SIZE are automatically uploaded as multithreaded- multipart, smaller files are uploaded using the traditional method. SIZE is in Mega-Bytes, default chunk size is 15MB, minimum allowed chunk size is 5MB, maximum is 5GB. --list-md5 Include MD5 sums in bucket listings (only for &#39;ls&#39; command). -H, --human-readable-sizes Print sizes in human readable form (eg 1kB instead of 1234). --ws-index=WEBSITE_INDEX Name of index-document (only for [ws-create] command) --ws-error=WEBSITE_ERROR Name of error-document (only for [ws-create] command) --expiry-date=EXPIRY_DATE Indicates when the expiration rule takes effect. (only for [expire] command) --expiry-days=EXPIRY_DAYS Indicates the number of days after object creation the expiration rule takes effect. (only for [expire] command) --expiry-prefix=EXPIRY_PREFIX Identifying one or more objects with the prefix to which the expiration rule applies. (only for [expire] command) --progress Display progress meter (default on TTY). --no-progress Don&#39;t display progress meter (default on non-TTY). --stats Give some file-transfer stats. --enable Enable given CloudFront distribution (only for [cfmodify] command) --disable Disable given CloudFront distribution (only for [cfmodify] command) --cf-invalidate Invalidate the uploaded filed in CloudFront. Also see [cfinval] command. --cf-invalidate-default-index When using Custom Origin and S3 static website, invalidate the default index file. --cf-no-invalidate-default-index-root When using Custom Origin and S3 static website, don&#39;t invalidate the path to the default index file. --cf-add-cname=CNAME Add given CNAME to a CloudFront distribution (only for [cfcreate] and [cfmodify] commands) --cf-remove-cname=CNAME Remove given CNAME from a CloudFront distribution (only for [cfmodify] command) --cf-comment=COMMENT Set COMMENT for a given CloudFront distribution (only for [cfcreate] and [cfmodify] commands) --cf-default-root-object=DEFAULT_ROOT_OBJECT Set the default root object to return when no object is specified in the URL. Use a relative path, i.e. default/index.html instead of /default/index.html or s3://bucket/default/index.html (only for [cfcreate] and [cfmodify] commands) -v, --verbose Enable verbose output. -d, --debug Enable debug output. --version Show s3cmd version (2.2.0) and exit. -F, --follow-symlinks Follow symbolic links as if they are regular files --cache-file=FILE Cache FILE containing local source MD5 values -q, --quiet Silence output on stdout --ca-certs=CA_CERTS_FILE Path to SSL CA certificate FILE (instead of system default) --ssl-cert=SSL_CLIENT_CERT_FILE Path to client own SSL certificate CRT_FILE --ssl-key=SSL_CLIENT_KEY_FILE Path to client own SSL certificate private key KEY_FILE --check-certificate Check SSL certificate validity --no-check-certificate Do not check SSL certificate validity --check-hostname Check SSL certificate hostname validity --no-check-hostname Do not check SSL certificate hostname validity --signature-v2 Use AWS Signature version 2 instead of newer signature methods. Helpful for S3-like systems that don&#39;t have AWS Signature v4 yet. --limit-rate=LIMITRATE Limit the upload or download speed to amount bytes per second. Amount may be expressed in bytes, kilobytes with the k suffix, or megabytes with the m suffix --no-connection-pooling Disable connection re-use --requester-pays Set the REQUESTER PAYS flag for operations -l, --long-listing Produce long listing [ls] --stop-on-error stop if error in transfer --content-disposition=CONTENT_DISPOSITION Provide a Content-Disposition for signed URLs, e.g., &quot;inline; filename=myvideo.mp4&quot; --content-type=CONTENT_TYPE Provide a Content-Type for signed URLs, e.g., &quot;video/mp4&quot;<br/> Commands: Make bucket s3cmd mb s3://BUCKET Remove bucket s3cmd rb s3://BUCKET List objects or buckets s3cmd ls [s3://BUCKET[/PREFIX]] List all object in all buckets s3cmd la Put file into bucket s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX] Get file from bucket s3cmd get s3://BUCKET/OBJECT LOCAL_FILE Delete file from bucket s3cmd del s3://BUCKET/OBJECT Delete file from bucket (alias for del) s3cmd rm s3://BUCKET/OBJECT Restore file from Glacier storage s3cmd restore s3://BUCKET/OBJECT Synchronize a directory tree to S3 (checks files freshness using size and md5 checksum, unless overridden by options, see below) s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR or s3://BUCKET[/PREFIX] s3://BUCKET[/PREFIX] Disk usage by buckets s3cmd du [s3://BUCKET[/PREFIX]] Get various information about Buckets or Files s3cmd info s3://BUCKET[/OBJECT] Copy object s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2] Modify object metadata s3cmd modify s3://BUCKET1/OBJECT Move object s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2] Modify Access control list for Bucket or Files s3cmd setacl s3://BUCKET[/OBJECT] Modify Bucket Policy s3cmd setpolicy FILE s3://BUCKET Delete Bucket Policy s3cmd delpolicy s3://BUCKET Modify Bucket CORS s3cmd setcors FILE s3://BUCKET Delete Bucket CORS s3cmd delcors s3://BUCKET Modify Bucket Requester Pays policy s3cmd payer s3://BUCKET Show multipart uploads s3cmd multipart s3://BUCKET [Id] Abort a multipart upload s3cmd abortmp s3://BUCKET/OBJECT Id List parts of a multipart upload s3cmd listmp s3://BUCKET/OBJECT Id Enable/disable bucket access logging s3cmd accesslog s3://BUCKET Sign arbitrary string using the secret key s3cmd sign STRING-TO-SIGN Sign an S3 URL to provide limited public access with expiry s3cmd signurl s3://BUCKET/OBJECT &lt;expiry_epoch|+expiry_offset&gt; Fix invalid file names in a bucket s3cmd fixbucket s3://BUCKET[/PREFIX] Create Website from bucket s3cmd ws-create s3://BUCKET Delete Website s3cmd ws-delete s3://BUCKET Info about Website s3cmd ws-info s3://BUCKET Set or delete expiration rule for the bucket s3cmd expire s3://BUCKET Upload a lifecycle policy for the bucket s3cmd setlifecycle FILE s3://BUCKET Get a lifecycle policy for the bucket s3cmd getlifecycle s3://BUCKET Remove a lifecycle policy for the bucket s3cmd dellifecycle s3://BUCKET List CloudFront distribution points s3cmd cflist Display CloudFront distribution point parameters s3cmd cfinfo [cf://DIST_ID] Create CloudFront distribution point s3cmd cfcreate s3://BUCKET Delete CloudFront distribution point s3cmd cfdelete cf://DIST_ID Change CloudFront distribution point parameters s3cmd cfmodify cf://DIST_ID Display CloudFront invalidation request(s) status s3cmd cfinvalinfo cf://DIST_ID[/INVAL_ID]<br/> For more information, updates and news, visit the s3cmd website: http://s3tools.org </span></pre> </edit-fold> <edit-fold setup> <h3 id="signup">Signup</h3> <p> I went to the <a href='https://login.linode.com/signup' target='_blank' rel='nofollow'>Linode signup page</a> and signed up with my email. Using third parties for authentication means they track you more easily, and introduces an unnecessary dependency. </p> <p> The email signup procedure requires a mobile phone for SMS-based MFA. I would rather use a <a href='https://datatracker.ietf.org/doc/html/rfc6238' target='_blank' rel='nofollow'>TOTP authenticator app</a> instead of SMS for MFA. </p> <div style=""> <picture> <source srcset="/blog/images/linode/signup.webp" type="image/webp"> <source srcset="/blog/images/linode/signup.png" type="image/png"> <img src="/blog/images/linode/signup.png" class=" liImg2 rounded shadow" /> </picture> </div> <p> I want to ensure that I limit my financial liability. One way of protecting myself is to limit the maximum amount that can be charged to the payment mechanism. Because PayPal has no maximum transaction limit, it is not a suitable choice for a service with unlimited financial liability. However, credit cards can have transaction limits set, and Google Pay has the following limits: </p> <div class='quote'> If you <a href='https://support.google.com/googlepay/answer/10623602' target='_blank' rel='nofollow'>set up your Google Pay balance to make contactless payments</a>, there are some transaction limits: <br><br> Maximum single transaction amount: $2,000 USD.<br> Daily maximum total transaction amount: $2,500 USD.<br> Up to 15 transactions per day.<br> Additional limits on the dollar amount or frequency of transactions may be imposed in accordance with the Google Pay Terms of Service. <br><br> <span style='font-style:normal;'> &nbsp;&ndash; From <a href='https://support.google.com/googlepay/answer/10187490#:~:text=If%20you%20set%20up%20your,to%2015%20transactions%20per%20day.' rel='nofollow' target='_blank'>Google Pay Help</a></span> </div> <p> <a href='https://support.google.com/googlepay/answer/10187490' target='_blank' rel='nofollow'>Here</a> are additional limits for Google Pay. Because of prior good experiences with how credit card processors handled fraud, I decided to use a credit card with a low limit instead of Google Pay. </p> </edit-fold> <edit-fold keys> <h3 id="keys">Generating Linode Access Keys</h3> <p> I followed the <a href='https://www.linode.com/docs/products/storage/object-storage/guides/access-keys/' target='_blank' rel='nofollow'>directions</a>: </p> <ol> <li>Logged into the <a href='https://cloud.linode.com/' target='_blank' rel='nofollow'>Cloud Manager</a>.</li> <li>Selected the <b>Object Storage</b> menu item in the sidebar and clicked on the <b>Access Keys</b> tab.</li> <li>Clicked on the <b>Create Access</b> Key button, which displays the <b>Create Access Key</b> panel.</li> <li>Typed in the name <code>mslinn</code> as the label for the new access key.</li> <li>Clicked the <kbd>Submit</kbd> button to create the access key.</li> <li>The new access key and its secret key are displayed. This is the only time that the secret key is visible. I stored the access key and the secret key in my password manager, <a href='https://lastpass.com' target='_blank' rel='nofollow'><code>lastpass.com</code></a>.</li> </ol> </edit-fold> <edit-fold s3cmd> <h3 id="s3cmd_cfg">Configuring <span class="code">s3cmd</span></h3> <p> <code>s3cmd</code> is a Python program that works with all S3-compatible APIs, such as those provided by AWS, Linode Storage, Google Cloud Storage and DreamHost DreamObjects. I intend to use its <a href='https://s3tools.org/s3cmd-sync' target='_blank' rel='nofollow'><code>sync</code></a> subcommand to synchronize the <code>www.mslinn.com</code> bucket contents in Linode Storage with the most recently generated version of this website by Jekyll. </p> <p> <code>s3cmd</code> needs to be <a href='https://s3tools.org/kb/item14.htm' target='_blank' rel='nofollow'>configured</a> for <a href='https://www.linode.com/docs/products/storage/object-storage/guides/s3cmd' target='_blank' rel='nofollow'>Linode</a> before it can be used. Some configuration settings for Linode are non-obvious. The following should work for most users, with the caution that the access key and secret key are of course unique for every user. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idebd4e74f4c91'><button class='copyBtn' data-clipboard-target='#idebd4e74f4c91' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd --configure <span class='unselectable'><br>Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options.<br/> Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. Access Key [asdfasdf]: </span>asdfasdfasdf <span class='unselectable'>Secret Key [asdfasdfasdf]: </span>asdfasdfasdf <span class='unselectable'>Default Region [US]:<br/> Use &quot;s3.amazonaws.com&quot; for S3 Endpoint and not modify it to the target Amazon S3. S3 Endpoint [s3.amazonaws.com]: </span>us-east-1.linodeobjects.com<br/> <span class='unselectable'>Use &quot;%(bucket)s.s3.amazonaws.com&quot; to the target Amazon S3. &quot;%(bucket)s&quot; and &quot;%(location)s&quot; vars can be used if the target S3 system supports dns based buckets. DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: </span>%(bucket)s.us-east-1.linodeobjects.com<br/> <span class='unselectable'>Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: Path to GPG program [/usr/bin/gpg]:<br/> When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP, and can only be proxied with Python 2.7 or newer Use HTTPS protocol [Yes]:<br/> On some networks all internet access must go through a HTTP proxy. Try setting it here if you can&#39;t connect to S3 directly HTTP Proxy server name:<br/> New settings: Access Key: </span>asdfasdfasdf <span class='unselectable'>Secret Key: </span>asdfasdfasdf <span class='unselectable'>Default Region: US S3 Endpoint: </span>https://us-east-1.linodeobjects.com <span class='unselectable'>DNS-style bucket+hostname:port template for accessing a bucket: </span>%(bucket)s.us-east-1.linodeobjects.com <span class='unselectable'>Encryption password: Path to GPG program: /usr/bin/gpg Use HTTPS protocol: True HTTP Proxy server name: HTTP Proxy server port: 0<br/> Test access with supplied credentials? [Y/n] n Save settings? [y/N] y Configuration saved to '/home/mslinn/.s3cfg' </span></pre> <p> I was unable to successfully test access. Two different errors appeared at different times: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd0f88f792e0f'>Please wait, attempting to list all buckets... ERROR: Test failed: 403 (SignatureDoesNotMatch) ERROR: Test failed: [Errno -2] Name or service not known<br/></pre> <p> Eventually I saved the configuration without testing as shown above. This created a file called <code>$HOME/.s3cfg</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>~/.s3cfg</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id16fe439db82c'>[default] access_key = asdfasdfasdf access_token = add_encoding_exts = add_headers = bucket_location = US ca_certs_file = cache_file = check_ssl_certificate = True check_ssl_hostname = True cloudfront_host = cloudfront.amazonaws.com connection_max_age = 5 connection_pooling = True content_disposition = content_type = default_mime_type = binary/octet-stream delay_updates = False delete_after = False delete_after_fetch = False delete_removed = False dry_run = False enable_multipart = True encoding = UTF-8 encrypt = False expiry_date = expiry_days = expiry_prefix = follow_symlinks = False force = False get_continue = False gpg_command = /usr/bin/gpg gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_passphrase = guess_mime_type = True host_base = us-east-1.linodeobjects.com host_bucket = %(bucket)s.us-east-1.linodeobjects.com human_readable_sizes = False invalidate_default_index_on_cf = False invalidate_default_index_root_on_cf = True invalidate_on_cf = False kms_key = limit = -1 limitrate = 0 list_md5 = False log_target_prefix = long_listing = False max_delete = -1 mime_type = multipart_chunk_size_mb = 15 multipart_copy_chunk_size_mb = 1024 multipart_max_chunks = 10000 preserve_attrs = True progress_meter = True proxy_host = proxy_port = 0 public_url_use_https = False put_continue = False recursive = False recv_chunk = 65536 reduced_redundancy = False requester_pays = False restore_days = 1 restore_priority = Standard secret_key = asdfasdfasdf send_chunk = 65536 server_side_encryption = False signature_v2 = False signurl_use_https = False simpledb_host = sdb.amazonaws.com skip_existing = False socket_timeout = 300 ssl_client_cert_file = ssl_client_key_file = stats = False stop_on_error = False storage_class = throttle_max = 100 upload_id = urlencoding_mode = normal use_http_expect = False use_https = True use_mime_magic = True verbosity = INFO website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/ website_error = website_index = index.html</pre> <p> According to the docs (<a href='https://www.linode.com/docs/products/storage/object-storage/guides/urls/' target='_blank' rel='nofollow'>Access Buckets and Files through URLs</a>), the configuration file needs <code>website_endpoint: http://%(bucket)s.website-[cluster-url]/</code>. However, this is an error; instead of <code>cluster-url</code>, which might be <code>https://us-east-1.linodeobjects.com</code>, <code>cluster-id</code> should be used, which for me was simply <code>us-east-1</code>. <span class="bg_yellow">Thus the endpoint for me, and everyone with a bucket at <code>us-east-1</code> in Newark, New Jersey should be:<br> <code>http://%(bucket)s.<span class="bg_yellow">website-</span>us-east-1.linodeobjects.com/</code></span> </p> <p> After editing the file to manually change the value of <code>website_endpoint</code>, I tried to list the buckets, and that worked: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7e44dae69dbe'><button class='copyBtn' data-clipboard-target='#id7e44dae69dbe' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd ls <span class='unselectable'>2022-07-01 17:32 s3://mslinn </span></pre> </edit-fold> <edit-fold linodecli> <h3 id="linode-cli"><span class="code">linode-cli</span></h3> <p> Next I tried <a href='https://github.com/linode/linode-cli' target='_blank' rel='nofollow'><code>linode-cli</code></a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id857ee0c334f0'><button class='copyBtn' data-clipboard-target='#id857ee0c334f0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pip install linode-cli <span class='unselectable'>Collecting linode-cli Downloading linode_cli-5.21.0-py2.py3-none-any.whl (204 kB) &#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473; 204.2/204.2 kB 4.8 MB/s eta 0:00:00 Collecting terminaltables Downloading terminaltables-3.1.10-py2.py3-none-any.whl (15 kB) Collecting PyYAML Downloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB) &#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473; 682.2/682.2 kB 16.8 MB/s eta 0:00:00 Requirement already satisfied: requests in /home/mslinn/venv/default/lib/python3.10/site-packages (from linode-cli) (2.28.0) Requirement already satisfied: charset-normalizer~=2.0.0 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests-&gt;linode-cli) (2.0.12) Requirement already satisfied: idna&lt;4,&gt;=2.5 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests-&gt;linode-cli) (3.3) Requirement already satisfied: urllib3&lt;1.27,&gt;=1.21.1 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests-&gt;linode-cli) (1.26.9) Requirement already satisfied: certifi&gt;=2017.4.17 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests-&gt;linode-cli) (2022.6.15) Installing collected packages: terminaltables, PyYAML, linode-cli Successfully installed PyYAML-6.0 linode-cli-5.21.0 terminaltables-3.1.10 </span></pre> <p> The documentation fails to mention that the first time <code>linode-cli</code> is executed, and every time it is invoked with the <code>configure</code> subcommand, it attempts to open a web browser. WSL and WSL2 will not respond appropriately unless an X client is already running, for example <a href='https://apps.microsoft.com/store/detail/gwsl/9NL6KD1H33V3' target='_blank' rel='nofollow'>GWSL</a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id358ac00e4dfd'><button class='copyBtn' data-clipboard-target='#id358ac00e4dfd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>linode-cli configure <span class='unselectable'><br>Welcome to the Linode CLI. This will walk you through some initial setup.<br/> The CLI will use its web-based authentication to log you in. If you prefer to supply a Personal Access Token, use <code>linode-cli configure --token</code><br/> Press enter to continue. This will open a browser and proceed with authentication. A browser should open directing you to this URL to authenticate:<br/> https://login.linode.com/oauth/authorize?client_id=asdfasdfasdf&amp;response_type=token&amp;scopes=*&amp;redirect_uri=http://localhost:45789<br/> If you are not automatically directed there, please copy/paste the link into your browser to continue..<br/><br/> Configuring mslinn<br/><br/> Default Region for operations. Choices are: 1 - ap-west 2 - ca-central 3 - ap-southeast 4 - us-central 5 - us-west 6 - us-southeast 7 - us-east 8 - eu-west 9 - ap-south 10 - eu-central 11 - ap-northeast<br/> Default Region (Optional): 7<br/> Default Type of Linode to deploy. Choices are: 1 - g6-nanode-1 2 - g6-standard-1 3 - g6-standard-2 4 - g6-standard-4 5 - g6-standard-6 6 - g6-standard-8 7 - g6-standard-16 8 - g6-standard-20 9 - g6-standard-24 10 - g6-standard-32 11 - g7-highmem-1 12 - g7-highmem-2 13 - g7-highmem-4 14 - g7-highmem-8 15 - g7-highmem-16 16 - g6-dedicated-2 17 - g6-dedicated-4 18 - g6-dedicated-8 19 - g6-dedicated-16 20 - g6-dedicated-32 21 - g6-dedicated-48 22 - g6-dedicated-50 23 - g6-dedicated-56 24 - g6-dedicated-64 25 - g1-gpu-rtx6000-1 26 - g1-gpu-rtx6000-2 27 - g1-gpu-rtx6000-3 28 - g1-gpu-rtx6000-4<br/> Default Type of Linode (Optional):<br/> Default Image to deploy to new Linodes. Choices are: 1 - linode/almalinux8 2 - linode/almalinux9 3 - linode/alpine3.12 4 - linode/alpine3.13 5 - linode/alpine3.14 6 - linode/alpine3.15 7 - linode/alpine3.16 8 - linode/arch 9 - linode/centos7 10 - linode/centos-stream8 11 - linode/centos-stream9 12 - linode/debian10 13 - linode/debian11 14 - linode/debian9 15 - linode/fedora34 16 - linode/fedora35 17 - linode/fedora36 18 - linode/gentoo 19 - linode/kali 20 - linode/debian11-kube-v1.20.15 21 - linode/debian9-kube-v1.20.7 22 - linode/debian9-kube-v1.21.1 23 - linode/debian11-kube-v1.21.12 24 - linode/debian9-kube-v1.22.2 25 - linode/debian11-kube-v1.22.9 26 - linode/debian11-kube-v1.23.6 27 - linode/opensuse15.3 28 - linode/opensuse15.4 29 - linode/rocky8 30 - linode/slackware14.2 31 - linode/slackware15.0 32 - linode/ubuntu16.04lts 33 - linode/ubuntu18.04 34 - linode/ubuntu20.04 35 - linode/ubuntu21.10 36 - linode/ubuntu22.04 37 - linode/centos8 38 - linode/slackware14.1 39 - linode/ubuntu21.04<br/> Default Image (Optional): Active user is now mslinn<br/> Config written to /home/mslinn/.config/linode-cli </span></pre> <p> <code>linode-cli</code> configuration created this file: </p> <div class='codeLabel unselectable' data-lt-active='false'>~/.config/linode-cli</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5eebee2d0a2a'>[DEFAULT] default-user = mslinn [mslinn] token = asdfasdfasdfasdfasdfasdf region = us-east</pre> <p> Right after I did the above, I was surprised to get the following email from Linode: </p> <div class="quote"> Hello,<br><br> This is a notification to inform you that a device has been trusted to skip authentication on your Linode Account (mslinn). The request came from the following IP address: 70.53.179.203. <br><br> The device will not be prompted for a username or password for 30 days. <br><br> If this action did not originate from you, we recommend logging in and changing your password immediately. Also, as an extra layer of security, we highly recommend enabling two-factor authentication on your account. Please see the link below for more information on how to enable two-factor authentication: <br><br> <a href='https://www.linode.com/docs/security/linode-manager-security-controls/#enable-two-factor-authentication' target='_blank' rel='nofollow'>Two-Factor Authentication</a> <br><br> If you have any questions or concerns, please do not hesitate to contact us 24/7 by opening a support ticket from within the Linode Manager, giving us a call, or emailing <a href='mailto:support@linode.com'>support@linode.com</a>. <br><br> <a href='https://www.linode.com/contact' target='_blank' rel='nofollow'>Contact</a> <br><br> Thank you,<br> The Linode.com Team<br> <br><br> <a href='mailto:support@linode.com'>Contact Us</a> </div> <p class="alert rounded shadow"> I began to suspect that Linode&rsquo;s documentation had major deficiencies at this point. Instead of merely configuring a command-line client, my IP was whitelisted unexpectedly. I have nothing polite to say about this. </p> <p> After poking around a bit, I discovered my <a href='https://cloud.linode.com/profile/tokens' target='_blank' rel='nofollow'>API Tokens page</a> on Linode. It seems that the <code>linode-cli</code> created a personal access token with a label containing the name of the computer used: <code>Linode CLI @ camille</code>. </p> <div style=""> <picture> <source srcset="/blog/images/linode/apiTokens.webp" type="image/webp"> <source srcset="/blog/images/linode/apiTokens.png" type="image/png"> <img src="/blog/images/linode/apiTokens.png" class=" liImg2 rounded shadow" /> </picture> </div> <p> Clicking on the <kbd>View Scopes</kbd> button above displays the permissions granted to the token: </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/linode/cliPermissions.webp" type="image/webp"> <source srcset="/blog/images/linode/cliPermissions.png" type="image/png"> <img src="/blog/images/linode/cliPermissions.png" class="center halfsize liImg2 rounded shadow" /> </picture> </div> <p class="alert rounded shadow"> This personal access token is all-powerful. The <code>linode-cli</code> quietly created a personal access token with all permissions enabled, and did not warn the user, and did not say how to reduce or limit the permissions. <br><br> Security is enhanced when the minimum permission necessary is provided to accomplish necessary tasks. Instead, this token silently maximizes the financial risk to the person or entity who pays for the account. </p> </edit-fold> <edit-fold bucket> <h3 id="bucket">Creating the Website Bucket</h3> <p class="alert rounded shadow"> The name of the bucket must exactly match the name of the subdomain that the website is served from. If you try to serve content from a misnamed bucket, you will not be able to apply an SSL certificate; instead, the certificate will be issued by <code>linodeobjects.com</code>. </p> <p> I wanted to test using the subdomain <code>linode.mslinn.com</code>, and then if all was well, switch <code>www.mslinn.com</code> over to Linode Storage. Because buckets cannot be renamed, I had to make 2 buckets with identical contents, one called <code>linode.mslinn.com</code> and the other called <code>www.mslinn.com</code>. If I decide to stay with Linode, I will delete the bucket I am using for testing, called <code>linode.mslinn.com</code>, and run this website from the <code>www.mslinn.com</code> bucket. </p> <p> I created a website-enabled bucket called <code>www.mslinn.com</code> as follows: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id73d3f4077c45'><button class='copyBtn' data-clipboard-target='#id73d3f4077c45' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd mb --acl-public s3://www.mslinn.com <span class='unselectable'>Bucket 's3://www.mslinn.com/' created </span> <span class='unselectable'>$ </span>s3cmd ls <span class='unselectable'>2022-07-01 17:32 s3://www.mslinn.com </span> <span class='unselectable'>$ </span>s3cmd ws-create \ --ws-index=index.html \ --ws-error=404.html \ s3://www.mslinn.com <span class='unselectable'>Bucket 's3://www.mslinn.com/': website configuration created. </span></pre> <p> The <code>s3cmd ws-info</code> subcommand displays the Linode Object Storage bucket URL as the <b>Website endpoint</b>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id355d3b18d5d8'><button class='copyBtn' data-clipboard-target='#id355d3b18d5d8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd ws-info s3://www.mslinn.com <span class='unselectable'>Bucket s3://www.mslinn.com/: Website configuration Website endpoint: http://www.mslinn.com.website-us-east-1.linodeobjects.com/ Index document: index.html Error document: 404.html </span></pre> <h3 id="subdomain">Defining a CNAME for the Bucket</h3> <p> Although Linode provides a <a href='https://www.linode.com/docs/guides/dns-manager/' target='_blank' rel='nofollow'>DNS Manager</a>, while I am testing out Linode I wanted to continue using Namecheap for DNS, so I navigated to <a href='https://ap.www.namecheap.com/Domains/DomainControlPanel/mslinn.com/advancedns' target='_blank' rel='nofollow'><code>ap.www.namecheap.com/Domains/DomainControlPanel/mslinn.com/advancedns</code></a> and created CNAMEs like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcaebff3b6c46'><button class='copyBtn' data-clipboard-target='#idcaebff3b6c46' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>linode.mslinn.com CNAME linode.mslinn.com.<span class="bg_yellow">website-</span>us-east-1.linodeobjects.com www.mslinn.com CNAME www.mslinn.com.<span class="bg_yellow">website-</span>us-east-1.linodeobjects.com</pre> <p class="alert rounded shadow"> Warning: if the <code>CNAME</code> does not point to a URL with the word <code>website</code> in it, for example <code>www.mslinn.com.us-east-1.linodeobjects.com</code>, the default index file and the error file will not automatically be served when expected. </p> </edit-fold> <edit-fold cert> <h3 id="cert">Making a Free SSL Certificate</h3> <p> <a href='https://www.ssllabs.com/ssltest/' target='_blank' rel='nofollow'>Qualys / SSL Labs</a> rates the SSL security provided by AWS CloudFront using the AWS-generated SSL certificates with a &ldquo;B&rdquo; grade. </p> <div style=""> <picture> <source srcset="/blog/images/aws/sslReport.webp" type="image/webp"> <source srcset="/blog/images/aws/sslReport.png" type="image/png"> <img src="/blog/images/aws/sslReport.png" class=" liImg2 rounded shadow" /> </picture> </div> <p> Lets see if Linode Object Storage can host a more secure website when provided with a 4096-bit certificate generated by <code>certbot</code>/<code>letsencrypt</code>. The conclusion of this post will reveal the answer. </p> <p> Please read <a href='/blog/2022/06/15/certbot.html'>Creating and Renewing Letsencrypt Wildcard SSL Certificates</a> for details. </p> </edit-fold> <edit-fold linode_dns> <h4 id="linodeDns">Linode DNS Authentication</h4> <p> Eventually, if I stick with Linode, I could use the <a href='https://certbot-dns-linode.readthedocs.io/en/stable/' target='_blank' rel='nofollow'><code>certbot</code> DNS plugin for Linode</a>, but I will not use it right now because I am trialing Linode. Use of this plugin requires a configuration file containing Linode API credentials, obtained from your Linode account&rsquo;s <a href='https://cloud.linode.com/profile/tokens' target='_blank' rel='nofollow'>Applications &amp; API Tokens</a> page. I stored the credentials in <code>~/.certbot/linode.ini</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>~/.certbot/linode.ini</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id41240cc2f3ed'><button class='copyBtn' data-clipboard-target='#id41240cc2f3ed' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># Linode API credentials used by Certbot dns_linode_key = 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ64 dns_linode_version = [<blank>|3|4]</pre> <p> Here is how to create an SSL certificate <code>certbot</code> using Linode DNS authentication: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9e615c581ac6'><button class='copyBtn' data-clipboard-target='#id9e615c581ac6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>certbot certonly \ --dns-linode \ --dns-linode-credentials ~/.certbot/linode.ini \ --rsa-key-size 4096 \ -d mslinn.com \ -d *.mslinn.com \ --config-dir ~/.certbot/mslinn.com/config \ --logs-dir ~/.certbot/mslinn.com/logs \ --work-dir ~/.certbot/mslinn.com/work</pre> <h2 id="certUp">Installing the Certificate</h2> <p> Now I created 2 certificates: one with the full chain of responsibility (<code>fullchain.pem</code>) and one containing my private key (<code>privkey.pem</code>). I used bash environment variables called <code>CERT_DIR</code>, <code>CERT</code> and <code>KEY</code> to make the incantation convenient to type in. <code>CERT</code> and <code>KEY</code> contain the contents of the files <code>fullchain.pem</code> and <code>privkey.pem</code>, respectively. As you can see, the result was a valid (wildcard) SSL certificate. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida915dcbfa7e7'><button class='copyBtn' data-clipboard-target='#ida915dcbfa7e7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>CERT_DIR=/home/mslinn/.certbot/mslinn.com/config/live/mslinn.com <span class='unselectable'>$ </span>CERT="$( cat $CERT_DIR/fullchain.pem )" <span class='unselectable'>$ </span>KEY="$( cat $CERT_DIR/privkey.pem )" <span class='unselectable'>$ </span>linode-cli object-storage ssl-upload \ us-east-1 www.mslinn.com \ --certificate "$CERT" \ --private_key "$KEY" <div style="line-height: 100%;"><span class='unselectable'>┌──────┐ │ ssl │ ├──────┤ │ True │ └──────┘ </span></div></pre> </edit-fold> <edit-fold sync> <h3 id="sync">Syncing the Website Bucket</h3> <p> I wrote a bash script to build this Jekyll-generated website. Jekyll places the generated website in a directory called <code>_site</code>. I <a href='https://s3tools.org/s3cmd-sync' target='_blank' rel='nofollow'>synchronized</a> the contents of the freshly generated <code>mslinn.com</code> website, stored in <code>_site/</code>, with the bucket as follows. Note the trailing slash on <code>_site<span class="bg_yellow">/</span></code>, <a href='https://s3tools.org/s3cmd-sync' target='_blank' rel='nofollow'>it is significant</a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6bdb01bb81f9'><button class='copyBtn' data-clipboard-target='#id6bdb01bb81f9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd sync \ --acl-public --delete-removed --guess-mime-type --quiet \ _site<span class="bg_yellow">/</span> \ s3://www.mslinn.com <span class='unselectable'>INFO: No cache file found, creating it. INFO: Compiling list of local files... INFO: Running stat() and reading/calculating MD5 values on 2157 files, this may take some time... INFO: [1000/2157] INFO: [2000/2157] INFO: Retrieving list of remote files for s3://www.mslinn.com/_site ... INFO: Found 2157 local files, 0 remote files INFO: Verifying attributes... INFO: Summary: 2137 local files to upload, 20 files to remote copy, 0 remote files to delete ... lots more output ... Done. Uploaded 536630474 bytes in 153.6 seconds, 3.33 MB/s. </span></pre> <p> Originally, I did not provide the <code>--acl-public</code> option to <code>s3cmd sync</code>. That meant the files in the bucket were all private &ndash; the opposite of what is required for a website. Here is the incantation for making all the files in the bucket publicly readable: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5e98a9d74fbd'><button class='copyBtn' data-clipboard-target='#id5e98a9d74fbd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd setacl s3://www.mslinn.com/ \ --acl-public --recursive --quiet</pre> <p> Linode&rsquo;s S3 implementation faithfully mirrors a problem that AWS S3 has. CSS files are automatically assigned the MIME type <code>text/plain</code>, instead of <code>text/css</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1ae52a2b6ae5'><button class='copyBtn' data-clipboard-target='#id1ae52a2b6ae5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -sI http://linode.mslinn.com/assets/css/style.css | \ grep Content-Type <span class='unselectable'>Content-Type: text/plain </span></pre> <p> I re-uploaded the CSS files with the proper mime type using this incantation: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3b04cee94ac7'><button class='copyBtn' data-clipboard-target='#id3b04cee94ac7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cd _site <span class='unselectable'>$ </span>find . -name *.css -exec \ s3cmd put -P --mime-type=text/css {} s3://www.mslinn.com/{} \;</pre> <p> It should be possible to provide a MIME-type mapping somehow! Linode, if you are reading this, please provide an extension to the S3 functionality, so customers could go beyond AWS's bugs. </p> </edit-fold> <h2 id="result">The Result</h2> <p> Success! The same content can be fetched with 3 different URLs: </p> <ul> <li> <a href='https://https://linode.mslinn.com' target='_blank' rel='nofollow'><code>https://linode.mslinn.com</code></a> uses the free wildcard certificate that was created above. </li> <li> <a href='http://linode.mslinn.com.website-us-east-1.linodeobjects.com' target='_blank' rel='nofollow'><code>http://linode.mslinn.com.website-us-east-1.linodeobjects.com</code></a> works without an SSL certificate. </li> <li> <a href='https://linode.mslinn.com.website-us-east-1.linodeobjects.com' target='_blank' rel='nofollow'><code>https://linode.mslinn.com.website-us-east-1.linodeobjects.com</code></a> uses a certificate from <code>us-east-1.linodeobjects.com</code>, which shows as an error, as it should. The error would be prevented if the certificate was a wildcard certificate issued by <code>linodeobjects.com</code> instead. </li> </ul> <p> Qualys / SSL Labs rated the site on Linode Storage with the Let&rsquo;s Encrypt SSL certificate: and gave it the same rating as when the site was hosted on AWS S3 / CloudFront. </p> <div style=""> <picture> <source srcset="/blog/images/linode/sslReportLinode.webp" type="image/webp"> <source srcset="/blog/images/linode/sslReportLinode.png" type="image/png"> <img src="/blog/images/linode/sslReportLinode.png" class=" liImg2 rounded shadow" /> </picture> </div> <p> The site feels quite responsive. <span style='font-size: 3em; '>&#x1F601;</span> </p> Considering Microsoft Azure for Static Websites 2022-06-20T00:00:00-04:00 https://mslinn.github.io/blog/2022/06/20/azure <p> After realizing that <a href='/blog/2022/06/02/azure-security.html'>AWS does not integrate security with real-time billing</a>, and being presented with a huge bill after my account was hijacked, I decided to see if Microsoft Azure offered me better financial security. </p> <p> In particular, I am looking for two things: </p> <ol> <li> <b>A more secure authentication mechanism</b> &ndash; If AWS credentials are compromised, they can be used from anywhere in the world. </li> <li> <b>Restricting scope and scale of authorized services</b> &ndash; With AWS, if an IAM user has a role that allows EC2 instances to be launched, any number of any size EC2 instances can be launched. There is no way to cap the number or type of EC2 instances, and AWS real-time billing is not integrated with the authentication mechanism. Furthermore, AWS has a peculiar set of busy work tasks for victims of account hijacking that appears to be targeted more towards increasing support revenue than address root issues. This means that the bad guys have no limit to the financial damage they can incur on their victims. </li> </ol> <p class="alert rounded shadow"> For PaaS vendors such as AWS, Azure, Digital Ocean, Cloudflare, ScaleWay, etc: &ldquo;pay-as-you-go&rdquo; is shorthand for &ldquo;there is nothing you can do to limit your financial liability&rdquo;. </p> <p> I set out to discover if Azure also suffers from these same problems. </p> <h2 id="policies_roles">Azure Roles and Policies</h2> <p> The <a href='https://docs.microsoft.com/en-us/azure/virtual-machines/security-policy' target='_blank' rel='nofollow'>Azure Virtual Machines Documentation</a> contains the following passage: </p> <div class="quote"> <h3 style="margin-top: 0;">Policies</h3> <p> <a href='https://docs.microsoft.com/en-us/azure/governance/policy/overview' target='_blank' rel='nofollow'>Azure policies</a> can be used to define the desired behavior for your organization's <a href='https://docs.microsoft.com/en-us/azure/virtual-machines/windows/policy' target='_blank' rel='nofollow'>Windows VMs</a> and <a href='https://docs.microsoft.com/en-us/azure/virtual-machines/linux/policy' target='_blank' rel='nofollow'>Linux VMs</a>. By using policies, an organization can enforce various conventions and rules throughout the enterprise. Enforcement of the desired behavior can help mitigate risk while contributing to the success of the organization. </p> <h3>Azure role-based access control</h3> <p> Using <a href='https://docs.microsoft.com/en-us/azure/role-based-access-control/overview' target='_blank' rel='nofollow'>Azure role-based access control (Azure RBAC)</a>, you can segregate duties within your team and grant only the amount of access to users on your VM that they need to perform their jobs. Instead of giving everybody unrestricted permissions on the VM, you can allow only certain actions. You can configure access control for the VM in the <a href='https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal' target='_blank' rel='nofollow'>Azure portal</a>, using the <a href='https://docs.microsoft.com/en-us/cli/azure/role' target='_blank' rel='nofollow'>Azure CLI</a>, or <a href='https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-powershell' target='_blank' rel='nofollow'>Azure PowerShell</a>. </p> </div> <p> Ekran Systems publishes a <a href='https://www.ekransystem.com/en/blog/rbac-vs-abac' target='_blank' rel='nofollow'>good description</a> of various flavors of RBAC, and the more complex ABAC. Okta, Inc. also publishes a <a href='https://www.okta.com/identity-101/role-based-access-control-vs-attribute-based-access-control/' target='_blank' rel='nofollow'>good overview of RBAC and ABAC</a>. I have no connection with either company. </p> <p> I soon discovered <a href='https://docs.microsoft.com/en-us/azure/governance/policy/overview#azure-policy-and-azure-rbac' target='_blank' rel='nofollow'>the following passage</a> in the Azure documentation, which appeared to exactly match the second item in the wish list at the top of this article: </p> <p class="quote"> Azure RBAC focuses on managing user <a href='https://docs.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations' target='_blank' rel='nofollow'>actions</a> at different scopes. If control of an action is required, then Azure RBAC is the correct tool to use. Even if an individual has access to perform an action, if the result is a non-compliant resource, Azure Policy still blocks the create or update. <br><br> The combination of Azure RBAC and Azure Policy provides full scope control in Azure. </p> <p class="alert rounded shadow"> <a href='https://docs.aws.amazon.com/prescriptive-guidance/latest/saas-multitenant-api-access-authorization/access-control-types.html' target='_blank' rel='nofollow'>AWS RBAC</a> does not provide functionality equivalent to that provided by Microsoft Azure RBAC plus Azure Policy, even when combined with other AWS functionality. </p> <h2 id="rbacAgain">No Limits On Other Azure Services</h2> <p> The financial limitations that Azure allows you to impose are only available for virtual machines. Users are subject to unlimited financial liability from other Azure services. It seems I have <strike>two</strike>three choices: </p> <ol> <li>Figure out how to set up RBAC and accept that other services might run up a huge bill.</li> <li>Look for a fixed-price hosting service</li> <li>Give <a href="#spendLimit">Azure Spending Limit</a> a try</li> </ol> <p> Before I sign off today, I encountered a blocking issue with Azure that I'd like to mention. </p> <h2 id="ad">Active Directory?</h2> <p> I want to update my website by running a command-line command that runs something analogous to <a href='https://linux.die.net/man/1/rsync' target='_blank' rel='nofollow'><code>rsync</code></a>. I spent quite some time looking for how this might be done with Azure. </p> <p> Before long, it seemed that Active Directory was the best way forward, however setting it up is daunting. Apparently a <a href='https://docs.microsoft.com/en-us/azure/active-directory/develop/app-objects-and-service-principals' target='_blank' rel='nofollow'>service principal</a> is better suited for scripts that run on-premises, like my home office. I got the impression that I need to register an app; unsure what app might be required for hosting a web site on Azure Blob Storage with Azure CDN. </p> <p> These Active Directory docs look like a lot of abstract, generalized information that is way more complex than I need. I found a <a href='https://docs.microsoft.com/en-us/azure/storage/blobs/authorize-access-azure-active-directory' target='_blank' rel='nofollow'>streamlined article</a> for my use case. Not sure what this means: &ldquo;Only storage accounts created with the Azure Resource Manager deployment model support Azure AD authorization.&rdquo; This streamlined document is not very streamlined. For me, this issue is a significant barrier to adoption. </p> <h2 id="spendLimit">Update: Azure Spending Limit</h2> <p> I just bumped into an article entitled <a href='https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/spending-limit' target='_blank' rel='nofollow'>Azure spending limit</a>. Perhaps this might be useful. I tried to find out more from Microsoft, but at first they said I would have to subscribe to an expensive support plan before anyone would speak to me. That is not how I wish to proceed. Later, I opened a ticket inquiring about Azure customers facing unlimited financial liability. </p> <h2>Update 2022-06-29</h2> <p> The ticket was escalated twice, then Microsoft sent me the following official response (emphasis is mine): </p> <p class="quote"> Correct, at this moment that feature of setting limits does not exist, <b class="bg_yellow">Azure is not able to safeguard customers from unlimited financial liability</b>. <br><br> We are a support team that we provide help with current account or billing issues, in addition to that, I would like to mention that Azure products are constantly being updated, also Azure is constantly improving to offer better services to our customers. For this reason, I would like to invite you to leave your feedback in the Azure feedback forum (<a href='https://feedback.azure.com/d365community/forum/79b1327d-d925-ec11-b6e6-000d3a4f06a4' target='_blank' rel='nofollow'>General Feedback · Community (azure.com)</a>), so your feedback is forwarded through the right channel and to the Microsoft internal resources that work on it. </p> <p> I responded by indicating that I would close my Azure account. </p> <p class="alert rounded shadow"> While the practice of imposing unlimited financial liability on customers is common today for PaaS vendors, it is unnecessary and predatory. If in the future I need to use a product or service that incurs unlimited financial liability for a client, I will require the client to accept the liability. </p> Creating and Renewing Letsencrypt Wildcard SSL Certificates 2022-06-15T00:00:00-04:00 https://mslinn.github.io/blog/2022/06/15/certbot <p> The process of making wildcard SSL certificates can be hard to get right at first. However, I use wildcard SSL certificates for most of my internet domains, so it is important to me. This blog post distills the essence of what I have found to be important when creating these certificates. </p> <edit-fold cert> <h2 id="cert">Making a Free Wildcard SSL Certificate</h2> <p> <a href='https://certbot.eff.org/' target='_blank' rel='nofollow'><code>Certbot</code></a> powers <a href='https://www.eff.org/' target='_blank' rel='nofollow'>EFF</a>&rsquo;s <a href='https://letsencrypt.org/' target='_blank' rel='nofollow'>Letsencrypt</a> capability. Source code is on <a href='https://github.com/certbot/certbot' target='_blank' rel='nofollow'>GitHub</a>. Install <code>certbot</code> on Debian distros such as Ubuntu as follows: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id797cc500c569'><button class='copyBtn' data-clipboard-target='#id797cc500c569' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install certbot <span class='unselectable'>Preparing to unpack .../06-python3-zope.event_4.4-3_all.deb ... Unpacking python3-zope.event (4.4-3) ... Selecting previously unselected package python3-zope.component. Preparing to unpack .../07-python3-zope.component_4.3.0-3_all.deb ... Unpacking python3-zope.component (4.3.0-3) ... Selecting previously unselected package python3-certbot. Preparing to unpack .../08-python3-certbot_1.21.0-1build1_all.deb ... Unpacking python3-certbot (1.21.0-1build1) ... Selecting previously unselected package python3-icu. Preparing to unpack .../09-python3-icu_2.8.1-0ubuntu2_amd64.deb ... Unpacking python3-icu (2.8.1-0ubuntu2) ... Selecting previously unselected package certbot. Preparing to unpack .../10-certbot_1.21.0-1build1_all.deb ... Unpacking certbot (1.21.0-1build1) ... Setting up python3-configargparse (1.5.3-1) ... Setting up python3-requests-toolbelt (0.9.1-1) ... Setting up python3-parsedatetime (2.6-2) ... Setting up python3-icu (2.8.1-0ubuntu2) ... Setting up python3-zope.event (4.4-3) ... Setting up python3-zope.hookable (5.1.0-1build1) ... Setting up python3-josepy (1.10.0-1) ... Setting up python3-zope.component (4.3.0-3) ... Setting up python3-acme (1.21.0-1) ... Setting up python3-certbot (1.21.0-1build1) ... Setting up certbot (1.21.0-1build1) ... Created symlink /etc/systemd/system/timers.target.wants/certbot.timer → /lib/systemd/system/certbot.timer. Processing triggers for man-db (2.10.2-1) ... Scanning processes... Scanning processor microcode... Scanning linux images... Failed to retrieve available kernel versions. Failed to check for processor microcode upgrades. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. </span></pre> <p> <code>certbot</code> has 2 subcommands of interest: <code>certonly</code> (used when creating a certificate for the first time), and <code>renew</code> (used when updating a pre-existing certificate). The creation process requires the user to do things before it can complete, so it can only be run interactively. </p> <p class="alert shadow rounded"> The files and directories creating by the process of creating a new SSL certificate should not be deleted. Contrary to what the Letsencrypt <code>certbot</code> documentation says, I have found that these files and directories allow the renewal process to proceed without requiring user interaction, so it can be scripted. </p> </edit-fold> <edit-fold help> <h2 id="help"><span class='code'>Certbot</span> Help Messages</h2> <p> Here is the help message for the <code>certbot certonly</code> subcommand: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idea5704d371eb'><button class='copyBtn' data-clipboard-target='#idea5704d371eb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>certbot certonly --help <span class='unselectable'>- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...<br/> Certbot can obtain and install HTTPS/TLS/SSL certificates. By default, it will attempt to use a webserver both for obtaining and installing the certificate. The most common SUBCOMMANDS and flags are:<br/> obtain, install, and renew certificates: (default) run Obtain &amp; install a certificate in your current webserver certonly Obtain or renew a certificate, but do not install it renew Renew all previously obtained certificates that are near expiry enhance Add security enhancements to your existing configuration -d DOMAINS Comma-separated list of domains to obtain a certificate for<br/> (the certbot apache plugin is not installed) --standalone Run a standalone webserver for authentication (the certbot nginx plugin is not installed) --webroot Place files in a server&#39;s webroot folder for authentication --manual Obtain certificates interactively, or using shell script hooks<br/> -n Run non-interactively --test-cert Obtain a test certificate from a staging server --dry-run Test &quot;renew&quot; or &quot;certonly&quot; without saving any certificates to disk<br/> manage certificates: certificates Display information about certificates you have from Certbot revoke Revoke a certificate (supply --cert-name or --cert-path) delete Delete a certificate (supply --cert-name)<br/> manage your account: register Create an ACME account unregister Deactivate an ACME account update_account Update an ACME account --agree-tos Agree to the ACME server&#39;s Subscriber Agreement -m EMAIL Email address for important account notifications<br/> More detailed help:<br/> -h, --help [TOPIC] print this message, or detailed help on a topic; the available TOPICS are:<br/> all, automation, commands, paths, security, testing, or any of the subcommands or plugins (certonly, renew, install, register, nginx, apache, standalone, webroot, etc.) -h all print a detailed help page including all topics --version print the version number - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </span></pre> <p> Here is the help message for the <code>certbot certonly</code> subcommand: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5ceba56a73d2'><button class='copyBtn' data-clipboard-target='#id5ceba56a73d2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>certbot renew --help <span class='unselectable'>- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -<br/> certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...<br/> Certbot can obtain and install HTTPS/TLS/SSL certificates. By default, it will attempt to use a webserver both for obtaining and installing the certificate. The most common SUBCOMMANDS and flags are:<br/> obtain, install, and renew certificates: (default) run Obtain &amp; install a certificate in your current webserver certonly Obtain or renew a certificate, but do not install it renew Renew all previously obtained certificates that are near expiry enhance Add security enhancements to your existing configuration -d DOMAINS Comma-separated list of domains to obtain a certificate for<br/> (the certbot apache plugin is not installed) --standalone Run a standalone webserver for authentication (the certbot nginx plugin is not installed) --webroot Place files in a server&#39;s webroot folder for authentication --manual Obtain certificates interactively, or using shell script hooks<br/> -n Run non-interactively --test-cert Obtain a test certificate from a staging server --dry-run Test &quot;renew&quot; or &quot;certonly&quot; without saving any certificates to disk<br/> manage certificates: certificates Display information about certificates you have from Certbot revoke Revoke a certificate (supply --cert-name or --cert-path) delete Delete a certificate (supply --cert-name)<br/> manage your account: register Create an ACME account unregister Deactivate an ACME account update_account Update an ACME account show_account Display account details --agree-tos Agree to the ACME server&#39;s Subscriber Agreement -m EMAIL Email address for important account notifications<br/> More detailed help:<br/> -h, --help [TOPIC] print this message, or detailed help on a topic; the available TOPICS are:<br/> all, automation, commands, paths, security, testing, or any of the subcommands or plugins (certonly, renew, install, register, nginx, apache, standalone, webroot, etc.) -h all print a detailed help page including all topics --version print the version number - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </span></pre> <p> Complete documentation for <code>certbot</code> is <a href='https://eff-certbot.readthedocs.io/en/stable/intro.html' target='_blank' rel='nofollow'>here</a>. </p> </edit-fold> <edit-fold dns> <h2 id="genericDns">Generic DNS Authentication</h2> <p> I used <code>certbot</code> to create a special file within the website as follows, without using <code>sudo</code>. I want a <a href='https://en.wikipedia.org/wiki/Wildcard_certificate' target='_blank' rel='nofollow'>wildcard SSL certificate</a>, which <a href='https://certbot.eff.org/faq#does-let-s-encrypt-issue-wildcard-certificates' target='_blank' rel='nofollow'>requires certbot to use DNS authentication</a>. </p> <p> BTW, the following options include two <code>-d</code> options, one for the domain apex (<code>mslinn.com</code>) and one for the subdomains (<code>*.mslinn.com</code>) </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7ccdd7cac810'><button class='copyBtn' data-clipboard-target='#id7ccdd7cac810' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>certbot certonly \ --manual \ --agree-tos \ --preferred-challenges dns-01 \ --rsa-key-size 4096 \ -d mslinn.com -d *.mslinn.com \ --config-dir ~/.certbot/mslinn.com/config \ --logs-dir ~/.certbot/mslinn.com/logs \ --work-dir ~/.certbot/mslinn.com/work <span class='unselectable'>Saving debug log to /home/mslinn/.certbot/mslinn.com/logs/letsencrypt.log Enter email address (used for urgent renewal and security notices) (Enter &#39;c&#39; to cancel): </span> mslinn@mslinn.com <br/><span class='unselectable'>- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Would you be willing, once your first certificate is successfully issued, to share your email address with the Electronic Frontier Foundation, a founding partner of the Let&#39;s Encrypt project and the non-profit organization that develops Certbot? We&#39;d like to send you email about our work encrypting the web, EFF news, campaigns, and ways to support digital freedom. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (Y)es/(N)o: </span>n <span class='unselectable'>Account registered. Requesting a certificate for mslinn.com and *.mslinn.com<br/> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Please deploy a DNS TXT record under the name:<br/> _acme-challenge.mslinn.com.<br/> with the following value:<br/> n6o3qMw5N7qxAzV4uNQePKxJjaw0f-Bo32CybWr-lhE<br/> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Press Enter to Continue (This must be set up in addition to the previous challenges; do not remove, replace, or undo the previous challenge tasks yet. Note that you might be asked to create multiple distinct TXT records with the same name. This is permitted by DNS standards.) Before continuing, verify the TXT record has been deployed. Depending on the DNS provider, this may take some time, from a few seconds to multiple minutes. You can check if it has finished deploying with aid of online tools, such as the Google Admin Toolbox: https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.mslinn.com. Look for one or more bolded line(s) below the line ';ANSWER'. It should show the value(s) you&rsquo;ve just added. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Press Enter to Continue Successfully received certificate. Certificate is saved at: /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/fullchain.pem Key is saved at: /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/privkey.pem This certificate expires on 2022-09-29. These files will be updated when the certificate renews. NEXT STEPS: - This certificate will not be renewed automatically. Autorenewal of --manual certificates requires the use of an authentication hook script (--manual-auth-hook) but one was not provided. To renew this certificate, repeat this same certbot command before the certificate&rsquo;s expiry date. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - If you like Certbot, please consider supporting our work by: * Donating to ISRG / Let&rsquo;s Encrypt: https://letsencrypt.org/donate * Donating to EFF: https://eff.org/donate-le - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </span></pre> <p> I made a bash script (<code>~/.local/bin/sslCert</code>) to create SSL certificates, which performs the same steps as the above, but it works for any site, and can renew certificates as well as create them: </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,sslCert' download='sslCert' title='Click on the file name to download the file'>~/.local/bin/sslCert</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id0701fa3d4eb5">#!/bin/bash function help &#123; echo "Creates or updates an wildcard SSL certificate Usage: $(basename $0) DOMAIN Example: $(basename $0) scalacourses.com The relevant files are stored in ~/.certbot/DOMAIN/ Creating a new certificate requires an interactive user session; updating a certificate can be done in a cron job. " exit 1 &#125; if [ -z "$1" ]; then help; fi DOMAIN="$1" if [ -d "$HOME/.certbot/$DOMAIN" ]; then # Renew existing certificate CMD=renew OPTIONS=--force-renew else # Make new certificate (interactively) CMD=certonly unset OPTIONS fi certbot "$CMD" $OPTIONS \ --agree-tos \ --config-dir "$HOME/.certbot/$DOMAIN/config" \ -d "$DOMAIN" -d "*.$DOMAIN" \ --email mslinn@mslinn.com \ --logs-dir "$HOME/.certbot/$DOMAIN/logs" \ --manual \ --preferred-challenges dns-01 \ --rsa-key-size 4096 \ --work-dir "$HOME/.certbot/$DOMAIN/work" </pre> <p> Here is the help message for the script: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idae5f69ef2e08'><button class='copyBtn' data-clipboard-target='#idae5f69ef2e08' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sslCert <span class='unselectable'>Creates or updates an wildcard SSL certificate Usage: sslCert DOMAIN Example: sslCert scalacourses.com The relevant files are stored in ~/.certbot/DOMAIN/ Creating a new certificate requires an interactive user session; updating a certificate can be done in a cron job. </span></pre> <p> The following <code>crontab</code> entry causes the script to run every 60 days. </p> <div class='codeLabel unselectable' data-lt-active='false'>crontab</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id22bec971e575'><button class='copyBtn' data-clipboard-target='#id22bec971e575' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>0 0 1 */2 * .local/bin/sslCert mslinn.com</pre> <p> The next time the script was run, output looked like: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida241897dff6a'><button class='copyBtn' data-clipboard-target='#ida241897dff6a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sslCert mslinn.com <span class='unselectable'>Saving debug log to /home/mslinn/.certbot/mslinn.com/logs/letsencrypt.log Renewing an existing certificate for mslinn.com and *.mslinn.com<br/> Successfully received certificate. Certificate is saved at: /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/fullchain.pem Key is saved at: /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/privkey.pem This certificate expires on 2022-12-13. These files will be updated when the certificate renews.<br/> NEXT STEPS: - This certificate will not be renewed automatically. Autorenewal of --manual certificates requires the use of an authentication hook script (--manual-auth-hook) but one was not provided. To renew this certificate, repeat this same certbot command before the certificate&#39;s expiry date.<br/> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - If you like Certbot, please consider supporting our work by: * Donating to ISRG / Let&#39;s Encrypt: https://letsencrypt.org/donate * Donating to EFF: https://eff.org/donate-le - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </span></pre> <span style='font-size: 3em; float: right; margin-left: 5px;'>&#x1F601;</span> <p> Subsequent blog posts will discuss how to install the new wildcard SSL certificate for various scenarios. </p> Considering Cloudflare R2 for Static Websites 2022-06-09T00:00:00-04:00 https://mslinn.github.io/blog/2022/06/09/cloudflare-r2 <p> After deciding to <a href='/blog/2022/05/26/aws-hijacking.html'>close my AWS account</a>, I started looking for alternatives to host my static web sites. Without getting into my selection criteria, my short list of possible hosting companies was <a href='https://azure.microsoft.com/' target='_blank' rel='nofollow'>Microsoft Azure</a>, <a href='https://www.cloudflare.com' target='_blank' rel='nofollow'>Cloudflare</a>, <a href='https://www.digitalocean.com/' target='_blank' rel='nofollow'>Digital Ocean</a>, <a href='https://www.linode.com' target='_blank' rel='nofollow'>Linode</a>, and <a href='https://www.netlify.com/' target='_blank' rel='nofollow'>Netlify</a>. Many other options exist. </p> <p> Cloudflare has a free tier that has no time limit. It includes an S3 work-alike called R2, SSL and a built-in world-wide CDN that works automatically. 250 GB of storage and 1TB/month of transfer are provided at no charge, forever. There are also no ingress or egress charges. <a href='https://www.fool.com/investing/2022/05/02/this-tumbling-cloud-stock-is-still-way-too-expensi/' target='_blank' rel='nofollow'>Cloudflare&rsquo;s edge network</a> now spans 275 cities around the world, with nearly all Internet users within 50 milliseconds of a Cloudflare server. </p> <p> This blog post documents my experience with Cloudflare. If you don't care about details, and want to know my verdict on Cloudflare R2, <a href='#verdict'>skip to the end</a>. </p> <p class="alert rounded shadow"> For PaaS vendors such as AWS, Azure, Digital Ocean, Cloudflare, ScaleWay, etc: &ldquo;pay-as-you-go&rdquo; is shorthand for &ldquo;there is nothing you can do to limit your financial liability&rdquo;. </p> <h2 id="history">This is what I did</h2> <p> After creating an account, I followed the directions at <a href='https://developers.cloudflare.com/r2/get-started/' target='_blank' rel='nofollow'>R2 get started guide</a>. </p> <h2 id="wrangler">Wrangler</h2> <p> The directions told me to set up <a href='https://github.com/cloudflare/wrangler2' target='_blank' rel='nofollow'>Wrangler v2</a>, a command-line interface for transferring files to R2. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5bd0caa2f267'><button class='copyBtn' data-clipboard-target='#id5bd0caa2f267' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>npm install -g wrangler <span class='unselectable'>npm WARN config global `--global`, `--local` are deprecated. Use `--location=global` instead. npm WARN deprecated rollup-plugin-inject@3.0.2: This package has been deprecated and is no longer maintained. Please use @rollup/plugin-inject.<br/> added 56 packages, and audited 57 packages in 8s<br/> 10 high severity vulnerabilities<br/> To address issues that do not require attention, run: npm audit fix<br/> To address all issues, run: npm audit fix --force<br/> Run `npm audit` for details. </span></pre> <p> The above messages do not inspire confidence. Wrangler is written using Node. I believe that Node has more security issues than any other computer language, particularly in <a href='https://turingpoint.de/en/blog/node-package-manager-security-everything-about-npm-package-security/' target='_blank' rel='nofollow'>package management</a>. <a href='https://cheatsheetseries.owasp.org/cheatsheets/NPM_Security_Cheat_Sheet.html' target='_blank' rel='nofollow'>The OWasp recommendations</a> do not address the <a href='https://news.ycombinator.com/item?id=29245080' target='_blank' rel='nofollow'>fundamental security vulnerabilities</a> in the Node package management infrastructure. </p> <h2 id="rclone">RClone</h2> <p> I looked for alternatives to Wrangler and found <a href='https://rclone.org/docs/' target='_blank' rel='nofollow'>RClone</a>. RClone is a command-line program to manage files on cloud storage. It has <a href='https://rclone.org/docs/#subcommands' target='_blank' rel='nofollow'>many subcommands</a>, including two types of sync. Although the RClone documentation does not mention Cloudflare, the <a href='https://developers.cloudflare.com/r2/examples/rclone/' target='_blank' rel='nofollow'>Cloudflare docs described how to set up RClone</a>. </p> <h2 id="limits">Limits</h2> <p> <a href='https://developers.cloudflare.com/workers/platform/limits/' target='_blank' rel='nofollow'>Platform limits</a> are important. Not only are the technical limits important for defining inputs and outputs, users should be particularly interested in spend limits, so they are not subject to unlimited financial liability. </p> <p class="quote"> Workers Paid plan is separate from any other Cloudflare plan (Free, Professional, Business) you may have. If you are an Enterprise customer, reach out to your account team to confirm pricing details. <br><br> Only requests that hit a Worker will count against your limits and your bill. Since Cloudflare Workers runs before the Cloudflare cache, the caching of a request still incurs costs. See definitions and behavior after a limit is hit in the limits article . <br><br> &nbsp;&ndash; <a href='https://developers.cloudflare.com/workers/platform/pricing/#fine-print' target='_blank' rel='nofollow'>Cloudflare Workers Pricing Fine Print</a> </p> <h2 id="setup">Setup</h2> <div style=""> <picture> <source srcset="/blog/images/cloudflare/bucket0.webp" type="image/webp"> <source srcset="/blog/images/cloudflare/bucket0.png" type="image/png"> <img src="/blog/images/cloudflare/bucket0.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <div style=""> <picture> <source srcset="/blog/images/cloudflare/analytics0.webp" type="image/webp"> <source srcset="/blog/images/cloudflare/analytics0.png" type="image/png"> <img src="/blog/images/cloudflare/analytics0.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <div style=""> <picture> <source srcset="/blog/images/cloudflare/billing0.webp" type="image/webp"> <source srcset="/blog/images/cloudflare/billing0.png" type="image/png"> <img src="/blog/images/cloudflare/billing0.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <div style=""> <picture> <source srcset="/blog/images/cloudflare/workers0.webp" type="image/webp"> <source srcset="/blog/images/cloudflare/workers0.png" type="image/png"> <img src="/blog/images/cloudflare/workers0.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <div style=""> <picture> <source srcset="/blog/images/cloudflare/workers1.webp" type="image/webp"> <source srcset="/blog/images/cloudflare/workers1.png" type="image/png"> <img src="/blog/images/cloudflare/workers1.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <h2 id="cache">Caching</h2> <p> <a href='https://dash.cloudflare.com/de87d0e0cbc74c70e8feb87e9671cdd0/mslinn.com/caching/configuration' target='_blank' rel='nofollow'>Cache control</a>. </p> <h2 id="pages">Pages</h2> <p class="quote"> Cloudflare Pages is a JAMstack platform for frontend developers to collaborate and deploy websites. <br><br> &nbsp;&ndash; <a href='https://pages.cloudflare.com' target='_blank' rel='nofollow'><code>pages.cloudflare.com</code></a> </p> <p> Cloudflare Pages supports <a href='https://developers.cloudflare.com/pages/framework-guides/deploy-a-jekyll-site/' target='_blank' rel='nofollow'>Jekyll sites</a>. However, it looks like Cloudflare Pages builds the site in the cloud. While this might be a useful mechanism for many, my Jekyll builds need to access my local machine, and use my Jekyll plugins. </p> <p> <a href='https://developers.cloudflare.com/workers/platform/sites/start-from-existing' target='_blank' rel='nofollow'>Cloudflare Workers Sites</a> suits my use case. The <a href='https://developers.cloudflare.com/workers/platform/sites/start-from-existing' target='_blank' rel='nofollow'>Start From Existing</a> documentation looks appropriate, except it is written for using Wrangler, which I view as a security threat. </p> <h2 id="verdict">The Verdict: No to Cloudflare</h2> <p class="alert rounded shadow"> CloudFlare does not offer a spend limit for accounts on paid plans. This is unacceptable. I tried to remove my credit card, but found I could not. I then deleted my user account, and saw: </p> <p class="warning shadow rounded"> It could take up to 12 months to delete your information completely. </p> <p> There is no way to reframe this as an example of how Cloudflare is looking out for the best interests of their customers. </p> Montréal International vs. Bill 96 2022-05-27T00:00:00-04:00 https://mslinn.github.io/blog/2022/05/27/mi-bill96 <p> Last night I attended the 25th Annual General Meeting of <a href='https://www.montrealinternational.com/en/' target='_blank' rel='nofollow'>Montréal International</a>, an energetic organization dedicated to attracting foreign business to establish a presence in the greater Montréal region. My key takeaways were: </p> <ol> <li> I spoke with about 20 Montréal International employees; all of them were bilingual or trilingual, intelligent, motivated and well-informed. Their talent and commitment is palpable. Clearly, the hiring process is successful. </li> <li> The event was marred by long-winded, self-congratulatory speeches by senior management, and former senior managers who were invited to a panel discussion. This lack of discipline caused the event to significantly run overtime. </li> <li> The proceedings were conducted entirely in French. For an organization that calls itself &ldquo;Montréal International&rdquo;, this is unforgivable. </li> <li> I asked every employee I spoke with if <a href='https://qcgn.ca/bill-96/' target='_blank' rel='nofollow'>Quebec&rsquo;s new Bill 96</a> impacted their goals. All of them expressed grave concern, and said that their mandate to attract foreign business had become extremely difficult as a result. However, not one mention was made by anyone on stage about Bill 96. </li> </ol> <p> Montréal International&rsquo;s mandate is severely impacted by Quebec's Bill 96. If any organization has an intrinsic mandate to demonstrate public leadership towards repealing or modifying Bill 96 so it respects basic human rights, it would be Montréal International. Instead, Montréal International&rsquo;s leadership, which consists entirely of old white French-speaking alpha males, is complicit in their silence, and by their actions. </p> Limit Your Financial Vulnerability From AWS Account Hijacking 2022-05-26T00:00:00-04:00 https://mslinn.github.io/blog/2022/05/26/aws-hijacking <p class="alert shadow rounded"> I am a free agent, independent and critical. <a href='/softwareexpert/index.html'>You can buy my time as a software expert</a>, but not my opinion. Just because Amazon / AWS hired me in the past to help them against a patent troll does not mean they own me. Quite the contrary. </p> <p> In the time you take a quick shower, your AWS account can be highjacked and tens of thousands of dollars in fees can be incurred. EC2 is the service that provides the greatest financial risk. You can take steps to limit your liability, and this blog post shows you how. </p> <h2 id="nolimits">AWS IAM Users and Roles Have No Budget Limitations</h2> <div class="pullQuote"> AWS customers are exposed to unlimited financial liability </div> <p> I encountered two main issues when attempting to secure against AWS account hijacking: </p> <ol> <li> <b>AWS does not provide a mechanism to guard against launching expensive services.</b> For example, if an IAM user has a role that allows EC2 instances to be launched, then that user can launch an unlimited number of EC2 instances of any size. <br><br> The most expensive Amazon EC2 is currently the <a href='https://aws.amazon.com/ec2/instance-types/p4/' target='_blank' rel='nofollow'><code>p4de.24xlarge</code></a>, which costs $40.96 USD per hour on-demand in the US East (N. Virginia) region. The instance comes with 96 vCPUs, 1152 GiB of RAM and eight 1TB NVMe SSDs, and has eight NVIDIA A100 Tensor Core GPUs. <br><br> Using stolen credentials that allow EC2 instances to be launched, an attacker using scripts could spin up an armada of <code>p4de.24xlarge</code> instances and incur eye-popping costs in minutes. </li> <li> AWS Budgets provides a convenient way to shut down pre-designated services when a total budgetary amount is exceeded. <b>However, the AWS AMI security model is not integrated into real-time cost monitoring.</b> Thus, there is no way to automatically shut down newly launched service instances that exceed a pre-authorized budgetary amount. </li> </ol> <h2 id="orElse">Shared Security Responsibility</h2> <p> The <a href='https://aws.amazon.com/agreement/' target='_blank' rel='nofollow'>AWS Customer Agreement</a> and <a href='https://aws.amazon.com/compliance/shared-responsibility-model/' target='_blank' rel='nofollow'>Shared Responsibility Model</a> describe how AWS shares responsibility for security with users. If you follow the AWS security recommendations, described next, you will not be held accountable for extra charges if your AWS account is hijacked. </p> <p> The AWS Security Blog published a nice overview, entitled <a href='https://aws.amazon.com/blogs/security/getting-started-follow-security-best-practices-as-you-configure-your-aws-resources/' target='_blank' rel='nofollow'>Getting Started: Follow Security Best Practices as You Configure Your AWS Resources</a>. Some of the same information is also provided in <a href='https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html' target='_blank' rel='nofollow'>Security best practices in IAM</a>. </p> <p class="alert rounded shadow"> Even though the AWS instructions do not prevent bad guys taking over your account, follow them anyway so you won't be held liable for the costs incurred if your AWS account is hijacked. </p> <h2 id="recommendations">AWS Security Recommendations</h2> <p> Security needs to have depth. Any single measure can fail, and given enough scale, all measures will eventually fail. Layer your security measures so that one failure, no matter how grave, will not be fatal. </p> <p> AWS recommends the following. I have highlighted what AWS personnel have described to me as being the quickest and easiest path; this blog post walks through those steps to the extent that I have been able. </p> <ol> <li> Set up at least two of the following services to monitor cost and usage: <ol> <li> <a href='https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-managing-costs.html' target='_blank' rel='nofollow'><span class="bg_yellow">Managing your costs with AWS Budgets </span></a> </li> <li> <a href='https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/gs_monitor_estimated_charges_with_cloudwatch.html#gs_creating_billing_alarm' target='_blank' rel='nofollow'><span class="bg_yellow">Create a billing alarm Using CloudWatch</span></a>. </li> <li> <a href='https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html' target='_blank' rel='nofollow'>CloudTrail User Guide</a>. </li> <li> <a href='https://docs.aws.amazon.com/waf/latest/developerguide/what-is-aws-waf.html' target='_blank' rel='nofollow'>Web Application Firewall (WAF)</a>. </li> <li> <a href='https://docs.aws.amazon.com/awssupport/latest/user/get-started-with-aws-trusted-advisor.html' target='_blank' rel='nofollow'>Trusted Advisor</a>. <span class="bg_yellow">I found that Trusted Advisor was somewhat easier to use than CloudWatch.</span> </li> </ol> For more information about managing your AWS cost and usage, see the <a href='https://docs.aws.amazon.com/cost-management/latest/userguide/what-is-costmanagement.html' target='_blank' rel='nofollow'>AWS Cost Management User Guide</a>. </li> <li>Set up at least one of the following security best practices: <ol> <li> <a href='https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa_enable.html' target='_blank' rel='nofollow'><span class="bg_yellow">Using multi-factor authentication (MFA) in AWS</span></a>. </li> <li> <a href='https://docs.aws.amazon.com/securityhub/index.html' target='_blank' rel='nofollow'>AWS Security Hub</a>. </li> <li> <a href='https://docs.aws.amazon.com/guardduty/index.html' target='_blank' rel='nofollow'>Amazon GuardDuty</a>. </li> </ol> </ol> <h2 id="easiestFirst">Do The Easiest Things First</h2> <p> I usually prefer to do the easiest things first. <a href='https://www.lifehack.org/articles/productivity/easy-tasks-difficult-tasks-first-which-one-more-productive.html' target='_blank' rel='nofollow'>Not everyone agrees.</a> Doing even one highlighted item above provides a significant security improvement, so why wait? </p> <div class="quote"> “The best is the enemy of the good.”<br> &nbsp;&ndash; Voltaire.<br><br> “Better a diamond with a flaw than a pebble without.”<br> &nbsp;&ndash; Confucius.<br><br> “Striving to better, oft we mar what’s well.”<br> &nbsp;&ndash; Shakespeare. </div> <h2 id="root">Root Credentials</h2> <p> If a bad guy has your root credentials, they can change budget limits. For greatest security, <a href='https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/security_credentials' target='_blank' rel='nofollow'>delete your root credentials</a> by selecting <b>Access Keys (access key ID and secret access key)</b> as shown in the image below. However, if you do so, <a href='https://docs.aws.amazon.com/general/latest/gr/root-vs-iam.html#aws_tasks-that-require-root' target='_blank' rel='nofollow'>many things become impossible</a>. </p> <div style=""> <picture> <source srcset="/blog/images/aws/rootKeys.webp" type="image/webp"> <source srcset="/blog/images/aws/rootKeys.png" type="image/png"> <img src="/blog/images/aws/rootKeys.png" class=" liImg2 rounded shadow" /> </picture> </div> <h3 id="mfa">Enabling MFA</h3> <div style="text-align: right;"> <a href="https://www.amazon.com/Yubico-YubiKey-NFC-Authentication-USB/dp/B07HBD71HL" target="_blank" ><picture> <source srcset="/blog/images/aws/yubikey.webp" type="image/webp"> <source srcset="/blog/images/aws/yubikey.png" type="image/png"> <img src="/blog/images/aws/yubikey.png" title="Yubikey NFC" class="right liImg2 rounded shadow" alt="Yubikey NFC" /> </picture></a> </div> <p> The easiest highlighted item above is to enable multifactor authentication (MFA). </p> <p> I use a virtual MFA device, <a href='https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa.html#enable-virt-mfa-for-root' target='_blank' rel='nofollow'>enabled for my root account</a> Google Authenticator for <a href='https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2' target='_blank' rel='nofollow'>Android</a> and <a href='https://apps.apple.com/us/app/google-authenticator/id388497605' target='_blank' rel='nofollow'>iOS</a>. Google Authenticator features a <a href='https://tools.ietf.org/html/rfc6238' target='_blank' rel='nofollow'>RFC 6238</a> standards-based TOTP (time-based one-time password) algorithm. </p> <p> I use a <a href='https://www.amazon.com/Yubico-YubiKey-NFC-Authentication-USB/dp/B07HBD71HL' target='_blank' rel='nofollow'>YubiKey</a> to provide MFA for the AMI user ID that I use to log into the AWS console for everyday work. </p> <p class="pullQuote"> Only use root credentials when absolutely necessary. </p> <h2 id="trustedAdvisor">Trusted Advisor</h2> <p> After adding MFA to root credentials, I found that using <a href='https://docs.aws.amazon.com/awssupport/latest/user/get-started-with-aws-trusted-advisor.html' target='_blank' rel='nofollow'>AWS Trusted Advisor</a> was the next easiest thing to try to make my AWS account secure. </p> <p> Once you visit the <a href='https://console.aws.amazon.com/trustedadvisor/home' target='_blank' rel='nofollow'>Trusted Advisor Console</a>, and agree to enable Trusted Advisor, <a href='https://docs.aws.amazon.com/awssupport/latest/user/security-trusted-advisor.html' target='_blank' rel='nofollow'>IAM permissions are added to the AWS IAM user</a> that you are logged in as. </p> <p> Usage seems straightforward; however, I found the information for securing S3 buckets puzzling at first: </p> <div style=""> <picture> <source srcset="/blog/images/aws/trustedAdvisor1.webp" type="image/webp"> <source srcset="/blog/images/aws/trustedAdvisor1.png" type="image/png"> <img src="/blog/images/aws/trustedAdvisor1.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <p> The column labeled <b>ACL Allows List</b> means that buckets flagged with <b>Yes</b> have an ACL that allows objects to be listed. AWS recommends that S3 buckets not allow objects to be listed by the public. To correct this: </p> <ol> <li>Click on the bucket name.</li> <li>Click on the <b>Permissions</b> tab in the page that opens next.</li> <li>Scroll down to <b>Access Control List</b>.</li> <li>Click on the <kbd>Edit</kbd> button.</li> <li>Disable the items marked in red, as shown below.</li> </ol> <div style=""> <picture> <source srcset="/blog/images/aws/trustedAdvisor2.webp" type="image/webp"> <source srcset="/blog/images/aws/trustedAdvisor2.png" type="image/png"> <img src="/blog/images/aws/trustedAdvisor2.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <p> Referring back to the previous screenshot, the <b>Policy Allows Access</b> column flags all S3 buckets used to serve webpages as insecure. To serve HTML pages and assets stored in an S3 bucket, the permissions must be set to allow objects to be read by everyone. Trusted Advisor flags these buckets as insecure, which does not make sense to me. </p> <p> Perhaps there is a better security policy for when webpages are served via CloudFront, but as is often the case with AWS documentation, it is difficult to piece together how to optimally configure S3 permissions with CloudFront options. </p> <h3 id="evolution">Best Practices Evolve With Technology</h3> <p> The AWS documentation is written as if the products described have always existed in their present state. In the course of researching this article, I discovered that AWS provides revision histories for their services. For example, the revision history of S3 is documented in the <a href='https://docs.aws.amazon.com/AmazonS3/latest/userguide/WhatsNew.html' target='_blank' rel='nofollow'>AWS S3 User Guide</a>, and the CloudFront revision history is documented in the <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/WhatsNew.html' target='_blank' rel='nofollow'>AWS CloudFront Developer Guide</a>. Revision histories are useful for experienced technologists to review periodically, so they can keep up with best practices as they evolve. </p> <p> I started using AWS S3 to serve websites in 2013, and at that time, CloudFront did not exist yet. As the two services matured, best practices, which were at first unknown, evolved and interactions between them became more complex. In particular, CloudFront <code>http</code>/<code>https</code> promotion, how origins are specified, and S3 permissions, all interact in ways that were not well documented for several years. This was a source of security problems and unwanted downtime for me. </p> <div style="text-align: right;"> <picture> <source srcset="/blog/images/aws/roadToHell.webp" type="image/webp"> <source srcset="/blog/images/aws/roadToHell.png" type="image/png"> <img src="/blog/images/aws/roadToHell.png" class="right liImg2 rounded shadow" style="max-height: 400px;" /> </picture> </div> <p class="alert rounded shadow"> I have 15 buckets that are used to serve web pages. On my <a href='https://us-east-1.console.aws.amazon.com/support/home?region=us-east-1&skipRegion=true#/' target='_blank' rel='nofollow'>AWS Support Center console</a>, Trusted Advisor shows that &ldquo;You have a yellow check affecting 15 of 24 resources&rdquo;; however, the Trusted Advisor console displays those items as red triangles with exclamation marks. <br><br> This is confusing, and became frustrating when an AWS security support technician suggested I disregard anything that seemed annoying. For an IT/security person, that advice is the quickest way to hell that I know. </p> <h3 id="rough">Rough Spots That Need Love</h3> <p> While researching this article I found <a href='https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/getting-started-cloudfront-overview.html#getting-started-cloudfront-distribution' target='_blank' rel='nofollow'>Use an Amazon CloudFront distribution to serve a static website</a>, which says: </p> <div class="quote clear"> Under <b>Origin Settings</b>, for <b>Origin Domain Name</b>, choose the Amazon S3 bucket that you created previously. <span class="bg_yellow">For <b>S3 bucket access</b>, select <b>Yes, use OAI (bucket can restrict access to only CloudFront)</b></span>. For the <b>Origin access identity</b>, you can choose from the list, or choose <b>Create new OAI</b> (both will work). For <b>Bucket policy</b>, select <b>Yes, update the bucket policy</b>. </div> <p> However, I found conflicting information in <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3.html' target='_blank' rel='nofollow'>Restricting access to Amazon S3 content by using an origin access identity (OAI)</a>. Right near the top of that article, it says: </p> <div class="quote"> <h2>Important</h2> <p> If you use an Amazon S3 bucket configured as a website endpoint, you must set it up with CloudFront as a custom origin. <span class="bg_yellow">You can’t use the origin access identity feature described in this topic.</span> However, you can restrict access to content on a custom origin by setting up custom headers and configuring your origin to require them. For more information, see <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-overview.html#forward-custom-headers-restrict-access' target='_blank' rel='nofollow'>Restricting access to files on custom origins</a>. </p> </div> <p> As if this is not confusing enough, <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/DownloadDistS3AndCustomOrigins.html#concept_S3Origin_website' target='_blank' rel='nofollow'>Using various origins with CloudFront distributions</a> states that in order for Amazon S3 redirects to work, a non-standard format for the origin must be used:<br><br> <code>http://<span style="color: red;">bucket-name</span>.s3-website-<span style="color: red;">region</span>.amazonaws.com</code> <br><br> As I discussed in a <a href='/jekyll/10500-redirects.html'>previous blog post</a>, AWS S3 buckets support two types of redirects, but there is no indication of which type might or might not work with the non-standard origin format. <br><br> Also, there is no mention of whether <code>https</code> would work or not, or if these disparate instructions even work together. As usual, there is lots of uncertainty about how things actually work on AWS, and for how long before things change. </p> <p> As per standard operating procedure with AWS, one would have to muck about extensively to determine which of the conflicting statements above is generally prevalent, and under what circumstances. I expect that one article is newer than the other, and it likely supercedes the older article somewhat to some degree under as-yet unknown circumstances. If I get lucky, the Trusted Advisor warnings will go away. Trusted Advisor should reference whichever article is currently correct. </p> <h3 id="2022-05-31">Update 2022-05-31</h3> <p> I received the following email from Trusted Advisor. Unfortunately, the contents of the email were different from what I found on the Trusted Advisor console. Perhaps this is because a higher-level, and more expensive, service contract would be required to see those messages. If so, then this is not the way to gain happy customers. The cost of the higher service level would be 4 times more than my total AWS costs at present, clearly not something that I would want to do. </p> <div class="quote"> Dear AWS customer:<br><br> AWS Trusted Advisor currently shows alerts for 3 checks (1 red and 2 yellow) and $0 of potential monthly savings based on your usage.<br><br> Here is a summary of status changes for this week:<br><br> The alert severity of these checks has improved:<br><br> From red to yellow:<br> &nbsp;&nbsp;&nbsp;Amazon S3 Bucket Permissions<br><br> From red to green:<br> &nbsp;&nbsp;&nbsp;VPC<br> &nbsp;&nbsp;&nbsp;Security Groups - Specific Ports Unrestricted<br><br> From yellow to green:<br> &nbsp;&nbsp;&nbsp;VPC Internet Gateways<br><br> The alert status of 5 checks has not changed. For details and recommendations, visit AWS Trusted Advisor.<br><br> Sign in to the Trusted Advisor console to view your check results.<br><br> Best,<br><br> AWS Support<br> Was this report helpful? <a href='https://www.amazon.com/gp/f.html' target='_blank' rel='nofollow'>Send us your feedback.</a> </div> <p> To add insult to injury, the final "Send us your feedback" just links to a generic form for AWS sales, where one can <a href='https://writingexplained.org/idiom-dictionary/go-pound-sand' target='_blank' rel='nofollow'>pound sand</a>. </p> <p class="alert rounded shadow"> There is abundant evidence to demonstrate that whereas <code>amazon.com</code> is consumer-centric, AWS marches to a very different drummer. Perhaps small businesses are not presently viewed by AWS as a significant growth opportunity. My personal experience is that the current risk/value proposition provided by AWS is no longer attractive for that demographic. </p> <h3 id="2022-05-31">Update 2022-06-03</h3> <p> The credit card I used to pay for my AWS usage had a very high limit. In retrospect that was unwise. </p> <p> AWS took weeks to agree to refund the charge on my credit card, and then they said it would take another week for the reversed charges to appear. This meant that if I only paid for the other charges on that card, the month-to-month credit card interest on the fraudulent AWS charges would have been problematic. </p> <p> Just before the credit card billing period ended, I called Visa and asked what my options were. Apparently I am not the first person who had this problem with AWS. The answer was immediate: report the charges as fraud. OK, that made sense. This meant my card was immediately cancelled, the charges were forgiven by Visa, and a new card with a new number would arrive in about a week. </p> <h2 id="budgets">Managing Costs With AWS Budgets</h2> <p> Short-lived hijacking is the most serious financial threat, and AWS Budgets was seemingly not designed to guard against that. Ticking off this item will enable AWS to absolve you of financial responsibility for being hacked, but you should not expect that it will do much to slow down an attacker. </p> <p> AWS Budgets are easy to set up in a passive manner, but they are clumsy to work with and not effective for preventing new services from being launched that would exceed the budget. The cost limiting functionality is not automatic, it must be set up manually. AWS documentation does not provide detailed how-to instructions for common scenarios. Tens of thousands of words of documentation must be digested before the full capabilities can be harnessed. </p> <p> Another significant issue is that only three types of actions are supported: </p> <ol> <li> <b>IAM Policy</b> &ndash; I am unclear on how to set this up so the bad guys cannot launch new services, and I am not certain this is possible. </li> <li> <a href='https://us-east-1.console.aws.amazon.com/organizations/v2/home/policies/service-control-policy' target='_blank' rel='nofollow'><b>Service Control Policy</b></a> &ndash; Hopefully this could prevent new services from being launched that would exceed the budget; however, my head exploded when I researched this. <div style=""> <picture> <source srcset="/blog/images/aws/serviceControlPolicies1.webp" type="image/webp"> <source srcset="/blog/images/aws/serviceControlPolicies1.png" type="image/png"> <img src="/blog/images/aws/serviceControlPolicies1.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> Attempting to use this option to prevent new, expensive services from being launched would seem to introduce lots of complexity. Perhaps AWS did not consider this use case, or deemed it unimportant. </li> <li> <b>Automate Instances to Stop for EC2 or RDS</b> &ndash; If you run a <a href='https://aws.amazon.com/ec2/instance-types/' target='_blank' rel='nofollow'><code>t3.nano</code></a> instance, for example, bad guys can only incur a certain amount of financial damage. Instead of shutting down the EC2 or RDS instances that I want to run, my main concern is to prevent bad guys from launching expensive new services. This option is therefore unsuitable for my use case. </li> </ol> <p> If AWS Budgets provides an easy way of preventing new services to be launched that would exceed the budget, I could not find it after several hours of reading. </p> <h3 id="passive">Passively Monitoring Budget Spend</h3> <p> Simply follow <a href='https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-create.html' target='_blank' rel='nofollow'>the directions</a> and accept the defaults. Daily budgets do not support enabling forecasted alerts, or daily budget planning, so select <b>Monthly Budgets</b>. There is no need to set up SNS alerts or AWS Chatbot alerts, email notification works just as well for most users. </p> <h3 id="passive">Attaching Actions</h3> <p> Following are my notes from my attempt to enable cost limiting. I was unable to attach an action that prevented expensive new services from being started, and it is unclear if resolving the problem will, in fact, provide the desired benefit. Perhaps someone reading these notes will be able to tell me about an approach that might work for a reasonable amount of effort. </p> <p class="alert rounded shadow"> Because AWS will absolve you of financial liability if you create a Budget without any actions, don't stress about the pointless exercise involved in making a Budget that cannot protect you. Just do it and give AWS more time to figure out a better way of preventing account hijacking. <br><br> You can stop reading this article now, unless the details interest you. </p> <p class="alert rounded shadow"> For PaaS vendors such as AWS, Azure, Digital Ocean, Cloudflare, ScaleWay, etc: &ldquo;pay-as-you-go&rdquo; is shorthand for &ldquo;there is nothing you can do to limit your financial liability&rdquo;. </p> <ol> <li>When you get to the step labeled <b>Attach actions - Optional</b>, choose <b>Add Action</b>.</li> <li> You must choose between using an existing IAM role for AWS Budgets to use, or you can create a new IAM role expressly for this purpose. IAM roles are free. I like the idea of a dedicated IAM role, so I clicked on <a href='https://us-east-1.console.aws.amazon.com/iamv2/home?region=us-east-1#/roles' target='_blank' rel='nofollow'>manually create an IAM role</a>.<br> <div style=""> <picture> <source srcset="/blog/images/aws/budgetIamRole1.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole1.png" type="image/png"> <img src="/blog/images/aws/budgetIamRole1.png" class=" halfsize liImg2 rounded shadow" /> </picture> </div> </li> <li> A new browser tab opened, and I clicked on the blue <kbd>Create role</kbd> button. </li> <li> Now a confusing page appeared: <div style=""> <picture> <source srcset="/blog/images/aws/budgetIamRole2.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole2.png" type="image/png"> <img src="/blog/images/aws/budgetIamRole2.png" class=" liImg2 rounded shadow" /> </picture> </div> </li> <li> I selected the default <b>Trusted entity type</b>, AWS Service, then from the pulldown menu labeled <b>Use cases for other AWS services:</b> I selected <b>Budgets</b>. </li> <li> A new radio button appeared underneath the pull-down, also labeled <b>Budgets</b>, and I clicked on that. This does not win any usability awards. </li> <li> I clicked on the button labeled <kbd>Next</kbd>. </li> <li> <div class="quote" style="min-width: 35rem;"> Managed policy name: <code><a href='https://docs.aws.amazon.com/cost-management/latest/userguide/billing-permissions-ref.html#budget-managedIAM-SSM' target='_blank' rel='nofollow'><code>AWSBudgetsActionsRolePolicyForResourceAdministrationWithSSM</code></a></code><br><br> This managed policy is focused on specific actions that AWS Budgets takes on your behalf when completing a specific action. This policy gives AWS Budgets broad permission to control AWS resources. For example, starts and stops Amazon EC2 or Amazon RDS instances by running AWS Systems Manager (SSM) scripts.<br><br> &nbsp;&ndash; From <a href='https://docs.aws.amazon.com/cost-management/latest/userguide/billing-permissions-ref.html#budget-managedIAM-SSM' target='_blank' rel='nofollow'>Using identity-based policies (IAM policies) for AWS Cost Management</a> </div> On the page, I applied <code>AWSBudgetsActionsRolePolicyForResourceAdministrationWithSSM</code>, then I clicked on <kbd>Next</kbd>. <div style=""> <picture> <source srcset="/blog/images/aws/budgetIamRole3.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole3.png" type="image/png"> <img src="/blog/images/aws/budgetIamRole3.png" class=" liImg2 rounded shadow" /> </picture> </div> </li> <li> On the final page, I named the IAM role <b>Budget</b> and clicked on <kbd>Create role</kbd>. <div style=""> <picture> <source srcset="/blog/images/aws/budgetIamRole4.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole4.png" type="image/png"> <img src="/blog/images/aws/budgetIamRole4.png" class=" liImg2 rounded shadow" /> </picture> </div> </li> <li> Back in the <b>Billing Management Console</b>, I refreshed the list of available IAM roles, then selected the new role called <b>Budget</b>. <div style=""> <picture> <source srcset="/blog/images/aws/budgetIamRole5.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole5.png" type="image/png"> <img src="/blog/images/aws/budgetIamRole5.png" class=" halfsize liImg2 rounded shadow" /> </picture> </div> </li> <li> I selected <b>Service Control Policy</b> from the pull-down that appeared next, labeled <b>Which action type should be applied when the budget threshold has been exceeded?</b>. I have no idea how to proceed, or even if going in this direction might provide the desired results. <div style=""> <picture> <source srcset="/blog/images/aws/serviceControlPolicies2.webp" type="image/webp"> <source srcset="/blog/images/aws/serviceControlPolicies2.png" type="image/png"> <img src="/blog/images/aws/serviceControlPolicies2.png" class=" liImg2 rounded shadow" /> </picture> </div> </li> </ol> Microsoft Clarity Lets Me Watch You Click and Scroll 2022-03-31T00:00:00-04:00 https://mslinn.github.io/blog/2022/03/31/clarity <p> I spent a fair bit of time designing the plugins for this Jekyll-powered website for <a href='https://moz.com/beginners-guide-to-seo' target='_blank' rel='nofollow'>SEO</a>. These plugins are published as open source; click on the word Jekyll in the sentence above this paragraph to learn more. You are welcome! </p> <p> I am currently getting strong month-over-month audience growth, and the rate of growth continues to accelerate. I do not use Google Analytics because it slows down page load time dramatically. I do use <a href='https://search.google.com' target='_blank' rel='nofollow'>Google Search Console</a>, however, and recently started trying out <a href='https://www.bing.com/webmasters/' target='_blank' rel='nofollow'>Microsoft Bing Webmaster Tools</a>. </p> <h2 id="clarity">Microsoft Clarity and Hotjar</h2> <p> Earlier today, I stumbled across <a href='https://clarity.microsoft.com/' target='_blank' rel='nofollow'>Microsoft Clarity</a>, a free product that I knew nothing about. I was curious to know what benefit it might provide. One of the things it can do is provide videos of actual user sessions as they interact with the website. </p> <p> Check out this video! </p> <style>.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }</style><div class='embed-container'> <iframe title="YouTube video player" width="640" height="390" src="//www.youtube.com/embed/N-Tip0Y4GMU" frameborder="0" allowfullscreen></iframe></div> <p style="margin-top: 1em"> Microsoft Clarity lets me watch movies of users clicking and scrolling through my website; spooky yet very informative. </p> <p> I have since learned that <a href='https://www.hotjar.com/' target='_blank' rel='nofollow'>Hotjar</a> is similar to Microsoft Clarity. </p> <h2 id="behavior">Online Behavior Matches Real-World Behavior</h2> <p> The user in the above video read a bit about my experience as a software expert witness, then straightaway downloaded my resume. They were on the website for just over one minute. Although they did call me, their online behavior showed a lack of urgency, and their &lsquo;real-world&rsquo; behavior mirrored what I saw in the movie. </p> <p> A week later another party visited my site, and spent 80 minutes carefully reading 3 pages, among others. Their &lsquo;real-world&rsquo; behavior also matched their online behavior, in that they exhibited a sense of urgency towards engaging a software expert. </p> <h2 id="downloads">Tracking Downloads And Other Behavior</h2> <p> Microsoft Clarity does not consider a download as a click. It does not even notice downloads. For me, downloads are what I care about most. If you never download my resume, you probably are not a candidate for hiring me as a <a href='/softwareexpert/index.html'>software expert</a>. I want to track downloads, not clicks. Clicks are nice, but there is a direct relationship between resume downloads and signing contracts with legal firms who represent new clients. </p> <p> AWS <a href='https://aws.amazon.com/premiumsupport/knowledge-center/view-iam-history/' target='_blank' rel='nofollow'>CloudTrail</a> and <a href='https://aws.amazon.com/aws-cost-management/aws-cost-optimization/monitor-track-and-analyze/' target='_blank' rel='nofollow'>CloudWatch</a> can provide download details and much more. For example, user IP addresses and geographic location can be captured when a monitored event occurs. </p> <h2 id="others">Many Websites Perform Surveillance</h2> <p> Wired Magazine published an article on a similar type of surveillance (keyloggers) on May 11, 2022: <a href='https://www.wired.com/story/leaky-forms-keyloggers-meta-tiktok-pixel-study/' target='_blank' rel='nofollow'>Thousands of Popular Websites See What You Type—Before You Hit Submit</a>. I paraphrased two sentences from that article: </p> <ul class="quote"> <li>1.8% of websites studied gathered an EU user's email address without their consent, and a staggering 2.95% logged a US user's email.</li> <li>For US users, 8.4% of sites may have been leaking data to Meta, Facebook’s parent company, and 7.4% of sites may be impacted for EU users.</li> </ul> Make a Visual Studio Code Extension 2022-03-01T00:00:00-05:00 https://mslinn.github.io/blog/2022/03/01/make-vscode-extension <p> I want to be able to move pages in my Jekyll-powered website without breaking links from the outside. The <a href='https://github.com/jekyll/jekyll-redirect-from' target='_blank' rel='nofollow'><code>jekyll-redirect-from</code></a> Jekyll plugin generates little HTML files that contain <code>http-meta-refresh</code> client-side redirects as desired. I wanted an easy way to inject the names of those redirect pages into the Jekyll front matter. Since I usually author my website using Visual Studio Code, a custom Visual Studio Code extension seems like a good way to create the redirects. </p> <h2 id="requirements">Extension Requirements</h2> <p> All the extension has to do is discover the URL path component of the page currently being edited, and write an entry into the front matter. For example, I have set up Jekyll such that given a page at <code>collections/_posts/2022/2022-02-21-jekyll-debugging.html</code>, it would deploy to <code>/blog/2022/02/21/jekyll-debugging.html</code>. The required front matter would be: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idec7272340910'><button class='copyBtn' data-clipboard-target='#idec7272340910' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>redirect_from: - /blog/2022/02/21/jekyll-debugging.html</pre> <p> With that in mind, if I want to move a published post to another location without breaking it, the extension should: </p> <ol> <li>Present a new menu option when a right-click on a file in the side bar, or a right-click on a file name tab in the editor.</li> <li> When the menu item is selected, obtain the relative path to the file within the project directory (for example: <code>collections/_posts/2022/2022-02-21-jekyll-debugging.html</code>) </li> <li> Convert the relative path to the deployed relative path (for example, <code>/blog/2022/02/21/jekyll-debugging.html</code>) </li> <li> Write the entry into the front matter of the file. For example: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id657c2f047fde'><button class='copyBtn' data-clipboard-target='#id657c2f047fde' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>redirect_from:<br> - /blog/2022/02/21/jekyll-debugging.html</pre> </li> </ol> <h2 id="background">Background</h2> The <a href='https://code.visualstudio.com/api/get-started/your-first-extension' target='_blank' rel='nofollow'>Microsoft documentation</a> describes how to write an Visual Studio Code extension. <h2 id="setup">Setup</h2> <p> Visual Studio Code extensions are written in JavaScript (actually, node.js) or TypeScript. Ensure that a version of node.js has been installed: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id28b62445f07f'><button class='copyBtn' data-clipboard-target='#id28b62445f07f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>which node <span class='unselectable'>/home/mslinn/.nvm/versions/node/v17.3.1/bin/node </span></pre> <p> Ensure you have <a href='properly'>set up your global package manager</a> for node.js, then type: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3dfb96e181b9'><button class='copyBtn' data-clipboard-target='#id3dfb96e181b9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>npm install -g <a href='https://github.com/yeoman/yo/' target='_blank' rel='nofollow'>yo</a> <a href='https://github.com/microsoft/vscode-generator-code' target='_blank' rel='nofollow'>generator-code</a> <span class='unselectable'>&nbsp; npm WARN deprecated har-validator@5.1.5: this library is no longer supported npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142 added 872 packages, and audited 873 packages in 57s 15 vulnerabilities (13 moderate, 2 high) To address issues that do not require attention, run: npm audit fix To address all issues (including breaking changes), run: npm audit fix --force Run `npm audit` for details. </span></pre> <p> The <code>yo</code> module is the culprit; lets address its vulnerabilities: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id44d842af8b2e'><button class='copyBtn' data-clipboard-target='#id44d842af8b2e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>npm install -g npm-check-updates <span class='unselectable'>added 270 packages, and audited 271 packages in 9s found 0 vulnerabilities </span></pre> <h2 id="generate">Generating the Skeleton</h2> <p> I decided to use <a href='https://www.typescriptlang.org/' target='_blank' rel='nofollow'>TypeScript</a> instead of JavaScript for the extension. TypeScript code converts to JavaScript, however it uses type inference and integrates better with some editors. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8b1a9a5813d0'><button class='copyBtn' data-clipboard-target='#id8b1a9a5813d0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mkdir redirect_generator <span class='unselectable'>$ </span>cd redirect_generator/ <span class='unselectable'>$ </span>$ yo code <span class='unselectable'>? ========================================================================== We&rsquo;re constantly looking for ways to make yo better! May we anonymously report usage statistics to improve the tool over time? More info: https://github.com/yeoman/insight &amp; http://yeoman.io ========================================================================== </span> No <span class='unselectable'>_-----_ ╭──────────────────────────╮ | | │ Welcome to the Visual │ |--(o)--| │ Studio Code Extension │ `---------´ │ generator! │ ( _´U`_ ) ╰──────────────────────────╯ /___A___\ / | ~ | __'.___.'__ ´ ` |° ´ Y ` ? What type of extension do you want to create? </span> New Extension (TypeScript) <span class='unselectable'>? What's the name of your extension? </span> redirect_generator <span class='unselectable'>? What's the identifier of your extension? </span> redirect-generator ? What's the description of your extension? Injects the URL of a redirect page into Jekyll front matter <span class='unselectable'>? Initialize a git repository? </span>Yes <span class='unselectable'>? Bundle the source code with webpack? </span>No <span class='unselectable'>? Which package manager to use? </span>npm <span class='unselectable'>Writing in /mnt/f/work/jekyll/redirect_generator/redirect-generator... create redirect-generator/.vscode/extensions.json create redirect-generator/.vscode/launch.json create redirect-generator/.vscode/settings.json create redirect-generator/.vscode/tasks.json create redirect-generator/package.json create redirect-generator/tsconfig.json create redirect-generator/.vscodeignore create redirect-generator/vsc-extension-quickstart.md create redirect-generator/README.md create redirect-generator/CHANGELOG.md create redirect-generator/src/extension.ts create redirect-generator/src/test/runTest.ts create redirect-generator/src/test/suite/extension.test.ts create redirect-generator/src/test/suite/index.ts create redirect-generator/.eslintrc.json Changes to package.json were detected. Running npm install for you to install the required dependencies. added 203 packages, and audited 204 packages in 29s found 0 vulnerabilities Your extension redirect-generator has been created! To start editing with Visual Studio Code, use the following commands: code redirect-generator Open vsc-extension-quickstart.md inside the new extension for further instructions on how to modify, test and publish your extension. For more information, also visit http://code.visualstudio.com and follow us @code. ? Do you want to open the new folder with Visual Studio Code? Open with `code` _-----_ ╭───────────────────────╮ | | │ Bye from us! │ |--(o)--| │ Chat soon. │ `---------´ │ Yeoman team │ ( _´U`_ ) │ http://yeoman.io │ /___A___\ /╰───────────────────────╯ | ~ | __'.___.'__ ´ ` |° ´ Y ` </span></pre> <p> The instructions in the web page assume the <a href='https://dictionary.cambridge.org/dictionary/english/happy-path' target='_blank' rel='nofollow'>happy path</a> is the only possibility. Instead: </p> <ol> <li>Do not work on the extension in a workspace that has any other projects in it.</li> <li>The online instructions assume that TypeScript was used, not JavaScript, so the file types are assumed to be <code>.ts</code>. </ol> <p> The instructions in <code>vsc-extension-quickstart.md</code> do not make assumptions about TypeScript. </p> <h2 id="debug">Debugging the Extension</h2> <p> I found the procedure awkward at first: </p> <ol> <li>Press <kbd>F5</kbd> to activate the first defined launcher</li> <li>Once the new debug VSCode instance settles down, click in it and type <kbd>Ctrl</kbd>-<kbd>Shift</kbd>-<kbd>P</kbd>, then type <code>redirect</code></li> <li>Any breakpoints in the original VSCode instance will trigger.</li> <li>Once the breakpoint is cleared, a message saying <b>Add redirecct from redirect_generator!</b> will appear in the debugged instance of VSCode.</li> </ol> <h2 id="anatomy">Extension Anatomy</h2> <p> I moved on to the next part of the tutorial, <a href='https://code.visualstudio.com/api/get-started/extension-anatomy' target='_blank' rel='nofollow'>Extension Anatomy</a>. </p> Node.js, NVM, NPM and Yarn 2022-03-01T00:00:00-05:00 https://mslinn.github.io/blog/2022/03/01/node-package-managers <p> <a href='https://nodejs.dev/learn/a-brief-history-of-nodejs' target='_blank' rel='nofollow'>Node.js</a>, a JavaScript runtime, is not my favorite programming environment. Its historical disregard for <a href='https://cheatsheetseries.owasp.org/cheatsheets/Nodejs_Security_Cheat_Sheet.html' target='_blank' rel='nofollow'>security</a> and stability is problematic. However, it has significant traction, especially amongst younger software technologists, and is used by many Rubyists for asset pipelines. With reluctance, I put together these notes for installing Node on Ubuntu/WSL using a version manager. </p> <h2 id="nvm">NVM</h2> <p> NVM is a <code>node.js</code> version manager. It provides easy installation, the ability to switch between versions of <code>node.js</code>, and retains globally installed packages for each version. </p> <p> Install <code>node.js</code> with <code>nvm</code>, the Node Version Manager. This allows you to install and maintain many different independent versions of <code>node.js</code>, and their associated packages, at the same time. </p> <p> <code>Nvm</code> is installed as follows: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9d93811c4061'><button class='copyBtn' data-clipboard-target='#id9d93811c4061' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash <span class='unselectable'>&nbsp; % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 14926 100 14926 0 0 84806 0 --:--:-- --:--:-- --:--:-- 85291 => nvm is already installed in /home/mslinn/.nvm, trying to update using git => => Compressing and cleaning up git repository => nvm source string already in /home/mslinn/.bashrc => bash_completion source string already in /home/mslinn/.bashrc => Close and reopen your terminal to start using nvm or run the following to use it now: export NVM_DIR="$HOME/.nvm" [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" # This loads nvm bash_completion </span></pre> <p> View the currently installed versions of <code>node.js</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id209bb0290fa6'><button class='copyBtn' data-clipboard-target='#id209bb0290fa6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>nvm list <span class='unselectable'>-> v17.3.1 default -> node (-> v17.3.1) iojs -> N/A (default) unstable -> N/A (default) node -> stable (-> v17.3.1) (default) stable -> 17.3 (-> v17.3.1) (default) lts/* -> lts/gallium (-> N/A) lts/argon -> v4.9.1 (-> N/A) lts/boron -> v6.17.1 (-> N/A) lts/carbon -> v8.17.0 (-> N/A) lts/dubnium -> v10.24.1 (-> N/A) lts/erbium -> v12.22.9 (-> N/A) lts/fermium -> v14.18.3 (-> N/A) lts/gallium -> v16.13.2 (-> N/A) </span></pre> <editor-fold versions> <p> View the very long list of available versions of <code>node.js</code> like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida6170b8b0058'><button class='copyBtn' data-clipboard-target='#ida6170b8b0058' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>nvm list-remote <span class='unselectable'>&nbsp; v0.1.14 v0.1.15 v0.1.16 v0.1.17 v0.1.18 v0.1.19 v0.1.20 v0.1.21 v0.1.22 v0.1.23 v0.1.24 v0.1.25 v0.1.26 v0.1.27 v0.1.28 v0.1.29 v0.1.30 v0.1.31 v0.1.32 v0.1.33 v0.1.90 v0.1.91 v0.1.92 v0.1.93 v0.1.94 v0.1.95 v0.1.96 v0.1.97 v0.1.98 v0.1.99 v0.1.100 v0.1.101 v0.1.102 v0.1.103 v0.1.104 v0.2.0 v0.2.1 v0.2.2 v0.2.3 v0.2.4 v0.2.5 v0.2.6 v0.3.0 v0.3.1 v0.3.2 v0.3.3 v0.3.4 v0.3.5 v0.3.6 v0.3.7 v0.3.8 v0.4.0 v0.4.1 v0.4.2 v0.4.3 v0.4.4 v0.4.5 v0.4.6 v0.4.7 v0.4.8 v0.4.9 v0.4.10 v0.4.11 v0.4.12 v0.5.0 v0.5.1 v0.5.2 v0.5.3 v0.5.4 v0.5.5 v0.5.6 v0.5.7 v0.5.8 v0.5.9 v0.5.10 v0.6.0 v0.6.1 v0.6.2 v0.6.3 v0.6.4 v0.6.5 v0.6.6 v0.6.7 v0.6.8 v0.6.9 v0.6.10 v0.6.11 v0.6.12 v0.6.13 v0.6.14 v0.6.15 v0.6.16 v0.6.17 v0.6.18 v0.6.19 v0.6.20 v0.6.21 v0.7.0 v0.7.1 v0.7.2 v0.7.3 v0.7.4 v0.7.5 v0.7.6 v0.7.7 v0.7.8 v0.7.9 v0.7.10 v0.7.11 v0.7.12 v0.8.0 v0.8.1 v0.8.2 v0.8.3 v0.8.4 v0.8.5 v0.8.6 v0.8.7 v0.8.8 v0.8.9 v0.8.10 v0.8.11 v0.8.12 v0.8.13 v0.8.14 v0.8.15 v0.8.16 v0.8.17 v0.8.18 v0.8.19 v0.8.20 v0.8.21 v0.8.22 v0.8.23 v0.8.24 v0.8.25 v0.8.26 v0.8.27 v0.8.28 v0.9.0 v0.9.1 v0.9.2 v0.9.3 v0.9.4 v0.9.5 v0.9.6 v0.9.7 v0.9.8 v0.9.9 v0.9.10 v0.9.11 v0.9.12 v0.10.0 v0.10.1 v0.10.2 v0.10.3 v0.10.4 v0.10.5 v0.10.6 v0.10.7 v0.10.8 v0.10.9 v0.10.10 v0.10.11 v0.10.12 v0.10.13 v0.10.14 v0.10.15 v0.10.16 v0.10.17 v0.10.18 v0.10.19 v0.10.20 v0.10.21 v0.10.22 v0.10.23 v0.10.24 v0.10.25 v0.10.26 v0.10.27 v0.10.28 v0.10.29 v0.10.30 v0.10.31 v0.10.32 v0.10.33 v0.10.34 v0.10.35 v0.10.36 v0.10.37 v0.10.38 v0.10.39 v0.10.40 v0.10.41 v0.10.42 v0.10.43 v0.10.44 v0.10.45 v0.10.46 v0.10.47 v0.10.48 v0.11.0 v0.11.1 v0.11.2 v0.11.3 v0.11.4 v0.11.5 v0.11.6 v0.11.7 v0.11.8 v0.11.9 v0.11.10 v0.11.11 v0.11.12 v0.11.13 v0.11.14 v0.11.15 v0.11.16 v0.12.0 v0.12.1 v0.12.2 v0.12.3 v0.12.4 v0.12.5 v0.12.6 v0.12.7 v0.12.8 v0.12.9 v0.12.10 v0.12.11 v0.12.12 v0.12.13 v0.12.14 v0.12.15 v0.12.16 v0.12.17 v0.12.18 iojs-v1.0.0 iojs-v1.0.1 iojs-v1.0.2 iojs-v1.0.3 iojs-v1.0.4 iojs-v1.1.0 iojs-v1.2.0 iojs-v1.3.0 iojs-v1.4.1 iojs-v1.4.2 iojs-v1.4.3 iojs-v1.5.0 iojs-v1.5.1 iojs-v1.6.0 iojs-v1.6.1 iojs-v1.6.2 iojs-v1.6.3 iojs-v1.6.4 iojs-v1.7.1 iojs-v1.8.1 iojs-v1.8.2 iojs-v1.8.3 iojs-v1.8.4 iojs-v2.0.0 iojs-v2.0.1 iojs-v2.0.2 iojs-v2.1.0 iojs-v2.2.0 iojs-v2.2.1 iojs-v2.3.0 iojs-v2.3.1 iojs-v2.3.2 iojs-v2.3.3 iojs-v2.3.4 iojs-v2.4.0 iojs-v2.5.0 iojs-v3.0.0 iojs-v3.1.0 iojs-v3.2.0 iojs-v3.3.0 iojs-v3.3.1 v4.0.0 v4.1.0 v4.1.1 v4.1.2 v4.2.0 (LTS: Argon) v4.2.1 (LTS: Argon) v4.2.2 (LTS: Argon) v4.2.3 (LTS: Argon) v4.2.4 (LTS: Argon) v4.2.5 (LTS: Argon) v4.2.6 (LTS: Argon) v4.3.0 (LTS: Argon) v4.3.1 (LTS: Argon) v4.3.2 (LTS: Argon) v4.4.0 (LTS: Argon) v4.4.1 (LTS: Argon) v4.4.2 (LTS: Argon) v4.4.3 (LTS: Argon) v4.4.4 (LTS: Argon) v4.4.5 (LTS: Argon) v4.4.6 (LTS: Argon) v4.4.7 (LTS: Argon) v4.5.0 (LTS: Argon) v4.6.0 (LTS: Argon) v4.6.1 (LTS: Argon) v4.6.2 (LTS: Argon) v4.7.0 (LTS: Argon) v4.7.1 (LTS: Argon) v4.7.2 (LTS: Argon) v4.7.3 (LTS: Argon) v4.8.0 (LTS: Argon) v4.8.1 (LTS: Argon) v4.8.2 (LTS: Argon) v4.8.3 (LTS: Argon) v4.8.4 (LTS: Argon) v4.8.5 (LTS: Argon) v4.8.6 (LTS: Argon) v4.8.7 (LTS: Argon) v4.9.0 (LTS: Argon) v4.9.1 (Latest LTS: Argon) v5.0.0 v5.1.0 v5.1.1 v5.2.0 v5.3.0 v5.4.0 v5.4.1 v5.5.0 v5.6.0 v5.7.0 v5.7.1 v5.8.0 v5.9.0 v5.9.1 v5.10.0 v5.10.1 v5.11.0 v5.11.1 v5.12.0 v6.0.0 v6.1.0 v6.2.0 v6.2.1 v6.2.2 v6.3.0 v6.3.1 v6.4.0 v6.5.0 v6.6.0 v6.7.0 v6.8.0 v6.8.1 v6.9.0 (LTS: Boron) v6.9.1 (LTS: Boron) v6.9.2 (LTS: Boron) v6.9.3 (LTS: Boron) v6.9.4 (LTS: Boron) v6.9.5 (LTS: Boron) v6.10.0 (LTS: Boron) v6.10.1 (LTS: Boron) v6.10.2 (LTS: Boron) v6.10.3 (LTS: Boron) v6.11.0 (LTS: Boron) v6.11.1 (LTS: Boron) v6.11.2 (LTS: Boron) v6.11.3 (LTS: Boron) v6.11.4 (LTS: Boron) v6.11.5 (LTS: Boron) v6.12.0 (LTS: Boron) v6.12.1 (LTS: Boron) v6.12.2 (LTS: Boron) v6.12.3 (LTS: Boron) v6.13.0 (LTS: Boron) v6.13.1 (LTS: Boron) v6.14.0 (LTS: Boron) v6.14.1 (LTS: Boron) v6.14.2 (LTS: Boron) v6.14.3 (LTS: Boron) v6.14.4 (LTS: Boron) v6.15.0 (LTS: Boron) v6.15.1 (LTS: Boron) v6.16.0 (LTS: Boron) v6.17.0 (LTS: Boron) v6.17.1 (Latest LTS: Boron) v7.0.0 v7.1.0 v7.2.0 v7.2.1 v7.3.0 v7.4.0 v7.5.0 v7.6.0 v7.7.0 v7.7.1 v7.7.2 v7.7.3 v7.7.4 v7.8.0 v7.9.0 v7.10.0 v7.10.1 v8.0.0 v8.1.0 v8.1.1 v8.1.2 v8.1.3 v8.1.4 v8.2.0 v8.2.1 v8.3.0 v8.4.0 v8.5.0 v8.6.0 v8.7.0 v8.8.0 v8.8.1 v8.9.0 (LTS: Carbon) v8.9.1 (LTS: Carbon) v8.9.2 (LTS: Carbon) v8.9.3 (LTS: Carbon) v8.9.4 (LTS: Carbon) v8.10.0 (LTS: Carbon) v8.11.0 (LTS: Carbon) v8.11.1 (LTS: Carbon) v8.11.2 (LTS: Carbon) v8.11.3 (LTS: Carbon) v8.11.4 (LTS: Carbon) v8.12.0 (LTS: Carbon) v8.13.0 (LTS: Carbon) v8.14.0 (LTS: Carbon) v8.14.1 (LTS: Carbon) v8.15.0 (LTS: Carbon) v8.15.1 (LTS: Carbon) v8.16.0 (LTS: Carbon) v8.16.1 (LTS: Carbon) v8.16.2 (LTS: Carbon) v8.17.0 (Latest LTS: Carbon) v9.0.0 v9.1.0 v9.2.0 v9.2.1 v9.3.0 v9.4.0 v9.5.0 v9.6.0 v9.6.1 v9.7.0 v9.7.1 v9.8.0 v9.9.0 v9.10.0 v9.10.1 v9.11.0 v9.11.1 v9.11.2 v10.0.0 v10.1.0 v10.2.0 v10.2.1 v10.3.0 v10.4.0 v10.4.1 v10.5.0 v10.6.0 v10.7.0 v10.8.0 v10.9.0 v10.10.0 v10.11.0 v10.12.0 v10.13.0 (LTS: Dubnium) v10.14.0 (LTS: Dubnium) v10.14.1 (LTS: Dubnium) v10.14.2 (LTS: Dubnium) v10.15.0 (LTS: Dubnium) v10.15.1 (LTS: Dubnium) v10.15.2 (LTS: Dubnium) v10.15.3 (LTS: Dubnium) v10.16.0 (LTS: Dubnium) v10.16.1 (LTS: Dubnium) v10.16.2 (LTS: Dubnium) v10.16.3 (LTS: Dubnium) v10.17.0 (LTS: Dubnium) v10.18.0 (LTS: Dubnium) v10.18.1 (LTS: Dubnium) v10.19.0 (LTS: Dubnium) v10.20.0 (LTS: Dubnium) v10.20.1 (LTS: Dubnium) v10.21.0 (LTS: Dubnium) v10.22.0 (LTS: Dubnium) v10.22.1 (LTS: Dubnium) v10.23.0 (LTS: Dubnium) v10.23.1 (LTS: Dubnium) v10.23.2 (LTS: Dubnium) v10.23.3 (LTS: Dubnium) v10.24.0 (LTS: Dubnium) v10.24.1 (Latest LTS: Dubnium) v11.0.0 v11.1.0 v11.2.0 v11.3.0 v11.4.0 v11.5.0 v11.6.0 v11.7.0 v11.8.0 v11.9.0 v11.10.0 v11.10.1 v11.11.0 v11.12.0 v11.13.0 v11.14.0 v11.15.0 v12.0.0 v12.1.0 v12.2.0 v12.3.0 v12.3.1 v12.4.0 v12.5.0 v12.6.0 v12.7.0 v12.8.0 v12.8.1 v12.9.0 v12.9.1 v12.10.0 v12.11.0 v12.11.1 v12.12.0 v12.13.0 (LTS: Erbium) v12.13.1 (LTS: Erbium) v12.14.0 (LTS: Erbium) v12.14.1 (LTS: Erbium) v12.15.0 (LTS: Erbium) v12.16.0 (LTS: Erbium) v12.16.1 (LTS: Erbium) v12.16.2 (LTS: Erbium) v12.16.3 (LTS: Erbium) v12.17.0 (LTS: Erbium) v12.18.0 (LTS: Erbium) v12.18.1 (LTS: Erbium) v12.18.2 (LTS: Erbium) v12.18.3 (LTS: Erbium) v12.18.4 (LTS: Erbium) v12.19.0 (LTS: Erbium) v12.19.1 (LTS: Erbium) v12.20.0 (LTS: Erbium) v12.20.1 (LTS: Erbium) v12.20.2 (LTS: Erbium) v12.21.0 (LTS: Erbium) v12.22.0 (LTS: Erbium) v12.22.1 (LTS: Erbium) v12.22.2 (LTS: Erbium) v12.22.3 (LTS: Erbium) v12.22.4 (LTS: Erbium) v12.22.5 (LTS: Erbium) v12.22.6 (LTS: Erbium) v12.22.7 (LTS: Erbium) v12.22.8 (LTS: Erbium) v12.22.9 (LTS: Erbium) v12.22.10 (Latest LTS: Erbium) v13.0.0 v13.0.1 v13.1.0 v13.2.0 v13.3.0 v13.4.0 v13.5.0 v13.6.0 v13.7.0 v13.8.0 v13.9.0 v13.10.0 v13.10.1 v13.11.0 v13.12.0 v13.13.0 v13.14.0 v14.0.0 v14.1.0 v14.2.0 v14.3.0 v14.4.0 v14.5.0 v14.6.0 v14.7.0 v14.8.0 v14.9.0 v14.10.0 v14.10.1 v14.11.0 v14.12.0 v14.13.0 v14.13.1 v14.14.0 v14.15.0 (LTS: Fermium) v14.15.1 (LTS: Fermium) v14.15.2 (LTS: Fermium) v14.15.3 (LTS: Fermium) v14.15.4 (LTS: Fermium) v14.15.5 (LTS: Fermium) v14.16.0 (LTS: Fermium) v14.16.1 (LTS: Fermium) v14.17.0 (LTS: Fermium) v14.17.1 (LTS: Fermium) v14.17.2 (LTS: Fermium) v14.17.3 (LTS: Fermium) v14.17.4 (LTS: Fermium) v14.17.5 (LTS: Fermium) v14.17.6 (LTS: Fermium) v14.18.0 (LTS: Fermium) v14.18.1 (LTS: Fermium) v14.18.2 (LTS: Fermium) v14.18.3 (LTS: Fermium) v14.19.0 (Latest LTS: Fermium) v15.0.0 v15.0.1 v15.1.0 v15.2.0 v15.2.1 v15.3.0 v15.4.0 v15.5.0 v15.5.1 v15.6.0 v15.7.0 v15.8.0 v15.9.0 v15.10.0 v15.11.0 v15.12.0 v15.13.0 v15.14.0 v16.0.0 v16.1.0 v16.2.0 v16.3.0 v16.4.0 v16.4.1 v16.4.2 v16.5.0 v16.6.0 v16.6.1 v16.6.2 v16.7.0 v16.8.0 v16.9.0 v16.9.1 v16.10.0 v16.11.0 v16.11.1 v16.12.0 v16.13.0 (LTS: Gallium) v16.13.1 (LTS: Gallium) v16.13.2 (LTS: Gallium) v16.14.0 (Latest LTS: Gallium) v17.0.0 v17.0.1 v17.1.0 v17.2.0 v17.3.0 -> v17.3.1 v17.4.0 v17.5.0 v17.6.0 </span></pre> </editor-fold> <h2 id="npm">NPM</h2> <p> <code>Npm</code> is a package manager for the JavaScript programming language. It helps install libraries, plugins, frameworks and applications. <code>npm</code> is the default package manager for Node.js. It consists of a command-line client, also called <code>npm</code>, and an online database of public and paid-for private packages called the <code>npm</code> registry. </p> <p> <code>Npm</code> fetches dependencies from the <code>npm</code> registry for every <code>npm install</code> command. </p> <p> <code>Npm</code> generates a <code>package-lock.json</code> file. The layout of this file is a trade-off between determinism and simplicity. The same <code>node_modules/</code> folder will be generated, even from different <code>npm</code> versions. Every dependency will have a version number associated with it in the <code>package-lock</code> file. </p> <h3 id="npm_install">Install npm with a node version manager</h3> <p> Follow <a href='https://docs.npmjs.com/resolving-eacces-permissions-errors-when-installing-packages-globally#reinstall-npm-with-a-node-version-manager' target='_blank' rel='nofollow'>these instructions</a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcca6afaeffbc'><button class='copyBtn' data-clipboard-target='#idcca6afaeffbc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mkdir ~/.npm-global <span class='unselectable'>$ </span>npm config set prefix ~/.npm-global <span class='unselectable'>$ </span>cat >> ~/.bashrc &lt;&lt;EOF export PATH="$HOME/.npm-global/bin:$PATH" export NVM_DIR="$HOME/.nvm" [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" 1>&2 # Loads nvm EOF <span class='unselectable'>$ </span>source ~/.bashrc</pre> <h3 id="global_install">Install A Global Package</h3> <p> To install a global package, the command template for <code>npm</code> is: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf5064c02c186'><button class='copyBtn' data-clipboard-target='#idf5064c02c186' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>npm install -g package_name[@version_number]</pre> <p> For example: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5e1d909663b8'><button class='copyBtn' data-clipboard-target='#id5e1d909663b8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>npm install -g broken-link-checker <span class='unselectable'>npm WARN deprecated calmcard@0.1.1: no longer maintained npm WARN deprecated nopter@0.3.0: try optionator npm WARN deprecated uuid@2.0.3: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. npm WARN deprecated urlobj@0.0.11: use universal-url, minurl, relateurl, url-relation added 104 packages, and audited 105 packages in 5s </span></pre> <h2 id="yarn">Yarn</h2> <p> <a href='https://yarnpkg.com/' target='_blank' rel='nofollow'>Yarn</a> stands for Yet Another Resource Negotiator and it is a package manager like <code>npm</code>. It was developed by Facebook and is now open-source. The intention behind developing yarn was to fix performance and security concerns with <code>npm</code>. Yarn generates a <code>yarn.lock</code> file, which helps easy merges. The merges are predictable. Yarn caches every package it has downloaded, so it never needs to download the same package again. It also does almost everything concurrently to maximize resource utilization. This means faster installs. Yarn guarantees that any installation that works on one system will work exactly the same on another system, unlike <code>npm</code>. </p> <p> The <a href='https://www.npmjs.com/package/yarn' target='_blank' rel='nofollow'><code>yarn</code> package</a> on <code>npmjs</code> notes that Yarn was inspired by Bundler, Cargo and npm, so similarities exist. </p> <p> Yarn introduces the zero-install concept, which means that a project should be able to be used as soon as it is cloned. Use Plug&lsquo;n&lsquo;Play to resolve dependencies via the cache folder and not from node_modules. The cache folder is by default stored within your project folder, in <code>.yarn/cache</code> </p> <h2 id="npm_yarn">Npm vs. Yarn</h2> <p> When using <code>npm install</code>, dependencies are installed sequentially, one after another. The output logs in the terminal are informative but a bit hard to read. To install the packages with Yarn, run the <code>yarn</code> command. Yarn installs packages in parallel, which is one of the reasons it's quicker than npm. </p> <p> See the article titled <a href='https://www.geeksforgeeks.org/difference-between-npm-and-yarn/' target='_blank' rel='nofollow'>Difference between npm and yarn</a>. </p> <h2 id="install_yarn">Installing Yarn</h2> The <a href='https://yarnpkg.com/getting-started/install' target='_blank' rel='nofollow'>official docs</a> say to add the following to <code>.gitignore</code> for any git project that uses <code>yarn</code>: <div class='codeLabel unselectable' data-lt-active='false'>.gitignore</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide61d78977e9d'><button class='copyBtn' data-clipboard-target='#ide61d78977e9d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>.yarn/* !.yarn/cache !.yarn/patches !.yarn/plugins !.yarn/releases !.yarn/sdks !.yarn/versions</pre> <p> Install <code>yarn</code> as follows: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf4bba5dd259d'><button class='copyBtn' data-clipboard-target='#idf4bba5dd259d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>corepack enable <span class='unselectable'>$ </span>npm i -g corepack <span class='unselectable'>$ </span>yarn init -2</pre> <p> Yarn stores dependencies locally, and if the proper version is present locally, it is fetched from the disk during a <code>yarn add</code> command. </p> <p> To install a global package, the command template for yarn is: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb6c1db680a00'><button class='copyBtn' data-clipboard-target='#idb6c1db680a00' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yarn global add package_name@version_number</pre> Iterating Slim Language Templates 2022-02-22T00:00:00-05:00 https://mslinn.github.io/blog/2022/02/22/testing-slim <video controls autoplay="1" width="100%" preload="auto" class="shadow rounded" style="margin-bottom: 1em;"> <source src="https://user-images.githubusercontent.com/485818/155521347-856ca755-cb89-4bc7-97ce-fa69d091cf7a.mp4" type="video/mp4"> Your browser does not support the video tag. Please use another browser to view this video. </video> <p> <a href='https://github.com/slim-template/slim#configuring-slim' target='_blank' rel='nofollow'>Slim</a> is a cool way to generate a potentially complex HTML block with a minimum of characters. Most often, Slim is thought of as an alternative to Ruby on Rail&rsquo;s ERB; however, it is useful in a broad range of contexts. </p> <p> The Slim Language Explorer transforms the experience of learning the Slim Language from something painfully awkward, to something pleasant. </p> <h2 id="explorer">I Have a Thing About Explorers</h2> <p> Over the years, I&rsquo;ve made about a dozen Explorers, starting with SMX Explorer in 1994 for The Internet Factory. They had the world&rsquo;s first programmable web server. I wrote the technical manual for using the programming language of their server; this was the first online publication with interactive code examples. </p> <p> <a href='/resume/history/jspexplorer/developerWorks/'>IBM published an article about JSP Explorer</a> in 2001. My <a href='/blog/index.html#Zamples'>Zamples</a> startup in 2001 was all about the world&rsquo;s first multi-lingual &ldquo;Try it!&rdquo; button, which executed the displayed code written one of many languages, some of which were pre-packaged with various popular libraries. Zamples was essentially a next-generation JSP Explorer. </p> <p> If you are a web developer, then you have seen and probably used explorers before. Ever try <a href='https://codepen.io/' target='_blank' rel='nofollow'>Code Pen</a>? I invented that technology. </p> <p> Anyway, here it is, yet another explorer, this one packaged as a git project for Ubuntu &ndash; for the Slim Language! Squint and it looks like my last name. </p> <h2 id="slim_explorer">The Slim Language Explorer</h2> <p> When you are trying to figure out how to express HTML in Slim, however, you inevitably rerun the generation process over and over. The <a href='https://github.com/mslinn/slim_explorer' target='_blank'>Slim Language Explorer</a> consists of a small Ruby program that launches Slim and displays the results of evaluating the Slim expression. </p> <h2 id="reload">Live Reload</h2> <p> The output is regenerated whenever a file is modified, created or deleted within the <code>watched</code> directory of the project. This allows you to edit the Slim expression and/or modify the YAML data, and view the updated output each time a change is made. </p> <h2 id="ruby">Embed Ruby Code</h2> <p> The template below uses a Ruby filter, shown highlighted in yellow, where Ruby code can be inserted. Everything indented after <code>Ruby:</code> is parsed as Ruby code. Gems and other code can be <code>require</code>d. Note that methods defined in a Ruby filter must be <a href='https://github.com/slim-template/slim/issues/835' target='_blank' rel='nofollow'>defined as class methods</a>, which means that when defining them, their names must be prefixed with <code>self</code>. Any <code>Module</code>s that might be <code>include</code>d would also need similar handling, or they would need to be <code>include</code>d into a <code>class</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>template.slim</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4b3bdaed65ff'><button class='copyBtn' data-clipboard-target='#id4b3bdaed65ff' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class="bg_yellow">ruby:</span> require 'slugify' def <span class="bg_yellow">self.</span>padding 'padding: 0 5px 3px 5px' end def <span class="bg_yellow">self.</span>boxed(contents) "&lt;div style='border: thin solid grey; #{padding};'>#{contents}&lt;/div>" end doctype html html head title = heading body h1 = heading p = message ul li: a href="mailto:#{email}" #{name} li style="margin-top: 3px; #{padding}; background-color: #{background_color}; color: #{color};" = 'Green and white slugs ... yuck!'.slugify ==boxed 'Help, I am stuck inside this computer!'</pre> <p> The above template contains references to local variables, defined in the Ruby filter, and variables passed to it which <code>slim_explorer</code> read from a YAML file. </p> <h2 id="data">YAML Data</h2> <p> Most templates require data to inflate them. Slim Language Explorer gets data from <code>template.yaml</code>, shown below. Go ahead and change it while experimenting. </p> <div class='codeLabel unselectable' data-lt-active='false'>template.slim</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc8932ed5c621'><button class='copyBtn' data-clipboard-target='#idc8932ed5c621' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>environment: - email: mslinn@mslinn.com - heading: Testing, 1-2-3, Testing ... - message: World peace begins with you and me. - name: Michael Slinn - color: white - background_color: green</pre> <p> The name of the top-most property (<code>environment</code>) does not matter. </p> <h2 id="install">Installation</h2> <h3 id="Ruby">Ruby Development Support</h3> <p> The official instructions for installing full Ruby <a href='https://www.ruby-lang.org/en/documentation/installation/' target='_blank' rel='nofollow'>are here</a>, although the instructions are incomplete, terse and dated. Be sure to include the development tools. For Ubuntu, this is what I recommend: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd42f411f1e7f'><button class='copyBtn' data-clipboard-target='#idd42f411f1e7f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install libruby ruby-dev</pre> <h3 id="clone">Clone slim_explorer</h3> <p> Clone the <a href='https://github.com/mslinn/slim_explorer' target='_blank'><code>slim_explorer</code> git repo</a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfcdabc28071e'><button class='copyBtn' data-clipboard-target='#idfcdabc28071e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git clone https://github.com/mslinn/slim_explorer.git</pre> <p> Move to the newly cloned directory: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id478b28bb2f2b'><button class='copyBtn' data-clipboard-target='#id478b28bb2f2b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cd slim_explorer</pre> <h3 id="gems">Install Project Gems</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb90c898a7071'><button class='copyBtn' data-clipboard-target='#idb90c898a7071' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>bundle install</pre> <h2 id="repl">Slim Language REPL</h2> <p> In some sense, this is a REPL for the Slim Language. </p> <h3 id="commandMode">Command-Line Mode</h3> <p> The video above demonstrates command-line mode running in Visual Studio Code. </p> <ol> <li> To use the Slim Language Explorer, start a bash shell and type: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf2f8409d4461'><button class='copyBtn' data-clipboard-target='#idf2f8409d4461' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>./slim_explorer</pre> The Slim expression(s) stored in <code>watched/template.slim</code> are evaluated and stored into <code>www/raw.htm</code> using the key and values stored in file <code>watched/scope.yaml</code>. </li> <li> Use the editor of your choice to modify any file in the <code>watched/</code> directory. </li> <li> The contents of <code>watched/template.slim</code> are re-evaluated, and <code>www/raw.html</code> is updated after each change is saved. This process continues until interrupted. </li> </ol> <h3 id="webMode">Web Server Mode</h3> <ol> <li> Start a bash shell and type: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id72dda60677ed'><button class='copyBtn' data-clipboard-target='#id72dda60677ed' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>./slim_explorer serve</pre> The Slim expression(s) stored in <code>watched/template.slim</code> are evaluated and stored into <code>www/index.html</code> using the keys and values stored in file <code>watched/scope.yaml</code>. </li> <li> Open <code>www/index.html</code> in your favorite web browser, either from the file, or at <a href='https://http://localhost:3030/index.html' target='_blank' rel='nofollow'><code>http://localhost:3030/index.html</code></a>. This is what it looks like:<br/> <div style=""> <picture> <source srcset="https://github.com/mslinn/slim_explorer/raw/master/doc/server_mode.png" type="image/webp"> <source srcset="https://github.com/mslinn/slim_explorer/raw/master/doc/server_mode.png" type="image/png"> <img src="https://github.com/mslinn/slim_explorer/raw/master/doc/server_mode.png" class=" liImg2 rounded shadow" /> </picture> </div> </li> <li> Use the editor of your choice to modify any file in the <code>watched</code> directory. </li> <li> The contents of <code>watched/template.slim</code> are re-evaluated and <code>www/index.html</code> is updated each time a file in the <code>watched</code> directory is saved, created or deleted. </li> </ol> <h2 id="video">About the Video</h2> <p> You can see the generated HTML change as the Slim expression or the YAML data is modified. </p> <p> To make the video, I installed the <a href='https://marketplace.visualstudio.com/items?itemName=tht13.html-preview-vscode' target='_blank' rel='nofollow'>Visual Studio Code HTML Preview extension</a> by Thomas Haakon Townsend, which had 1,649,338 installations. While this is a generally useful extension, it enables the instantaneous display of the generated HTML in a render pane. The extension can be installed via the command line if you like: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5ddc46d1414e'><button class='copyBtn' data-clipboard-target='#id5ddc46d1414e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>code --install-extension tht13.html-preview-vscode@0.2.5</pre> Fun With Python Enums 2022-02-10T00:00:00-05:00 https://mslinn.github.io/blog/2022/02/10/python-3.4-enums <p> This blog post demonstrates how to define additional properties for <a href='https://docs.python.org/3/library/enum.html' target='_blank' rel='nofollow'>Python 3 enums</a>. Defining an additional property in a Python enum can provide a simple way to provide string values. The concept is then expanded to demonstrate composition, an important concept for functional programming. This post concludes with a demonstration of dynamic dispatch in Python, by further extending an enum. </p> <h2 id="enum">Adding Properties to Python Enums</h2> <p> Searching for <a href='https://www.google.com/search?q=python+enum+string+value' target='_blank' rel='nofollow'><code>python enum string value</code></a> yields some complex and arcane ways to approach the problem. </p> <p> Below is a short example of a Python enum that demonstrates a simple way to provide lower-case string values for enum constants: a new property, <code>to_s</code>, is defined. This property provides the string representation that is required. You could define other properties and methods to suit the needs of other projects. </p> <div class='codeLabel unselectable' data-lt-active='false'>cad_enums.py</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5142b0473f09'><button class='copyBtn' data-clipboard-target='#id5142b0473f09' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>"""Defines enums"""<br/> from enum import Enum, auto<br/> class EntityType(Enum): """Types of entities""" SITE = auto() GROUP = auto() COURSE = auto() SECTION = auto() LECTURE = auto()<br/> <span class="bg_yellow"> @property def to_s(self) -> str: """:return: lower-case name of this instance""" return self.name.lower()</span></pre> <p> Adding the following to the bottom of the program allows us to demonstrate it: </p> <div class='codeLabel unselectable' data-lt-active='false'>cad_enums.py (part 2)</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id54353b595509'><button class='copyBtn' data-clipboard-target='#id54353b595509' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>if __name__ == "__main__": # Just for demonstration print("Specifying individual values:") print(f" {EntityType.SITE.value}: {EntityType.SITE.to_s}") print(f" {EntityType.GROUP.value}: {EntityType.GROUP.to_s}") print(f" {EntityType.COURSE.value}: {EntityType.COURSE.to_s}") print(f" {EntityType.SECTION.value}: {EntityType.SECTION.to_s}") print(f" {EntityType.LECTURE.value}: {EntityType.LECTURE.to_s}") print("\nIterating through all values:") for entity_type in EntityType: print(f" {entity_type.value}: {entity_type.to_s}")</pre> <p> Running the program produces this output: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbdecf6b90455'><button class='copyBtn' data-clipboard-target='#idbdecf6b90455' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cad_enums.py <span class='unselectable'>Specifying individual values: 1: site 2: group 3: course 4: section 5: lecture Iterating through all values: 1: site 2: group 3: course 4: section 5: lecture </span></pre> <div class="right" style="font-size: 3em;">&#128513;</div> <p> Easy! </p> <h2 id="constructor">Constructing Enums</h2> <p> Enum constructors work the same as other Python class constructors. There are several ways to make a new instance of a Python enum. Let's try two ways by using the <a href='https://docs.python.org/3/tutorial/interpreter.html' target='_blank' rel='nofollow'>Python interpreter</a>. Throughout this blog post I've inserted a blank line between Python interpreter prompts for readability. </p> <div class='codeLabel unselectable' data-lt-active='false'>Python</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbbc2e78aebeb'><button class='copyBtn' data-clipboard-target='#idbbc2e78aebeb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>python <span class='unselectable'>Python 3.9.7 (default, Sep 10 2021, 14:59:43) [GCC 11.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> </span>from cad_enums import EntityType <span class='unselectable'>>>> </span># Specify the desired enum constant value symbolically <span class='unselectable'>>>> </span><span class="bg_yellow">gtype = EntityType.GROUP</span> <span class='unselectable'>>>> </span>print(gtype) <span class='unselectable'>EntityType.GROUP </span> <span class='unselectable'>>>> </span># Specify the desired enum constant value numerically <span class='unselectable'>>>> </span><span class="bg_yellow">stype = EntityType(1)</span> <span class='unselectable'>>>> </span>print(stype) <span class='unselectable'>EntityType.SITE </span></pre> <h2 id="ordering">Enum Ordering</h2> <p> A program I am working on needs to obtain the parent <code>EntityType</code>. By 'parent' I mean the <code>EntityType</code> with the next lowest numeric value. For example, the parent of <code>EntityType.GROUP</code> is <code>EntityType.SITE</code>. We can obtain a parent enum by computing its numeric value by adding the following method to the <code>EntityType</code> class definition. </p> <div class='codeLabel unselectable' data-lt-active='false'>Python</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1e45f981e0bc'><button class='copyBtn' data-clipboard-target='#id1e45f981e0bc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>@property def parent(self) -> <span class="bg_yellow">'EntityType'</span>: """:return: entity type of parent; site has no parent""" return EntityType(max(self.value - 1, 1))</pre> <p> The <span class="bg_yellow">return type</span> above is enclosed in quotes (<code>'EntityType'</code>) to keep Python's type checker happy, because this is a <a href='https://www.python.org/dev/peps/pep-0484/#forward-references' target='_blank' rel='nofollow'>forward reference</a>. This is a forward reference because the type is referenced before it is fully compiled. </p> <p> The complete enum class definition is now: </p> <div class='codeLabel unselectable' data-lt-active='false'>cad_enums.py</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3bd8638202f4'><button class='copyBtn' data-clipboard-target='#id3bd8638202f4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>"""Defines enums"""<br/> from enum import Enum, auto<br/> class EntityType(Enum): """Types of entities""" SITE = auto() GROUP = auto() COURSE = auto() SECTION = auto() LECTURE = auto()<br/> @property def to_s(self) -> str: """:return: lower-case name of this instance""" return self.name.lower()<br> @property def parent(self) -> 'EntityType': """:return: entity type of parent; site has no parent""" return EntityType(max(self.value - 1, 1))</pre> <p> Lets try out the new <code>parent</code> property in the Python interpreter. </p> <div class='codeLabel unselectable' data-lt-active='false'>Python</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf90119cef6b0'><button class='copyBtn' data-clipboard-target='#idf90119cef6b0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>EntityType.LECTURE.parent <span class='unselectable'>&lt;EntityType.SECTION: 4> </span> <span class='unselectable'>>>> </span>EntityType.SECTION.parent <span class='unselectable'>&lt;EntityType.COURSE: 3> </span> <span class='unselectable'>>>> </span>EntityType.COURSE.parent <span class='unselectable'>&lt;EntityType.GROUP: 2> </span> <span class='unselectable'>>>> </span>EntityType.GROUP.parent <span class='unselectable'>&lt;EntityType.SITE: 1> </span> <span class='unselectable'>>>> </span>EntityType.SITE.parent <span class='unselectable'>&lt;EntityType.SITE: 1> </span></pre> <h2 id="compose">Enum Composition</h2> <p> Like methods and properties in all other Python classes, enum methods and properties compose if they return an instance of the class. Composition is also known as <a href='https://en.wikipedia.org/wiki/Method_chaining' target='_blank' rel='nofollow'>method chaining</a>, and also can apply to class properties. Composition is an essential practice of a functional programming style. </p> <p> The <code>parent</code> property returns an instance of the <code>EntityType</code> enum class, so it can be composed with any other property or method of that class, for example the <code>to_s</code> property shown earlier. </p> <div class='codeLabel unselectable' data-lt-active='false'>Python</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6e21433e8ba4'><button class='copyBtn' data-clipboard-target='#id6e21433e8ba4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>EntityType.LECTURE.parent.to_s <span class='unselectable'>'section' </span> <span class='unselectable'>>>> </span>EntityType.SECTION.parent.to_s <span class='unselectable'>'course' </span> <span class='unselectable'>>>> </span>EntityType.COURSE.parent.to_s <span class='unselectable'>'group' </span> <span class='unselectable'>>>> </span>EntityType.GROUP.parent.to_s <span class='unselectable'>'site' </span> <span class='unselectable'>>>> </span>EntityType.SITE.parent.to_s <span class='unselectable'>'site' </span></pre> <h2 id="dispatch">Dynamic Dispatch</h2> <p> The <a href='https://docs.python.org/3/library/typing.html#callable' target='_blank' rel='nofollow'>Python documentation</a> might lead someone to assume that writing <a href='https://en.wikipedia.org/wiki/Dynamic_dispatch' target='_blank' rel='nofollow'>dynamic dispatch</a> code is more complex than it actually is. </p> <p> To summarize the documentation, all Python classes, methods and instances are callable. <a href='https://www.tutorialsteacher.com/python/callable-method' target='_blank' rel='nofollow'><code>Callable</code> functions</a> have type <code>Callable[[InputArg1Type, InputArg2Type], ReturnType]</code>. If you do not want any type checking, write <code>Callable[..., Any]</code>. However, this is not very helpful information for dynamic dispatch. Fortunately, working with <code>Callable</code> is very simple. </p> <p class="notepaper"> You can pass around any Python class, constructor, function or method, and later provide it with the usual arguments. Invocation just works. </p> <p> Let me show you how easy it is to write dynamic dispatch code in Python, let's construct one of five classes, depending on the value of an enum. First, we need a class definition for each enum value: </p> <div class='codeLabel unselectable' data-lt-active='false'>Python</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id341ca03729f4'><button class='copyBtn' data-clipboard-target='#id341ca03729f4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># pylint: disable=too-few-public-methods class BaseClass(): """Demo only""" class TestLecture(BaseClass): """This constructor has type Callable[[int, str], TestLecture]""" def __init__(self, id_: int, action: str): print(f"CadLecture constructor called with id {id_} and action {action}") class TestSection(BaseClass): """This constructor has type Callable[[int, str], TestSection]""" def __init__(self, id_: int, action: str): print(f"CadSection constructor called with id {id_} and action {action}") class TestCourse(BaseClass): """This constructor has type Callable[[int, str], TestCourse]""" def __init__(self, id_: int, action: str): print(f"CadCourse constructor called with id {id_} and action {action}") class TestGroup(BaseClass): """This constructor has type Callable[[int, str], TestGroup]""" def __init__(self, id_: int, action: str): print(f"CadGroup constructor called with id {id_} and action {action}") class TestSite(BaseClass): """This constructor has type Callable[[int, str], TestSite]""" def __init__(self, id_: int, action: str): print(f"CadSite constructor called with id {id_} and action {action}")</pre> <p> Now lets add another method, called <code>construct</code>, to <code>EntityType</code> that invokes the appropriate constructor according to the value of an <code>EntityType</code> instance: </p> <div class='codeLabel unselectable' data-lt-active='false'>Python</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcb072346a0ef'><button class='copyBtn' data-clipboard-target='#idcb072346a0ef' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>@property def construct(self) -> Callable: """:return: the appropriate Callable for each enum value""" if self == EntityType.LECTURE: return TestLecture if self == EntityType.SECTION: return TestSection if self == EntityType.COURSE: return TestCourse if self == EntityType.GROUP: return TestGroup return TestSite</pre> <p class="callForWorkRHS"> Using named arguments makes your code resistant to problems that might sneak in due to parameters changing over time. </p> <p> I favor using named arguments at all times; it avoids many problems. As code evolves, arguments might be added or removed, or even reordered. </p> <p class="clear"> Let's test out dynamic dispatch in the Python interpreter. A class specific to each <code>EntityType</code> value is constructed by invoking the appropriate <code>Callable</code> and passing it named arguments <a href='https://stackoverflow.com/a/28091085/553865' target='_blank' rel='nofollow'><code>id_</code></a> and <code>action</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Python</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4b81cfa5e193'><button class='copyBtn' data-clipboard-target='#id4b81cfa5e193' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>EntityType.LECTURE.construct(id_=55, action="gimme_lecture") <span class='unselectable'>TestLecture constructor called with id 55 and action gimme_lecture &lt;entity_types.TestLecture object at 0x7f9aac690070> </span> <span class='unselectable'>>>> </span>EntityType.SECTION.construct(id_=13, action="gimme_section") <span class='unselectable'>TestSection constructor called with id 13 and action gimme_section &lt;entity_types.TestSection object at 0x7f9aac5c1730> </span> <span class='unselectable'>>>> </span>EntityType.COURSE.construct(id_=40, action="gimme_course") <span class='unselectable'>TestCourse constructor called with id 40 and action gimme_course &lt;entity_types.TestCourse object at 0x7f9aac6900a0> </span> <span class='unselectable'>>>> </span>EntityType.GROUP.construct(id_=103, action="gimme_group") <span class='unselectable'>TestGroup constructor called with id 103 and action gimme_group &lt;entity_types.TestGroup object at 0x7f9aac4c6b20> </span> <span class='unselectable'>>>> </span>EntityType.SITE.construct(id_=1, action="gimme_site") <span class='unselectable'>TestSite constructor called with id 1 and action gimme_site &lt;entity_types.TestSite object at 0x7f9aac5c1730> </span></pre> <p> Because these factory methods return the newly created instance of the desired type, the string representation is printed on the console after the method finishes outputting its processing results, for example: <code>&lt;entity_types.TestLecture object at 0x7f9aac690070></code>. </p> <p> Using enums to construct class instances and/or invoke methods (aka dynamic dispatch) is super powerful. It rather resembles generics, actually, even though <a href='https://docs.python.org/3/library/typing.html#building-generic-types' target='_blank' rel='nofollow'>Python's support for generics</a> is still in its infancy. </p> <p> The complete Python program discussed in this post is <a href='/blog/python/entity_types.py'>here</a>. </p> Linking Directories on NTFS and Ext4 Volumes 2022-02-07T00:00:00-05:00 https://mslinn.github.io/blog/2022/02/07/wsl-volumes <p> Sometimes I need to insert some code into a program that depends on the type of format that a drive volume has. For example, today I need to either make a Windows junction to connect two directories on NTFS volumes. On the other hand, if one or both directories were on other types of volumes, I would have to connect the directories using a Linux symlink. </p> <p> All of the bash scripts shown in this blog post are meant to run in a Bash shell running on WSL or WSL2. </p> <h2 id="possible">Use Windows Junctions When Possible</h2> <p> When working on WSL, Windows junctions are more desirable than Linux hard links and symlinks because junctions are visible in Windows and also in WSL. Unlike Linux hard links, which only work within a single volume, Windows junctions can span two volumes. Linux symlinks are only visible from WSL; a symlinked directory only appears as a useless file when viewed from Windows. </p> <div style="text-align: right;"> <picture> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.webp" type="image/webp"> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.png" type="image/png"> <img src="/blog/images/wsl-volumes/windowsFileMgr.png" class="right liImg2 rounded shadow" /> </picture> </div> <p> Both directories need to be on NTFS volumes to make a Windows junction between them. Junctions are permitted within a single NTFS volume, or between two NTFS volumes. Linux symlinks can be used on all volume types, but only work properly when viewed from Linux. </p> <p> Windows junctions are shown with a small arrow icon in Windows File Manager. In the image above, the <code>curriculum</code> directory is a junction. </p> <h2 id="volumeType">Determining a Volume Type</h2> <p> I wrote the <code>volumeType</code> bash function to obtain the type of the volume that contains a file or directory. Linux <code>ext4</code> volumes have partition type <code>ext4</code>. NTFS volumes have partition type <code>9p</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide04076e184c8'><button class='copyBtn' data-clipboard-target='#ide04076e184c8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>function volumeType { # Usually returns volume types ext4 or 9p (for NTFS) df -Th "$1" | tail -n 1 | awk '{print $2}' }</pre> <p> Here are examples of using <code>volumeType</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id366b454ba6a3'><button class='copyBtn' data-clipboard-target='#id366b454ba6a3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>volumeType /mnt/c # Under WSL/WSL2 this is usually NTFS <span class='unselectable'>9p </span> <span class='unselectable'>$ </span>volumeType / # For Ubuntu this defaults to ext4 <span class='unselectable'>ext4 </span></pre> <p> All of the remaining scripts on this page either return a value (indicating <code>true</code>), or they do not return anything (indicating <code>false</code>). </p> <p> Two more bash functions test if a file or directory is part of an NTFS or ext4 volume: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id360141be2a29'><button class='copyBtn' data-clipboard-target='#id360141be2a29' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>function isNTFS { if [ "$( volumeType "$1" )" == 9p ]; then echo yes; fi } function isExt4 { if [ "$( volumeType "$1" )" == ext4 ]; then echo yes; fi }</pre> <p> Here are examples of using <code>isNTFS</code> and <code>isExt4</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida5405b2b5e10'><button class='copyBtn' data-clipboard-target='#ida5405b2b5e10' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>isNTFS /mnt/c <span class='unselectable'>yes </span> <span class='unselectable'>$ </span>isExt4 /mnt/c <span class='unselectable'>$ </span>isNTFS / <span class='unselectable'>$ </span>isExt4 / <span class='unselectable'>yes </span></pre> <h2 id="junctions">Windows Junctions</h2> <p> The <code>bothOnNTFS</code> bash function indicates if both of the paths passed to it are on NTFS volumes. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide8cee7ce7b1f'><button class='copyBtn' data-clipboard-target='#ide8cee7ce7b1f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>function bothOnNTFS { if [ "$( isNTFS "$1" )" ] && [ "$( isNTFS "$2" )" ]; then echo yes; fi }</pre> <p> Let's try out <code>bothOnNTFS</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida7ab818b339f'><span class='unselectable'>$ </span>bothOnNTFS /mnt/c /mnt/f <span class='unselectable'>yes </span> <span class='unselectable'>$ </span>bothOnNTFS /mnt/c /</pre> <p> <code>bothOnNTFS</code> lets us decide how to connect two directories. If they are both on NTFS volumes, we can connect them using a Windows junction; otherwise we'll need to symlink them. </p> <h2 id="connect">Connecting Via a Windows Junction or Linux Symlink</h2> <p> We could either make a Windows junction using the <a href='https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/mklink' target='_blank' rel='nofollow'><code>mklink</code></a> command, or we could make a Linux symlink using the <a href='https://man7.org/linux/man-pages/man1/ln.1.html' target='_blank' rel='nofollow'><code>ln -s</code></a> command. Notice how the order of parameters between <code>mklink</code> is the reverse of the order of the Linux <code>ln</code> command. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8492c3242457'><button class='copyBtn' data-clipboard-target='#id8492c3242457' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>if [ $( bothOnNTFS "$cadenzaCurriculum" . ) ]; then WINDOWS_PATH="$( wslpath -w "$cadenzaCurriculum/site_$TITLE" )" echo "Making Windows junction from $cadenzaCurriculum/site_$TITLE to curriculum/" cmd.exe /C mklink /j curriculum "$WINDOWS_PATH" else echo "Symlinking $cadenzaCurriculum/site_$TITLE to curriculum/" ln -s "$cadenzaCurriculum/site_$TITLE" curriculum fi</pre> When the above code ran it produced: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb1f6890f6554'>Making Windows junction from /mnt/f/work/cadenzaHome/cadenzaCurriculum/site_ScalaCourses.com to curriculum/ Junction created for curriculum <<===>> F:\work\cadenzaHome\cadenzaCurriculum\site_ScalaCourses.com</pre> Handcrafted Dynamic DNS for AWS Route53 and Namecheap 2022-01-30T00:00:00-05:00 https://mslinn.github.io/blog/2022/01/30/ddns-route53 <p> Now that I have fiber-optic internet service in my apartment, with 500 GB/s upload and download, I thought I would save money by hosting my <a href='https://scalacourses.com' target='_blank' rel='nofollow'><code>scalacourses.com</code></a> website on an Ubuntu server that runs here, instead of AWS. My home IP address is quite stable, and only changes when the fiber modem boots up. The modem is branded as a <a href='https://support.bell.ca/internet/products/home-hub-4000-modem' target='_blank' rel='nofollow'>Bell Home Hub 4000</a>, but I believe it is actually made by Arris (formerly known as Motorola). </p> <p> It makes little sense to pay the commercial cost of dedicated dynamic DNS services (typically $55 USD / year) when it is so easy to automate, and the operational cost is less than one cent per year. </p> <p> I wrote two little scripts that automatically check my public IP address, and modifies the DNS record for my home IP address whenever the IP address changes. One script is for sites that use DNS provided by AWS Route53, and the other is for DNS provided by Namecheap. </p> <p> The approach shown here could be used for all DNS servers that have a command-line interface. This was originally written for AWS Route53, but when I <a href='/blog/2022/05/26/aws-hijacking.html'>moved off AWS</a> I made a version for <a href='https://www.namecheap.com/support/knowledgebase/article.aspx/29/11/how-to-dynamically-update-the-hosts-ip-with-an-http-request/' target='_blank' rel='nofollow'>Namecheap</a>. Alternative DNS providers include <a href='https://docs.microsoft.com/en-us/cli/azure/network/dns?view=azure-cli-latest' target='_blank' rel='nofollow'>Azure DNS</a>, <a href='https://developers.cloudflare.com/cloudflare-one/tutorials/cli' target='_blank' rel='nofollow'>Cloudflare DNS</a>, <a href='https://support.dnsmadeeasy.com/support/solutions/articles/47001119947-the-ddns-shell-script' target='_blank' rel='nofollow'>DNSMadeEasy</a>, <a href='https://developer.dnsimple.com/libraries/' target='_blank' rel='nofollow'>DNSimple</a>, <a href='https://cloud.google.com/sdk/gcloud/reference/dns' target='_blank' rel='nofollow'>Google Cloud DNS</a>, and <a href='https://github.com/ultradns/dns_sprockets' target='_blank' rel='nofollow'>UltraDNS</a>. </p> <h2 id="fwd">Forwarding HTTP Requests</h2> <p> I added some entries to the modem so incoming HTTP traffic on ports 80 and 443 would be forwarded to ports 9000 and 9443 on my home server, <code>gojira</code>. </p> <div style=""> <picture> <source srcset="/blog/images/ddns/homeHubPortForward.webp" type="image/webp"> <source srcset="/blog/images/ddns/homeHubPortForward.png" type="image/png"> <img src="/blog/images/ddns/homeHubPortForward.png" class=" liImg2 rounded shadow" /> </picture> </div> <h2 id="usage">Using the Scripts</h2> <p> The version for AWS Route53 is called <code>dynamicDnsAws</code>, and the version for Namecheap is called <code>dynamicDnsNamecheap</code>. </p> <p> The scripts save the IP address to a file, and periodically compare the saved value to the current value. Then the scripts modify the DNS record for a specified subdomain whenever the value of the public IP address changes. </p> <h3 id="aws">Using the AWS Script</h3> <p> Here is the help information for the script: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd72cedeca4a3'><button class='copyBtn' data-clipboard-target='#idd72cedeca4a3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>dynamicDnsAws <span class='unselectable'>dynamicDnsAws - Maintains a dynamic DNS record in AWS Route53<br> Saves data in '/home/mslinn/.dynamicDnsAws'<br> Syntax: dynamicDnsAws [OPTIONS] SUB_DOMAIN DOMAIN<br> OPTIONS: -v Verbose mode<br> Example usage: dynamicDnsAws www scalacourses.com dynamicDnsAws -v www scalacourses.com </span></pre> <p> Here is a sample usage: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf5bdce725fc0'><button class='copyBtn' data-clipboard-target='#idf5bdce725fc0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>dynamicDnsAws www scalacourses.com <span class='unselectable'>{ "ChangeInfo": { "Id": "/change/C075751811HI18SH4L8L0", "Status": "PENDING", "SubmittedAt": "2022-01-30T21:10:09.261Z", "Comment": "UPSERT a record for www.scalacourses.com" } } </span></pre> <h3 id="aws">Using the Namecheap Script</h3> <p> Here is the help information for the script: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida0c8254429ea'><button class='copyBtn' data-clipboard-target='#ida0c8254429ea' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>dynamicDnsNamecheap <span class='unselectable'>dynamicDnsNamecheap - Maintains two Namecheap dynamic DNS records<br> Saves data in '/home/mslinn/.dynamicDns'<br> Syntax: dynamicDnsNamecheap [OPTIONS] DOMAIN PASSWORD<br> OPTIONS: -v Verbose mode<br> Example usage: dynamicDnsNamecheap mydomain.com asdfasdfasdfasdfasdf dynamicDnsNamecheap -v mydomain.com asdfasdfasdfasdf </span></pre> <p> Here is sample usage: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2b36c434038d'><button class='copyBtn' data-clipboard-target='#id2b36c434038d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>dynamicDnsNamecheap scalacourses.com asdfasdfasdfasdfasdf <span class='unselectable'>&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-16&quot;?&gt; &lt;interface-response&gt; &lt;Command&gt;SETDNSHOST&lt;/Command&gt; &lt;Language&gt;eng&lt;/Language&gt; &lt;IP&gt;142.126.4.220&lt;/IP&gt; &lt;ErrCount&gt;0&lt;/ErrCount&gt; &lt;errors /&gt; &lt;ResponseCount&gt;0&lt;/ResponseCount&gt; &lt;responses /&gt; &lt;Done&gt;true&lt;/Done&gt; &lt;debug&gt;&lt;![CDATA[]]&gt;&lt;/debug&gt; &lt;/interface-response&gt;&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-16&quot;?&gt; &lt;interface-response&gt; &lt;Command&gt;SETDNSHOST&lt;/Command&gt; &lt;Language&gt;eng&lt;/Language&gt; &lt;IP&gt;142.126.4.220&lt;/IP&gt; &lt;ErrCount&gt;0&lt;/ErrCount&gt; &lt;errors /&gt; &lt;ResponseCount&gt;0&lt;/ResponseCount&gt; &lt;responses /&gt; &lt;Done&gt;true&lt;/Done&gt; &lt;debug&gt;&lt;![CDATA[]]&gt;&lt;/debug&gt; &lt;/interface-response&gt; </span></pre> <h2 id="crontab">Invoking the Scripts from Crontab</h2> <p> A personal <code>crontab</code> can be modified by typing: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3e14de2c0cd1'><button class='copyBtn' data-clipboard-target='#id3e14de2c0cd1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>crontab -e</pre> <p> I pasted in the following into <code>crontab</code> on my Ubuntu server, running at home. These lines invoke the <code>dynamicDnsNamecheap</code> script via <code>crontab</code> every 5 minutes. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7044815a08ed'><button class='copyBtn' data-clipboard-target='#id7044815a08ed' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>*/5 * * * * /path/to/dynamicDnsNamecheap my_domain.com asdfasdfasdfasdfasdf</pre> <p> One the above is saved, <code>crontab</code> will run the script every 5 minutes. </p> <h2 id="source">Script Source Codes</h2> <p> Here are the bash scripts: </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,dynamicDnsAws' download='dynamicDnsAws' title='Click on the file name to download the file'>dynamicDnsAws</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idc066eb096d8a">#!/bin/bash # Author: Mike Slinn mslinn@mslinn.com # Written 2022-01-30 export SAVE_FILE_NAME="$HOME/.dynamicDns" function help &#123; echo "$( basename $0 ) - Maintains a dynamic DNS record in AWS Route53 Saves data in '$SAVE_FILE_NAME' Syntax: $( basename $0) [OPTIONS] SUB_DOMAIN DOMAIN OPTIONS: -v Verbose mode Example usage: $( basename $0) my_subdomain mydomain.com $( basename $0) -v my_subdomain mydomain.com " exit 1 &#125; function upsert &#123; export HOSTED_ZONES="$( aws route53 list-hosted-zones )" export HOSTED_ZONE_RECORD="$( jq -r ".HostedZones[] | select(.Name == \"$DOMAIN.\")" &lt;&lt;&lt; "$HOSTED_ZONES" )" export HOSTED_ZONE_RECORD_ID="$( jq -r .Id &lt;&lt;&lt; "$HOSTED_ZONE_RECORD" )" aws route53 change-resource-record-sets \ --hosted-zone-id "$HOSTED_ZONE_RECORD_ID" \ --change-batch "&#123; \"Comment\": \"UPSERT a record for $SUBDOMAIN.$DOMAIN\", \"Changes\": [&#123; \"Action\": \"UPSERT\", \"ResourceRecordSet\": &#123; \"Name\": \"$SUBDOMAIN.$DOMAIN\", \"Type\": \"A\", \"TTL\": 300, \"ResourceRecords\": [&#123; \"Value\": \"$IP\"&#125;] &#125; &#125;] &#125;" echo "$IP" > "$SAVE_FILE_NAME" &#125; if [ "$1" == -v ]; then export VERBOSE=true shift fi if [ -z "$2" ]; then help; fi set -e export SUBDOMAIN="$1" export DOMAIN="$2" export IP="$( dig +short myip.opendns.com @resolver1.opendns.com )" if [ ! -f "$SAVE_FILE_NAME" ]; then if [ "$VERBOSE" ]; then echo "Creating $SAVE_FILE_NAME"; fi upsert; elif [ $( cat "$SAVE_FILE_NAME" ) != "$IP" ]; then if [ "$VERBOSE" ]; then echo "Updating $SAVE_FILE_NAME" echo "'$IP' was not equal to '$( cat "$SAVE_FILE_NAME" )'" fi upsert; else if [ "$VERBOSE" ]; then echo "No change necessary for $SAVE_FILE_NAME"; fi fi </pre> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,dynamicDnsNamecheap' download='dynamicDnsNamecheap' title='Click on the file name to download the file'>dynamicDnsNamecheap</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="ided8f65f4ab2b">#!/bin/bash # Author: Mike Slinn mslinn@mslinn.com # Modified from AWS version (dynameicDnsAws) 2022-06-30 # See https://www.namecheap.com/support/knowledgebase/article.aspx/36/11/how-do-i-start-using-dynamic-dns/ export SAVE_FILE_NAME="$HOME/.dynamicDns" function help &#123; echo "$( basename $0 ) - Maintains two Namecheap dynamic DNS records Saves data in '$SAVE_FILE_NAME' Syntax: $( basename $0) [OPTIONS] DOMAIN PASSWORD OPTIONS: -v Verbose mode Example usage: $( basename $0) mydomain.com asdfasdfasdfasdfasdf $( basename $0) -v mydomain.com asdfasdfasdfasdf " exit 1 &#125; function upsert &#123; curl "https://dynamicdns.park-your-domain.com/update?host=@&amp;domain=$DOMAIN&amp;password=$PASSWORD&amp;ip=$IP" curl "https://dynamicdns.park-your-domain.com/update?host=www&amp;domain=$DOMAIN&amp;password=$PASSWORD&amp;ip=$IP" echo "$IP" > "$SAVE_FILE_NAME" echo "" &#125; if [ "$1" == -v ]; then export VERBOSE=true shift fi if [ -z "$2" ]; then help; fi set -e export DOMAIN="$1" export PASSWORD="$2" export IP="$( dig +short myip.opendns.com @resolver1.opendns.com )" if [ ! -f "$SAVE_FILE_NAME" ]; then if [ "$VERBOSE" ]; then echo "Creating $SAVE_FILE_NAME"; fi upsert; elif [ "$( cat "$SAVE_FILE_NAME" )" != "$IP" ]; then if [ "$VERBOSE" ]; then echo "Updating $SAVE_FILE_NAME" echo "'$IP' was not equal to '$( cat "$SAVE_FILE_NAME" )'" fi upsert; else if [ "$VERBOSE" ]; then echo "No change necessary for $SAVE_FILE_NAME"; fi fi </pre> Trimming Media Files Can Be Surprisingly Subtle 2022-01-23T00:00:00-05:00 https://mslinn.github.io/blog/2022/01/23/trimming-media <p> When I use OBS Studio to record video from Cam Link 4K and audio from Pro Tools using RME TotalMix I get really big <code>mkv</code> files. I need to be able to trim the video file so the bits before and after the good stuff are discarded. </p> <p> Lots of StackOverflow conversations revolve around trimming video files. Dozens of PC and Mac programs exist to do that task, mostly low quality and / or bothersome to use. Other solutions are overkill, for example Adobe Premiere Pro and DaVinci Resolve. </p> <p> In this post I present a bash script that reduces file size by > 80%, while preserving quality and metadata, and cropping to specified time periods. </p> <p> For any programmers who might read this: </p> <ul> <li>Some of the <code>ffmpeg</code> options this script uses are not available in older versions.</li> <li> The most important thing to know about options that might be passed to <code>ffmpeg</code> is that if you want to force a new video encoding, simply do not specify the <code>-vcodec copy</code> option. Re-encoding means that arbitrary start and end times can be specified accurately when cropping. </li> </ul> <p> <a href='https://gist.github.com/mslinn/15a1308aa2ca04a418f404e42c7f32e0' target='_blank'>This is the script:</a> </p> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/15a1308aa2ca04a418f404e42c7f32e0.js"> </script> <p> Here is a sample session, which extracts the sequence beginning at <code>00:00:25.000</code> until <code>00:02:52.000</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfb758c12b58e'><button class='copyBtn' data-clipboard-target='#idfb758c12b58e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>crop 'VideoFile.mkv' 25 2:52 <span class='unselectable'>ffmpeg version 4.4-6ubuntu5 Copyright (c) 2000-2021 the FFmpeg developers built with gcc 11 (Ubuntu 11.2.0-7ubuntu1) configuration: --prefix=/usr --extra-version=6ubuntu5 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared libavutil 56. 70.100 / 56. 70.100 libavcodec 58.134.100 / 58.134.100 libavformat 58. 76.100 / 58. 76.100 libavdevice 58. 13.100 / 58. 13.100 libavfilter 7.110.100 / 7.110.100 libswscale 5. 9.100 / 5. 9.100 libswresample 3. 9.100 / 3. 9.100 libpostproc 55. 9.100 / 55. 9.100 Input #0, matroska,webm, from 'VideoFile.mkv': Metadata: ENCODER : Lavf58.29.100 Duration: 00:02:57.93, start: 0.000000, bitrate: 8137 kb/s Stream #0:0: Video: h264 (High), yuv420p(tv, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 1k tbn, 60 tbc (default) Metadata: DURATION : 00:02:57.933000000 Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp (default) Metadata: title : simple_aac_recording DURATION : 00:02:57.813000000 Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264)) Stream #0:1 -> #0:1 (copy) Press [q] to stop, [?] for help [libx264 @ 0x55fe8a8a6d40] using SAR=1/1 [libx264 @ 0x55fe8a8a6d40] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX [libx264 @ 0x55fe8a8a6d40] profile High, level 3.1, 4:2:0, 8-bit [libx264 @ 0x55fe8a8a6d40] 264 - core 160 r3011 cde9a93 - H.264/MPEG-4 AVC codec - Copyleft 2003-2020 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00 Output #0, matroska, to 'VideoFile.crop.mkv': Metadata: encoder : Lavf58.76.100 Stream #0:0: Video: h264 (H264 / 0x34363248), yuv420p(tv, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 30 fps, 1k tbn (default) Metadata: DURATION : 00:02:57.933000000 encoder : Lavc58.134.100 libx264 Side data: cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A Stream #0:1: Audio: aac (LC) ([255][0][0][0] / 0x00FF), 48000 Hz, stereo, fltp (default) Metadata: title : simple_aac_recording DURATION : 00:02:57.813000000 frame= 4410 fps= 56 q=-1.0 Lsize= 29863kB time=00:02:26.98 bitrate=1664.3kbits/s speed=1.85x video:26295kB audio:3489kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.266039% [libx264 @ 0x55fe8a8a6d40] frame I:18 Avg QP:19.94 size: 91473 [libx264 @ 0x55fe8a8a6d40] frame P:1111 Avg QP:22.35 size: 14426 [libx264 @ 0x55fe8a8a6d40] frame B:3281 Avg QP:26.70 size: 2820 [libx264 @ 0x55fe8a8a6d40] consecutive B-frames: 0.8% 0.0% 0.1% 99.1% [libx264 @ 0x55fe8a8a6d40] mb I I16..4: 9.1% 68.8% 22.1% [libx264 @ 0x55fe8a8a6d40] mb P I16..4: 0.1% 1.2% 0.4% P16..4: 41.3% 11.1% 8.7% 0.0% 0.0% skip:37.2% [libx264 @ 0x55fe8a8a6d40] mb B I16..4: 0.0% 0.2% 0.0% B16..8: 25.8% 2.6% 0.6% direct: 0.7% skip:70.0% L0:42.9% L1:51.0% BI: 6.1% [libx264 @ 0x55fe8a8a6d40] 8x8 transform intra:71.2% inter:72.8% [libx264 @ 0x55fe8a8a6d40] coded y,uvDC,uvAC intra: 81.3% 83.4% 41.2% inter: 6.8% 12.5% 0.3% [libx264 @ 0x55fe8a8a6d40] i16 v,h,dc,p: 27% 14% 15% 44% [libx264 @ 0x55fe8a8a6d40] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 15% 12% 7% 13% 9% 13% 7% 9% [libx264 @ 0x55fe8a8a6d40] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 13% 11% 10% 17% 11% 10% 6% 7% [libx264 @ 0x55fe8a8a6d40] i8c dc,h,v,p: 46% 21% 21% 11% [libx264 @ 0x55fe8a8a6d40] Weighted P-Frames: Y:0.5% UV:0.1% [libx264 @ 0x55fe8a8a6d40] ref P L0: 56.5% 10.1% 21.4% 12.0% 0.0% [libx264 @ 0x55fe8a8a6d40] ref B L0: 87.7% 8.9% 3.4% [libx264 @ 0x55fe8a8a6d40] ref B L1: 94.4% </span></pre> <p> The cropped file is quite a bit smaller than the original. I have not noticed any decrease in quality. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id87fc34e6360f'><button class='copyBtn' data-clipboard-target='#id87fc34e6360f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls -AlF <span class='unselectable'>total 1546136 -rw-r--r-- 1 mslinn mslinn <span class='bg_yellow'>30579665</span> Jan 23 13:03 'VideoFile.crop.mkv' -rwxrwxrwx 1 mslinn mslinn <span class='bg_yellow'>180988555</span> Jan 22 18:31 'VideoFile.mkv' </span></pre> Windows Diskpart Cooperates With Diskmgmt 2022-01-14T00:00:00-05:00 https://mslinn.github.io/blog/2022/01/14/diskpart <p> Recently, I wanted to use some old hard drives as backup media. That meant scrubbing all the partitions off the drives, and installing new partitions, which of course would be empty. </p> <div class="right warning" style="width: 45%"> <p> Warning &ndash; Working with a command line program for system-level operations, without having backed up the system, is like walking on a tightrope without a net. </p> <p style="margin-bottom: 0"> A mistake could inadvertently wipe out a different hard drive on the computer than you intended. Without the ability to restore the system disk from backup, your computer could become inoperable. </p> </div> <p> However, I was unable to repartition one of those old drives using the GUI-based Windows <code>diskmgmt.msc</code> disk manager. The drive had soft errors. For some reason, those errors made it impossible for <code>diskmgmt.msc</code> to scan the drive. </p> <p> The more powerful Windows <a href='https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/diskpart' target='_blank' rel='nofollow'><code>diskpart</code></a>, a command line program, was able to get the job done. </p> <p> This article <a href ="#gui">ends</a> by demonstrating how you can get benefit from the <code>diskmgmt</code> GUI even when you are using the <code>diskpart</code> command line interface. This is possible because every command you type into <code>diskpart</code> causes a Windows system event to be published, and because <code>diskmgmt</code> subscribes to those events, it is able to display the results as they happen. </p> <h2 id="starting">Starting <span class="code">Diskpart</span></h2> <p> Run <code>diskpart</code> as administrator as follows: </p> <ol> <li> Press the <kbd>Windows</kbd> key. Do not hold it down, just depress it once, as you would do for any other key, and let it go. </li> <li>Type <code>diskpart</code>.</li> <li>Use the mouse or arrow keys to select <b>Run as administrator</b>.</li> </ol> <div style="text-align: center;"> <picture> <source srcset="/blog/images/diskpart/diskpartLaunch.webp" type="image/webp"> <source srcset="/blog/images/diskpart/diskpartLaunch.png" type="image/png"> <img src="/blog/images/diskpart/diskpartLaunch.png" class="center halfsize liImg2 rounded shadow" /> </picture> </div> <p> A window should open up labeled <b>Microsoft DiskPart</b>. Let's start by listing all the <code>diskpart</code> commands. </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id00275467b170'><button class='copyBtn' data-clipboard-target='#id00275467b170' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>Microsoft DiskPart version 10.0.19041.964 Copyright (C) Microsoft Corporation. On computer: BEAR DISKPART> </span>help <span class='unselectable'>Microsoft DiskPart version 10.0.19041.964 ACTIVE - Mark the selected partition as active. ADD - Add a mirror to a simple volume. ASSIGN - Assign a drive letter or mount point to the selected volume. ATTRIBUTES - Manipulate volume or disk attributes. ATTACH - Attaches a virtual disk file. AUTOMOUNT - Enable and disable automatic mounting of basic volumes. BREAK - Break a mirror set. CLEAN - Clear the configuration information, or all information, off the disk. COMPACT - Attempts to reduce the physical size of the file. CONVERT - Convert between different disk formats. CREATE - Create a volume, partition or virtual disk. DELETE - Delete an object. DETAIL - Provide details about an object. DETACH - Detaches a virtual disk file. EXIT - Exit DiskPart. EXTEND - Extend a volume. EXPAND - Expands the maximum size available on a virtual disk. FILESYSTEMS - Display current and supported file systems on the volume. FORMAT - Format the volume or partition. GPT - Assign attributes to the selected GPT partition. HELP - Display a list of commands. IMPORT - Import a disk group. INACTIVE - Mark the selected partition as inactive. LIST - Display a list of objects. MERGE - Merges a child disk with its parents. ONLINE - Online an object that is currently marked as offline. OFFLINE - Offline an object that is currently marked as online. RECOVER - Refreshes the state of all disks in the selected pack. Attempts recovery on disks in the invalid pack, and resynchronizes mirrored volumes and RAID5 volumes that have stale plex or parity data. REM - Does nothing. This is used to comment scripts. REMOVE - Remove a drive letter or mount point assignment. REPAIR - Repair a RAID-5 volume with a failed member. RESCAN - Rescan the computer looking for disks and volumes. RETAIN - Place a retained partition under a simple volume. SAN - Display or set the SAN policy for the currently booted OS. SELECT - Shift the focus to an object. SETID - Change the partition type. SHRINK - Reduce the size of the selected volume. UNIQUEID - Displays or sets the GUID partition table (GPT) identifier or master boot record (MBR) signature of a disk. </span></pre> <h2 id="listing">Listing Drives, Partitions and Volumes</h2> <p> Listing the drives is generally a good first step. Let's discover the command for that: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb9a8bb60efdd'><button class='copyBtn' data-clipboard-target='#idb9a8bb60efdd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>list <span class='unselectable'>Microsoft DiskPart version 10.0.19041.964 DISK - Display a list of disks. For example, LIST DISK. PARTITION - Display a list of partitions on the selected disk. For example, LIST PARTITION. VOLUME - Display a list of volumes. For example, LIST VOLUME. VDISK - Displays a list of virtual disks. </span></pre> <p> OK, we can list disks, partitions, volumes and virtual disks. Let's list the disk drives. </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id936ae94faff4'><button class='copyBtn' data-clipboard-target='#id936ae94faff4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>list disk <span class='unselectable'>Disk ### Status Size Free Dyn Gpt -------- ------------- ------- ------- --- --- Disk 0 Online 1863 GB 1024 KB * Disk 1 Online 465 GB 1024 KB Disk 2 Online 1863 GB 1024 KB * Disk 3 Online 1863 GB 0 B * Disk 5 Online 931 GB 931 GB * </span></pre> <div style=""> <picture> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.webp" type="image/webp"> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.png" type="image/png"> <img src="/blog/images/diskpart/newerTechVoyagerS3.png" class=" right" style="width: 25%; height: auto;" /> </picture> </div> <p> At this point I inserted the old 5.25" SATA drive that I wanted to repurpose into a <a href='https://www.amazon.com/NewerTech-Enclosure-Interface-NWTU3S3HD-hot-swapping/dp/B007TTQQIA' target='_blank' rel='nofollow'>NewerTech Voyager S3</a> caddy, connected to my computer via a USB 3 cable, and Windows automatically mounted it. The caddy also accepts 2.5" SATA drives, such as those commonly found in laptops. </p> <p> Now I told <code>diskpart</code> to rescan the drives, and then I listed the volumes on all disks. </p> <div class="clear"></div> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3eac10b37299'><button class='copyBtn' data-clipboard-target='#id3eac10b37299' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>rescan <span class='unselectable'>Please wait while DiskPart scans your configuration... DiskPart has finished scanning your configuration. DISKPART> </span>list volume <span class='unselectable'>Volume ### Ltr Label Fs Type Size Status Info ---------- --- ----------- ----- ---------- ------- --------- -------- Volume 0 D DVD-ROM 0 B No Media Volume 1 C BEAR_C NTFS Partition 1861 GB Healthy Boot Volume 2 FAT32 Partition 100 MB Healthy System Volume 3 NTFS Partition 539 MB Healthy Hidden Volume 4 NTFS Partition 450 MB Healthy Hidden Volume 5 F Work NTFS Partition 1863 GB Healthy Volume 6 E BEAR_E NTFS Partition 1863 GB Healthy <span class="bg_yellow"> Volume 8 FAT32 Partition 512 MB Healthy Hidden</span> </span></pre> <p> <code>Diskpart</code> displayed the hidden volume (#8) in the drive in the caddy. This drive has <code>readonly</code> status set, which prevents its contents from being modified or deleted. That would be good if I wanted to use this drive as an archive, but instead I want to scrub it and write new information on it. The currently existing partitions on this drive cannot be erased until <code>readonly</code> is cleared. Lets remove <code>readonly</code> status from the drive now: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5779f8a570db'><button class='copyBtn' data-clipboard-target='#id5779f8a570db' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>select volume 8 <span class='unselectable'>Volume 8 is the selected volume. DISKPART> </span>attributes disk clear readonly <span class='unselectable'>Disk attributes cleared successfully. DISKPART> </span>rescan <span class='unselectable'>Please wait while DiskPart scans your configuration... DiskPart has finished scanning your configuration. DISKPART> </span>list volume <span class='unselectable'>Volume ### Ltr Label Fs Type Size Status Info ---------- --- ----------- ----- ---------- ------- --------- -------- Volume 0 D DVD-ROM 0 B No Media Volume 1 C BEAR_C NTFS Partition 1861 GB Healthy Boot Volume 2 FAT32 Partition 100 MB Healthy System Volume 3 NTFS Partition 539 MB Healthy Hidden Volume 4 NTFS Partition 450 MB Healthy Hidden Volume 5 F Work NTFS Partition 1863 GB Healthy Volume 6 E BEAR_E NTFS Partition 1863 GB Healthy Volume 8 FAT32 Partition 512 MB Healthy Hidden </span></pre> <h2 id="wipe">Wiping the Drive</h2> <p> Now it is time to wipe the drive clean, which removes all partitions. </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0f2469959222'><button class='copyBtn' data-clipboard-target='#id0f2469959222' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>select disk 5 <span class='unselectable'>Disk 5 is now the selected disk. DISKPART> </span>list disk <span class='unselectable'>Disk ### Status Size Free Dyn Gpt -------- ------------- ------- ------- --- --- Disk 0 Online 1863 GB 1024 KB * Disk 1 Online 465 GB 1024 KB Disk 2 Online 1863 GB 1024 KB * Disk 3 Online 1863 GB 0 B * Disk 5 Online 931 GB 931 GB * DISKPART> </span>clean <span class='unselectable'>DiskPart succeeded in cleaning the disk. </span></pre> <h2 id="setRO">Archiving a Drive</h2> <p> If instead of wiping the drive, I wanted to archive the drive, I would want to set the read-only status. To do that, first select the drive as before, then type: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id942745462657'><button class='copyBtn' data-clipboard-target='#id942745462657' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>list disk <span class='unselectable'># Output as shown above </span> <span class='unselectable'>DISKPART> </span>select disk N <span class='unselectable'>Disk N is now the selected disk. </span> <span class='unselectable'>DISKPART> </span>attributes disk set readonly</pre> <p> Now the drive's contents could not accidently be erased or modified. </p> <h2 id="partition">Create A New Partition</h2> <p> We need to create a new partition that spans the entire disk on the now-empty selected drive. The <code>create</code> command can do that. Let's look at the help before using the command: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id18f189350700'><button class='copyBtn' data-clipboard-target='#id18f189350700' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>create <span class='unselectable'>Microsoft DiskPart version 10.0.19041.964 PARTITION - Create a partition. VOLUME - Create a volume. VDISK - Creates a virtual disk file. DISKPART> </span>create partition <span class='unselectable'>Microsoft DiskPart version 10.0.19041.964 EFI - Create an EFI system partition. EXTENDED - Create an extended partition. LOGICAL - Create a logical drive. MSR - Create a Microsoft Reserved partition. PRIMARY - Create a primary partition. </span></pre> <p> To create a new partition that spans the entire disk on the now-empty selected drive and make it active: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4208752c732e'><button class='copyBtn' data-clipboard-target='#id4208752c732e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>create partition primary <span class='unselectable'>DiskPart succeeded in creating the specified partition. DISKPART> </span>select partition 1 <span class='unselectable'>Partition 1 is now the selected partition. DISKPART> </span>active <span class='unselectable'>DiskPart marked the current partition as active. </span></pre> <h2 id="format">Format A Volume</h2> <p> Let's format the entire selected drive as one volume. Like the <code>clean</code> command, the <code>format</code> command operates on the currently selected disk. First, let's look at the help: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc4af6c7b4cbd'><button class='copyBtn' data-clipboard-target='#idc4af6c7b4cbd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>help format <span class='unselectable'>Formats the specified volume for use with Windows. Syntax: FORMAT [[FS=<FS>] [REVISION=<X.XX>] | RECOMMENDED] [LABEL=<"label">] [UNIT=<N>] [QUICK] [COMPRESS] [OVERRIDE] [DUPLICATE] [NOWAIT] [NOERR] FS=<FS> Specifies the type of file system. If no file system is given, the default file system displayed by the FILESYSTEMS command is used. REVISION=<X.XX> Specifies the file system revision (if applicable). RECOMMENDED If specified, use the recommended file system and revision instead of the default if a recommendation exists. The recommended file system (if one exists) is displayed by the FILESYSTEMS command. LABEL=<"label"> Specifies the volume label. UNIT=<N> Overrides the default allocation unit size. Default settings are strongly recommended for general use. The default allocation unit size for a particular file system is displayed by the FILESYSTEMS command. NTFS compression is not supported for allocation unit sizes above 4096. QUICK Performs a quick format. COMPRESS NTFS only: Files created on the new volume will be compressed by default. OVERRIDE Forces the file system to dismount first if necessary. All opened handles to the volume would no longer be valid. DUPLICATE UDF Only: This flag applies to UDF format, version 2.5 or higher. This flag instructs the format operation to duplicate the file system meta-data to a second set of sectors on the disk. The duplicate meta-data is used by applications, for example repair or recovery applications. If the primary meta-data sectors are found to be corrupted, the file system meta-data will be read from the duplicate sectors. NOWAIT Forces the command to return immediately while the format process is still in progress. If NOWAIT is not specified, DiskPart will display format progress in percentage. NOERR For scripting only. When an error is encountered, DiskPart continues to process commands as if the error did not occur. Without the NOERR parameter, an error causes DiskPart to exit with an error code. A volume must be selected for this operation to succeed. Examples: FORMAT FS=NTFS LABEL="New Volume" QUICK COMPRESS FORMAT RECOMMENDED OVERRIDE </span></pre> <p> <code>Diskpart</code> automatically chooses the optimal file system for the selected drive if you do not specify it. Choices include FAT, FAT32 and NTFS. To quick format the selected drive using the optimal file system: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1a0d4ca7e674'><button class='copyBtn' data-clipboard-target='#id1a0d4ca7e674' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>format quick</pre> <p> The default is to fully format the selected drive, which is what you want if the selected drive is suspect: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id615b7ab19c50'><button class='copyBtn' data-clipboard-target='#id615b7ab19c50' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>format</pre> <p> You could specify multiple parameters, for example: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd6e4f5d31ae8'><button class='copyBtn' data-clipboard-target='#idd6e4f5d31ae8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>format fs=ntfs label="My Drive" quick</pre> <h2 id="assign">Assigning a Drive Letter</h2> <p> Once the drive was formatted, I assigned the selected drive the letter <code>G</code>. You do not normally need to perform this step, unless you want to <a href='https://www.groovypost.com/howto/assign-permanent-letter-removable-usb-drive-windows/' target='_blank' rel='nofollow'>define a default letter for this drive</a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id657076d561f0'><button class='copyBtn' data-clipboard-target='#id657076d561f0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>assign letter=G</pre> <h2 id="gui"><span class="code">Diskmgmt</span> GUI Shows Progress</h2> <p> Having a GUI continuously report the current state of the drive maintenance you are performing using a command-line interface is a good practice. </p> <p> The GUI-based Windows disk manager, <code>diskmgmt.msc</code>, can show the instantaneous progress of all <code>diskpart</code> commands, working on every drive, including creating and deleting partitions, volumes, formatting and much more. For example, formatting progress can be seen here: </p> <div style=""> <picture> <source srcset="/blog/images/diskpart/diskUI.webp" type="image/webp"> <source srcset="/blog/images/diskpart/diskUI.png" type="image/png"> <img src="/blog/images/diskpart/diskUI.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <p> Run <code>diskmgmt.msc</code> by: </p> <ol> <li>Press the <kbd>Windows</kbd> key once and let go.</li> <li>Type <code>diskmgmt</code></li> <li>Press <kbd>Enter</kbd></li> </ol> <h2 id="exit"><span class="code">Exit</span></h2> <p> To exit <code>diskpart</code>, either type <code>Exit</code> or press <kbd>Ctrl</kbd>-<kbd>C</kbd>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida9413bc631f0'><button class='copyBtn' data-clipboard-target='#ida9413bc631f0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>exit</pre> <p> The drive is ready for its next assignment! </p> WSL / WSL 2 Backup and Restore 2022-01-10T00:00:00-05:00 https://mslinn.github.io/blog/2022/01/10/wsl-backup <editor-fold intro> <p> I needed to back up my WSL2 installation before <a href='https://www.microsoft.com/en-us/software-download/windows10' target='_blank' rel='nofollow'>reinstalling Windows 10</a>, as one must do every 6 months if you work your machine like a developer. It had been years since I had last refreshed this machine, and it was now bluescreening several times a day. Reboots took forever. </p> <div class="pullQuote"> <div class="right" style="font-size: 3em;">&#128513;</div> <p> This story detail the various approaches I attempted; all failed until I was able to <a href="#duplicate">duplicate</a> the original Ubuntu instance using <span class="code">LxRunOffline</span>. </p> </div> <p> I wanted to retain my WSL2 instance. When refreshing Windows you must reinstall all your programs, and you lose all the WSL/WSL2 instances. The refresh decommissions them, then moves them into a hidden directory tree, along with the rest of the stuff stored in your old Windows 10 profile directory tree. </p> <p> I also wanted to be able to replicate my WSL2 instance reliably and easily. It was very important to me to be able to administer WSL instances separately from my Windows 10 profile. </p> <p> Only recently has it become possible to work with WSL/WSL2 images on non-system drives. Most online documentation is now out of date in this regard. Running Windows off a different drive than the drive that a WSL/WSL2 client OS runs from can provide a dramatic performance boost for some applications. </p> <p> I use Dell laptops, because I love their onsite warranty price and features. However, sometimes it feels like Dell's solution to every problem is to replace the motherboard. That means Windows 10 feels a bit uncomfortable, and it wonders if it has been pirated. Refreshing Windows solves that problem, and blows away the standard WSL/WSL2 image. Again. And again. And again! Dell replaced the motherboard 4 times in 2 years for one of my laptops. </p> <p> Having the WSL/WSL2 image elsewhere in the filesystem means that reinstalling Windows is not as traumatic. </p> <p> Backing up WSL/WSL2 can be problematic. Restoring it is even more delicate. Knowing this, I decided to backup and test the restoration process before refreshing Windows and potentially losing my working system. </p> <p> I have tried several times before to make this work. As I said, it had been years since I allowed the OS in my main workstation to be refreshed. The reason I resisted doing proper maintenance was because I wanted to preserve my WSL2 Ubuntu image. Until today, I met with frustrating failures every time I attempted to use the standard tools. Today I succeeded, via new software, hence the publication of my notes. Who wants to read stories about all the things that do not work? </p> <p> Following is my experience. I began by following the directions in <a href='https://www.windowscentral.com/how-backup-windows-subsystem-linux-wsl-distribution' target='_blank' rel='nofollow'>How to back up a Windows Subsystem for Linux (WSL) distribution</a>, published 18 Feb 2021 by Windows Central. It is a nice story, but I've tried this stuff on a variety of computers, and this is not Microsoft's most robust code base as yet. Read on and I will tell you of reality as I found it. </p> </editor-fold> <editor-fold wslOptions> <h2 id="wslOptions">WSL Command-Line Options</h2> <p> First let's look at the command-line options for the <code>wsl</code> command. </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id26c1b0225d7b'><button class='copyBtn' data-clipboard-target='#id26c1b0225d7b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>Microsoft Windows [Version 10.0.19044.1415] (c) Microsoft Corporation. All rights reserved. </span> <span class='unselectable'>C:\Users\Mike Slinn> </span>wsl -? <span class='unselectable'>Invalid command line option: -? Copyright (c) Microsoft Corporation. All rights reserved. Usage: wsl.exe [Argument] [Options...] [CommandLine] Arguments for running Linux binaries: If no command line is provided, wsl.exe launches the default shell. --exec, -e &lt;CommandLine> Execute the specified command without using the default Linux shell. -- Pass the remaining command line as is. Options: --cd &lt;Directory> Sets the specified directory as the current working directory. If ~ is used the Linux user&rsquo;s home path will be used. If the path begins with a / character, it will be interpreted as an absolute Linux path. Otherwise, the value must be an absolute Windows path. --distribution, -d &lt;Distro> Run the specified distribution. --user, -u &lt;UserName> Run as the specified user. Arguments for managing Windows Subsystem for Linux: --help Display usage information. --install [Options] Install additional Windows Subsystem for Linux distributions. For a list of valid distributions, use &quot;wsl --list --online&quot;. Options: --distribution, -d [Argument] Downloads and installs a distribution by name. Arguments: A valid distribution name (not case sensitive). Examples: wsl --install -d Ubuntu wsl --install --distribution Debian --set-default-version &lt;Version> Changes the default install version for new distributions. --shutdown Immediately terminates all running distributions and the WSL 2 lightweight utility virtual machine. --status Show the status of Windows Subsystem for Linux. --update [Options] If no options are specified, the WSL 2 kernel will be updated to the latest version. Options: --rollback Revert to the previous version of the WSL 2 kernel. Arguments for managing distributions in Windows Subsystem for Linux: --export &lt;Distro> &lt;FileName> Exports the distribution to a tar file. The filename can be - for standard output. --import &lt;Distro> &lt;InstallLocation> &lt;FileName> [Options] Imports the specified tar file as a new distribution. The filename can be - for standard input. Options: --version &lt;Version> Specifies the version to use for the new distribution. --list, -l [Options] Lists distributions. Options: --all List all distributions, including distributions that are currently being installed or uninstalled. --running List only distributions that are currently running. --quiet, -q Only show distribution names. --verbose, -v Show detailed information about all distributions. --online, -o Displays a list of available distributions for install with 'wsl --install'. --set-default, -s &lt;Distro> Sets the distribution as the default. --set-version &lt;Distro> &lt;Version> Changes the version of the specified distribution. --terminate, -t &lt;Distro> Terminates the specified distribution. --unregister &lt;Distro> Unregisters the distribution and deletes the root filesystem. </span></pre> </editor-fold> <editor-fold wsBackup> <h2 id="wslBackup">Backing Up With WSL</h2> <p> Now let's use the <code>wsl</code> command to back up the Ubuntu VM in WSL2. </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7f8158360ff3'><button class='copyBtn' data-clipboard-target='#id7f8158360ff3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --export Ubuntu ubuntuBear_2021-01-10.tar</pre> <p> Well, well, it backed up without any problem in about an hour! Color me <s>surprised</s>happy. File size was about 50 GB. </p> <p> Alright, let's try importing the image now. We'll locate the new image at <code>f:\ubuntuBear</code>. It used to be that WSL did not support moving or installing a distro to non-system drives. No longer! </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4f4517f47753'><button class='copyBtn' data-clipboard-target='#id4f4517f47753' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --import Ubuntu f:\ubuntuBear ubuntuBear_2021-01-10.tar <span class='unselectable'>A distribution with the supplied name already exists. </span></pre> <p> The error message &ldquo;A distribution with the supplied name already exists&rdquo; makes sense because I backed up a WSL2 instance called <code>Ubuntu</code> and instance names must be unique. Let's import under the name <code>UbuntuBear</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf57a1c57aabc'><button class='copyBtn' data-clipboard-target='#idf57a1c57aabc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --import UbuntuBear f:\ubuntuBear ubuntuBear_2021-01-10.tar <span class='unselectable'>Unspecified error </span></pre> <p> Ahh, the dreaded <code>Unspecified error</code> that <code>wsl import</code> is infamous for. <a href='https://github.com/microsoft/WSL/issues/4735#issuecomment-800103777' target='_blank' rel='nofollow'>Google brought me</a> to a potential solution, <a href='https://github.com/DDoSolitary/LxRunOffline' target='_blank' rel='nofollow'><code>LxRunOffline</code></a>. </p> </editor-fold> <editor-fold lxRunOffInstall> <h2 id="LxRunOfflineInstall">Installing <span class="code">LxRunOffline</span></h2> <p> I downloaded the binaries in <a href='https://github.com/DDoSolitary/LxRunOffline/releases/download/v3.5.0/LxRunOffline-v3.5.0-msvc.zip' target='_blank' rel='nofollow'><code>LxRunOffline-v3.5.0-msvc.zip</code></a> directly from <a href='https://github.com/DDoSolitary/LxRunOffline/releases' target='_blank' rel='nofollow'>GitHub</a> into <code>C:\Program Files\LxRunOffline</code>. </p> <p> Now add <code>C:\Program Files\LxRunOffline</code> to the Windows <code>PATH</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id111eca34b410'><button class='copyBtn' data-clipboard-target='#id111eca34b410' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>setx PATH "%PATH%;C:\Program Files\LxRunOffline" SUCCESS: Specified value was saved. %}</pre> <p> I then ran the following in an administrative shell to register the DLL: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc4cb9b91af14'><button class='copyBtn' data-clipboard-target='#idc4cb9b91af14' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>regsvr32 "C:\Program Files\LxRunOffline\LxRunOfflineShellExt.dll"</pre> </editor-fold> <editor-fold LxRunOffline> <h2 id="LxRunOffline" class="code">LxRunOffline Help Info</h2> <p> Let's progressively discover how this command-line program can be used. First I'll just type the command name, which causes the program to list its top-level actions. </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id03b48196665a'><button class='copyBtn' data-clipboard-target='#id03b48196665a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline <span class='unselectable'>[ERROR] No action is specified. Supported actions are: l, list List all installed distributions. gd, get-default Get the default distribution, which is used by bash.exe. sd, set-default Set the default distribution, which is used by bash.exe. i, install Install a new distribution. ui, uninstall Uninstall a distribution. rg, register Register an existing installation directory. ur, unregister Unregister a distribution but not delete the installation directory. m, move Move a distribution to a new directory. d, duplicate Duplicate an existing distribution in a new directory. e, export Export a distribution&quot;s filesystem to a .tar.gz file, which can be imported by the "install" command. r, run Run a command in a distribution. di, get-dir Get the installation directory of a distribution. gv, get-version Get the filesystem version of a distribution. ge, get-env Get the default environment variables of a distribution. se, set-env Set the default environment variables of a distribution. ae, add-env Add to the default environment variables of a distribution. re, remove-env Remove from the default environment variables of a distribution. gu, get-uid Get the UID of the default user of a distribution. su, set-uid Set the UID of the default user of a distribution. gk, get-kernelcmd Get the default kernel command line of a distribution. sk, set-kernelcmd Set the default kernel command line of a distribution. gf, get-flags Get some flags of a distribution. See https://docs.microsoft.com/en-us/previous-versions/windows/desktop/api/wslapi/ne-wslapi-wsl_distribution_flags for details. sf, set-flags Set some flags of a distribution. See https://docs.microsoft.com/en-us/previous-versions/windows/desktop/api/wslapi/ne-wslapi-wsl_distribution_flags for details. s, shortcut Create a shortcut to launch a distribution. ec, export-config Export configuration of a distribution to an XML file. ic, import-config Import configuration of a distribution from an XML file. sm, summary Get general information of a distribution. version Get version information about this LxRunOffline.exe. </span></pre> <p> Alright, let's get information about the <code>i</code> (<code>install</code>) option. </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id821c6e41ab2a'><button class='copyBtn' data-clipboard-target='#id821c6e41ab2a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline i <span class='unselectable'>[ERROR] the option '-d' is required but missing Options: -n arg Name of the distribution -d arg The directory to install the distribution into. -f arg The tar file containing the root filesystem of the distribution to be installed. If a file of the same name with a .xml extension exists and "-c" isn&quot;t specified, that file will be imported as a config file. -r arg The directory in the tar file to extract. This argument is optional. -c arg The config file to use. This argument is optional. -v arg (=2) The version of filesystem to use, latest available one if not specified. -s Create a shortcut for this distribution on Desktop. </span></pre> </editor-fold> <p> I'll use the <code>-d</code> option to specify the directory to install into, as well as the <code>-f</code> and <code>-n</code> options. </p> <editor-fold LxRunOfflineImport> <h2 id="LxRunOfflineImport"><span class="code">LxRunOffline</span> Import</h2> <p> I feel brave, let's try importing for real now: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3e61dee6a46f'><button class='copyBtn' data-clipboard-target='#id3e61dee6a46f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline i -d f:/ubuntuBear -n UbuntuBear -f ubuntuBear_2021-01-10.tar <span class='unselectable'>[ERROR] The distro "UbuntuBear" already exists. </span></pre> <p> Some debris remains from the failed import (remember the <code>Unspecified error</code> a moment ago?). I will delete the b0rked <code>UbuntuBear</code> instance in a moment. Let's call this newly cloned linux instance <code>UbuntuBear2</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd599aa22bbc5'><button class='copyBtn' data-clipboard-target='#idd599aa22bbc5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline i -d f:/ubuntuBear -n UbuntuBear2 -f ubuntuBear_2021-01-10.tar <span class='unselectable'>[WARNING] Ignoring an unsupported file "var/lib/docker/volumes/backingFsBlockDev" of type 0060000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-17041-8893592-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10716-846285-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1899-75308257-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10151-8749190-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-18789-129803847-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-811-127664322-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12447-115564106-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-24480-65839207-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-23230-107496852-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-23230-107496852-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-24480-65839207-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-14211-5836039-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19778-110293600-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1969-34761241-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-17041-8893592-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1899-75308257-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-7642-17996-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10151-8749190-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19082-8772885-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12857-5829248-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-26692-70009588-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12447-115564106-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-22582-1181861-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-14211-5836039-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1969-34761241-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-7642-17996-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10716-846285-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-11424-57667328-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-28497-49814437-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-18789-129803847-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-2234-80045-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19082-8772885-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12857-5829248-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-22582-1181861-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-26692-70009588-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10990-5814132-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-811-127664322-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10990-5814132-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-11424-57667328-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-2234-80045-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19778-110293600-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-28497-49814437-in" of type 0010000. [WARNING] Ignoring an unsupported file "dev/random" of type 0020000. [WARNING] Ignoring an unsupported file "dev/tty" of type 0020000. [WARNING] Ignoring an unsupported file "dev/full" of type 0020000. [WARNING] Ignoring an unsupported file "dev/urandom" of type 0020000. [WARNING] Ignoring an unsupported file "dev/ptmx" of type 0020000. [WARNING] Ignoring an unsupported file "dev/zero" of type 0020000. [WARNING] Ignoring an unsupported file "dev/console" of type 0020000. [WARNING] Ignoring an unsupported file "dev/null" of type 0020000. [WARNING] Ignoring an unsupported file "dev/mapper/control" of type 0020000. [ERROR] Couldn&quot;t create the file "\\?\f:\ubuntuBear\rootfs\home\mslinn\.atom\packages\markdown-preview-plus\spec\fixtures\subdir\�cc�nt�d.md". Reason: The file exists. </span></pre> <p> Most of the warnings do not seem to be important. I do not care about Docker anyway. Also, now I know that temporary files and <code>dev</code> nodes should all be deleted before running this program. Even better, the program that created the tar should be modified to not attempt to replicate any temporary files. </p> <p> I don't understand the problem with the atom package, but I'll try to delete all of the atom settings from the tar, along with the other problematic directories, and then retry. </p> </editor-fold> <editor-fold cleanup> <h2 id="cleanup">Cleaning Up WSL</h2> <p> First let's see the VMs registered with WSL: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc253d6f68bf3'><button class='copyBtn' data-clipboard-target='#idc253d6f68bf3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl -l <span class='unselectable'>Windows Subsystem for Linux Distributions: Ubuntu (Default) UbuntuBear UbuntuBear2 </span></pre> <p> Let's delete the debris remaining from the failed <code>UbuntuBear</code> and <code>UbuntuBear2</code> imports: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id29f147c1ae7c'><button class='copyBtn' data-clipboard-target='#id29f147c1ae7c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --unregister UbuntuBear <span class='unselectable'>Unregistering... </span> <span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --unregister UbuntuBear2 <span class='unselectable'>Unregistering... </span></pre> <p> Attempting to delete the directory created by LxRunOffline hit a corrupted directory. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id143c3f104a3d'><button class='copyBtn' data-clipboard-target='#id143c3f104a3d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>del /s /q f:\ubuntuBear <span class='unselectable'>Deleted file - f:\ubuntuBear\rootfs\init Deleted file - f:\ubuntuBear\rootfs\lib Deleted file - f:\ubuntuBear\rootfs\lib32 Deleted file - f:\ubuntuBear\rootfs\lib64 Deleted file - f:\ubuntuBear\rootfs\libx32 Deleted file - f:\ubuntuBear\rootfs\sbin Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.wget-hsts Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.Xauthority Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zcompdump Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zcompdump-Bear-5.8 Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zcompdump-localhost-5.8 Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zshenv Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zshrc Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zshrc.pre-oh-my-zsh Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zsh_history Deleted file - f:\ubuntuBear\rootfs\home\mslinn\ancient_warmth_workspace.code-workspace Deleted file - f:\ubuntuBear\rootfs\home\mslinn\bear2.zip Deleted file - f:\ubuntuBear\rootfs\home\mslinn\bear3.zip Deleted file - f:\ubuntuBear\rootfs\home\mslinn\bearDirs.tar Deleted file - f:\ubuntuBear\rootfs\home\mslinn\dead.letter Deleted file - f:\ubuntuBear\rootfs\home\mslinn\django_bash_completion Deleted file - f:\ubuntuBear\rootfs\home\mslinn\jekyll_workspace.code-workspace Deleted file - f:\ubuntuBear\rootfs\home\mslinn\msp.txt Deleted file - f:\ubuntuBear\rootfs\home\mslinn\nodesource_setup.sh Deleted file - f:\ubuntuBear\rootfs\home\mslinn\package-lock.json Deleted file - f:\ubuntuBear\rootfs\home\mslinn\package.json Deleted file - f:\ubuntuBear\rootfs\home\mslinn\worldPeaceMusicCollective.png Deleted file - f:\ubuntuBear\rootfs\home\mslinn\worldPeaceMusicCollectiveBordered.png Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.vscode-server\extensions\wix.vscode-import-cost-2.15.0\package.json Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.vscode-server\extensions\wix.vscode-import-cost-2.15.0\pom.xml Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.vscode-server\extensions\wix.vscode-import-cost-2.15.0\README.md The file or directory is corrupted and unreadable. </span></pre> <p> &ldquo;The file or directory is corrupted and unreadable.&rdquo; That needs to be dealt with right away! I fixed the directory errors in drive F like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddfca3878aca9'><button class='copyBtn' data-clipboard-target='#iddfca3878aca9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Program Files> </span>chkdsk /F F: <span class='unselectable'>The type of the file system is NTFS. Chkdsk cannot run because the volume is in use by another process. Chkdsk may run if this volume is dismounted first. ALL OPENED HANDLES TO THIS VOLUME WOULD THEN BE INVALID. </span>y <span class='unselectable'>Chkdsk cannot dismount the volume because it is a system drive or there is an active paging file on it. Would you like to schedule this volume to be checked the next time the system restarts? (Y/N)</span> y <span class='unselectable'>This volume will be checked the next time the system restarts. </span></pre> <p> I rebooted the system, and it fixed the errors on drive F:. </p> </editor-fold> <editor-fold prune> <h2 id="prune">Pruning the tar</h2> <p> First lets back up the tar that we want to prune. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb0655b2e833f'><button class='copyBtn' data-clipboard-target='#idb0655b2e833f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pushd '/mnt/c/Users/Mike Slinn/' <span class='unselectable'>$ </span>ls -alF *.tar <span class='unselectable'>-rwxr--r-- 1 mslinn mslinn 524482560 Jan 10 18:45 'bear_ubuntu_2021-01-10.tar'* -rwxr--r-- 1 mslinn mslinn 51464028160 Jan 10 21:11 'ubuntuBear_2021-01-10.tar'* </span> <span class='unselectable'>$ </span>cp ubuntuBear_2021-01-10.tar ubuntuBearPruned_2021-01-10.tar</pre> <p> Now lets try to prune out all the problematic, and unnecessary, files. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id721ff351f91b'><button class='copyBtn' data-clipboard-target='#id721ff351f91b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>tar -f ubuntuBearPruned_2021-01-10.tar \ --delete dev/* \ --delete tmp/* \ --delete home/mslinn/.atom/* \ --delete var/lib/docker/* \ --delete var/sitesUbuntu/* \ --delete var/work/* \ --delete var/tmp/* <span class='unselectable'>tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.fuseoverlayfs.opaque' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.fuseoverlayfs.opaque' tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: dev/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: tmp/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: home/mslinn/.atom/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: var/lib/docker/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: var/sitesUbuntu/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: var/work/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: var/tmp/*: Not found in archive tar: Exiting with failure status due to previous errors </span></pre> <p> I think the error messages just indicate that the tar was made by <a href='https://www.freebsd.org/cgi/man.cgi?tar(1)' target='_blank' rel='nofollow'>BSD-TAR</a>, while Ubuntu uses <a href='https://www.gnu.org/software/tar/' target='_blank' rel='nofollow'>GNU-TAR</a>. I installed <code>bsdtar</code> like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb61f53b06a7e'><button class='copyBtn' data-clipboard-target='#idb61f53b06a7e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt-get install <a href='https://packages.ubuntu.com/hirsute/libarchive-tools' target='_blank' rel='nofollow'>libarchive-tools</a> <span class='unselectable'>Reading package lists... Done Building dependency tree... Done Reading state information... Done The following NEW packages will be installed: libarchive-tools 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 57.1 kB of archives. After this operation, 207 kB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu hirsute/universe amd64 libarchive-tools amd64 3.4.3-2 [57.1 kB] Fetched 57.1 kB in 0s (167 kB/s) Selecting previously unselected package libarchive-tools. (Reading database ... 244498 files and directories currently installed.) Preparing to unpack .../libarchive-tools_3.4.3-2_amd64.deb ... Unpacking libarchive-tools (3.4.3-2) ... Setting up libarchive-tools (3.4.3-2) ... Processing triggers for man-db (2.9.4-2) ... </span></pre> <p> I <a href='https://stackoverflow.com/a/56031400/553865' target='_blank' rel='nofollow'>tried again</a> using <code>bsdtar</code>, which does not support GNU tar's <code>--delete</code> option. Unfortunately, I only got a 1KB file out, no matter what I did. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id87a06a441f1b'><button class='copyBtn' data-clipboard-target='#id87a06a441f1b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>bsdtar -cvf ubuntuBearPruned2_2021-01-10.tar \ --exclude 'dev/*' \ --exclude 'tmp/*' \ --exclude 'home/mslinn/.atom/*' \ --exclude 'var/lib/docker/*' \ --exclude 'var/sitesUbuntu/*' \ --exclude 'var/work/*' \ --exclude 'var/tmp/*' \ @ubuntuBear_2021-01-10.tar</pre> <p> Maybe there is a bug. Maybe there is bad documentation. I do not think I made an error. Life is short. Bugs are rampant. <a href='https://en.wikipedia.org/wiki/Illegitimi_non_carborundum' target='_blank' rel='nofollow'>Illegitimi non carborundum!</a> </p> <p> I give up on this direction. Let's try to win some other way! </p> </editor-fold> <editor-fold duplicate> <h2 id="duplicate">Success Duplicating the Ubuntu Instance</h2> <p> Let&rsquo;s try duplicating the Ubuntu instance with <code>LxRunOffline</code>. First the <code>Ubuntu</code> VM must be terminated. </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3c7729689862'><button class='copyBtn' data-clipboard-target='#id3c7729689862' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl -t Ubuntu <span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline duplicate -n Ubuntu -N UbuntuWsl2 -d f:\UbuntuWsl2</pre> <p> The above ran for a couple of hours and concluded without error. <code>LxRunOffline duplicate</code> automatically registers the new instance. </p> <p> The <code>F:\UbuntuWsl2</code>directory was 239 GB. That's a fair-sized VM. The directory only contains one file, called <code>ext4.vhdx</code>. </p> <p> Let's see the Ubuntu instances that are currently set: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc018e1ea496e'><button class='copyBtn' data-clipboard-target='#idc018e1ea496e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --list <span class='unselectable'>Windows Subsystem for Linux Distributions: Ubuntu (Default) UbuntuWsl2 </span></pre> <div class="right" style="font-size: 3em;">&#128513;</div> <p> That is what I wanted to see! I started the new <code>UbuntuWsl2</code> Ubuntu instance like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfee395f54627'><button class='copyBtn' data-clipboard-target='#idfee395f54627' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl -d UbuntuWsl2</pre> <p> To set the default Ubuntu instance I typed: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id181f21cc0a86'><button class='copyBtn' data-clipboard-target='#id181f21cc0a86' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --setdefault UbuntuWsl2</pre> <p> Let's verify that worked: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd2ca40764fb3'><button class='copyBtn' data-clipboard-target='#idd2ca40764fb3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --list <span class='unselectable'>Windows Subsystem for Linux Distributions: UbuntuWsl2 (Default) Ubuntu </span></pre> <p> I then deleted the huge tar files that I had created in earlier steps. I never needed them because I used <code>LxRunOffline duplicate</code>. </p> </editor-fold> <editor-fold USB3 SSD> <h2 id="usb3">Ubuntu on 500GB USB3 SSD</h2> <div style="text-align: right;"> <a href="https://www.amazon.com/gp/product/B08GTXVG9P" target="_blank" ><picture> <source srcset="/blog/images/wslBackup/sandiskExtreme.webp" type="image/webp"> <source srcset="/blog/images/wslBackup/sandiskExtreme.png" type="image/png"> <img src="/blog/images/wslBackup/sandiskExtreme.png" class="right liImg2 rounded shadow" /> </picture></a> </div> <p> I have a <a href='https://www.amazon.com/gp/product/B08GTXVG9P' target='_blank' rel='nofollow'>Sandisk Extreme 500GB USB3 SSD drive</a>. This tiny, light, and very portable storage device should be perfect for holding my Ubuntu development system. How cool it would be to be able to drop it into a small pocket! </p> <p> Better yet, the <a href='https://www.quora.com/Can-a-USB-3-0-external-SSD-be-faster-than-an-internal-laptop%E2%80%99s-HDD' target='_blank' rel='nofollow'>performance of portable USB3 SSD drives</a> blows away traditional hard drives. </p> <p> For example, assume that your SSD drive appears as drive <code>X</code>. Also assume that <code>LxRunOffline.exe</code> and <code>LxRunOffline.dll</code> have been copied to the root directory of the SSD drive. To register your portable Ubuntu you would simply type: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf78048cfaa56'><button class='copyBtn' data-clipboard-target='#idf78048cfaa56' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\> </span>X:LxRunOffline register X:\MyUbuntu -n UbuntuMSlinn</pre> <div class="pullQuote"> You can walk up to any recently updated Windows 10 machine, plug your tiny USB3 SSD drive into a USB3 or USB3.1 port, run the registration command, and boom! you are productive with great storage performance. </div> <div class="right" style="font-size: 3em;">&#128513;</div> <h2 id="duplicate">Duplicating to SSD</h2> <p> Again, the <code>Ubuntu</code> VM must be terminated. </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide79a719854db'><button class='copyBtn' data-clipboard-target='#ide79a719854db' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl -t Ubuntu <span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline duplicate -n Ubuntu -N UbuntuWsl2Extreme -d I:\UbuntuWsl2Extreme</pre> <p> The above ran for a couple of hours and concluded without error. <code>LxRunOffline duplicate</code> automatically registers the new instance. </p> <p> Let's see the Ubuntu instances that are currently set: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id20a019ea895b'><button class='copyBtn' data-clipboard-target='#id20a019ea895b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --list <span class='unselectable'>Windows Subsystem for Linux Distributions: Ubuntu (Default) UbuntuWsl2 UbuntuWsl2Extreme </span></pre> <div class="right" style="font-size: 3em;">&#128513;</div> <h2 id="startStop">Starting and Stopping Portable VMs</h2> <p> You can start the WSL VM called <code>UbuntuMSlinn</code> by using <code>wsl -d</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3f8aed16914f'><button class='copyBtn' data-clipboard-target='#id3f8aed16914f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\> </span>wsl -d UbuntuMSlinn</pre> <p> You can stop it by using <code>wsl -t</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc4b90597d7e9'><button class='copyBtn' data-clipboard-target='#idc4b90597d7e9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\> </span>wsl -t UbuntuMSlinn</pre> </editor-fold> <editor-fold issues> <h2 id="issues">Microsoft WSL Issue Reporting</h2> <p> In the course of writing this blog post I found the <a href='https://github.com/microsoft/WSL/blob/master/CONTRIBUTING.md#8-collect-wsl-logs' target='_blank' rel='nofollow'>web page</a> for reporting a WSL issue, so that Microsoft can look at it. </editor-fold> Streaming Solo to Facebook From OBS Studio 2022-01-07T00:00:00-05:00 https://mslinn.github.io/blog/2022/01/07/streaming-facebook <p> Briefly: </p> <ol> <li>Set up Facebook for streaming</li> <li>OBS Studio initiates streaming</li> <li>Musician performs</li> <li>Helper monitors stream and moderates comments on Facebook Messenger</li> </ol> <h2 id="pageStream">Streaming From a Facebook Page</h2> <p> I prefer to stream from a specific page on Facebook, <a href='https://www.facebook.com/mslinnmusic' target='_blank' rel='nofollow'><code>facebook.com/mslinnmusic</code></a>, so my live streams can be found at <a href='https://www.facebook.com/mslinnmusic/live_videos/' target='_blank' rel='nofollow'><code>facebook.com/mslinnmusic/live_videos/</code></a>. </p> <h2 id="fbSetup">Setting Up Facebook For Streaming</h2> <p> Go to <a href='https://www.facebook.com/live/producer/' target='_blank' rel='nofollow'>Facebook Live Producer</a> and check out the various areas on screen. We'll change a few settings. </p> <div style=""> <picture> <source srcset="/blog/images/facebookStream/settingsStream.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/settingsStream.png" type="image/png"> <img src="/blog/images/facebookStream/settingsStream.png" class=" liImg2 rounded shadow" style="float: left; margin-right: 1em; width: 300px; height: auto;" /> </picture> </div> <div style=""> <picture> <source srcset="/blog/images/facebookStream/settingsViewing.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/settingsViewing.png" type="image/png"> <img src="/blog/images/facebookStream/settingsViewing.png" class=" liImg2 rounded shadow" style="float: left; margin-right: 1em; width: 300px; height: auto;" /> </picture> </div> <div style=""> <picture> <source srcset="/blog/images/facebookStream/settingsComments.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/settingsComments.png" type="image/png"> <img src="/blog/images/facebookStream/settingsComments.png" class=" liImg2 rounded shadow" style="width: 300px; height: auto;" /> </picture> </div> <div style=""> <picture> <source srcset="/blog/images/facebookStream/keyUrl.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/keyUrl.png" type="image/png"> <img src="/blog/images/facebookStream/keyUrl.png" class=" liImg2 rounded shadow" style="height: auto; width: 615px;" /> </picture> </div> <div style="float: left; margin-right: 1em; max-width: 300px"> <div style=""> <picture> <source srcset="/blog/images/facebookStream/ticketsGoLive.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/ticketsGoLive.png" type="image/png"> <img src="/blog/images/facebookStream/ticketsGoLive.png" class=" liImg2 rounded shadow" style="height: auto; width: 300px;" /> </picture> </div> I pixelated the stream key in the image to the right for security. </div> <div style=""> <picture> <source srcset="/blog/images/facebookStream/keyUrl2.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/keyUrl2.png" type="image/png"> <img src="/blog/images/facebookStream/keyUrl2.png" class=" liImg2 rounded shadow" style="width: 300px; height: auto;" /> </picture> </div> <ol> <li>Enable <b>Use a persistent stream key</b></li> <li>Copy the Facebook stream key to the clipboard.</li> <li>Enable <b>Publish as a test broadcast</b> if desired.</li> </ol> <h2 id="obsSetup">Setting Up OBS Studio For Streaming</h2> <ol> <li> Paste the Facebook persistent stream key into OBS Studio by going to <b>Settings</b> / <b>Stream</b> / <b>Facebook Live</b> / <b>Stream Key</b>.. </li> <li>Press <kbd>OK</kbd>.</li> </ol> <p> Chats from the audience will appear in Facebook Messenger. </p> <h2 id="obsSetup">Streaming</h2> <p> The Facebook streaming key is specific to the computer that is streaming. If you use more than one computer to stream from a given Facebook account, you will need to get a fresh stream key each time you stream from a different computer. </p> <p> To start streaming: </p> <ol> <li>Click on the Facebook Live Video button <div style=""> <picture> <source srcset="/blog/images/facebookStream/fbLiveVideoButton.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/fbLiveVideoButton.png" type="image/png"> <img src="/blog/images/facebookStream/fbLiveVideoButton.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> </li> <li> Wait for a couple of seconds while the Facebook Live Producer page loads any settings that you may have previously set. </li> <li> <div style=""> <picture> <source srcset="/blog/images/facebookStream/topLeft.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/topLeft.png" type="image/png"> <img src="/blog/images/facebookStream/topLeft.png" class=" liImg2 rounded shadow" style="height: auto; width: 300px;" /> </picture> </div> Fill in the title of the stream, and write a short description. </li> <li>Click on the <b>Start Streaming</b> button in OBS Studio.</li> <li> Facebook will show the stream image at the bottom right of the Facebook Live Producer web page. <div style=""> <picture> <source srcset="/blog/images/facebookStream/fbLivePreview.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/fbLivePreview.png" type="image/png"> <img src="/blog/images/facebookStream/fbLivePreview.png" class=" liImg2 rounded shadow" style="height: auto; width: 300px;" /> </picture> </div> Attention: you are not yet streaming! </li> <li> Click on the blue <b>Go live</b> button at the bottom left of the Facebook Live Producer web page. <div style=""> <picture> <source srcset="/blog/images/facebookStream/fbGoLive.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/fbGoLive.png" type="image/png"> <img src="/blog/images/facebookStream/fbGoLive.png" class=" liImg2 rounded shadow" style="height: auto; width: 300px;" /> </picture> </div> You are now live.</li> <li>Check Facebook Messenger for audience feedback.</li> <li>View the stream, again, for me, this is at <a href='https://www.facebook.com/mslinnmusic/live_videos/' target='_blank' rel='nofollow'><code>https://www.facebook.com/mslinnmusic/live_videos/</code></a></li> </ol> OBS Studio Streaming Using Nvidia GTX 1660/1650 GPUs 2021-12-29T00:00:00-05:00 https://mslinn.github.io/blog/2021/12/29/obs-s1650 <p> OBS Studio running on my desktop PC was able to record audio and video at any resolution and frame rate that I desired, but whenever I tried to stream to YouTube, the video did not work. Instead of live video, just the first frame was sent; a still image. </p> <p> To help me diagnose the problem, I ran <code>UserBenchMark</code>, available for free from <a href='https://www.userbenchmark.com' target='_blank' rel='nofollow'><code>userbenchmark.com</code></a>. The benchmark showed that my PC's video card, an old EVGA GeForce GTX 760, was mismatched to the rest of the computer; it was by far the weakest link. The video card was able to drive a 4K monitor and a 1080p monitor simultaneously, for displaying static windows, but it was unsuitable for gaming and streaming live video. </p> <h2 id="recommend">OBS Studio Recommendations</h2> <p> Nvidia video cards are <a href='https://obsproject.com/forum/threads/best-gpu-for-rendering-previews-advice-needed.118869/' target='_blank' rel='nofollow'>strongly recommended by the OBS Studio developers</a>, &ldquo;preferably a Turing-based unit&rdquo;, and that the best value cards were <a href='https://gpu.userbenchmark.com/Nvidia-GTX-1660/Rating/4038' target='_blank' rel='nofollow'>GTX 1660</a> (all flavors) and GTX 1650 Super GPUs. Currently, <a href='https://www.digitaltrends.com/computing/what-to-expect-from-gpus-2022/' target='_blank' rel='nofollow'>video cards are in extremely short supply</a>, so prices for used units are higher than those for new units because new units are often not available. </p> <p> I bought a used EVGA GeForce GTX 1660 video card (model <a href='https://www.evga.com/products/Specs/GPU.aspx?pn=338d9f6e-477c-4d81-9432-64bffa3f9513' target='_blank' rel='nofollow'><code>06G-P4-1067-RX</code></a>) here in Montreal for $550 (CAD) and popped it into my computer. After installing drivers from <a href='https://https://www.nvidia.com/Download/index.aspx' target='_blank' rel='nofollow'><code>https://www.nvidia.com/Download/index.aspx</code></a>, and rebooting, OBS Studio was able to stream live without straining the PC. I also noticed that the Windows system performance was smoother, and more stable than it had been with the older video card. </p> <h2 id="fbStream">Facebook Live Stream</h2> <p> This is the <a href='https://www.facebook.com/100070935496310/videos/1163736614368165' target='_blank' rel='nofollow'>Facebook Live Stream</a> I tested with: </p> <iframe src="https://www.facebook.com/plugins/video.php?height=314&href=https%3A%2F%2Fwww.facebook.com%2Fmslinnmusic%2Fvideos%2F1163736614368165%2F&show_text=true&width=560&t=0" class="shadow rounded" width="560" height="429" style="border:none;overflow:hidden" scrolling="no" frameborder="0" allowfullscreen="true" allow="autoplay; clipboard-write; encrypted-media; picture-in-picture; web-share" allowFullScreen="true"></iframe> <h2 id="oclock">Over/Under- Clocking</h2> <p> There is no need to overclock this video card. </p> <p> Overclocking causes more power to be consumed, which means more heat must be dissipated, which means fans must work harder, which means the computer is louder. </p> <p> The fans on this video card only spin when needed, and they only spin as fast as required when needed. Fans are noisy, so the recording room is quieter when the video card is not working very hard. </p> <p> I want the computer I use for streaming to be as quiet as possible. This video card barely breathes hard as it streams live, so underclocking might be a way to further reduce power usage, and hence reduce the need for fans to spin. </p> AI / ML System Behavior Reflects the Society That Produced It 2021-12-26T00:00:00-05:00 https://mslinn.github.io/blog/2021/12/26/ai-society <p> Artificial intelligence systems, including machine learning systems, are primarily software-driven. Like all software, AI and ML systems reflect the societies that produced them. <a href='https://www.thoughtworks.com/insights/articles/demystifying-conways-law' target='_blank' rel='nofollow'>Conway&rsquo;s Law</a> explains how the internal organization of software reflects the organization that created it. </p> <p> I postulate that the behavior of AI & ML systems reflect the society that the systems are embedded in. Do you find the AI systems are <a href='https://www.bruegel.org/blog-post/dark-side-artificial-intelligence-manipulation-human-behaviour' target='_blank' rel='nofollow'>instrusive or exploitive</a>? Look in the collective mirror for the society it was created by. </p> <p class="alert rounded shadow"> If the populace is viewed as merely something for corporations to exploit, and this view is accepted at enough levels in society, that becomes the status quo. In my opinion, the USA in 2022 has definitely reached that point. </p> <div class="quoteCite shadow rounded"> <h2 style="margin-top: 0">Conway&rsquo;s Law</h2> Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure. <br><br> &nbsp;&ndash; Melvin E. Conway, 1967 </div> <div class="quoteCite shadow rounded"> <h2 style="margin-top: 0">Allan Kelly’s Corollary</h2> Organisational design is system design. <br><br> &nbsp;&ndash; Allan Kelly, 2005 </div> <p> The <a href='https://www.hbs.edu/ris/Publication%20Files/16-124_7ae90679-0ce6-4d72-9e9d-828872c7af49.pdf' target='_blank' rel='nofollow'>The Mirroring Hypothesis: Theory, Evidence and Exceptions</a> by Lyra J. Colfer and Carliss Y. Baldwin was published by the Harvard Business School in 2016. </p> <div class="quoteCite shadow rounded"> A large-scale system is one that stretches across time and space. A large organization similarly stretches across time and space. Conway’s Law ties the two. How we organize defines how we think collectively, and thus what we make collectively...<br><br> ... To build software, first build a community...<br><br> Software is essentially about people, and the larger-scale we make it, the more that truth becomes visible. One program reflects how one person thinks. A large-scale application reflects how many people think together. <br><br> &nbsp;&ndash; Pieter Hintjes, <a href='http://hintjens.com/blog:73' target='_blank' rel='nofollow'>Sex in Title, and Other Stories</a>, Dec 16, 2013 </div> <p> Do you even care? If so, please be the change you want to see in the world. </p> <div class='quote'> <div style="text-align: right;"> <picture> <source srcset="/blog/images/gandhi.webp" type="image/webp"> <source srcset="/blog/images/gandhi.png" type="image/png"> <img src="/blog/images/gandhi.png" class="right quartersize liImg2 rounded shadow" /> </picture> </div> We but mirror the world. All the tendencies present in the outer world are to be found in the world of our body. If we could change ourselves, the tendencies in the world would also change. As a man changes his own nature, so does the attitude of the world change towards him. This is the divine mystery supreme. A wonderful thing it is and the source of our happiness. We need not wait to see what others do. <br><br> <span style='font-style:normal;'> &nbsp;&ndash; From <a href='https://commonground.ca/be-the-change-you-want-to-see-in-the-world/' rel='nofollow' target='_blank'>Mahatma Gandhi</a></span> </div> Spring-Breezifier: Solving COVID-19 With HVAC 2021-12-22T00:00:00-05:00 https://mslinn.github.io/blog/2021/12/22/covid-hvac <p> When the COVID-19 pandemic began, the world was unprepared and no one knew how the virus was transmitted. As the medical profession lurched awkwardly into action, it gave advice that later turned out to be incorrect. As time went by, important details about how the virus is transmitted were revealed, and the previous medical advice had become politicized. The new advice has not yet been generally adopted as policy and is currently unknown to most people. </p> <div class="quoteCite shadow rounded" style="height: 230px;"> <div style=""> <picture> <source srcset="/blog/images/covidHvac/mencken.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/mencken.png" type="image/png"> <img src="/blog/images/covidHvac/mencken.png" class=" rounded" style="float:right; maxwidth; 25%; width: 130px; height: 180px; margin-top: 0;" /> </picture> </div> For every complex problem there is an answer that is clear, simple, and wrong. <br><br> &nbsp;&ndash; <a href='https://www.britannica.com/biography/H-L-Mencken' target='_blank' rel='nofollow'>H. L. Mencken</a> 1880-1956. </div> <h2 id="about">About This Article</h2> <p> This article's intent is to show how the COVID-19 pandemic could be dealt with simply, cheaply, and with a minimum of aggravation by raising awareness of the potential role of <a href='https://www.cdc.gov/infectioncontrol/guidelines/environmental/background/air.html#c3' target='_blank' rel='nofollow'>HVAC</a> (heating, ventilation and air conditioning) equipment. </p> <p> The author is an electrical engineer with no health-related or HVAC-related qualifications. However, he can read and write, ideas often come to mind when presented with new information. </p> <div class="formalNotice rounded shadow"> After initially publishing this blog post, an old friend showed me a YouTube video made by a consulting civil engineer who specializes in this topic. <a href="#youtube2">You can skip to it if you are impatient</a> and want to know in-depth technical specifics. </div> <p> All the new information referenced in this article is generally available, from qualified and reputable sources, and this article just disseminates it. </p> <p> This article concludes with actionable suggestions. </p> <h2 id="trans">COVID-19 Is Primarily Transmitted Via Aerosols</h2> <p> The most significant bit of information about COVID-19, which was not generally accepted at the beginning of the pandemic, is that it is <a href='https://www.nature.com/articles/d41586-021-00251-4' target='_blank' rel='nofollow'>mostly transmitted via aerosols</a>; that is, via small airborne droplets only a few microns in diameter. (One micron is a millionth of a meter, or one twenty-five thousandth of an inch). Aerosol particles can hang in the air for hours and travel hundreds of feet. </p> <p> The Centers for Disease Control and Prevention (CDC) officially recently officially recognized that SARS-CoV-2 (the virus that causes COVID-19) is airborne, meaning it is highly transmissible through the air. </p> <div class="quoteCite shadow rounded"> <div style="text-align: right;"> <picture> <source srcset="/blog/images/covidHvac/cdc.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/cdc.png" type="image/png"> <img src="/blog/images/covidHvac/cdc.png" class="right " style="maxwidth; 25%; width: 130px; height: auto;" /> </picture> </div> Exposure occurs in three principal ways: <ol> <li>inhalation of very fine respiratory droplets and aerosol particles</li> <li> deposition of respiratory droplets and particles on exposed mucous membranes in the mouth, nose, or eye by direct splashes and sprays </li> <li> touching mucous membranes with hands that have been soiled either directly by virus-containing respiratory fluids or indirectly by touching surfaces with virus on them. </li> </ol> <p> People release respiratory fluids during exhalation (e.g., quiet breathing, speaking, singing, exercise, coughing, sneezing) in the form of droplets across a spectrum of sizes. These droplets carry virus and transmit infection. To stay healthy, avoid these droplets. </p> <p> The largest droplets settle out of the air rapidly, within seconds to minutes. The smallest very fine droplets, and aerosol particles formed when these fine droplets rapidly dry, are small enough that they can remain suspended in the air for minutes to hours. </p> &nbsp;&ndash;From the CDC <a href='https://www.cdc.gov/coronavirus/2019-ncov/science/science-briefs/sars-cov-2-transmission.html' target='_blank' rel='nofollow'>Scientific Brief: SARS-CoV-2 Transmission</a>, Updated May 7, 2021. </div> <p> The idea that physical distancing of 6 feet (2 meters) might help keep people healthy is an example of bad science from a study <a href='https://www.businessinsider.com/6-foot-distancing-rule-is-outdated-oxford-mit-new-system-2020-8' target='_blank' rel='nofollow'>80 years ago</a> that was not debunked until recently. </p> <div class="quoteCite shadow rounded"> <div style="text-align: right;"> <picture> <source srcset="/blog/images/covidHvac/epaLogo.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/epaLogo.png" type="image/png"> <img src="/blog/images/covidHvac/epaLogo.png" class="right quartersize " /> </picture> </div> Transmission of COVID-19 from inhalation of virus in the air can occur at distances greater than six feet. Particles from an infected person can move throughout an entire room or indoor space. The particles can also linger in the air after a person has left the room – they can remain airborne for hours in some cases. <br><br> &nbsp;&ndash;From the US Environmental Protection Agency (EPA): <a href='https://www.epa.gov/coronavirus/indoor-air-and-coronavirus-covid-19' target='_blank' rel='nofollow'>Indoor Air and Coronavirus (COVID-19)</a>, web page was updated December 15, 2021. </div> <p> Distance between people is not a significant transmissibility factor. Imagine two people, back to back, one (downwind) very sick with COVID-19, and the other (healthy) upwind, and the wind is blowing at 20 mph (32 km/h). The sick person will not infect the healthy person, unless they exchange positions. Just control the air that is breathed, and all airborne viruses such as COVID-19 will be controled. </p> <div class="quoteCite shadow rounded"> <div style="text-align: right;"> <picture> <source srcset="/blog/images/covidHvac/ucsdMedicine.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/ucsdMedicine.png" type="image/png"> <img src="/blog/images/covidHvac/ucsdMedicine.png" class="right " style="maxwidth; 25%; width: 130px; height: auto;" /> </picture> </div> What we learned during the pandemic is that aerosols were one of the main drivers in spreading the COVID-19 virus and that their importance in the transmission of many other respiratory pathogens has been systematically underappreciated. <br><br> &nbsp;&ndash; Dr. Robert Schooley, Professor in the Department of Medicine at the University of California San Diego (November 22, 2021), quoted from <a href='https://ucsdnews.ucsd.edu/pressrelease/covid-gets-airborne' target='_blank' rel='nofollow'>COVID Gets Airborne &ndash; UC San Diego develops computer model to aid understanding of how viruses travel through the air</a> </div> <h2 id="virus">Controlling Virus Infiltration</h2> <p> We would do well to remember that the COVID-19 virus causes sickness. Without exposure to the virus people would not get sick. The risk that a person might catch the disease is directly related to the number of viruses inhaled per unit of time. For example, the more viruses someone inhales in an hour, the more likely they will get sick. </p> <p> <a href='https://www.statnews.com/2020/04/14/how-much-of-the-coronavirus-does-it-take-to-make-you-sick/' target='_blank' rel='nofollow'>The amount of virus necessary to make a person sick is called the infectious dose.</a> All the countermeasures that have been used (hand washing, physical distancing, masks, curfews, etc.) are intended to reduce the number of viruses people are exposed to per unit time. Pandemic policy treats countermeasures as proxies for virus transmission vectors. By not updating pandemic policy as new information becomes available, we loose the ability to control the spread of the virus. </p> <h2 id="hepa">HEPA Filters</h2> <p> HEPA filters on HVAC equipment are designed to remove these aerosols. They come in a huge variety of sizes, shapes and air flow capacities. If you could just breathe the pure air emitted from a fan that had a HEPA filter, you would not get sick from any airborne virus. Any number of people could gather safely, if each of them received an adequate supply of pure air in their face at all times. </p> <div class="quoteCite shadow rounded"> <div style="text-align: right;"> <picture> <source srcset="/blog/images/covidHvac/epaLogo.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/epaLogo.png" type="image/png"> <img src="/blog/images/covidHvac/epaLogo.png" class="right quartersize " /> </picture> </div> HEPA is a type of pleated mechanical air filter. It is an acronym for “high efficiency particulate air filter” (as officially defined by the U.S. Dept. of Energy). This type of air filter can theoretically remove at least 99.97% of dust, pollen, mold, bacteria, and any airborne particles with a size of 0.3 microns (µm). <br><br> &nbsp;&ndash;<a href='https://www.epa.gov/indoor-air-quality-iaq/what-hepa-filter-1' target='_blank' rel='nofollow'>US Environment Protection Agency</a> </div> <p> HEPA filters are inexpensive and are readily available. They are not new. In fact, <a href='https://www.apcfilters.com/the-history-of-hepa-filters/' target='_blank' rel='nofollow'>HEPA filter technology was created in the 1940s</a> by the US Army Chemical Corps and National Defense Research Committee as part of the Manhattan Project. HEPA technology was declassified after World War II and became available for commercial and personal use. HEPA filters play a key role in the research and development of modern pharmaceuticals, aerospace engineering, and computer chip manufacturing. </p> <div class="quoteCite shadow rounded"> HEPA air cleaners, which remain little-used in Canadian hospitals, are a cheap and easy way to reduce risk from airborne pathogens. <br><br> &nbsp;&ndash;From Nature Magazine, October 6, 2021: <a href='https://www.nature.com/articles/d41586-021-02669-2' target='_blank' rel='nofollow'>Real-world data show that filters clean COVID-causing virus from air &ndash; An inexpensive type of portable filter efficiently screened SARS-CoV-2 and other disease-causing organisms from hospital air.</a> </div> <p> It is easy to attach a HEPA filter to a fan, or to replace an existing HVAC filter with a HEPA filter. Here is a very <a href='https://www.jefftk.com/p/ceiling-air-purifier' target='_blank' rel='nofollow'>scientific approach to a home-made solution</a>. </p> <div style=""> <a href="https://youtu.be/kH5APw_SLUU?t=106" target="_blank" ><picture> <source srcset="/blog/images/covidHvac/hepaFan.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/hepaFan.png" type="image/png"> <img src="/blog/images/covidHvac/hepaFan.png" title="Dr. Jeffrey E. Terrell, director of the Michigan Sinus Center, demonstrates how to build an air purifier with a HEPA filter for about $25 with parts from your local hardware store." class=" fullsize liImg2 rounded shadow" alt="Dr. Jeffrey E. Terrell, director of the Michigan Sinus Center, demonstrates how to build an air purifier with a HEPA filter for about $25 with parts from your local hardware store." /> </picture></a> <figcaption class="fullsize" style="width: 100%; text-align: center;"> <a href="https://youtu.be/kH5APw_SLUU?t=106" target="_blank" > Dr. Jeffrey E. Terrell, director of the Michigan Sinus Center, demonstrates how to build an air purifier with a HEPA filter for about $25 with parts from your local hardware store. </a> </figcaption> </figure> </div> <p> One of the comments on the above video, from Rick Rude in 2014, was: </p> <div class="quoteCite shadow rounded"> Better to pull air thru the filter, if you push air through the filter dust that builds up will get blown off and go back into the room. Also, if the fan is pulling air through, the filter will stick to the fan and won't need as much tape to hold it on. Always handle used filters with care because handling them rough will release concentrated dust back into your air. If possible, always take air cleaners or vacuum cleaners outside to change the filters or bags. </div> <h2 id="youtube2">How to Identify and Rectify Poorly Ventilated Indoor Spaces Using Engineering Controls</h2> <p> <a href='https://www.linkedin.com/in/elfstrom/' target='_blank' rel='nofollow'>David Elfstrom, P. Eng.</a>, is a independent civil engineer based in Simcoe, Ontario who consults on the efficient use of energy in buildings. During the pandemic he has been drawing attention to the importance of ventilation, filtration, and overall indoor environmental quality. Earlier this year Mr. Elfstrom completed a ventilation and air pathways assessment of an apartment building in outbreak for a public health unit. This presentation was part of Passive Buildings Canada's 2021 Annual General Meeting. </p> <style>.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }</style><div class='embed-container'> <iframe title="YouTube video player" width="640" height="390" src="//www.youtube.com/embed/9J96F32tv1s" frameborder="0" allowfullscreen></iframe></div> <p style="margin-top: 1em;"> Mr. Elfstrom also co-authored the <a href='https://docs.google.com/document/d/17tKk8Da8tnchtnp9ZRe7fPazGAmXtvoA-n4GZcY0_fQ/edit' target='_blank' rel='nofollow'>Masks4Canada Room Ventilation/Filtration Guide and Tip Sheet</a>. </p> <h2 id="hvac">N95, KF94, FFP2, and 9152 Masks and Their Cousins</h2> <p> If you need to move about in a crowd, or enter a space that contained people a few hours ago, you need to wear a properly fitting face mask that filters out the tiny aerosol particles that contain the COVID-19 virus. <a href='https://www.travelawaits.com/2559161/n95-vs-kn95-vs-kf94-masks-for-travel/' target='_blank' rel='nofollow'>N95 masks and their cousins</a> (KN95, KF94, etc.) do the job because they filter particles as small as 0.3 microns. This is similar to the particle size filtered by HEPA filters. Caution: KN95 is a self-reported test standard, and lacks strict government regulation by China, resulting in many underperforming and often flat-out fake masks. </p> <p> The US National Personal Protective Technology Laboratory (NPPTL), which is part of the US National Institute for Occupational Safety and Health (NIOSH), which itself is part of the CDC, evaluated various masks and published the results as <a href='https://www.cdc.gov/niosh/npptl/respirators/testing/NonNIOSHresults.html' target='_blank' rel='nofollow'>NPPTL Respirator Assessments to Support the COVID-19 Response</a>. </p> <p> The following types of masks do <b>not</b> reliably filter tiny aerosols, so they do not adequately protect you from COVID-19 variants such as Omicron: </p> <ol> <li> <a href='https://www.theguardian.com/commentisfree/2021/dec/27/best-masks-covid-tests-cloth-surgical-respirators' target='_blank' rel='nofollow'>Surgical masks</a> (what most people wear these days) </li> <li>Masks with activated charcoal</li> <li>Vented masks</li> <li>Cloth masks</li> <li>Gaiters</li> <li>Ill-fitting masks that do not cover and seal the mouth and nose</li> </ol> <iframe width="690" height="388" src="https://www.youtube.com/embed/WE5Uo3F2TdU" frameborder="0" allowfullscreen class="rounded shadow liImg"></iframe> <h2 id="other">Inoculations and Pills</h2> <div style="text-align: right;"> <a href="https://www.reuters.com/business/healthcare-pharmaceuticals/us-fda-set-authorize-pfizer-merck-covid-19-pills-this-week-bloomberg-news-2021-12-21/" target="_blank" ><picture> <source srcset="/blog/images/covidHvac/covidPills.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/covidPills.png" type="image/png"> <img src="/blog/images/covidHvac/covidPills.png" title="COVID-19 pills" class="right quartersize liImg2 rounded shadow" alt="COVID-19 pills" /> </picture></a> </div> <p> Inoculations, including boosters, have proven to be very helpful. Pills from Pfizer Inc. and Merck & Co. for people sick with COVID-19 will also be very helpful once they become available. However, pills will only help sick people get better, they will not prevent getting sick. </p> <p> Until everyone in the entire world has somehow acquired an effective level of antibodies, COVID-19 will continue to mutate. </p> <h2 id="go">The Only Solution Available Today</h2> <p> Inoculations and masks are good, but they are not perfect preventative measures against COVID-19. Protection against infection is not 100%. At present, the only preventative measure against COVID-19 that would allow people to safely mingle in person would be to guarantee continuous streams of pure air directed individually at each person's face. </p> <p> This measure would also prevent the spread of <a href='https://bmcinfectdis.biomedcentral.com/articles/10.1186/s12879-019-3707-y' target='_blank' rel='nofollow'>all other diseases and their variants that are primarily spread via aerosols</a>, including coronoviruses, colds, flus, tuberculosis, MERS-CoV, measles, ebola and chickenpox. Pollen, dust and other particulates would also be removed, so athsma sufffers would feel relief from extended periods breathing pure air. </p> <p> The solution to living with COVID-19 is simple, safe, inexpensive and is not disruptive: </p> <div class="formalNotice rounded shadow"> <h2 class="centered" style="margin: 0; padding: 0">Live your social life with a pure breeze in your face.</h2> <div style=""> <picture> <source srcset="/blog/images/covidHvac/breeze-dandelion.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/breeze-dandelion.png" type="image/png"> <img src="/blog/images/covidHvac/breeze-dandelion.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <div style="clear:both; font-size: 0"></div> </div> <p> Life as we knew it before the pandemic started 2 years ago could resume, mostly. </p> <h2 id="hvac">HVAC Upgrades Are Key</h2> <p> The engineering group that encompasses HVAC is ASHRAE. ASHRAE was formed as the American Society of Heating, Refrigerating and Air-Conditioning Engineers by the merger in 1959 of American Society of Heating and Air-Conditioning Engineers (ASHAE), founded in 1894 and The American Society of Refrigerating Engineers (ASRE), founded in 1904. </p> <p> ASHRAE offers <a href='https://www.ashrae.org/file%20library/technical%20resources/covid-19/core-recommendations-for-reducing-airborne-infectious-aerosol-exposure.pdf' target='_blank' rel='nofollow'>Core Recommendations for Reducing Airborne Infectious Aerosol Exposure</a>. <a href='https://www.iso-aire.com/blog/what-are-the-differences-between-a-merv-13-and-a-hepa-filter' target='_blank' rel='nofollow'>MERV 13 or better</a> levels of performance are recommended, and HEPA filters surpass that specifation. In fact, all HEPA filters have a rating of a MERV 17 or higher. </p> <h2 id="entre">Entrepreneurs</h2> <p> This represents an opportunity for entrepreneurs. </p> <h3 id="restos">Restaurants</h2> <p> Imagine a restaurant that stays open with near-normal seating capacity because each table is equipped with a fan and ducting that blows a gentle breeze of pure air directly into the face of every patron. The air would recirculate within the restaurant, so there would be no need to upgrade the existing HVAC system because the HEPA filters located at each table would continuously clean the air within the restaurant. Those filters would need to be cleaned and disinfected daily. The wait staff would all wear N95 or similar masks, however each cashier could instead enjoy their own gentle breeeze of pure air. </p> <p> Cost to equip each seat in the restaurant could be less than $50. For do-it-yourself owners, the cost could approach $10 per seat. For example, a 100-seat restaurant might be able to retrofit 75 of those seats at a cost of less than $3750. They would never have to close again due to any pandemic caused by airborne aerosols... provided, of course, that local regulations recognized this approach. </p> <h3 id="stores">Retail Stores and Office Buildings</h2> <p> Enclosing sales clerks in stores behind clear plastic walls is wrong because it decreases air circulation. Instead, each those staff members should have a dedicated ducted fan that blows a gentle breeze of pure air into their face at all times. </p> <p> Similarly, queues of people awaiting their turn at checking out should have fans blowing pure air at their heads. The air would recirculate within the store, there would be no need to upgrade the existing HVAC system because the HEPA filters would continuously clean the air within the building. </p> <h2 id="domain">Spring-Breezifier Is Offered Into the Public Domain</h2> <p> I offer this idea to the world; I am an idea machine so I cannot act on most of them. Just for fun, let's call this idea the <i>Spring-Breezifier</i>. </p> <p> Designing and building Spring-Breezifiers seems like it might be a good high school or youth group project. HVAC installers in particular should be able to make short work of this idea; go ahead and make lots of money providing pure airflows by building custom Spring-Breezifiers for your customers, we all thank you! </p> <p> If anyone would like to talk to me about their Spring-Breezifier project, I would be happy to speak with them. </p> <h2 id="next">Next Steps</h2> <ol> <li> Medical HVAC specialists could suggest the necessary cubic feet per minute of air required per person sufficient to guarantee a continuous stream of pure air, and provide guidelines for designing ductwork to direct that air effectively. </li> <li> Build and test prototypes. Most of the effort will be in designing and building the ducting and/or the scaffolding. Adding inner baffles would raise the cost and weight, lower the noise and make the airflow less turbulent. Perhaps Spring-Breezifiers could be built entirely from off-the-shelf parts for some or even most installations. Large-format 3D printers for special circumstances might be helpful. </li> <li>Propose the inclusion of pure air streams into public spaces as a matter of public health policy.</li> </ol> Recording Solo With OBS Studio 2021-11-15T00:00:00-05:00 https://mslinn.github.io/blog/2021/11/15/obs-web <p> Making video recordings of yourself solo is a challenge in many ways. Not only do you need to start and stop recordings, but you also need to hear and see yourself as you appear in the video, and either adjust settings or adjust yourself. </p> <p> For me, one of the biggest issues is starting and stopping the recording without moving out of position. The space where I sit in order to play has a camera, lights and microphones, and all of that cannot be right in front of the computer that records the audio and video. I need another viewing surface, and a means of controlling the recording software. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/recordingSpace.webp" type="image/webp"> <source srcset="/blog/images/obs/recordingSpace.png" type="image/png"> <img src="/blog/images/obs/recordingSpace.png" class="center liImg2 rounded shadow" /> </picture> </div> <p> The software I usually need to run in order to make music videos includes RME TotalMix, Pro Tools, and OBS Studio. Another problem that solo recording artists face is that these programs must all run at the same time, and all of them require lots of real estate on your computer screen. </p> <h2 id="remote">RME TotalMix Remote and Avid EUCON</h2> <p> I have experimented with various ways of controlling the software, including controlling the recording computer from a touchscreen laptop, a first-generation iPad mini, an Android phone, and MIDI devices. <a href='https://www.rme-usa.com/totalmix-fx-remote.html' target='_blank' rel='nofollow'>RME TotalMix Remote</a>, <a href='https://cdn-www.avid.com/-/media/avid/files/products-pdf/artist-control/eucon_application_setup_v2_6.pdf' target='_blank' rel='nofollow'>Pro Tools EUCON</a> and <a href='https://avid.secure.force.com/pkb/articles/en_US/User_Guide/EuControl-Product-Guides' target='_blank' rel='nofollow'>Pro Tools EUControl</a> all work well on every device I have tried, for example: </p> <ul> <li>RME TotalMix Remote on my first-generation iPad mini</li> <li>RME TotalMix Remote on a touchscreen laptop</li> <li>Pro Tools EUControl and EUCON Client on touchscreen laptop</li> <li>Avid Control on the iPad mini</li> </ul> <p> It is also possible to use MIDI controllers that have knobs, slider and buttons to control arbitrary software using <a href='https://www.bome.com/products/miditranslator' target='_blank' rel='nofollow'>Bome MIDI Translator Pro</a>. That software is complex, but very capable. MIDI devices generally require cables, and setting them up can be time-consuming, and that setup is specific for each MIDI controller you might want to try. <a href='https://www.google.com/search?q=midi+over+wifi' target='_blank' rel='nofollow'>MIDI over WiFi</a> is interesting, but requires extra setup. Maybe I will dig into this topic one day. </p> <h2 id="ipad_mini">iPad Mini Is Too Small</h2> <p> The iPad mini works well, but it is awkward to run TotalMix Remote and Avid Control together on such a tiny device. I can run TotalMix remote on the iPad mini, and Avid Control on my Android phone, although the phone's screen is too small to be useful. </p> <p> Using a second laptop is better because multiple programs can be visible at the same time, and I don't have to think about which device to pick up in order to make an adjustment or start/stop recording. </p> <p> Perhaps using the <a href='https://support.apple.com/en-ca/HT207582' target='_blank' rel='nofollow'>iPadOS multitasking feature</a> on an iPad Pro might work as well as running several programs on a laptop, but an iPad Pro is as expensive as a laptop, and laptops are more versatile. </p> <h2 id="obs-different">OBS Studio Control Options</h2> <p> Two simple ways that I experimented with in order to control OBS Studio from the recording space were abandoned: </p> <ol> <li> Plugging in a second keyboard on the recording computer and using OBS Studio hotkeys to start and stop recording (difficult to see the computer screen, and the hot keys only work if OBS Studio is in the foreground.)</li> <li> Using a <a href='https://www.dell.com/en-us/shop/dell-24-touch-monitor-p2418ht/apd/210-alcs/monitors-monitor-accessories' target='_blank' rel='nofollow'>touch-sensitive computer monitor</a> on the recording computer, and mounting the monitor on a swivel arm (OBS Studio's buttons are too small to work reliably, half the time the wrong button is pushed). <div class="videoWrapper2 shadow" style="margin-top: 1em"> <iframe width="1267" height="722" src="https://www.youtube.com/embed/I346wSqhobw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> </div> </li> </ol> <p> <code>Obs-web</code> is a better way, because it provides a live web page that can be accessed from anywhere in your local area network. However, you need some technical ability because there is no user-friendly way to set up <code>obs-web</code>. Once setup, operation is very simple, however. </p> <p> <code>Obs-web</code> requires an OBS Studio plugin called <code>obs-websocket</code>. </p> <div class="formalNotice rounded shadow"> <div style="text-align: right;"> <picture> <source srcset="/blog/images/obs/obsws_logo_new.webp" type="image/webp"> <source srcset="/blog/images/obs/obsws_logo_new.png" type="image/png"> <img src="/blog/images/obs/obsws_logo_new.png" class="right quartersize " /> </picture> </div> <i>From <a href='https://discord.com/channels/715691013825364120/715692236901187674/911388251791708190' target='_blank' rel='nofollow'>Discord #announcements (obs-websocket)</a></i><br> @everyone Hey there, update for you regarding obs-websocket. We've moved to the OBS Project! <https://github.com/obsproject/obs-websocket> This is exciting news. It means that obs-websocket is on the roadmap to be included as a default plugin in OBS Studio.<br><br> I'll share our current roadmap, which may change at any point:<br><br> <b>**Til the end of 2021:**</b> Release obs-websocket version 5.0.0 as an independent plugin, officially beginning the transition period from 4.x to 5.x.<br><br> <b>**A few months into 2022:**</b> Merge obs-websocket as a submodule to obs-studio, and include it by default in the next major OBS release. This will overwrite obs-websocket 4.x, officially deprecating those versions. <br><br> As for our financial contributors over at Open Collective, we're working to get all of that data transferred to the OBS Project's page, where we will have a project dedicated to obs-websocket. Palakis and I are looking forward to obs-websocket's future in OBS. We will be keeping our roles as lead maintainers of the plugin. </div> <h2 id="obs-web">Installing Obs-web</h2> <p> Using a mobile device or another computer to control OBS Studio is possible using an OBS Studio plugin called <a href='https://github.com/obsproject/obs-websocket' target='_blank' rel='nofollow'>OBS-websocket</a>, and a client called <a href='https://github.com/Niek/obs-web' target='_blank' rel='nofollow'>obs-web</a>. Some technical experience with computers is required for this approach, but it works quite well. </p> <ol> <li> OBS-websocket was installed easily on the computer that is running OBS Studio by using the provided installer program. Once installed, the OBS Studio Tools menu had a new item called <b>Websockets Server Settings</b>. <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/obs-websocket-menu-addition.webp" type="image/webp"> <source srcset="/blog/images/obs/obs-websocket-menu-addition.png" type="image/png"> <img src="/blog/images/obs/obs-websocket-menu-addition.png" class="center liImg2 rounded shadow" /> </picture> </div> </li> <li> Clicking on the new OBS Studio Tools menu allows you to configure OBS-websocket. Notice that I disabled <b>Enable authentication</b>. <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/obs-websocket-config.webp" type="image/webp"> <source srcset="/blog/images/obs/obs-websocket-config.png" type="image/png"> <img src="/blog/images/obs/obs-websocket-config.png" class="center liImg2 rounded shadow" /> </picture> </div> </li> <li> On the same computer, download OBS-web from <a href='https://github.com/Niek/obs-web' target='_blank' rel='nofollow'>this page</a> and click on <b>Download latest build here</b>. <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/obs-web-download.webp" type="image/webp"> <source srcset="/blog/images/obs/obs-web-download.png" type="image/png"> <img src="/blog/images/obs/obs-web-download.png" class="center halfsize liImg2 rounded shadow" /> </picture> </div> </li> <li> I extracted the contents of the download, provided as a zip file, to <code>E:\media\obs-web-gh-pages</code>. </li> <li> I used Python 3 to start a small web server on port 4321 that serves the OBS-web files like this: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id16ac451a5bdf'><button class='copyBtn' data-clipboard-target='#id16ac451a5bdf' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn></span>python -m http.server \ --directory E:\media\obs-web-gh-pages \ 4321 <span class='unselectable'>Serving HTTP on :: port 4321 (http://[::]:4321/) ... ::1 - - [15/Nov/2021 10:49:52] "GET / HTTP/1.1" 304 - ::1 - - [15/Nov/2021 10:49:53] "GET /service-worker.js HTTP/1.1" 304 - </span></pre> The above incantation works on Windows, Mac and Linux. You only need to adjust the directory that the OBS-web files are stored in. </li> </ol> <h2 id="usingObsWeb">Using Obs-web</h2> <p> There are 2 small web servers now running on the computer with OBS Studio: the Python web server I started on the command line, and the OBS-web plugin for OBS Studio. This is how all the various bits of software work together: </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/obs-web-flow.webp" type="image/webp"> <source srcset="/blog/images/obs/obs-web-flow.png" type="image/png"> <img src="/blog/images/obs/obs-web-flow.png" class="center liImg2 rounded shadow" /> </picture> </div> <p> The computer running OBS Studio had IP address <code>192.168.1.77</code>. Knowing the IP address of that computer allows it and its programs to be accessed from other computers in the same network. <a href='https://www.tp-link.com/us/support/faq/838/' target='_blank' rel='nofollow'>Discover your computer's IP address.</a> </p> <!-- <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/obs-web-not-connected.webp" type="image/webp"> <source srcset="/blog/images/obs/obs-web-not-connected.png" type="image/png"> <img src="/blog/images/obs/obs-web-not-connected.png" class="center liImg2 rounded shadow" /> </picture> </div> <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/obs-web-not-connected-small.webp" type="image/webp"> <source srcset="/blog/images/obs/obs-web-not-connected-small.png" type="image/png"> <img src="/blog/images/obs/obs-web-not-connected-small.png" class="center liImg2 rounded shadow" /> </picture> </div> <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/obs-web-connecting.webp" type="image/webp"> <source srcset="/blog/images/obs/obs-web-connecting.png" type="image/png"> <img src="/blog/images/obs/obs-web-connecting.png" class="center liImg2 rounded shadow" /> </picture> </div> --> <p> To control OBS Studio from any computer or device in my local area network, I just need to point a web browser on the device to the <code>http://192.168.1.77:4321/#192.168.1.77:4444</code>. The buttons displayed on the web browser can be clicked on using a mouse, or by a finger if the device is touch-senstive. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/obs-web-connected.webp" type="image/webp"> <source srcset="/blog/images/obs/obs-web-connected.png" type="image/png"> <img src="/blog/images/obs/obs-web-connected.png" class="center liImg2 rounded shadow" /> </picture> </div> <p> The 3 scenes I defined in OBS Studio are shown as blue and green buttons labeled <b>Default</b>, <b>Sony ILCE-7SM3</b> and <b>ProTools</b>. I can click on the red button labeled <b>Start recording</b>. Works really well. </p> Using an HDMI Splitter with OBS Studio 2021-11-13T00:00:00-05:00 https://mslinn.github.io/blog/2021/11/13/hdmi-splitter <p> <a href='/blog/2021/11/12/external-video-monitor.html'>A previous blog post</a> discussed the benefits of using an external monitor attached to your camera. HDMI is the most common OBS Studio is a wonderful tool for aggregating media into a real-time stream, and/or making a permanent recording. If you use an <a href='https://www.amazon.com/gp/product/B07XCZC6SP' target='_blank' rel='nofollow'>HDMI splitter</a>, the video from the camera can be viewed on a large monitor right in front of you, and the video can also be sent to the computer that is running OBS Studio. </p> <div style="text-align: center;"> <a href="https://www.amazon.com/gp/product/B07XCZC6SP" target="_blank" ><picture> <source srcset="/blog/media/videoSplitter.webp" type="image/webp"> <source srcset="/blog/media/videoSplitter.png" type="image/png"> <img src="/blog/media/videoSplitter.png" title="A 4K Video Splitter" class="center halfsize " alt="A 4K Video Splitter" /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://www.amazon.com/gp/product/B07XCZC6SP" target="_blank" > A 4K Video Splitter </a> </figcaption> </figure> </div> <p> The video splitter shown works with video resolutions up to 4K @ 60 Hz, which includes 4K @ 30 Hz and all 1080p video variants. When used in tandem with an HDMI to USB converter, for example a <a href='https://www.elgato.com/en/cam-link-4k' target='_blank' rel='nofollow'>CamLink 4K</a>, the HDMI video is converted into a video stream that OBS Studio can accept, either 1080p up to 60 Hz or 4K at 30 Hz. </p> <div style="text-align: center;"> <a href="https://www.elgato.com/en/cam-link-4k" target="_blank" ><picture> <source srcset="/blog/media/camlink-packaging.webp" type="image/webp"> <source srcset="/blog/media/camlink-packaging.png" type="image/png"> <img src="/blog/media/camlink-packaging.png" title="This HDMI to USB 3 adaptor is powered via USB.<br>No additional power is required." class="center halfsize " alt="This HDMI to USB 3 adaptor is powered via USB.<br>No additional power is required." /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://www.elgato.com/en/cam-link-4k" target="_blank" > This HDMI to USB 3 adaptor is powered via USB.<br>No additional power is required. </a> </figcaption> </figure> </div> <p> Here is a conceptual diagram of the signal flow between the camera, video splitter, CamLink and the computer running OBS Studio: </p> <div style="text-align: center;"> <picture> <source srcset="/blog/media/video-monitor-and-camlink.webp" type="image/webp"> <source srcset="/blog/media/video-monitor-and-camlink.png" type="image/png"> <img src="/blog/media/video-monitor-and-camlink.png" title="Signal paths with HDMI splitter and CamLink." class="center liImg2 rounded shadow" alt="Signal paths with HDMI splitter and CamLink." /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> Signal paths with HDMI splitter and CamLink. </figcaption> </figure> </div> <p> I have had the video splitter only one day so far, and it has worked well. The splitter takes about 8 seconds after being plugged in before it generates video output. Check out this video saved from OBS Studio using the above configuration: </p> <div class="videoWrapper2 shadow" style="margin-bottom: 1em"> <iframe width="1267" height="722" src="https://www.youtube.com/embed/eGYhGHl3xV0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> </div> <p> The video signal is noticably degraded by passing throught the video splitter. Compare the above video quality to the video below, made with the same equipment but without the video splitter: </p> <div class="videoWrapper2 shadow"> <iframe width="1267" height="722" src="https://www.youtube.com/embed/lkNe7gBP4YE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> </div> External Video Monitors For Cameras 2021-11-12T00:00:00-05:00 https://mslinn.github.io/blog/2021/11/12/external-video-monitor <p> When you are making a video recording of yourself, and no-one is helping you, the first problem you are likely to encounter is the need to see yourself. There are many reasons for needing to see yourself, including the need to ensure that: </p> <ul> <li>The camera is pointing at the right spot.</li> <li>The zoom level is correct.</li> <li>The camera is focused on the right area.</li> <li>The depth of field is correct.</li> <li>You look the way you want.</li> </ul> <h2 id="flip">Flip Monitors</h2> <p> Some cameras have a small flip-out viewscreen that can be seen from the front of the camera. That is only useful if you are very close to the camera, which means that most of the above reasons are unsatisfied. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/media/HR_G7X_MARKII_3QBACKLCD_CL.0.webp" type="image/webp"> <source srcset="/blog/media/HR_G7X_MARKII_3QBACKLCD_CL.0.png" type="image/png"> <img src="/blog/media/HR_G7X_MARKII_3QBACKLCD_CL.0.png" title="Canon Powershot G7x Mark II, showing its flip-up viewscreen" class="center halfsize liImg2 rounded shadow" alt="Canon Powershot G7x Mark II, showing its flip-up viewscreen" /> </picture> <figcaption class="halfsize" style="width: 100%; text-align: center;"> Canon Powershot G7x Mark II, showing its flip-up viewscreen </figcaption> </figure> </div> <p> You need a larger monitor, and it would be best if that monitor was located in the direction that you will be looking while making the recording. Quite often, this means the monitor should be directly under or over the camera. </p> <h2 id="overkill">Overkill</h2> <p> If you search online for solutions to this problem, you might get the impression that you need to spend a lot of money on external camera monitors that are 5" or 8" wide. Many articles and videos imply that you might even want to build a cage around the camera to hold both the monitor and extra batteries. This can cost hundreds or even thousands of dollars, but there is no need to go that way if you are shooting video indoors, for example, in a music studio. Not only that, but an 8" wide monitor is too small to be useful when working solo. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/media/atomos.webp" type="image/webp"> <source srcset="/blog/media/atomos.png" type="image/png"> <img src="/blog/media/atomos.png" title="Atomos Ninja V attached to a camera" class="center halfsize liImg2 rounded shadow" alt="Atomos Ninja V attached to a camera" /> </picture> <figcaption class="halfsize" style="width: 100%; text-align: center;"> Atomos Ninja V attached to a camera </figcaption> </figure> </div> <h2 id="computer">Computer Monitors</h2> <p> You can instead attach a computer monitor or a good quality TV to your camera. They are much larger, and some computer monitors and TVs provide powered USB ports for devices such as the camera, a music stand light, etc. This reduces the amount of wires and power plugs necessary. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/media/externalMonitor.webp" type="image/webp"> <source srcset="/blog/media/externalMonitor.png" type="image/png"> <img src="/blog/media/externalMonitor.png" title="A computer monitor attached to a camera via HDMI" class="center liImg2 rounded shadow" alt="A computer monitor attached to a camera via HDMI" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> A computer monitor attached to a camera via HDMI </figcaption> </figure> </div> <p> Most cameras provide a micro HDMI port, and most computer monitors accept fullsize HDMI connections. An <a href='https://www.amazon.com/AmazonBasics-High-Speed-Micro-HDMI-HDMI-Cable/dp/B014I8TZXW' target='_blank' rel='nofollow'>micro HDMI to fullsize HDMI adaptor cable</a> is all you need to connect the camera to the computer monitor. Note that most cameras do not provide audio on the HDMI output port. </p> <div style="text-align: center;"> <a href="https://www.amazon.com/AmazonBasics-High-Speed-Micro-HDMI-HDMI-Cable/dp/B014I8TZXW" target="_blank" ><picture> <source srcset="/blog/media/microHdmiAdapter.webp" type="image/webp"> <source srcset="/blog/media/microHdmiAdapter.png" type="image/png"> <img src="/blog/media/microHdmiAdapter.png" title="Micro-HDMI and HDMI connectors" class="center quartersize liImg2 rounded shadow" style="padding-left: 1em; padding-right: 1em; padding-top: 1em;" alt="Micro-HDMI and HDMI connectors" /> </picture></a> <figcaption class="quartersize" style="width: 100%; text-align: center;"> <a href="https://www.amazon.com/AmazonBasics-High-Speed-Micro-HDMI-HDMI-Cable/dp/B014I8TZXW" target="_blank" > Micro-HDMI and HDMI connectors </a> </figcaption> </figure> </div> <h2 id="usb">USB Ports</h2> <p> The least expensive computer monitors do not have USB ports, and some USB ports provide more power than others. Read monitor specifications before you buy. </p> <p> For this to work, the computer monitor needs to have <a href='https://en.wikipedia.org/wiki/USB_hardware' target='_blank' rel='nofollow'>USB type A ports</a> as shown below, either on the back of the monitor, or on the side. There is no need to connect the USB type B port to anything if you just want to provide power via USB type A connectors: </p> <div style="text-align: center;"> <picture> <source srcset="/blog/media/monitor-usb.webp" type="image/webp"> <source srcset="/blog/media/monitor-usb.png" type="image/png"> <img src="/blog/media/monitor-usb.png" title="USB ports on the back of a video monitor" class="center liImg2 rounded shadow" alt="USB ports on the back of a video monitor" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> USB ports on the back of a video monitor </figcaption> </figure> </div> <p> The USB ports should provide at least 1 amp <i>each</i>. Some USB devices require more power than that. Some manufacturers do not publish the power provided by their computer monitor's USB ports, so try with your actual USB devices before you buy, or at least have the option to return the monitor if the USB ports do not supply enough power. </p> <h2 id="mySetup">My Setup</h2> <p> I use the <a href='https://support.d-imaging.sony.co.jp/app/iemobile/en/' target='_blank' rel='nofollow'>Sony Imaging Edge Remote</a> camera app for my Android phone to control my <a href='https://www.sony.ca/en/electronics/interchangeable-lens-cameras/ilce-7m3-body-kit' target='_blank' rel='nofollow'>Sony Alpha 7 Mark iii camera</a> remotely. With that software I can adjust aperture and other parameters. I prefer to use the <a href='https://www.sony.ca/en/electronics/interchangeable-lens-cameras-tripods-remotes/rmt-p1bt' target='_blank' rel='nofollow'>Sony Wireless Remote Commander</a> to control recording because I can feel the physical controls, so I can operate it without looking. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/media/video-monitor-for-camera.webp" type="image/webp"> <source srcset="/blog/media/video-monitor-for-camera.png" type="image/png"> <img src="/blog/media/video-monitor-for-camera.png" title="Video monitor and power for camera" class="center liImg2 rounded shadow" alt="Video monitor and power for camera" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> Video monitor and power for camera </figcaption> </figure> </div> <p> I have several monitors that provide USB connections, including a BenQ BL3201PH 32" 4K Monitor, an ASUS ProArt Display PA248Q and an ASUS PA238Q. I use the PA238Q for monitoring the camera. The monitor is not attached to a computer, and I locate it as required for the scene I am shooting. </p> <div style="text-align: center;"> <a href="https://www.asus.com/Displays-Desktops/Monitors/ProArt/ProArt-Display-PA248Q/" target="_blank" ><picture> <source srcset="/blog/media/ASUS_PA248Q.webp" type="image/webp"> <source srcset="/blog/media/ASUS_PA248Q.png" type="image/png"> <img src="/blog/media/ASUS_PA248Q.png" title="ASUS ProArt Display PA248Q video monitor<br>with 4 USB 3 type A connectors on the side" class="center halfsize liImg2 rounded shadow" alt="ASUS ProArt Display PA248Q video monitor<br>with 4 USB 3 type A connectors on the side" /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://www.asus.com/Displays-Desktops/Monitors/ProArt/ProArt-Display-PA248Q/" target="_blank" > ASUS ProArt Display PA248Q video monitor<br>with 4 USB 3 type A connectors on the side </a> </figcaption> </figure> </div> <div style="text-align: center;"> <a href="https://www.asus.com/Displays-Desktops/Monitors/ProArt/ProArt-Display-PA238Q/" target="_blank" ><picture> <source srcset="/blog/media/ASUS_PA238Q.webp" type="image/webp"> <source srcset="/blog/media/ASUS_PA238Q.png" type="image/png"> <img src="/blog/media/ASUS_PA238Q.png" title="ASUS ProArt Display PA238Q video monitor<br> has 2 USB 2 type A connectors on the side, and 2 more underneath" class="center halfsize liImg2 rounded shadow" alt="ASUS ProArt Display PA238Q video monitor<br> has 2 USB 2 type A connectors on the side, and 2 more underneath" /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://www.asus.com/Displays-Desktops/Monitors/ProArt/ProArt-Display-PA238Q/" target="_blank" > ASUS ProArt Display PA238Q video monitor<br> has 2 USB 2 type A connectors on the side, and 2 more underneath </a> </figcaption> </figure> </div> <p> The UBS ports on my ASUS ProArt Display PA238Q provide enough power to run my Sony Alpha 7 Mark iii camera, which is nice, but not enough to run my <a href='https://www.amazon.com/gp/product/B07XCZC6SP' target='_blank' rel='nofollow'>HDMI splitter</a>. This just means that I must use a dedicated power supply for the HDMI splitter. I will talk about how I use the HDMI splitter in another blog post. </p> Sending DAW Output to OBS Studio Using RME TotalMix 2021-11-08T00:00:00-05:00 https://mslinn.github.io/blog/2021/11/08/totalmix-daw-obs <p> <a href='https://www.rme-usa.com/totalmix-fx.html' target='_blank' rel='nofollow'>RME TotalMix</a> provides routing and mixing functions in software for RME audio interfaces. I use an <a href='https://archiv.rme-audio.de/en/products/fireface_ufx.php' target='_blank' rel='nofollow'>RME UFX</a> on my main <a href='https://en.wikipedia.org/wiki/Digital_audio_workstation' target='_blank' rel='nofollow'>DAW</a>, and an <a href='https://archiv.rme-audio.de/en/products/fireface_uc.php' target='_blank' rel='nofollow'>RME UC</a> for mobile recordings. TotalMix works with all RME audio interfaces, including the popular <a href='https://rme-audio.de/babyface-pro-fs.html' target='_blank' rel='nofollow'>BabyFace</a>. </p> <p> This blog post briefly outlines how to capture a live performance on video, using your studio-quality microphones while enjoying the real-time effects and mixing capability of your favorite DAW software. You can also use this setup to make a high quality video capture of karaoke, that is, singing along or playing along with pre-recorded music. The recorded music could either be mixed using your DAW, or by providing it to OBS Studio as a Media Source. </p> <p> Nothing described in this blog post is specific to any particular DAW software; I mostly use Pro Tools and Ableton Live, but Cakewalk, etc would work just as well with these instructions. I tested this with Windows 10. Apparently TotalMix on Mac does not mute some inputs properly when loopback is enabled, or something like that; I am unclear exactly what the problem is. </p> <p> No changes to the system hardware or software are necessary to send DAW output to OBS Studio if you use an RME audio interface, most especially no physical or virtual cables are necessary. </p> <div class="formalNotice shadow rounded"> <h2 id="but" style="margin-top: 0">What If I Do Not Have an RME Audio Interface?</h2> If you have an audio interface made by another manufacturer then you will probably need to use a virtual audio cable, such as <a href='https://vb-audio.com/Cable/' target='_blank' rel='nofollow'>VB-Cable or HIFI-CABLE & ASIO-Bridge</a>. Both programs are available for Windows and Mac. Online help for those programs are available <a href='https://forum.vb-audio.com/index.php?sid=94158a3f6725fd801b7b52ea044c6ccb' target='_blank' rel='nofollow'>on the VB-Audio forums</a>. </div> <p> The way for TotalMix to route the DAW output channels to another stereo input is called ‘loopback’. <a href='https://www.youtube.com/watch?v=m-Zeruz-9Zk' target='_blank' rel='nofollow'>This YouTube video</a> shows how to configure TotalMix to use loopback, but unfortunately the video stops short of actually showing how to work with the routed audio. Read on and I will tell you the rest of the story. It is quick and easy! </p> <h2 id="signals">Signal Paths</h2> <p> The following diagram shows the important signal paths for this setup. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/totalmixObs/rmeLoopbackSignalPaths.webp" type="image/webp"> <source srcset="/blog/images/totalmixObs/rmeLoopbackSignalPaths.png" type="image/png"> <img src="/blog/images/totalmixObs/rmeLoopbackSignalPaths.png" class="center liImg2 rounded shadow" /> </picture> </div> <h2 id="steps">Step by Step</h2> <p> As shown in the video, set up an unused TotalMix output which will be sent to OBS Studio. Let’s use the AES output for that. The steps are: </p> <ol> <li> Open the AES tools (click on the little wrench icon) and enable loopback, like this: <div style="text-align: center;"> <picture> <source srcset="/blog/images/totalmixObs/totalMixAes.webp" type="image/webp"> <source srcset="/blog/images/totalmixObs/totalMixAes.png" type="image/png"> <img src="/blog/images/totalmixObs/totalMixAes.png" class="center liImg2 rounded shadow" /> </picture> </div> Unfortunately, TotalMix will not show loopback signal at AES input. (Hey, RME, please do something about this!) </li> <li style="clear: both"> In OBS Studio, define a new Audio Input Capture source that captures the output of the TotalMix AES loopback. First press the <b>+</b> but1ton, then click on <b>Audio Input Capture</b>: <div style="text-align: center;"> <picture> <source srcset="/blog/images/totalmixObs/totalMixAesDefine.webp" type="image/webp"> <source srcset="/blog/images/totalmixObs/totalMixAesDefine.png" type="image/png"> <img src="/blog/images/totalmixObs/totalMixAesDefine.png" class="center liImg2 rounded shadow" /> </picture> </div> </li> <li style="clear: both"> I called the new OBS Studio source <b>AES</b>: <div style="text-align: center;"> <picture> <source srcset="/blog/images/totalmixObs/totalMixAesDefineName.webp" type="image/webp"> <source srcset="/blog/images/totalmixObs/totalMixAesDefineName.png" type="image/png"> <img src="/blog/images/totalmixObs/totalMixAesDefineName.png" class="center liImg2 rounded shadow" /> </picture> </div> </li> <li style="clear: both"> To complete the definition of the new OBS Studio source, select the input source whose name starts with <b>AES (RME</b>: <div style="text-align: center;"> <picture> <source srcset="/blog/images/totalmixObs/obsStudioAesInput.webp" type="image/webp"> <source srcset="/blog/images/totalmixObs/obsStudioAesInput.png" type="image/png"> <img src="/blog/images/totalmixObs/obsStudioAesInput.png" class="center liImg2 rounded shadow" /> </picture> </div> </li> <li style="clear: both"> Set the mix levels in OBS Studio. <div style="text-align: center;"> <picture> <source srcset="/blog/images/totalmixObs/obsStudioMixLevels.webp" type="image/webp"> <source srcset="/blog/images/totalmixObs/obsStudioMixLevels.png" type="image/png"> <img src="/blog/images/totalmixObs/obsStudioMixLevels.png" class="center liImg2 rounded shadow" /> </picture> </div> </li> </ol> <p> That is all, you are done! The processed audio from your DAW will now be mixed with the other audio and video streams you set up. I use a <a href='https://www.elgato.com/en/cam-link-4k' target='_blank' rel='nofollow'>Camlink 4K</a> to stream video coming from my <a href='https://www.sony.ca/en/electronics/interchangeable-lens-cameras/ilce-7m3-body-kit' target='_blank' rel='nofollow'>Sony Alpha 7 Mark iii camera</a>. OBS Studio mixes all the audio and video streams, and the combined live media stream can be sent to Instagram, YouTube, Facebook etc, and/or it can be saved as an mp4 or mkv. I prefer to save as mkv. </p> <h2 id="hot">Hot Key</h2> <p> BTW, I set the hot key for recording the mix using OBS Studio to <kbd>Ctrl</kbd>-<kbd>Shift</kbd>-<kbd>Z</kbd>. You can find these settings at <b>File</b> / <b>Settings</b> / <b>HotKeys</b>: </p> <div style=""> <picture> <source srcset="/blog/images/totalmixObs/obsStudioHotKeys.webp" type="image/webp"> <source srcset="/blog/images/totalmixObs/obsStudioHotKeys.png" type="image/png"> <img src="/blog/images/totalmixObs/obsStudioHotKeys.png" class=" liImg2 rounded shadow" /> </picture> </div> <h2 id="testing">Test Recording</h2> <p> Here is a quick test recording that I made using the above setup, of an extemporaneous composition, while writing this blog post. Notice the echo and reverb effects on the audio; those were added by Pro Tools. I wore headphones so I could listen to the audio without worrying about feedback or adding extra echo. </p> <div class="videoWrapper2 shadow"> <iframe width="1267" height="722" src="https://www.youtube.com/embed/lkNe7gBP4YE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> </div> Extracting Audio from an MP4 as 32-bit WAV 2021-11-04T00:00:00-04:00 https://mslinn.github.io/blog/2021/11/04/mp4-to-wav <p> My <a href='https://www.sony.ca/en/electronics/interchangeable-lens-cameras/ilce-7m3-body-kit' target='_blank' rel='nofollow'>Sony Alpha 7 Mark iii camera</a> creates mp4 files with good quality stereo audio. I wanted to extract the audio to a 32-bit wav file, so I could work on it further in Pro Tools. Here is a bash script I wrote to do that: </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,mp4ToWav' download='mp4ToWav' title='Click on the file name to download the file'>mp4ToWav</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="ida4c45f8f4a11">#!/bin/bash # $1 Input file path function help &#123; if [ "$1" ]; then printf "$1\n\n"; fi echo "$(basename $0) - Extract audio stream from an mp4 file and save as 32-bit wav Usage: $(basename $0) filename " exit 1 &#125; if [ -z "$1" ]; then help "Error: no media file name specified"; fi if [ ! -f "$1" ]; then help "Error: '$1' not found"; fi filename="$( basename -- "$1" )" path="$( dirname "$1" )" extension="$&#123;filename##*.&#125;" filename="$&#123;filename%.*&#125;" ffmpeg \ -i "$1" \ -vn \ -acodec pcm_f32le \ -ar 44100 \ -ac 2 \ "$path/$filename.wav" </pre> <p> This is a sample usage: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id09ee152f4c01'><button class='copyBtn' data-clipboard-target='#id09ee152f4c01' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mp4ToWav "Video Files/Descending C to G djembe" ffmpeg version 4.3.2-0+deb11u1ubuntu1 Copyright (c) 2000-2021 the FFmpeg developers built with gcc 10 (Ubuntu 10.2.1-20ubuntu1) configuration: --prefix=/usr --extra-version=0+deb11u1ubuntu1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared libavutil 56. 51.100 / 56. 51.100 libavcodec 58. 91.100 / 58. 91.100 libavformat 58. 45.100 / 58. 45.100 libavdevice 58. 10.100 / 58. 10.100 libavfilter 7. 85.100 / 7. 85.100 libavresample 4. 0. 0 / 4. 0. 0 libswscale 5. 7.100 / 5. 7.100 libswresample 3. 7.100 / 3. 7.100 libpostproc 55. 7.100 / 55. 7.100 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x55fc3d8d1f80] st: 0 edit list: 1 Missing key frame while searching for timestamp: 1001 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x55fc3d8d1f80] st: 0 edit list 1 Cannot find an index entry before timestamp: 1001. Guessed Channel Layout for Input Stream #0.1 : stereo Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Video Files/Descending C to G djembe.mp4': Metadata: major_brand : XAVC minor_version : 16785407 compatible_brands: XAVCmp42iso2 creation_time : 2021-10-31T19:00:25.000000Z Duration: 00:09:06.55, start: 0.000000, bitrate: 51575 kb/s Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709/bt709/iec61966-2-4), 1920x1080 [SAR 1:1 DAR 16:9], 49492 kb/s, 59.94 fps, 59.94 tbr, 60k tbn, 119.88 tbc (default) Metadata: creation_time : 2021-10-31T19:00:25.000000Z handler_name : Video Media Handler encoder : AVC Coding Stream #0:1(und): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, stereo, s16, 1536 kb/s (default) Metadata: creation_time : 2021-10-31T19:00:25.000000Z handler_name : Sound Media Handler Stream #0:2(und): Data: none (rtmd / 0x646D7472), 491 kb/s (default) Metadata: creation_time : 2021-10-31T19:00:25.000000Z handler_name : Timed Metadata Media Handler timecode : 07:09:43:54 Stream mapping: Stream #0:1 -> #0:0 (pcm_s16be (native) -> pcm_f32le (native)) Press [q] to stop, [?] for help Output #0, wav, to 'Video Files/Descending C to G djembe.wav': Metadata: major_brand : XAVC minor_version : 16785407 compatible_brands: XAVCmp42iso2 ISFT : Lavf58.45.100 Stream #0:0(und): Audio: pcm_f32le ([3][0][0][0] / 0x0003), 44100 Hz, stereo, flt, 2822 kb/s (default) Metadata: creation_time : 2021-10-31T19:00:25.000000Z handler_name : Sound Media Handler encoder : Lavc58.91.100 pcm_f32le size= 188304kB time=00:09:06.55 bitrate=2822.4kbits/s speed= 153x video:0kB audio:188304kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000059%</pre> <p> The original mp4 was 3.4GB, and the output wav was 188MB. </p> Sony Alpha 7 Mark iii Camera Media Encodings 2021-11-03T00:00:00-04:00 https://mslinn.github.io/blog/2021/11/03/sony-a7iii-encodings <style> dt, dd { font-style: normal } </style> <p> The documentation for the <a href='https://www.sony.ca/en/electronics/interchangeable-lens-cameras/ilce-7m3-body-kit' target='_blank' rel='nofollow'>Sony Alpha 7 Mark iii camera</a> does not properly describe the differences in encodings between the various video formats available. The settings trade off quality versus file size. I want to know which settings to use for various purposes. To this end, I made a short video clip for each camera setting and examined its properties. This blog post details my findings. </p> <div style=""> <a href="https://www.sony.ca/en/electronics/interchangeable-lens-cameras/ilce-7m3-body-kit" target="_blank" ><picture> <source srcset="/blog/images/sonyA7iii.webp" type="image/webp"> <source srcset="/blog/images/sonyA7iii.png" type="image/png"> <img src="/blog/images/sonyA7iii.png" class=" liImg2 rounded shadow" /> </picture></a> </div> <p> Sony engineers have no doubt chosen these settings after careful deliberation and testing. Unfortunately, product documentation does not discuss when these settings should be used. </p> <p> To be specific, after working through all of this material, I still do not know when should I use the various formats. All I know for sure is that higher bit rate settings make larger video clips, and that there might be a quality difference. The degree to which the quality difference is perceptible is unknown. I would like guidelines in order to make intelligent decisions. Here are the specific formats that need to be rationalized: </p> <p style="margin-left: 2em"> <code>4K_30p_100M</code> vs. <code>4K_30p_60M</code><br> <code>4K_24p_100M</code> vs. <code>4K_24p_60M</code><br> <code>HD_120p_100M</code> vs. <code>HD_120p_60M</code><br> <code>HD_60p_50M</code> vs. <code>HD_60p_25M</code><br> <code>HD_30p_50M</code> vs. <code>HD_30p_16M</code> </p> <h2 id="background">Background</h2> <p> Here are a few key aspects from the Sony product specifications, to which I have added comments <i>in italics</i>: </p> <dl> <dt>Recording Format</dt> <dd> XAVC S, AVCHD format Ver. 2.0 compliant.<br> <i> The documentation is rather terse; the above means that two recording formats are available: XAVC, a proprietary Sony media format, and AVCHD, described <a href="#avchd">later in this document</a>. <a href='https://en.wikipedia.org/wiki/XAVC' target='_blank' rel='nofollow'>Wikipedia</a> says &ldquo;XAVC supports resolutions up to 3840 × 2160, uses MP4 as the container format, and uses either AAC or LPCM for the audio.&rdquo; </i> </dd> <dt>Video Compression</dt> <dd> XAVC S: MPEG-4 AVC/H.264, AVCHD: MPEG-4 AVC/H.264<br> <i>Again, the above means that the camera supports two types of video compression: XAVC S and AVCHD.</i> </dd> <dt>Audio Recording Format</dt> <dd> XAVC S: LPCM 2ch, AVCHD: Dolby® Digital (AC-3) 2ch, Dolby® Digital Stereo Creator.<br> <i>I understand this to mean that the camera supports 3 recording formats: <a href='https://en.wikipedia.org/wiki/Dolby_Digital' target='_blank' rel='nofollow'>Dolby AC-3</a> (a lossy format), AVCHD, and Dolby® Digital Stereo Creator (which I did not find).</i> <dd> <dt>Clean HDMI Output</dt> <dd> <i>This is important when streaming (via HDMI).</i><br> 3840 x 2160 (30p),<br> 3840 x 2160 (25p),<br> 3840 x 2160 (24p),<br> 1920 x 1080 (60p),<br> 1920 x 1080 (60i),<br> 1920 x 1080 (50p),<br> 1920 x 1080 (50i),<br> 1920 x 1080 (24p),<br> YCbCr 4:2:2 8bit / RGB 8bit.<br> <i>25p (25 frames per second) is noticably less smooth than 50p or 60p. I think that the extra resolution of 4K is not as noticable as the choppy video that it introduces, so my choice for streaming with this camera is 1920 x 1080 (60p). I am unclear about "YCbCr 4:2:2 8bit / RGB 8bit"; does that apply to 3840 x 2160 (25p), or to all the settings, or is it a separate setting?</i> </dd> <dt>Color Space</dt> <dd> xvYCC standard (x.v.Colour when connected via HDMI cable) compatible with TRILUMINOS Color.<br> <i>This means that color is somewhat degraded when streamed via HDMI. I would like to understand how much it is degraded.</i> </dd> </dl> <h2 id="mediaDump">Script to Examine Media Format</h2> <p> I wrote this bash script to examine video clips recorded at different settings. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,mediaDump' download='mediaDump' title='Click on the file name to download the file'>mediaDump</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id5bb2f6626cd9">#!/bin/bash function help &#123; if [ "$1" ]; then printf "$1\n\n"; fi echo "$(basename $0) - Dump information about a media file Usage: $(basename $0) filename " exit 1 &#125; if [ -z "$1" ]; then help "Error: no media file name specified"; fi ffprobe \ -hide_banner \ -loglevel fatal \ -show_error \ -show_format \ -show_streams \ -show_private_data \ -print_format json \ "$1" </pre> <h3 id="mediaDumpExample">Using the Script</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6c9ce701d262'><button class='copyBtn' data-clipboard-target='#id6c9ce701d262' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mediaDump 4K_24p_60M.mp4 <span class='unselectable'>{ "streams": [ { "index": 0, "codec_name": "h264", "codec_long_name": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10", "profile": "High", "codec_type": "video", "codec_time_base": "1001/48000", "codec_tag_string": "avc1", "codec_tag": "0x31637661", "width": 3840, "height": 2160, "coded_width": 3840, "coded_height": 2160, "closed_captions": 0, "has_b_frames": 1, "sample_aspect_ratio": "1:1", "display_aspect_ratio": "16:9", "pix_fmt": "yuv420p", "level": 51, "color_range": "tv", "color_space": "bt709", "color_transfer": "iec61966-2-4", "color_primaries": "bt709", "chroma_location": "left", "refs": 1, "is_avc": "true", "nal_length_size": "4", "r_frame_rate": "24000/1001", "avg_frame_rate": "24000/1001", "time_base": "1/24000", "start_pts": 0, "start_time": "0.000000", "duration_ts": 108108, "duration": "4.504500", "bit_rate": "53533087", "bits_per_raw_sample": "8", "nb_frames": "108", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0 }, "tags": { "creation_time": "2021-11-03T17:02:12.000000Z", "language": "und", "handler_name": "Video Media Handler", "encoder": "AVC Coding" } }, { "index": 1, "codec_name": "pcm_s16be", "codec_long_name": "PCM signed 16-bit big-endian", "codec_type": "audio", "codec_time_base": "1/48000", "codec_tag_string": "twos", "codec_tag": "0x736f7774", "sample_fmt": "s16", "sample_rate": "48000", "channels": 2, "bits_per_sample": 16, "r_frame_rate": "0/0", "avg_frame_rate": "0/0", "time_base": "1/48000", "start_pts": 0, "start_time": "0.000000", "duration_ts": 216240, "duration": "4.505000", "bit_rate": "1536000", "nb_frames": "216240", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0 }, "tags": { "creation_time": "2021-11-03T17:02:12.000000Z", "language": "und", "handler_name": "Sound Media Handler" } }, { "index": 2, "codec_type": "data", "codec_tag_string": "rtmd", "codec_tag": "0x646d7472", "r_frame_rate": "0/0", "avg_frame_rate": "0/0", "time_base": "1/24000", "start_pts": 0, "start_time": "0.000000", "duration_ts": 108108, "duration": "4.504500", "bit_rate": "196411", "nb_frames": "108", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0 }, "tags": { "creation_time": "2021-11-03T17:02:12.000000Z", "language": "und", "handler_name": "Timed Metadata Media Handler", "timecode": "07:28:14:16" } } ], "format": { "filename": "4K_24p_60M.mp4", "nb_streams": 3, "nb_programs": 0, "format_name": "mov,mp4,m4a,3gp,3g2,mj2", "format_long_name": "QuickTime / MOV", "start_time": "0.000000", "duration": "4.505000", "size": "33559173", "bit_rate": "59594535", "probe_score": 100, "tags": { "major_brand": "XAVC", "minor_version": "16785407", "compatible_brands": "XAVCmp42iso2", "creation_time": "2021-11-03T17:02:12.000000Z" } } } </span></pre> <p> The following attributes, which were of interest to me, had the same values for all the video clips I tested: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id41c44d51d9f7'><button class='copyBtn' data-clipboard-target='#id41c44d51d9f7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>streams[0].codec_name: h264 streams[0].codec_long_name: H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 streams[1].codec_name: pcm_s16be streams[1].codec_long_name: PCM signed 16-bit big-endian streams[1].sample_rate: 48000 format.format_long_name: QuickTime / MOV</pre> <p> In other words, the audio codec (<code>pcm_s16be</code>) and video format (<code>QuickTime / MOV</code>) were the same for all recording settings. The audio for the sample video clips was always recorded as 16-bits at 48 KHz, which is DVD quality. The only significant differences between the video clips are the bit rate of the H.264 codec and the video resolution. </p> <p> I wrote another bash script called <code>mediaSummary</code> that simply output the details of interest. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,mediaSummary' download='mediaSummary' title='Click on the file name to download the file'>mediaSummary</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="ida306d092d0ab">#!/bin/bash function help &#123; if [ "$1" ]; then printf "$1\n\n"; fi echo "$(basename $0) - Dump selected information about a media file Usage: $(basename $0) filename " exit 1 &#125; if [ -z "$1" ]; then help "Error: no media file name specified"; fi LC_NUMERIC=en_US.utf8 JSON="$( mediaDump "$1" )" STREAM0="$( jq '.streams[0]' &lt;&lt;&lt; "$JSON" )" codec_bit_rate0="$( jq -r .bit_rate &lt;&lt;&lt; "$STREAM0" )" codec_bit_rate0_formatted="$( printf "%'d" $codec_bit_rate0 )" echo "codec_bit_rate: $codec_bit_rate0_formatted"</pre> <h2 id="output">Media Formats Tested</h2> <p> I used the following menu sequence to set the camera to various media settings: </p> <h3 id="4k_settings">4K Video Settings</h3> <p> 4K video files are much larger than HD video files, and require significantly more processing power for post-production. </p> <p> MENU → Movie1 → File Format → XAVC S 4K<br> MENU → Movie1 → Record Setting → 30p 100M, 30p 60M, 24p 100M and 24p 60M. </p> <h3 id="4k_details">4K Video Format Details</h3> <p> All of the 4K formats had the same resolution: 3840 x 2160 pixels. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcf3c444b8fc1'><button class='copyBtn' data-clipboard-target='#idcf3c444b8fc1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mediaSummary 4K_24p_100M.mp4 codec_bit_rate: 95,404,278</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide0ee326b5a03'><button class='copyBtn' data-clipboard-target='#ide0ee326b5a03' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mediaSummary 4K_24p_60M.mp4 codec_bit_rate: 53,533,087</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id146e20d3eabf'><button class='copyBtn' data-clipboard-target='#id146e20d3eabf' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mediaSummary 4K_30p_100M.mp4 codec_bit_rate: 96,398,171</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc8edf503c5eb'><button class='copyBtn' data-clipboard-target='#idc8edf503c5eb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mediaSummary 4K_30p_60M.mp4 codec_bit_rate: 56,046,286</pre> <h3 id="hd">HD Video Settings</h3> <p> HD, also known as 1080p, is the most common video resolution on the internet today: 1920 x 1280 pixels. </p> <p> MENU → Movie1 → File Format → XAVC S HD<br> MENU → Movie1 → Record Setting → 60p 50M / 60p 25M, 30p 50M, 30p 16M, 24p 50M, 120p 100M and 120p 60M </p> <h3 id="hd_details">HD Video Format Details</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4536ca483799'><button class='copyBtn' data-clipboard-target='#id4536ca483799' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ mediaSummary HD_120p_100M.mp4 codec_bit_rate: 95,430,311</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4cc4496467da'><button class='copyBtn' data-clipboard-target='#id4cc4496467da' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ mediaSummary HD_120p_60M.mp4 codec_bit_rate: 56,137,995</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb9b8e8a20900'><button class='copyBtn' data-clipboard-target='#idb9b8e8a20900' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ mediaSummary HD_60p_50M.mp4 codec_bit_rate: 47,991,727</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id602ea25fe7c3'><button class='copyBtn' data-clipboard-target='#id602ea25fe7c3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ mediaSummary HD_60p_25M.mp4 codec_bit_rate: 24,176,270</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2598d17d0967'><button class='copyBtn' data-clipboard-target='#id2598d17d0967' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ mediaSummary HD_30p_50M.mp4 codec_bit_rate: 46,589,694</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id800391ae42ef'><button class='copyBtn' data-clipboard-target='#id800391ae42ef' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ mediaSummary HD_30p_16M.mp4 codec_bit_rate: 15,483,830</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5723157601af'><button class='copyBtn' data-clipboard-target='#id5723157601af' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ mediaSummary HD_24p_50M.mp4 codec_bit_rate: 47,529,910</pre> <h3 id="avchd">AVCHD Video Format</h3> <p class="warning"> Warning! <a href='https://www.easefab.com/avchd-tips/import-mts-to-davinci-resolve.html' target='_blank' rel='nofollow'>Davinci Resolve has issues with this format.</a> </p> <p> Because I use Davinci Resolve I did not test AVCHD. Wikipedia says: </p> <p class="quote"> Developed jointly by Sony and Panasonic, the format was introduced in 2006 primarily for use in high definition consumer camcorders.<br><br> For video compression, AVCHD uses the H.264/MPEG-4 AVC standard, supporting a variety of standard, high definition, and stereoscopic (3D) video resolutions. For audio compression, it supports both Dolby AC-3 (Dolby Digital) and uncompressed linear PCM audio. Stereo and multichannel surround (5.1) are both supported. <br><br> This format is compatible with Blu-ray. </p> <p> To enable AVCHD, use this menu sequence: </p> <p> MENU → Movie1 → File Format → AVCHD </p> <p> Jonny Elwyn has a good article entitled <a href='https://www.premiumbeat.com/blog/avchd-editing-workflow/' target='_blank' rel='nofollow'>Should Editors Transcode AVCHD to ProRes in Premiere?</a> </p> Disappointing Scala 3 Installation Experience 2021-05-19T00:00:00-04:00 https://mslinn.github.io/blog/2021/05/19/installing-scala-3.0 <p> I run <a href='https://scalacourses.com' target='_blank' rel='nofollow'>ScalaCourses.com</a>, an online Scala training web site. After years of hype, Scala 3 is now available. </p> <h2 id="prime">Scala 3: Not Yet Ready for Production</h2> <p> <a href='https://www.infoq.com/news/2021/03/scala3/' target='_blank' rel='nofollow'>Scala 3</a> was eight years in the making. You would never know that from the horrible installation process and the disappointing installation instructions. </p> <p> It is going to take quite a while before <a href='https://scalatimes.com/d374aea433' target='_blank' rel='nofollow'>Scala 3</a>, which was known as Dotty before it was released, can be trusted in production. According to the <a href='https://github.com/lampepfl/dotty' target='_blank' rel='nofollow'>Dotty GitHub project</a>, the only published future milestone is <a href='https://github.com/lampepfl/dotty/milestones' target='_blank' rel='nofollow'>v3.1.0, which has no due date</a>. Given that Scala 3 uses an entirely new build process, and an entirely new (and nonstandard) installation process, and that the internals of the Scala compiler were almost completely replaced, I doubt that version will be stable enough for use on production projects. </p> <h2 id="choices">Installation Choices</h2> <p> Installation cholices include: </p> <ul> <li>Command-line Scala has limited use cases, but is nice to have around for occassional experimentation.</li> <li> <a href='https://www.scala-lang.org/blog/2021/04/08/scala-3-in-sbt.html' target='_blank' rel='nofollow'>SBT</a> (which features an enhanced Scala REPL) is very helpful for interactively developing code, as well as for building and testing. </li> <li> <a href='https://www.jetbrains.com/help/idea/discover-intellij-idea-for-scala.html' target='_blank' rel='nofollow'>IntelliJ</a> provides the best Scala coding productivity and code quality. </li> <li> <a href='https://shunsvineyard.info/2020/11/20/setting-up-vs-code-for-scala-development-on-wsl/' target='_blank' rel='nofollow'>VSCode</a> has been playing catch-up but is not full-featured yet. </li> </ul> <h2 id="install">Installation Transcript</h2> <p> The following is the transcript of how I installed command-line Scala on Ubuntu 20.10 running under WSL2. </p> <h3 class="numbered" id="remove_scala2">Remove Scala 2</h3> <p> This step is not required. Scala 2 and Scala 3 can easily co-exist on the same system because their names are different. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id388187ac7934'><button class='copyBtn' data-clipboard-target='#id388187ac7934' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt remove scala <span class='unselectable'>Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be REMOVED: scala 0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded. After this operation, 666 MB disk space will be freed. Do you want to continue? [Y/n] (Reading database ... 236024 files and directories currently installed.) Removing scala (2.13.4-400) ... Processing triggers for man-db (2.9.3-2) ... </span></pre> <h3 class="numbered" id="cs">Install Coursier</h3> <p> Coursier is a multithreaded downloader for project dependencies, and it now also downloads Scala 3. SBT uses coursier internally. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id553a19b08f80'><button class='copyBtn' data-clipboard-target='#id553a19b08f80' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -fLo cs https://git.io/coursier-cli-"$(uname | tr LD ld)" <span class='unselectable'>&nbsp; % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 144 100 144 0 0 285 0 --:--:-- --:--:-- --:--:-- 6000 100 57.1M 100 57.1M 0 0 3656k 0 0:00:15 0:00:15 --:--:-- 4092k </span> <span class='unselectable'>$ </span>mv cs ~/.local/bin/ <span class='unselectable'>$ </span>chmod a+x ~/.local/bin/cs <span class='unselectable'>$ </span>cs install cs <span class='unselectable'>https://repo1.maven.org/maven2/io/get-coursier/apps/maven-metadata.xml 100.0% [##########] 1.8 KiB (8.5 KiB / s) https://repo1.maven.org/maven2/io/get-coursier/coursier-cli_2.12/maven-metadata.xml No new update since 2021-03-23 14:35:16 Wrote cs Warning: /home/mslinn/.local/share/coursier/bin is not in your PATH To fix that, add the following line to ~/.bashrc export PATH="$PATH:/home/mslinn/.local/share/coursier/bin" </span> <span class='unselectable'>$ </span> export PATH="$PATH:/home/mslinn/.local/share/coursier/bin"</pre> <p> Now let's test Coursier: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbf350f342f24'><button class='copyBtn' data-clipboard-target='#idbf350f342f24' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cs <span class='unselectable'>Coursier 2.0.16 Usage: cs [options] [command] [command-options] Available commands: bootstrap, channel, complete, fetch, get, install, java, java-home, launch, list, publish, resolve, setup, uninstall, update, search Type cs command --help for help on an individual command </span></pre> <p> This is the Coursier help message for the <code>install</code> subcommand: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida822f6f3008f'><button class='copyBtn' data-clipboard-target='#ida822f6f3008f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cs install --help <span class='unselectable'>Command: install Usage: cs install --cache &lt;string?&gt; Cache directory (defaults to environment variable COURSIER_CACHE, or ~/.cache/coursier/v1 on Linux and ~/Library/Caches/Coursier/v1 on Mac) --mode | -m &lt;offline|update-changing|update|missing|force&gt; Download mode (default: missing, that is fetch things missing from cache) --ttl | -l &lt;duration&gt; TTL duration (e.g. &quot;24 hours&quot;) --parallel | -n &lt;int&gt; Maximum number of parallel downloads (default: 6) --checksum &lt;checksum1,checksum2,...&gt; Checksum types to check - end with none to allow for no checksum validation if no checksum is available, example: SHA-256,SHA-1,none --retry-count &lt;int&gt; Retry limit for Checksum error when fetching a file --cache-file-artifacts | --cfa &lt;bool&gt; Flag that specifies if a local artifact should be cached. --follow-http-to-https-redirect &lt;bool&gt; Whether to follow http to https redirections --credentials &lt;host(realm) user:pass|host user:pass&gt; Credentials to be used when fetching metadata or artifacts. Specify multiple times to pass multiple credentials. Alternatively, use the COURSIER_CREDENTIALS environment variable --credential-file &lt;string*&gt; Path to credential files to read credentials from --use-env-credentials &lt;bool&gt; Whether to read credentials from COURSIER_CREDENTIALS (env) or coursier.credentials (Java property), along those passed with --credentials and --credential-file --quiet | -q &lt;counter&gt; Quiet output --verbose | -v &lt;counter&gt; Increase verbosity (specify several times to increase more) --progress | -P &lt;bool&gt; Force display of progress bars --log-changing &lt;bool&gt; Log changing artifacts --log-channel-version | --log-index-version | --log-jvm-index-version &lt;bool&gt; Log app channel or JVM index version --graalvm-home &lt;string?&gt; --graalvm-option &lt;string*&gt; --graalvm-default-version &lt;string?&gt; --install-dir | --dir &lt;string?&gt; --install-platform &lt;string?&gt; Platform for prebuilt binaries (e.g. &quot;x86_64-pc-linux&quot;, &quot;x86_64-apple-darwin&quot;, &quot;x86_64-pc-win32&quot;) --install-prefer-prebuilt &lt;bool&gt; --only-prebuilt &lt;bool&gt; Require prebuilt artifacts for native applications, don&#39;t try to build native executable ourselves --repository | -r &lt;maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|ivy:$pattern&gt; Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots) --default-repositories &lt;bool&gt; --proguarded &lt;bool?&gt; --channel &lt;org:name&gt; Channel for apps --default-channels &lt;bool&gt; Add default channels --contrib &lt;bool&gt; Add contrib channel --file-channels &lt;bool&gt; Add channels read from the configuration directory --jvm &lt;string?&gt; --jvm-dir &lt;string?&gt; --system-jvm &lt;bool?&gt; --local-only &lt;bool&gt; --update &lt;bool&gt; --jvm-index &lt;string?&gt; --repository | -r &lt;maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|scala-integration|scala-nightlies|ivy:$pattern|jitpack|clojars|jcenter|apache:$repo&gt; Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots) --no-default &lt;bool&gt; Do not add default repositories (~/.ivy2/local, and Central) --sbt-plugin-hack &lt;bool&gt; Modify names in Maven repository paths for sbt plugins --drop-info-attr &lt;bool&gt; Drop module attributes starting with &#39;info.&#39; - these are sometimes used by projects built with sbt --channel &lt;org:name&gt; Channel for apps --default-channels &lt;bool&gt; Add default channels --contrib &lt;bool&gt; Add contrib channel --file-channels &lt;bool&gt; Add channels read from the configuration directory --repository | -r &lt;maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|scala-integration|scala-nightlies|ivy:$pattern|jitpack|clojars|jcenter|apache:$repo&gt; Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots) --no-default &lt;bool&gt; Do not add default repositories (~/.ivy2/local, and Central) --sbt-plugin-hack &lt;bool&gt; Modify names in Maven repository paths for sbt plugins --drop-info-attr &lt;bool&gt; Drop module attributes starting with &#39;info.&#39; - these are sometimes used by projects built with sbt --channel &lt;org:name&gt; Channel for apps --default-channels &lt;bool&gt; Add default channels --contrib &lt;bool&gt; Add contrib channel --file-channels &lt;bool&gt; Add channels read from the configuration directory --env &lt;bool&gt; --disable-env | --disable &lt;bool&gt; --setup &lt;bool&gt; --user-home &lt;string?&gt; --add-channel &lt;string*&gt; (deprecated) --force | -f &lt;bool&gt; </span></pre> <h3 class="numbered" id="s3">Install Scala 3</h3> <p> The Scala 3 compiler and REPL are separate programs: <code>scala3-compiler</code> and <code>scala3-repl</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id53d44dc7aeca'><button class='copyBtn' data-clipboard-target='#id53d44dc7aeca' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cs install scala3-compiler <span class='unselectable'>https://repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.0.0/scala3-compiler_3-3.0.0.pom 100.0% [##########] 4.8 KiB (79.7 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.0.0/tasty-core_3-3.0.0.pom 100.0% [##########] 3.5 KiB (69.5 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.0.0/scala3-library_3-3.0.0.pom 100.0% [##########] 3.6 KiB (53.9 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.0.0/scala3-interfaces-3.0.0.pom 100.0% [##########] 3.4 KiB (65.9 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.0.0/scala3-interfaces-3.0.0.jar 100.0% [##########] 3.4 KiB (113.9 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.0.0/tasty-core_3-3.0.0.jar 100.0% [##########] 71.9 KiB (192.7 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.0.0/scala3-library_3-3.0.0.jar 100.0% [##########] 1.1 MiB (1.8 MiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.0.0/scala3-compiler_3-3.0.0.jar 100.0% [##########] 14.7 MiB (3.7 MiB / s) Wrote scala3-compiler </span> <span class='unselectable'>$ </span>cs install scala3-repl <span class='unselectable'>https://repo1.maven.org/maven2/io/get-coursier/apps/maven-metadata.xml No new update since 2021-05-14 04:42:19 Wrote scala3-repl </span></pre> <h3 class="numbered" id="repl3">Run Scala 3 REPL</h3> <p> The part is easy! </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddd8c7b24a5b7'><button class='copyBtn' data-clipboard-target='#iddd8c7b24a5b7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>scala3-repl --version <span class='unselectable'>Scala code runner version 3.0.0 -- Copyright 2002-2021, LAMP/EPFL </span> <span class='unselectable'>$ </span>scala3-repl <span class='unselectable'>scala&gt; </span></pre> <h2 id="sbt">Easily Run Scala REPL With SBT</h2> <p> If you do not mind directories called <code>project/</code> and <code>target/</code> being created in your current directory, and you have already <a href='https://www.scala-sbt.org/download.html' target='_blank' rel='nofollow'>installed sbt</a>, you can get a REPL powered by Scala 3 like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide32c50cb5337'><button class='copyBtn' data-clipboard-target='#ide32c50cb5337' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sbt "-Dsbt.version=1.5.2" ++3.0.0! console <span class='unselectable'>[info] welcome to sbt 1.5.2 (Ubuntu Java 11.0.11) [info] loading global plugins from /home/mslinn/.sbt/1.0/plugins [info] loading project definition from /var/work/ancientWarmth/ancientWarmth/project [info] set current project to ancientwarmth (in build file:/var/work/ancientWarmth/ancientWarmth/) [info] Forcing Scala version to 3.0.0 on all projects. [info] Reapplying settings... [info] set current project to ancientwarmth (in build file:/var/work/ancientWarmth/ancientWarmth/) [info] Updating [info] Resolved dependencies [info] Updating https://repo1.maven.org/maven2/org/scala-lang/scaladoc_3/3.0.0/scaladoc_3-3.0.0.pom 100.0% [##########] 6.1 KiB (82.6 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-tasty-inspector_3/3.0.0/scala3-tasty-inspector_3-3.0.0.pom 100.0% [##########] 3.6 KiB (80.8 KiB / s) [info] Resolved dependencies [info] Fetching artifacts of [info] Fetched artifacts of [info] Fetching artifacts of https://repo1.maven.org/maven2/org/scala-lang/scala3-tasty-inspector_3/3.0.0/scala3-tasty-inspector_3-3.0.0.jar 100.0% [##########] 16.6 KiB (338.1 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scaladoc_3/3.0.0/scaladoc_3-3.0.0.jar 100.0% [##########] 1.5 MiB (3.1 MiB / s) [info] Fetched artifacts of scala&gt; </span></pre> <p> Thanks to <a href='https://twitter.com/renghenKornel/status/1395684928440791040' target='_blank' rel='nofollow'>@renghen</a> for this tip. </p> <h2 id="sc">ScalaCourses</h2> <p> If you want to learn how to work effectively with Scala for functional and object-oriented programming, <a href='https://scalacourses.com' target='_blank' rel='nofollow'>ScalaCourses.com</a> is your best option. The course material is suitable for Scala 2 and Scala 3. Visit ScalaCourses.com to learn how to become a proficient Scala programmer. </p> OCI / Docker / AWS Lambda / Django / Buildah / podman 2021-04-29T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/29/buildah-podman-python-lambda <style> body { counter-reset: pcounter; } p.count:before { counter-increment: pcounter; content: counter(pcounter) ")\A0"; } </style> <editor-fold goal> <p> This blog post is a work in progress. Some of it may be incorrect, and some thoughts might lead nowhere. I am publicly posting it in this state so I can discuss it with others. This post will be improved as information becomes available. </p> <h2 id="goal">Goal</h2> <div style="text-align: right;"> <a href="https://podman.io" target="_blank" ><picture> <source srcset="/blog/images/buildahPodman/podman-logo-crop.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/podman-logo-crop.png" type="image/png"> <img src="/blog/images/buildahPodman/podman-logo-crop.png" class="right liImg2 rounded shadow" style="padding: 1em; height: 191px; width: auto;" /> </picture></a> </div> <div style="text-align: right;"> <a href="https://buildah.io/" target="_blank" ><picture> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.png" type="image/png"> <img src="/blog/images/buildahPodman/buildah-logo-crop.png" class="right liImg2 rounded shadow" style="padding: 0.73em; height: 191px; width: auto;" /> </picture></a> </div> <p> <a href='/blog/2021/04/28/buildah-podman.html'>As previously discussed</a>, Buildah is a drop-in replacement for using <code>docker build</code> and a <code>Dockerfile</code>. Buildah’s <code>build-using-dockerfile</code>, or <code>bud</code> argument makes it behave just like <code>docker build</code> does. </p> <p> The goal of this blog post is to use Buildah / <code>podman</code> to create an Open Container Initiative (OCI) container image with a Django app, including the Python 3.8 runtime installed. The Django app will start when the container is created. The code for the Django app will be stored on the local machine where its source code can be edited, and it will be mapped into the container from the host system. Changes made to the code from the host system will be immediately visible inside the container. </p> <h2 id="todo">TODO</h2> <p class="count"> Background: AWS publishes <a href='https://docs.aws.amazon.com/lambda/latest/dg/python-image.html' target='_blank' rel='nofollow'>Deploying Python with an AWS base image</a>, but that does not discuss running or testing. <a href='https://docs.aws.amazon.com/lambda/latest/dg/getting-started-create-function.html' target='_blank' rel='nofollow'>Create a Lambda function with the console</a> is a more complete article, but is focused on using the web browser console, using Docker, and Node.js. So many differences from the desired goal make the articles difficult to translate to AWS CLI, Buildah / <code>podman</code> and Python. </p> <p class="count"> Talk about the <a href='https://github.com/aws/aws-lambda-runtime-interface-emulator' target='_blank' rel='nofollow'>AWS Lambda Runtime Interface Emulator</a>, compare and contrast with the <a href='https://pypi.org/project/awslambdaric/' target='_blank' rel='nofollow'>AWS Lambda Python Runtime Interface Client</a>. </p> <p class="count"> Compare these <a href='https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html' target='_blank' rel='nofollow'>AWS Lambda Runtimes</a> with other, equivalant runtimes. </p> <p class="count"> OCI images are swapped in when AWS Lambda is invoked. Do larger images cost more to use? If so, discuss. </p> </editor-fold> <editor-fold main> <h2 id="main">Deploy Python Lambda function with Container Image</h2> <p> Consider this <code>Dockerfile</code>, which launches a Python 3.8 command-line application in a manner compatible with AWS Lambda: </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,Dockerfile' download='Dockerfile' title='Click on the file name to download the file'>Dockerfile</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idcc2ab4941de1">FROM public.ecr.aws/lambda/python:3.8 COPY app.py ./ CMD ["app.handler"] </pre> <p> Following is a small Python app called <code>app.py</code>, which will be launched by the <code>Dockerfile</code>. The Python app can be run as an AWS Lambda program because it implements the <code>handler</code> entry point. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,app.py' download='app.py' title='Click on the file name to download the file'>app.py</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="ida897dc388539">import sys def handler(event, context): return f"Hello from AWS Lambda using Python &#123;sys.version&#125;!" </pre> </editor-fold> <editor-fold build_hello> <h2 id="build">Build image</h2> <p> Buildah builds the image, just the same way that Docker would: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfa2582efdbb5'><button class='copyBtn' data-clipboard-target='#idfa2582efdbb5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah bud -t hello . <span class='unselectable'>STEP 1: FROM public.ecr.aws/lambda/python:3.8 Getting image source signatures Copying blob 03ac043af787 skipped: already exists Copying blob 420e64b38334 done Copying blob ff259f25b075 done Copying blob 3ff716981d54 done Copying blob 6b6e623a48a8 done Copying blob 9aa8f1e66d54 done Copying config 67dc3a2a54 done Writing manifest to image destination Storing signatures STEP 2: COPY app.py ./ STEP 3: CMD ["app.handler"] STEP 4: COMMIT hello Getting image source signatures Copying blob 683073d39306 skipped: already exists Copying blob 658871a69e1f skipped: already exists Copying blob 6fa16f35d11e skipped: already exists Copying blob d6fa53d6caa6 skipped: already exists Copying blob 61c062506436 skipped: already exists Copying blob 1c1d66a5fd95 skipped: already exists Copying blob 33af9dc6463a done Copying config 98862dfd20 done Writing manifest to image destination Storing signatures --&gt; 98862dfd208 98862dfd2087152ee821553d6cb1c033e735af06e5f11c814bcc9300fb65584e </span></pre> </editor-fold> <editor-fold deploy_local> <h2 id="deploy_local">Test Lambda function Locally</h2> <p> Before calling the Lambda API from a local container, first run the container. Containers default to running in the foreground, but the <code>-d</code> option causes a container to be run as a background process. This container is given the name <code>hello</code>, the external HTTP endpoint at 9000 is mapped to internal port 8080, and the latest version of the <code>hello</code> lambda function is run in the container. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idef03bec0edf1'><button class='copyBtn' data-clipboard-target='#idef03bec0edf1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman run \ -d \ --name hello \ -p 9000:8080 \ hello:latest <span class='unselectable'>d4d296e4c91d01c98d312e3f79599dca53990d95218e94bbdfbbac6a43cde9e8 </span></pre> <p> Call the local version of the Lambda API: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6a9ac373ebb4'><button class='copyBtn' data-clipboard-target='#id6a9ac373ebb4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl \ -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" \ -d '{}' <span class='unselectable'>"Hello from AWS Lambda using Python 3.8.9 (default, Apr 20 2021, 13:58:54) \n[GCC 7.3.1 20180712 (Red Hat 7.3.1-12)]!" </span></pre> <p> Stop the container called <code>hello</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc44fbccdabcd'><button class='copyBtn' data-clipboard-target='#idc44fbccdabcd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman stop hello <span class='unselectable'>96cc1b1ed92368a1165d6a6ad0b1e5544d4ac751b64e94df33bf2322e6d7b30c </span></pre> </editor-fold> <editor-fold create_repo> <h2 id="podman_tag">Create AWS ECR Repository</h2> <p> AWS provides a registry for OCI-compatible image repositories called the <a href='https://aws.amazon.com/ecr/' target='_blank' rel='nofollow'>AWS Elastic Container Registry (ECR)</a>. </p> <editor-fold create_repo_help> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide7a99b73f548'><button class='copyBtn' data-clipboard-target='#ide7a99b73f548' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ecr create-repository help CREATE-REPOSITORY() CREATE-REPOSITORY()<br/> NAME create-repository -<br/> DESCRIPTION Creates a repository. For more information, see Amazon ECR Repositories in the Amazon Elastic Container Registry User Guide .<br/> See also: AWS API Documentation<br/> See &#39;aws help&#39; for descriptions of global parameters.<br/> SYNOPSIS create-repository --repository-name &lt;value&gt; [--tags &lt;value&gt;] [--image-tag-mutability &lt;value&gt;] [--image-scanning-configuration &lt;value&gt;] [--cli-input-json &lt;value&gt;] [--generate-cli-skeleton &lt;value&gt;]<br/> OPTIONS --repository-name (string) The name to use for the repository. The repository name may be spec- ified on its own (such as nginx-web-app ) or it can be prepended with a namespace to group the repository into a category (such as project-a/nginx-web-app ).<br/> --tags (list) The metadata that you apply to the repository to help you categorize and organize them. Each tag consists of a key and an optional value, both of which you define. Tag keys can have a maximum character length of 128 characters, and tag values can have a maximum length of 256 characters.<br/> (structure) The metadata that you apply to a resource to help you categorize and organize them. Each tag consists of a key and an optional value, both of which you define. Tag keys can have a maximum character length of 128 characters, and tag values can have a maximum length of 256 characters.<br/> Key -&gt; (string) One part of a key-value pair that make up a tag. A key is a general label that acts like a category for more specific tag values.<br/> Value -&gt; (string) The optional part of a key-value pair that make up a tag. A value acts as a descriptor within a tag category (key).<br/> Shorthand Syntax:<br/> Key=string,Value=string ...<br/> JSON Syntax:<br/> [ { &quot;Key&quot;: &quot;string&quot;, &quot;Value&quot;: &quot;string&quot; } ... ]<br/> --image-tag-mutability (string) The tag mutability setting for the repository. If this parameter is omitted, the default setting of MUTABLE will be used which will al- low image tags to be overwritten. If IMMUTABLE is specified, all im- age tags within the repository will be immutable which will prevent them from being overwritten.<br/> Possible values:<br/> o MUTABLE<br/> o IMMUTABLE<br/> --image-scanning-configuration (structure) The image scanning configuration for the repository. This setting determines whether images are scanned for known vulnerabilities af- ter being pushed to the repository.<br/> scanOnPush -&gt; (boolean) The setting that determines whether images are scanned after be- ing pushed to a repository. If set to true , images will be scanned after being pushed. If this parameter is not specified, it will default to false and images will not be scanned unless a scan is manually started with the StartImageScan API.<br/> Shorthand Syntax:<br/> scanOnPush=boolean<br/> JSON Syntax:<br/> { &quot;scanOnPush&quot;: true|false }<br/> --cli-input-json (string) Performs service operation based on the JSON string provided. The JSON string follows the format provided by --gen- erate-cli-skeleton. If other arguments are provided on the command line, the CLI values will override the JSON-provided values. It is not possible to pass arbitrary binary values using a JSON-provided value as the string will be taken literally.<br/> --generate-cli-skeleton (string) Prints a JSON skeleton to standard output without sending an API request. If provided with no value or the value input, prints a sample input JSON that can be used as an argument for --cli-input-json. If provided with the value output, it validates the command inputs and returns a sample output JSON for that command.<br/> See &#39;aws help&#39; for descriptions of global parameters.<br/> EXAMPLES Example 1: To create a repository<br/> The following create-repository example creates a repository inside the specified namespace in the default registry for an account.<br/> aws ecr create-repository \ --repository-name project-a/nginx-web-app<br/> Output:<br/> { &quot;repository&quot;: { &quot;registryId&quot;: &quot;123456789012&quot;, &quot;repositoryName&quot;: &quot;sample-repo&quot;, &quot;repositoryArn&quot;: &quot;arn:aws:ecr:us-west-2:123456789012:repository/project-a/nginx-web-app&quot; } }<br/> For more information, see Creating a Repository in the Amazon ECR User Guide.<br/> Example 2: To create a repository configured with image tag immutabil- ity<br/> The following create-repository example creates a repository configured for tag immutability in the default registry for an account.<br/> aws ecr create-repository \ --repository-name sample-repo \ --image-tag-mutability IMMUTABLE<br/> Output:<br/> { &quot;repository&quot;: { &quot;registryId&quot;: &quot;123456789012&quot;, &quot;repositoryName&quot;: &quot;sample-repo&quot;, &quot;repositoryArn&quot;: &quot;arn:aws:ecr:us-west-2:123456789012:repository/sample-repo&quot;, &quot;imageTagMutability&quot;: &quot;IMMUTABLE&quot; } }<br/> For more information, see Image Tag Mutability in the Amazon ECR User Guide.<br/> Example 3: To create a repository configured with a scanning configura- tion<br/> The following create-repository example creates a repository configured to perform a vulnerability scan on image push in the default registry for an account.<br/> aws ecr create-repository \ --repository-name sample-repo \ --image-scanning-configuration scanOnPush=true<br/> Output:<br/> { &quot;repository&quot;: { &quot;registryId&quot;: &quot;123456789012&quot;, &quot;repositoryName&quot;: &quot;sample-repo&quot;, &quot;repositoryArn&quot;: &quot;arn:aws:ecr:us-west-2:123456789012:repository/sample-repo&quot;, &quot;imageScanningConfiguration&quot;: { &quot;scanOnPush&quot;: true } } }<br/> For more information, see Image Scanning in the Amazon ECR User Guide.<br/> OUTPUT repository -&gt; (structure) The repository that was created.<br/> repositoryArn -&gt; (string) The Amazon Resource Name (ARN) that identifies the repository. The ARN contains the arn:aws:ecr namespace, followed by the re- gion of the repository, AWS account ID of the repository owner, repository namespace, and repository name. For example, arn:aws:ecr:region:012345678910:repository/test .<br/> registryId -&gt; (string) The AWS account ID associated with the registry that contains the repository.<br/> repositoryName -&gt; (string) The name of the repository.<br/> repositoryUri -&gt; (string) The URI for the repository. You can use this URI for Docker push or pull operations.<br/> createdAt -&gt; (timestamp) The date and time, in JavaScript date format, when the reposi- tory was created.<br/> imageTagMutability -&gt; (string) The tag mutability setting for the repository.<br/> imageScanningConfiguration -&gt; (structure) The image scanning configuration for a repository.<br/> scanOnPush -&gt; (boolean) The setting that determines whether images are scanned after being pushed to a repository. If set to true , images will be scanned after being pushed. If this parameter is not speci- fied, it will default to false and images will not be scanned unless a scan is manually started with the StartImageScan API.<br/> <br/> CREATE-REPOSITORY()</pre> </editor-fold> <p> The following creates an AWS ECR image repository in called <code>hello</code> within the <code>test</code> namespace. <a href='https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-scanning.html' target='_blank' rel='nofollow'>Images are scanned</a> for known vulnerabilities after they are pushed to the repository. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbc9af3f2e1d9'><button class='copyBtn' data-clipboard-target='#idbc9af3f2e1d9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ecr create-repository \ --repository-name test/hello \ --image-scanning-configuration scanOnPush=true <span class='unselectable'>{ "repository": { "repositoryArn": "arn:aws:ecr:us-east-1:031372724784:repository/test/hello", "registryId": "031372724784", "repositoryName": "test/hello", "repositoryUri": "031372724784.dkr.ecr.us-east-1.amazonaws.com/test/hello", "createdAt": 1620232146.0, "imageTagMutability": "MUTABLE", "imageScanningConfiguration": { "scanOnPush": true } } } </span></pre> </editor-fold> <editor-fold podman_tag> <h2 id="podman_tag">Tag Image</h2> <p class="quote"> <b><code>podman tag</code></b> &ndash; Assigns a new image name to an existing image. A full name refers to the entire image name, including the optional tag after the <code>:</code>. If there is no tag provided, then podman will default to latest for both the image and the target-name. &nbsp; &ndash; From <code>man podman-tag</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida70407635400'><button class='copyBtn' data-clipboard-target='#ida70407635400' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>IMAGE_NAME=hello <span class='unselectable'>$ </span>IMAGE_VERSION=0.1 <span class='unselectable'>$ </span>podman tag $IMAGE_NAME:$IMAGE_VERSION \ $REGISTRY/$IMAGE_NAME:$IMAGE_VERSION <span class='unselectable'>$ </span>podman images <span class='unselectable'>REPOSITORY TAG IMAGE ID CREATED SIZE localhost/hello 0.1 98862dfd2087 39 minutes ago 622 MB 752246127823.dkr.ecr.us-east-1.amazonaws.com/hello latest 98862dfd2087 39 minutes ago 622 MB public.ecr.aws/lambda/python 3.8 67dc3a2a54fb 25 hours ago 622 MB 752246127823.dkr.ecr.us-east-1.amazonaws.com/ancientwarmth latest 5d18ea34fc30 28 hours ago 2.03 GB localhost/ancientwarmth latest 5d18ea34fc30 28 hours ago 2.03 GB &lt;none&gt; &lt;none&gt; 40ef32b39cf4 5 days ago 622 MB docker.io/library/amazonlinux latest 53ef897d731f 5 days ago 170 MB docker.io/amazon/aws-lambda-python 3.8 e12ea62c5582 9 days ago 622 MB docker.io/library/alpine latest 6dbb9cc54074 2 weeks ago 5.88 MB docker.io/lambci/lambda build-python3.8 714c659c9f6f 3 months ago 2.03 GB </span></pre> </editor-fold> <editor-fold push_ecr> <h2 id="push">Push Image to ECR</h2> <p> <code>Podman</code> will use the IAM credentials for the <code>dev</code> <a href='https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html' target='_blank' rel='nofollow'>profile</a> in <code>~/.aws/credentials</code> to log into that AWS account: </p> <div class='codeLabel unselectable' data-lt-active='false'>~/.aws/credentials</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb1a67f5ee5e2'><button class='copyBtn' data-clipboard-target='#idb1a67f5ee5e2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[default] aws_access_key_id = ******************** aws_secret_access_key = **************************************** region = us-east-1<br> [dev] aws_access_key_id = ******************** aws_secret_access_key = **************************************** region = us-east-1</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id259215752f94'><button class='copyBtn' data-clipboard-target='#id259215752f94' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>export AWS_PROFILE=dev <span class='unselectable'>$ </span>AWS_ACCOUNT="$( aws sts get-caller-identity \ --query Account \ --output text )" <span class='unselectable'>$ </span>AWS_REGION="$( aws configure get region )" <span class='unselectable'>$ </span>REGISTRY="$AWS_ACCOUNT.dkr.ecr.$AWS_REGION.amazonaws.com" <span class='unselectable'>$ </span>aws ecr get-login-password \ --region "$AWS_REGION" | \ podman login \ --password-stdin \ --username AWS \ "$REGISTRY" <span class='unselectable'>Login Succeeded! </span></pre> <p> Now that <code>podman</code> is logged into AWS, use <code>podman</code> push the image to AWS ECR: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idecb6928e614a'><button class='copyBtn' data-clipboard-target='#idecb6928e614a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman push test/$IMAGE_NAME \ $REGISTRY/$IMAGE_NAME:$IMAGE_VERSION <span class='unselectable'>Getting image source signatures Copying blob 692590faf2d1 [--------------------------------------] 8.0b / 8.2MiB Copying blob 397718cff58d [--------------------------------------] 8.0b / 206.2MiB Copying blob 9ca787b1c91c [--------------------------------------] 8.0b / 93.1MiB Copying blob ef26f5221b79 [--------------------------------------] 8.0b / 196.7MiB Copying blob 0a3f69c27a89 [--------------------------------------] 8.0b / 316.4MiB Copying blob 5b3cbb76df75 [--------------------------------------] 8.0b / 1.1GiB Copying blob e9cad39831b0 [--------------------------------------] 8.0b / 3.5KiB Error: Error copying image to the remote destination: Error writing blob: Error initiating layer upload to /v2/ancientwarmth/blobs/uploads/ in 752246127823.dkr.ecr.us-east-1.amazonaws.com: name unknown: The repository with name 'hello' does not exist in the registry with id '752246127823' </span></pre> <p> The results of an <a href='https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ecr/describe-image-scan-findings.html' target='_blank' rel='nofollow'>image scan</a> for the new repository can be retrieved as follows: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide6377c35f98b'><button class='copyBtn' data-clipboard-target='#ide6377c35f98b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ecr describe-image-scan-findings \ --repository-name test/hello \ --image-id imageTag=tag_name</pre> </editor-fold> <editor-fold aw> </editor-fold> <editor-fold buildah_python> <h2 id="buildah_python">Deploy Python Lambda function with Container Image</h2> <p> <code>Podman</code> can invoke the app using an OCI container with Amazon Linux 2 and Python 3.8: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida2c1162d9571'><button class='copyBtn' data-clipboard-target='#ida2c1162d9571' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman container run -ti \ public.ecr.aws/lambda/python:3.8 \ blog/docker/podman/app.py <span class='unselectable'>Trying to pull public.ecr.aws/lambda/python:3.8... Getting image source signatures Copying blob 1de4740de1c2 done Copying blob 03ac043af787 done Copying blob 2e2bb77ae2dc done Copying blob 842c9dce67e8 done Copying blob df513d38f4d9 done Copying blob 031c6369fb2b done Copying config e12ea62c55 done Writing manifest to image destination Storing signatures time="2021-05-02T23:38:30.971" level=info msg="exec '/var/runtime/bootstrap' (cwd=/var/task, handler=)" </span></pre> </editor-fold> Docker, OCI Images, Buildah and podman 2021-04-28T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/28/buildah-podman <editor-fold intro> <p> There are many ways to create and run Docker-compatible images. Docker is probably the worst option, mostly because it runs as a daemon, and all *nix daemons run with <code>root</code> privileges. Also, the <code>docker-ce</code> package lists <code>iptables</code> as a dependency, which needs <code>systemd</code> to be running normally, and WSL2 only partially supports <code>systemd</code>. </p> <p> <a href='https://www.capitalone.com/tech/cloud/container-runtime/' target='_blank' rel='nofollow'>A Comprehensive Container Runtime Comparison</a> provides helpful background information and an interesting historical viewpoint. </p> <h2 id="oci">Open Container Initiative (OCI)</h2> <div style=""> <a href="https://opencontainers.org/" target="_blank" ><picture> <source srcset="/blog/images/buildahPodman/oci_logo.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/oci_logo.png" type="image/png"> <img src="/blog/images/buildahPodman/oci_logo.png" class=" fullsize liImg2 rounded shadow" /> </picture></a> </div> <p> The latest evolution of Docker-compatible images, <a href='https://github.com/opencontainers/image-spec' target='_blank' rel='nofollow'>OCI image format</a> (not to be confused with <a href='https://www.oracle.com/ca-en/cloud/' target='_blank' rel='nofollow'>Oracle Cloud Infrastructure</a>), is compatible with: </p> <ul style="column-count: 2;"> <li><a href='https://aws.amazon.com/lambda/' target='_blank' rel='nofollow'>AWS Lambda</a></li> <li><a href='https://azure.microsoft.com/en-us/services/functions/' target='_blank' rel='nofollow'>Azure Functions</a></li> <li><a href='https://azure.microsoft.com/en-us/services/kubernetes-service/' target='_blank' rel='nofollow'>Azure Kubernetes Service</a></li> <li><a href='https://buildah.io/' target='_blank' rel='nofollow'>Buildah</a></li> <li><a href='https://buildpacks.io/' target='_blank' rel='nofollow'>Cloud Native Buildpacks</a></li> <li><a href='https://circleci.com/' target='_blank' rel='nofollow'>CircleCI</a></li> <li><a href='https://www.docker.com/' target='_blank' rel='nofollow'>Docker</a></li> <li><a href='https://dokku.com/' target='_blank' rel='nofollow'>Dokku</a></li> <li><a href='https://gitlab.com' target='_blank' rel='nofollow'>GitLab</a></li> <li><a href='https://cloud.google.com/container-registry/docs/image-formats' target='_blank' rel='nofollow'>Google Cloud</a></li> <li><a href='https://heroku.com' target='_blank' rel='nofollow'>Heroku</a></li> <li><a href='https://containerjournal.com/topics/container-management/what-is-knative-and-what-can-it-do-for-you/' target='_blank' rel='nofollow'>Knative</a></li> <li><a href='https://kubernetes.io/' target='_blank' rel='nofollow'>Kubernetes</a></li> <li><a href='https://podman.io/' target='_blank' rel='nofollow'><code>podman</code></a></li> <li><a href='https://github.com/containers/skopeo' target='_blank' rel='nofollow'><code>skopeo</code></a></li> <li><a href='https://spring.io/guides/topicals/spring-boot-docker/' target='_blank' rel='nofollow'>Spring Boot</a></li> <li><a href='https://cloud.google.com/tekton' target='_blank' rel='nofollow'>Tekton</a></li> </ul> <p> Supported OCI formats include: </p> <ul style="column-count: 2;"> <li>Docker containers schema 1</li> <li>Docker containers schema 2</li> <li>Pods (groups of containers)</li> <li>Images</li> <li>Volumes</li> </ul> <h2 id="three">Buildah, podman and skopeo</h2> <p> This blog post discusses 3 related open source projects from RedHat / IBM that provide an alternative to Docker: Buildah, <code>podman</code> and <code>skopeo</code>. These 3 projects share a common source code base, and are daemonless tools for managing Open Container Initiative (OCI) images. </p> <p> Paraphrasing the reasons expressed in <a href='https://developers.redhat.com/blog/2019/02/21/podman-and-buildah-for-docker-users/' target='_blank' rel='nofollow'>Podman and Buildah for Docker Users</a> for using <code>podman</code> instead of Docker, wherever <code>podman</code> is mentioned, read &ldquo;<code>podman</code>, Buildah and <code>skopeo</code>&rdquo;: </p> <p class="quoteCite" cite="From &ldquo;Podman and Buildah for Docker Users&rdquo;"> The Podman approach is simply to directly interact with the image registry, with the container and image storage, and with the Linux kernel through the <code>runC</code> container runtime process (not a daemon).<br><br> Running Podman as a normal user means that Podman will, by default, store images and containers in the user’s home directory. Podman uses a repository in the user’s home directory: <code>~/.local/share/containers</code> (instead of <code>/var/lib/docker</code>).<br><br> Despite the new locations for the local repositories, the images created by Docker and Podman are compatible with the OCI standard. Podman can push to and pull from popular container registries like Quay.io and Docker hub, as well as private registries. </p> <h2 id="buildah_vs_podman">Buildah vs. podman</h2> <p> <code>Podman</code> can build OCI containers interactively or in batch mode. You can either build using a <code>Dockerfile</code> using <code>podman build</code> (batch mode), or you can interactively run a container, make changes to the running image, and then <code>podman commit</code> those changes to a new image tag. </p> <p> Buildah was written before <code>podman</code>. Some of Buildah's source code for creating and managing container images was ported to <code>podman</code>. The <code>podman build</code> command is a subset of Buildah&rsquo;s functionality. </p> <p> <p> However, apparently the differences between the two programs are important: </p> <p class="quote"> Buildah builds OCI images. Confusingly, <code>podman build</code> can also be used to build Docker images also, but it’s incredibly slow and used up a lot of disk space by using the <code>vfs</code> storage driver by default. <code>buildah bud</code> (‘build using Dockerfile’) was much faster for me, and uses the overlay storage driver. <br><br> &nbsp; &ndash; From <a href='https://zwischenzugs.com/page/3/' target='_blank' rel='nofollow'>Goodbye Docker: Purging is Such Sweet Sorrow</a> by Ian Miell. </p> </editor-fold> <editor-fold podman> <h2 id="podman">podman</h2> <div style=""> <a href="https://podman.io" target="_blank" ><picture> <source srcset="/blog/images/buildahPodman/podman-logo.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/podman-logo.png" type="image/png"> <img src="/blog/images/buildahPodman/podman-logo.png" class=" liImg2 rounded shadow" style="padding: 1em" /> </picture></a> </div> <p> <code>Podman</code> supports developing, managing, and running OCI Containers on Linux systems, including WSL, without requiring <code>root</code> privilege. </p> <div class='codeLabel unselectable' data-lt-active='false'>shell Installation on Ubuntu</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbefd8d860f58'><button class='copyBtn' data-clipboard-target='#idbefd8d860f58' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install buildah podman skopeo</pre> <div class="pullQuote"> Podman commands are very nearly the same as Docker’s. </div> <p> Because <code>podman</code> is a drop-in replacement for <code>docker</code>, the following alias enables the <code>docker</code> command to invoke <code>podman</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>~/.bash_aliases</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id535760111908'><button class='copyBtn' data-clipboard-target='#id535760111908' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>alias docker=podman</pre> <p> As described in <a href='https://www.vultr.com/docs/how-to-install-and-use-podman-on-ubuntu-20-04' target='_blank' rel='nofollow'>How to Install and Use Podman on Ubuntu 20.04</a>, I added <code>'registry.access.redhat.com'</code> to the list of <code>registries</code> in <code>/etc/containers/registries.conf</code>. I also added <a href='https://gallery.ecr.aws/' target='_blank' rel='nofollow'><code>'gallery.ecr.aws'</code></a> and <a href='https://cloud.google.com/container-registry/docs/pushing-and-pulling#add-registry' target='_blank' rel='nofollow'><code>'gcr.io'</code></a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/containers/registries.conf</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7d7658381ac5'><button class='copyBtn' data-clipboard-target='#id7d7658381ac5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[registries.search] registries = ['docker.io', 'gallery.ecr.aws', 'gcr.io', 'quay.io', 'registry.access.redhat.com']</pre> </editor-fold> <editor-fold help> <h3 id="podmanHelp"><span class="code">podman</span> Help</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd21e12a03534'><button class='copyBtn' data-clipboard-target='#idd21e12a03534' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman --help <span class='unselectable'>Manage pods, containers and images<br/> Usage: podman [flags] podman [command]<br/> Available Commands: attach Attach to a running container auto-update Auto update containers according to their auto-update policy build Build an image using instructions from Containerfiles commit Create new image based on the changed container container Manage containers cp Copy files/folders between a container and the local filesystem create Create but do not start a container diff Display the changes to the object&#39;s file system events Show podman events exec Run a process in a running container export Export container&#39;s filesystem contents as a tar archive generate Generate structured data based on containers and pods. healthcheck Manage health checks on containers help Help about any command history Show history of a specified image image Manage images images List images in local storage import Import a tarball to create a filesystem image info Display podman system information init Initialize one or more containers inspect Display the configuration of object denoted by ID kill Kill one or more running containers with a specific signal load Load an image from container archive login Login to a container registry logout Logout of a container registry logs Fetch the logs of one or more containers manifest Manipulate manifest lists and image indexes mount Mount a working container&#39;s root filesystem network Manage networks pause Pause all the processes in one or more containers play Play a pod and its containers from a structured file. pod Manage pods port List port mappings or a specific mapping for the container ps List containers pull Pull an image from a registry push Push an image to a specified destination restart Restart one or more containers rm Remove one or more containers rmi Removes one or more images from local storage run Run a command in a new container save Save image to an archive search Search registry for image start Start one or more containers stats Display a live stream of container resource usage statistics stop Stop one or more containers system Manage podman tag Add an additional name to a local image top Display the running processes of a container unmount Unmounts working container&#39;s root filesystem unpause Unpause the processes in one or more containers unshare Run a command in a modified user namespace untag Remove a name from a local image version Display the Podman Version Information volume Manage volumes wait Block on one or more containers<br/> Flags: --cgroup-manager string Cgroup manager to use (&quot;cgroupfs&quot;|&quot;systemd&quot;) (default &quot;cgroupfs&quot;) --cni-config-dir string Path of the configuration directory for CNI networks --conmon string Path of the conmon binary -c, --connection string Connection to use for remote Podman service --events-backend string Events backend to use (&quot;file&quot;|&quot;journald&quot;|&quot;none&quot;) (default &quot;file&quot;) --help Help for podman --hooks-dir strings Set the OCI hooks directory path (may be set multiple times) (default [/usr/share/containers/oci/hooks.d]) --identity string path to SSH identity file, (CONTAINER_SSHKEY) --log-level string Log messages above specified level (debug, info, warn, error, fatal, panic) (default &quot;error&quot;) --namespace string Set the libpod namespace, used to create separate views of the containers and pods on the system --network-cmd-path string Path to the command for configuring the network -r, --remote Access remote Podman service (default false) --root string Path to the root directory in which data, including images, is stored --runroot string Path to the &#39;run directory&#39; where all state information is stored --runtime string Path to the OCI-compatible binary used to run containers, default is /usr/bin/runc --storage-driver string Select which storage driver is used to manage storage of images and containers (default is overlay) --storage-opt stringArray Used to pass an option to the storage driver --syslog Output logging information to syslog as well as the console (default false) --tmpdir string Path to the tmp directory for libpod state content. Note: use the environment variable &#39;TMPDIR&#39; to change the temporary storage location for container images, &#39;/var/tmp&#39;. --url string URL to access Podman service (CONTAINER_HOST) (default &quot;unix:/home/mslinn/.docker/run/podman/podman.sock&quot;) -v, --version Version of Podman<br/> Use &quot;podman [command] --help&quot; for more information about a command. </span></pre> </editor-fold> <editor-fold padman_info> <h3 id="padman_info"><span class="code">podman info</span></h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4063b9df0123'><button class='copyBtn' data-clipboard-target='#id4063b9df0123' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman info <span class='unselectable'>host: arch: amd64 buildahVersion: 1.15.2 cgroupVersion: v1 conmon: package: &#39;conmon: /usr/libexec/podman/conmon&#39; path: /usr/libexec/podman/conmon version: &#39;conmon version 2.0.20, commit: unknown&#39; cpus: 8 distribution: distribution: ubuntu version: &quot;20.10&quot; eventLogger: file hostname: Bear idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 kernel: 5.4.72-microsoft-standard-WSL2 linkmode: dynamic memFree: 897724416 memTotal: 6231638016 ociRuntime: name: runc package: &#39;containerd.io: /usr/bin/runc&#39; path: /usr/bin/runc version: |- runc version 1.0.0-rc93 commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec spec: 1.0.2-dev go: go1.13.15 libseccomp: 2.5.1 os: linux remoteSocket: path: /home/mslinn/.docker/run/podman/podman.sock rootless: true slirp4netns: executable: /bin/slirp4netns package: Unknown version: |- slirp4netns version 1.0.1 commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4 libslirp: 4.3.1 swapFree: 0 swapTotal: 0 uptime: 306h 23m 6.37s (Approximately 12.75 days) registries: search: - quay.io - docker.io - gallery.ecr.aws - registry.access.redhat.com store: configFile: /home/mslinn/.config/containers/storage.conf containerStore: number: 8 paused: 0 running: 0 stopped: 8 graphDriverName: overlay graphOptions: overlay.mount_program: Executable: /bin/fuse-overlayfs Package: Unknown Version: |- fusermount3 version: 3.9.3 fuse-overlayfs: version 1.0.0 FUSE library version 3.9.3 using FUSE kernel interface version 7.31 graphRoot: /home/mslinn/.local/share/containers/storage graphStatus: Backing Filesystem: extfs Native Overlay Diff: &quot;false&quot; Supports d_type: &quot;true&quot; Using metacopy: &quot;false&quot; imageStore: number: 4 runRoot: /home/mslinn/.docker/run/containers volumePath: /home/mslinn/.local/share/containers/storage/volumes version: APIVersion: 1 Built: 0 BuiltTime: Wed Dec 31 19:00:00 1969 GitCommit: &quot;&quot; GoVersion: go1.14.7 OsArch: linux/amd64 Version: 2.0.6 </span></pre> </editor-fold> <editor-fold padman_container_help> <h3 id="padman_container_help"><span class="code">podman container</span> Help</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idee5a29a46f48'><button class='copyBtn' data-clipboard-target='#idee5a29a46f48' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>man podman-container <span class='unselectable'>podman-container(1) General Commands Manual podman-container(1)<br/> NAME podman-container - Manage containers<br/> SYNOPSIS podman container subcommand<br/> DESCRIPTION The container command allows you to manage containers<br/> COMMANDS &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9516;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9516;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488; &#9474;Command &#9474; Man Page &#9474; Description &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;attach &#9474; podman-attach(1) &#9474; Attach to a running container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;checkpoint &#9474; podman-container-checkpoint(1) &#9474; Checkpoints one or more running &#9474; &#9474; &#9474; &#9474; containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;cleanup &#9474; podman-container-cleanup(1) &#9474; Cleanup the container&#39;s network &#9474; &#9474; &#9474; &#9474; and mountpoints. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;commit &#9474; podman-commit(1) &#9474; Create new image based on the &#9474; &#9474; &#9474; &#9474; changed container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;cp &#9474; podman-cp(1) &#9474; Copy files/folders between a &#9474; &#9474; &#9474; &#9474; container and the local &#9474; &#9474; &#9474; &#9474; filesystem. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;create &#9474; podman-create(1) &#9474; Create a new container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;diff &#9474; podman-diff(1) &#9474; Inspect changes on a container or &#9474; &#9474; &#9474; &#9474; image&#39;s filesystem. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;exec &#9474; podman-exec(1) &#9474; Execute a command in a running &#9474; &#9474; &#9474; &#9474; container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;exists &#9474; podman-container-exists(1) &#9474; Check if a container exists in &#9474; &#9474; &#9474; &#9474; local storage &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;export &#9474; podman-export(1) &#9474; Export a container&#39;s filesystem &#9474; &#9474; &#9474; &#9474; contents as a tar archive. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;init &#9474; podman-init(1) &#9474; Initialize a container &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;inspect &#9474; podman-inspect(1) &#9474; Display a container or image&#39;s &#9474; &#9474; &#9474; &#9474; configuration. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;kill &#9474; podman-kill(1) &#9474; Kill the main process in one or &#9474; &#9474; &#9474; &#9474; more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;list &#9474; podman-ps(1) &#9474; List the containers on the &#9474; &#9474; &#9474; &#9474; system.(alias ls) &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;logs &#9474; podman-logs(1) &#9474; Display the logs of a container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;mount &#9474; podman-mount(1) &#9474; Mount a working container&#39;s root &#9474; &#9474; &#9474; &#9474; filesystem. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;pause &#9474; podman-pause(1) &#9474; Pause one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;port &#9474; podman-port(1) &#9474; List port mappings for the &#9474; &#9474; &#9474; &#9474; container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;prune &#9474; podman-container-prune(1) &#9474; Remove all stopped containers &#9474; &#9474; &#9474; &#9474; from local storage. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;restart &#9474; podman-restart(1) &#9474; Restart one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;restore &#9474; podman-container-restore(1) &#9474; Restores one or more containers &#9474; &#9474; &#9474; &#9474; from a checkpoint. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;rm &#9474; podman-rm(1) &#9474; Remove one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;run &#9474; podman-run(1) &#9474; Run a command in a container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;runlabel &#9474; podman-container-runlabel(1) &#9474; Executes a command as described &#9474; &#9474; &#9474; &#9474; by a container image label. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;start &#9474; podman-start(1) &#9474; Starts one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;stats &#9474; podman-stats(1) &#9474; Display a live stream of one or &#9474; &#9474; &#9474; &#9474; more container&#39;s resource usage &#9474; &#9474; &#9474; &#9474; statistics. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;stop &#9474; podman-stop(1) &#9474; Stop one or more running &#9474; &#9474; &#9474; &#9474; containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;top &#9474; podman-top(1) &#9474; Display the running processes of &#9474; &#9474; &#9474; &#9474; a container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;unmount &#9474; podman-unmount(1) &#9474; Unmount a working container&#39;s &#9474; &#9474; &#9474; &#9474; root filesystem.(Alias unmount) &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;unpause &#9474; podman-unpause(1) &#9474; Unpause one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;wait &#9474; podman-wait(1) &#9474; Wait on one or more containers to &#9474; &#9474; &#9474; &#9474; stop and print their exit codes. &#9474; &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9524;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9524;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;<br/> SEE ALSO podman, podman-exec, podman-run<br/> podman-container(1) </span></pre> </editor-fold> <editor-fold podman_run_help> <h3 id="podman_help">Podman Run Help</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6fecdd46f75c'><button class='copyBtn' data-clipboard-target='#id6fecdd46f75c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman container run --help <span class='unselectable'>Run a command in a new container<br/> Description: Runs a command in a new container from the given image<br/> Usage: podman container run [flags] IMAGE [COMMAND [ARG...]]<br/> Examples: podman container run imageID ls -alF /etc podman container run --network=host imageID dnf -y install java podman container run --volume /var/hostdir:/var/ctrdir -i -t fedora /bin/bash<br/> Flags: --add-host strings Add a custom host-to-IP mapping (host:ip) (default []) --annotation strings Add annotations to container (key:value) -a, --attach strings Attach to STDIN, STDOUT or STDERR --authfile string Path of the authentication file. Use REGISTRY_AUTH_FILE environment variable to override --blkio-weight string Block IO weight (relative weight) accepts a weight value between 10 and 1000. --blkio-weight-device DEVICE_NAME:WEIGHT Block IO weight (relative device weight, format: DEVICE_NAME:WEIGHT) --cap-add strings Add capabilities to the container --cap-drop strings Drop capabilities from the container --cgroup-parent string Optional parent cgroup for the container --cgroupns string cgroup namespace to use --cgroups string control container cgroup configuration (&quot;enabled&quot;|&quot;disabled&quot;|&quot;no-conmon&quot;) (default &quot;enabled&quot;) --cidfile string Write the container ID to the file --conmon-pidfile string Path to the file that will receive the PID of conmon --cpu-period uint Limit the CPU CFS (Completely Fair Scheduler) period --cpu-quota int Limit the CPU CFS (Completely Fair Scheduler) quota --cpu-rt-period uint Limit the CPU real-time period in microseconds --cpu-rt-runtime int Limit the CPU real-time runtime in microseconds --cpu-shares uint CPU shares (relative weight) --cpus float Number of CPUs. The default is 0.000 which means no limit --cpuset-cpus string CPUs in which to allow execution (0-3, 0,1) --cpuset-mems string Memory nodes (MEMs) in which to allow execution (0-3, 0,1). Only effective on NUMA systems. -d, --detach Run container in background and print container ID --detach-keys [a-Z] Override the key sequence for detaching a container. Format is a single character [a-Z] or a comma separated sequence of `ctrl-&lt;value&gt;`, where `&lt;value&gt;` is one of: `a-cf`, `@`, `^`, `[`, `\`, `]`, `^` or `_` (default &quot;ctrl-p,ctrl-q&quot;) --device strings Add a host device to the container --device-cgroup-rule strings Add a rule to the cgroup allowed devices list --device-read-bps strings Limit read rate (bytes per second) from a device (e.g. --device-read-bps=/dev/sda:1mb) --device-read-iops strings Limit read rate (IO per second) from a device (e.g. --device-read-iops=/dev/sda:1000) --device-write-bps strings Limit write rate (bytes per second) to a device (e.g. --device-write-bps=/dev/sda:1mb) --device-write-iops strings Limit write rate (IO per second) to a device (e.g. --device-write-iops=/dev/sda:1000) --disable-content-trust This is a Docker specific option and is a NOOP --dns strings Set custom DNS servers --dns-opt strings Set custom DNS options --dns-search strings Set custom DNS search domains --entrypoint string Overwrite the default ENTRYPOINT of the image -e, --env stringArray Set environment variables in container (default [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,TERM=xterm]) --env-file strings Read in a file of environment variables --env-host Use all current host environment variables in container --expose strings Expose a port or a range of ports --gidmap strings GID map to use for the user namespace --group-add strings Add additional groups to join --health-cmd string set a healthcheck command for the container (&#39;none&#39; disables the existing healthcheck) --health-interval string set an interval for the healthchecks (a value of disable results in no automatic timer setup) (default &quot;30s&quot;) --health-retries uint the number of retries allowed before a healthcheck is considered to be unhealthy (default 3) --health-start-period string the initialization time needed for a container to bootstrap (default &quot;0s&quot;) --health-timeout string the maximum time allowed to complete the healthcheck before an interval is considered failed (default &quot;30s&quot;) -h, --hostname string Set container hostname --http-proxy Set proxy environment variables in the container based on the host proxy vars (default true) --image-volume string Tells podman how to handle the builtin image volumes (&quot;bind&quot;|&quot;tmpfs&quot;|&quot;ignore&quot;) (default &quot;bind&quot;) --init Run an init binary inside the container that forwards signals and reaps processes --init-path string Path to the container-init binary -i, --interactive Keep STDIN open even if not attached --ip string Specify a static IPv4 address for the container --ipc string IPC namespace to use --kernel-memory &lt;number&gt;[&lt;unit&gt;] Kernel memory limit (format: &lt;number&gt;[&lt;unit&gt;], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes)) -l, --label stringArray Set metadata on container --label-file strings Read in a line delimited file of labels --log-driver string Logging driver for the container --log-opt strings Logging driver options --mac-address string Container MAC address (e.g. 92:d0:c6:0a:29:33) -m, --memory &lt;number&gt;[&lt;unit&gt;] Memory limit (format: &lt;number&gt;[&lt;unit&gt;], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes)) --memory-reservation &lt;number&gt;[&lt;unit&gt;] Memory soft limit (format: &lt;number&gt;[&lt;unit&gt;], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes)) --memory-swap string Swap limit equal to memory plus swap: &#39;-1&#39; to enable unlimited swap --memory-swappiness int Tune container memory swappiness (0 to 100, or -1 for system default) (default -1) --mount stringArray Attach a filesystem mount to the container --name string Assign a name to the container --network string Connect a container to a network (default &quot;slirp4netns&quot;) --no-healthcheck Disable healthchecks on container --no-hosts Do not create /etc/hosts within the container, instead use the version from the image --oom-kill-disable Disable OOM Killer --oom-score-adj int Tune the host&#39;s OOM preferences (-1000 to 1000) --pid string PID namespace to use --pids-limit int Tune container pids limit (set 0 for unlimited, -1 for server defaults) --pod string Run container in an existing pod --pod-id-file string Read the pod ID from the file --privileged Give extended privileges to container -p, --publish strings Publish a container&#39;s port, or a range of ports, to the host (default []) -P, --publish-all Publish all exposed ports to random ports on the host interface --pull string Pull image before creating (&quot;always&quot;|&quot;missing&quot;|&quot;never&quot;) (default &quot;missing&quot;) -q, --quiet Suppress output information when pulling images --read-only Make containers root filesystem read-only --read-only-tmpfs When running containers in read-only mode mount a read-write tmpfs on /run, /tmp and /var/tmp (default true) --replace If a container with the same name exists, replace it --restart string Restart policy to apply when a container exits (&quot;always&quot;|&quot;no&quot;|&quot;on-failure&quot;) --rm Remove container (and pod if created) after exit --rmi Remove container image unless used by other containers --rootfs The first argument is not an image but the rootfs to the exploded container --seccomp-policy string Policy for selecting a seccomp profile (experimental) (default &quot;default&quot;) --security-opt stringArray Security Options --shm-size &lt;number&gt;[&lt;unit&gt;] Size of /dev/shm (format: &lt;number&gt;[&lt;unit&gt;], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes)) (default &quot;65536k&quot;) --sig-proxy Proxy received signals to the process (default true) --stop-signal string Signal to stop a container. Default is SIGTERM --stop-timeout uint Timeout (in seconds) to stop a container. Default is 10 (default 10) --subgidname string Name of range listed in /etc/subgid for use in user namespace --subuidname string Name of range listed in /etc/subuid for use in user namespace --sysctl strings Sysctl options --systemd string Run container in systemd mode (&quot;true&quot;|&quot;false&quot;|&quot;always&quot;) (default &quot;true&quot;) --tmpfs tmpfs Mount a temporary filesystem (tmpfs) into a container -t, --tty Allocate a pseudo-TTY for container --uidmap strings UID map to use for the user namespace --ulimit strings Ulimit options -u, --user string Username or UID (format: &lt;name|uid&gt;[:&lt;group|gid&gt;]) --userns string User namespace to use --uts string UTS namespace to use -v, --volume stringArray Bind mount a volume into the container --volumes-from strings Mount volumes from the specified container(s) -w, --workdir string Working directory inside the container </span></pre> </editor-fold> <editor-fold podman_run> <h2 id="podman_run">podman run</h2> <p> From <a href='https://chariotsolutions.com/blog/post/building-and-deploying-lambdas-from-a-docker-container/' target='_blank' rel='nofollow'>Building and Deploying Lambdas from a Docker Container</a> by Keith Gregory: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf59f2eea62e6'><button class='copyBtn' data-clipboard-target='#idf59f2eea62e6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman run \ -it \ --entrypoint /bin/bash \ --rm \ -v /tmp:/mnt \ amazon/aws-lambda-python:3.8 <span class='unselectable'>Trying to pull quay.io/amazon/aws-lambda-python:3.8... Requesting bear token: invalid status code from registry 405 (Method Not Allowed) Trying to pull docker.io/amazon/aws-lambda-python:3.8... Getting image source signatures Copying blob df513d38f4d9 skipped: already exists Copying blob 2e2bb77ae2dc skipped: already exists Copying blob 031c6369fb2b skipped: already exists Copying blob 03ac043af787 skipped: already exists Copying blob 842c9dce67e8 skipped: already exists Copying blob 1de4740de1c2 [--------------------------------------] 0.0b / 0.0b Copying config e12ea62c55 done Writing manifest to image destination Storing signatures bash-4.2# </span>pwd <span class='unselectable'>/var/task </span></pre> <h2 id="cleanup">Cleaning Up a Container</h2> <p> <code>podman container cleanup</code> is a good command to know about. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id047d0d71409b'><button class='copyBtn' data-clipboard-target='#id047d0d71409b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman container cleanup --help <span class='unselectable'>Cleanup network and mountpoints of one or more containers Description: podman container cleanup Cleans up mount points and network stacks on one or more containers from the host. The container name or ID can be used. This command is used internally when running containers, but can also be used if container cleanup has failed when a container exits. Usage: podman container cleanup [options] CONTAINER [CONTAINER...] Examples: podman container cleanup --latest podman container cleanup ctrID1 ctrID2 ctrID3 podman container cleanup --all Options: -a, --all Cleans up all containers --exec string Clean up the given exec session instead of the container -l, --latest Act on the latest container podman is aware of Not supported with the "--remote" flag --rm After cleanup, remove the container entirely --rmi After cleanup, remove the image entirely </span></pre> </editor-fold> <editor-fold buildah> <h2 id="builah">Buildah</h2> <div style=""> <a href="https://buildah.io/" target="_blank" ><picture> <source srcset="/blog/images/buildahPodman/buildah-logo.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/buildah-logo.png" type="image/png"> <img src="/blog/images/buildahPodman/buildah-logo.png" class=" liImg2 rounded shadow" style="padding: 1em" /> </picture></a> </div> <p> <a href='https://buildah.io/' target='_blank' rel='nofollow'>Buildah</a> is a drop-in replacement for using <code>docker build</code> and a <code>Dockerfile</code>. </p> <div class="quote"> Where Buildah really shines is in its native commands, which you can use to interact with container builds. Rather than using <code>build-using-dockerfile/bud</code> for each build, Buildah has commands to actually interact with the temporary container created during the build process. (Docker uses temporary, or intermediate containers, too, but you don’t really interact with them while the image is being built.) <br><br> Unlike <code>docker build</code>, Buildah doesn’t commit changes to a layer automatically for every instruction in the <code>Dockerfile</code> &ndash; it builds everything from top to bottom, every time. On the positive side, this means non-cached builds (for example, those you would do with automation or build pipelines) end up being somewhat faster than their Docker build counterparts, especially if there are many instructions. <br><br> &nbsp; &ndash; From <a href='https://opensource.com/article/18/6/getting-started-buildah' target='_blank' rel='nofollow'>Getting started with Buildah.</a>, published by <code>opensource.com</code> </div> <p> Some key Buildah subcommands: </p> <dl> <dt class="code">buildah bud</dt> <dd>Buildah’s <code>build-using-dockerfile</code>, or <code>bud</code> argument makes it behave just like <code>docker build</code> does.</dd> <dt class="code">buildah from</dt> <dd>Build up a container root filesystem from an image or from scratch.</dd> <dt class="code">buildah config</dt> <dd>Adjust defaults in the image's configuration blob.</dd> <dt class="code">buildah run</dt> <dd> <code>buildah run</code> is for running commands that build a container image. This is similar to <code>RUN</code> in a <code>Dockerfile</code>, and unlike <code>docker run</code>. </dd> <dt class="code">buildah commit</dt> <dd>Commit changes to the container to a new image.</dd> <dt class="code">buildah push</dt> <dd>Push images to registries (such a Quay) or a local <code>dockerd</code> instance.</dd> <dt class="code"></dt> <dd></dd> <dt class="code"></dt> <dd></dd> <dt class="code"></dt> <dd></dd> </dl> <h3 id="buildahHelp">Buildah Help</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf84d906b95f4'><button class='copyBtn' data-clipboard-target='#idf84d906b95f4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah -h <span class='unselectable'>A tool that facilitates building OCI images<br/> Usage: buildah [flags] buildah [command]<br/> Available Commands: add Add content to the container build-using-dockerfile Build an image using instructions in a Dockerfile commit Create an image from a working container config Update image configuration settings containers List working containers and their base images copy Copy content into the container from Create a working container based on an image help Help about any command images List images in local storage info Display Buildah system information inspect Inspect the configuration of a container or image login Login to a container registry logout Logout of a container registry manifest Manipulate manifest lists and image indexes mount Mount a working container&#39;s root filesystem pull Pull an image from the specified location push Push an image to a specified destination rename Rename a container rm Remove one or more working containers rmi Remove one or more images from local storage run Run a command inside of the container tag Add an additional name to a local image umount Unmount the root file system of the specified working containers unshare Run a command in a modified user namespace version Display the Buildah version information<br/> Flags: -h, --help help for buildah --log-level string The log level to be used. Either &quot;debug&quot;, &quot;info&quot;, &quot;warn&quot; or &quot;error&quot;. (default &quot;error&quot;) --registries-conf string path to registries.conf file (not usually used) --registries-conf-dir string path to registries.conf.d directory (not usually used) --root string storage root dir (default &quot;/var/lib/containers/storage&quot;) --runroot string storage state dir (default &quot;/var/run/containers/storage&quot;) --storage-driver string storage-driver --storage-opt strings storage driver option --userns-gid-map ctrID:hostID:length default ctrID:hostID:length GID mapping to use --userns-uid-map ctrID:hostID:length default ctrID:hostID:length UID mapping to use -v, --version version for buildah<br/> Use &quot;buildah [command] --help&quot; for more information about a command. </span></pre> </editor-fold> <editor-fold buildah_use> <h3 id="buildahUse">Buildah / <span class="code">Dockerfile</span> Compatibility</h3> <div style=""> <picture> <source srcset="/blog/images/buildahPodman/whales.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/whales.png" type="image/png"> <img src="/blog/images/buildahPodman/whales.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <p> <a href='https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/building_running_and_managing_containers/building-container-images-with-buildah_porting-containers-to-systemd-using-podman' target='_blank' rel='nofollow'>Buildah</a> can create an image from a Dockerfile by typing: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddff03472f992'><button class='copyBtn' data-clipboard-target='#iddff03472f992' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah bud -t hello .</pre> <p> &hellip;instead of: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id34695b3af4bb'><button class='copyBtn' data-clipboard-target='#id34695b3af4bb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker build -t hello .</pre> <p> Buildah can create an image called <code>hello</code> from the <code>Dockerfile</code> and the Python app by typing: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddd121beb8815'><button class='copyBtn' data-clipboard-target='#iddd121beb8815' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah bud -t hello . <span class='unselectable'>STEP 1: FROM public.ecr.aws/lambda/python:3.8 Getting image source signatures Copying blob 1de4740de1c2 done Copying blob 2e2bb77ae2dc done Copying blob df513d38f4d9 done Copying blob 03ac043af787 done Copying blob 031c6369fb2b done Copying blob 842c9dce67e8 done Copying config e12ea62c55 done Writing manifest to image destination Storing signatures STEP 2: COPY app.py ./ STEP 3: CMD [&quot;app.handler&quot;] STEP 4: COMMIT hello Getting image source signatures Copying blob 109f575f8e6a skipped: already exists Copying blob ff64b4f854ad skipped: already exists Copying blob dd66ad8702f4 skipped: already exists Copying blob d6fa53d6caa6 skipped: already exists Copying blob 80166c3283e5 skipped: already exists Copying blob 61f74564c3aa skipped: already exists Copying blob d95ebdc79761 done Copying config 40ef32b39c done Writing manifest to image destination Storing signatures --&gt; 40ef32b39cf 40ef32b39cf4ffd3d2e4e3426bec4a5ea168524f7f3fcfe863a378abd9794270 </span></pre> <p> Once the build is complete, the new image can be displayed with the <code>buildah images</code> command: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2e4b0a360026'><button class='copyBtn' data-clipboard-target='#id2e4b0a360026' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah images <span class='unselectable'>REPOSITORY TAG IMAGE ID CREATED SIZE localhost/hello latest 40ef32b39cf4 56 seconds ago 622 MB </span></pre> <p> The new image, tagged <code>hello:latest</code>, can be pushed to a remote image registry. This is easily accomplished with the <code>buildah push</code> command. </p> </editor-fold> <editor-fold buildah_push> <h3 id="buildah_push"><span class="code">buildah push</span> Help</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id38d8fdfb7d26'><button class='copyBtn' data-clipboard-target='#id38d8fdfb7d26' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>man buildah-push <span class='unselectable'>buildah-push(1) General Commands Manual buildah-push(1)<br/> NAME buildah-push - Push an image from local storage to elsewhere.<br/> SYNOPSIS buildah push [options] image [destination]<br/> DESCRIPTION Pushes an image from local storage to a specified destination, decompressing and recompessing layers as needed.<br/> imageID Image stored in local container/storage<br/> DESTINATION The DESTINATION is a location to store container images. If omitted, the source image parameter will be reused as destination.<br/> The Image &quot;DESTINATION&quot; uses a &quot;transport&quot;:&quot;details&quot; format. Multiple transports are supported:<br/> dir:path An existing local directory path storing the manifest, layer tarballs and signatures as individual files. This is a non-standardized format, primarily useful for debugging or noninvasive container inspection.<br/> docker://docker-reference An image in a registry implementing the &quot;Docker Registry HTTP API V2&quot;. By default, uses the authorization state in $XDG\_RUNTIME\_DIR/containers/auth.json, which is set using (buildah login). If the authorization state is not found there, $HOME/.docker/config.json is checked, which is set using (docker login). If docker-reference does not include a registry name, the image will be pushed to a registry running on local&#8208; host.<br/> docker-archive:path[:docker-reference] An image is stored in the docker save formatted file. docker-reference is only used when creating such a file, and it must not contain a digest.<br/> docker-daemon:docker-reference An image _dockerreference stored in the docker daemon internal storage. If _dockerreference does not begin with a valid registry name (a domain name containing &quot;.&quot; or the reserved name &quot;localhost&quot;) then the default registry name &quot;docker.io&quot; will be prepended. _dockerreference must contain either a tag or a digest. Alternatively, when reading images, the format can also be docker-daemon:algo:digest (an image ID).<br/> oci:path:tag An image tag in a directory compliant with &quot;Open Container Image Layout Specification&quot; at path.<br/> oci-archive:path:tag An image tag in a tar archive compliant with &quot;Open Container Image Layout Specification&quot; at path.<br/> If the transport part of DESTINATION is omitted, &quot;docker://&quot; is assumed.<br/> OPTIONS --authfile path<br/> Path of the authentication file. Default is ${XDG_RUNTIME_DIR}/containers/auth.json, which is set using buildah lo&#8208; gin. If the authorization state is not found there, $HOME/.docker/config.json is checked, which is set using docker login.<br/> --cert-dir path<br/> Use certificates at path (*.crt, *.cert, *.key) to connect to the registry. Default certificates directory is /etc/containers/certs.d.<br/> --creds creds<br/> The [username[:password]] to use to authenticate with the registry if required. If one or both values are not sup&#8208; plied, a command line prompt will appear and the value can be entered. The password is entered without echo.<br/> --digestfile Digestfile<br/> After copying the image, write the digest of the resulting image to the file.<br/> --disable-compression, -D<br/> Don&#39;t compress copies of filesystem layers which will be pushed.<br/> --encryption-key key<br/> The [protocol:keyfile] specifies the encryption protocol, which can be JWE (RFC7516), PGP (RFC4880), and PKCS7 (RFC2315) and the key material required for image encryption. For instance, jwe:/path/to/key.pem or pgp:admin@exam&#8208; ple.com or pkcs7:/path/to/x509-file.<br/> --format, -f<br/> Manifest Type (oci, v2s1, or v2s2) to use when saving image to directory using the &#39;dir:&#39; transport (default is manifest type of source)<br/> --quiet, -q<br/> When writing the output image, suppress progress output.<br/> --remove-signatures<br/> Don&#39;t copy signatures when pushing images.<br/> --sign-by fingerprint<br/> Sign the pushed image using the GPG key that matches the specified fingerprint.<br/> --tls-verify bool-value<br/> Require HTTPS and verify certificates when talking to container registries (defaults to true)<br/> EXAMPLE This example pushes the image specified by the imageID to a local directory in docker format.<br/> # buildah push imageID dir:/path/to/image<br/> This example pushes the image specified by the imageID to a local directory in oci format.<br/> # buildah push imageID oci:/path/to/layout:image:tag<br/> This example pushes the image specified by the imageID to a tar archive in oci format.<br/> # buildah push imageID oci-archive:/path/to/archive:image:tag<br/> This example pushes the image specified by the imageID to a container registry named registry.example.com.<br/> # buildah push imageID docker://registry.example.com/repository:tag<br/> This example pushes the image specified by the imageID to a container registry named registry.example.com and saves the digest in the specified digestfile.<br/> # buildah push --digestfile=/tmp/mydigest imageID docker://registry.example.com/repository:tag<br/> This example works like docker push, assuming registry.example.com/my_image is a local image.<br/> # buildah push registry.example.com/my_image<br/> This example pushes the image specified by the imageID to a private container registry named registry.example.com with authentication from /tmp/auths/myauths.json.<br/> # buildah push --authfile /tmp/auths/myauths.json imageID docker://registry.example.com/repository:tag<br/> This example pushes the image specified by the imageID and puts into the local docker container store.<br/> # buildah push imageID docker-daemon:image:tag<br/> This example pushes the image specified by the imageID and puts it into the registry on the localhost while turning off tls verification. # buildah push --tls-verify=false imageID docker://localhost:5000/my-imageID<br/> This example pushes the image specified by the imageID and puts it into the registry on the localhost using creden&#8208; tials and certificates for authentication. # buildah push --cert-dir /auth --tls-verify=true --creds=username:password imageID docker://local&#8208; host:5000/my-imageID<br/> ENVIRONMENT BUILD_REGISTRY_SOURCES<br/> BUILD_REGISTRY_SOURCES, if set, is treated as a JSON object which contains lists of registry names under the keys insecureRegistries, blockedRegistries, and allowedRegistries.<br/> When pushing an image to a registry, if the portion of the destination image name that corresponds to a registry is compared to the items in the blockedRegistries list, and if it matches any of them, the push attempt is denied. If there are registries in the allowedRegistries list, and the portion of the name that corresponds to the registry is not in the list, the push attempt is denied.<br/> TMPDIR The TMPDIR environment variable allows the user to specify where temporary files are stored while pulling and pushing images. Defaults to &#39;/var/tmp&#39;.<br/> FILES registries.conf (/etc/containers/registries.conf)<br/> registries.conf is the configuration file which specifies which container registries should be consulted when com&#8208; pleting image names which do not include a registry or domain portion.<br/> policy.json (/etc/containers/policy.json)<br/> Signature policy file. This defines the trust policy for container images. Controls which container registries can be used for image, and whether or not the tool should trust the images.<br/> SEE ALSO buildah(1), buildah-login(1), containers-policy.json(5), docker-login(1), containers-registries.conf(5)<br/> buildah June 2017 buildah-push(1) </span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id814e104cb117'><button class='copyBtn' data-clipboard-target='#id814e104cb117' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah run \ --entrypoint /var/lang/bin/pip \ --rm \ --user "$(id -u):$(id -g)" \ -v "$(pwd):/mnt" \ amazon/aws-lambda-python:3.8 \ install --target /mnt/build --upgrade psycopg2-binary</pre> </editor-fold> <editor-fold how_to> <div style="text-align: right;"> <a href="https://www.scholastic.ca/books/view/how-to-speak-dolphin" target="_blank" ><picture> <source srcset="/blog/images/buildahPodman/howToSpeakDolphin.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/howToSpeakDolphin.png" type="image/png"> <img src="/blog/images/buildahPodman/howToSpeakDolphin.png" class="right liImg2 rounded shadow" style="width: 25%; height: auto;" /> </picture></a> </div> <h2 id="howto">How To</h2> <p> The following was inspired by <a href='https://github.com/groda/big_data/blob/master/docker_for_beginners.md#recap-images-and-containers' target='_blank' rel='nofollow'>Recap: images and containers</a> from <b>Docker for beginners</b>. The equivalent commands for Docker alternatives are shown. </p> <h3 class="clear" id="ver">Check software version</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd7d43b431d5a'><span class='unselectable'>$ </span>docker -v<br><span class='unselectable'>Docker version 20.10.2, build 20.10.2-0ubuntu1~20.10.1 </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id84161756ed7a'><span class='unselectable'>$ </span>buildah -v<br><span class='unselectable'>buildah version 1.15.2 (image-spec 1.0.1, runtime-spec 1.0.2-dev) </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9d1049cd8243'><span class='unselectable'>$ </span>podman -v<br><span class='unselectable'>podman version 2.0.6 </span></pre> <div style="text-align: right;"> <picture> <source srcset="/blog/images/buildahPodman/aws_linux.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/aws_linux.png" type="image/png"> <img src="/blog/images/buildahPodman/aws_linux.png" class="right " style="width: 25%; height: auto; margin-bottom: 1em;" /> </picture> </div> <h3 id="dlimg">Download the Amazon Linux 2 image</h3> <p> AWS Lambda functions run under Amazon Linux. </p> <p> Each of these 3 commands does a very similar task, downloading a specific image. Docker uses different subdirectories for images than Buildah and <code>podman</code> do. </p> <div class="clear"> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2cc6cffa6869'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker pull amazonlinux <span class='unselectable'>Using default tag: latest latest: Pulling from library/amazonlinux 3c2c91c7c431: Pull complete Digest: sha256:06b9e2433e4e563e1d75bc8c71d32b76dc49a2841e9253746eefc8ca40b80b5e Status: Downloaded newer image for amazonlinux:latest docker.io/library/amazonlinux:latest </span></pre> </div> <p> Buildah works without complaint. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1a5ca7986422'><span class='unselectable'>$ </span>buildah pull amazonlinux <span class='unselectable'>53ef897d731f9a5673c083d0e86d7911f85d6e63bb2be2346b17bdbacdc58637 </span></pre> <p> <code>podman</code> seems to hiccup and then complete successfully. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id61aa9c6e433b'><span class='unselectable'>$ </span>podman pull amazonlinux <span class='unselectable'>Trying to pull quay.io/amazonlinux... error parsing HTTP 404 response body: invalid character &#39;&lt;&#39; looking for beginning of value: &quot;&lt;!DOCTYPE HTML PUBLIC \&quot;-//W3C//DTD HTML 3.2 Final//EN\&quot;&gt;\n&lt;title&gt;404 Not Found&lt;/title&gt;\n&lt;h1&gt;Not Found&lt;/h1&gt;\n&lt;p&gt;The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.&lt;/p&gt;\n&quot; Trying to pull docker.io/library/amazonlinux... Getting image source signatures Copying blob 3c2c91c7c431 [--------------------------------------] 0.0b / 0.0b Copying config 53ef897d73 done Writing manifest to image destination Storing signatures 53ef897d731f9a5673c083d0e86d7911f85d6e63bb2be2346b17bdbacdc58637 </span></pre> <div style="text-align: right;"> <picture> <source srcset="/blog/images/buildahPodman/bash-logo.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/bash-logo.png" type="image/png"> <img src="/blog/images/buildahPodman/bash-logo.png" class="right liImg2 rounded shadow" style="width: 35%; height: auto; margin-bottom: 1em;" /> </picture> </div> <h3 id="echo">Run a Bash Command in an OCI Container</h3> <p> Again, <code>Docker</code> must be run as root for this operation, this represents an unnecessary security risk. </p> <div class="clear"> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id77a463dc2768'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker container run amazonlinux echo 'Hello World!' <span class='unselectable'>Hello World! </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcbea7a0a8e81'><span class='unselectable'>$ </span>podman container run amazonlinux <a href='https://opensource.com/article/18/6/linux-version#how-to-find-the-linux-kernel-version' target='_blank' rel='nofollow'>cat /etc/os-release</a> <span class='unselectable'>VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" </span></pre> </div> <div style="text-align: right;"> <picture> <source srcset="/blog/images/buildahPodman/kodak-carousel-projector.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/kodak-carousel-projector.png" type="image/png"> <img src="/blog/images/buildahPodman/kodak-carousel-projector.png" class="right liImg2 rounded shadow" style="width: 35%; height: auto; margin-bottom: 1em;" /> </picture> </div> <h3 id="localImages">Show All Locally Available Images</h3> <p> Again, <code>Docker</code> must be run as root for this operation, this represents an unnecessary security risk. </p> <div class="clear"><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id094a6a38f6c2'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker images <span class='unselectable'>REPOSITORY TAG IMAGE ID CREATED SIZE amazonlinux latest 53ef897d731f 21 hours ago 163MB </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5f31c4935c2a'><span class='unselectable'>$ </span>podman images <span class='unselectable'>REPOSITORY TAG IMAGE ID CREATED SIZE localhost/hello latest 40ef32b39cf4 5 hours ago 622 MB docker.io/library/amazonlinux latest 53ef897d731f 21 hours ago 170 MB </span></pre> </div> <div style="text-align: right;"> <picture> <source srcset="/blog/images/buildahPodman/containerShip.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/containerShip.png" type="image/png"> <img src="/blog/images/buildahPodman/containerShip.png" class="right liImg2 rounded shadow" style="width: 35%; height: auto; margin-bottom: 1em;" /> </picture> </div> <h3 id="listCont">List OCI Containers</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2e6939d3d477'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker container ls<br><span class='unselectable'>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id26f9f84810c4'><span class='unselectable'>$ </span>podman container ls<br><span class='unselectable'>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES </span></pre> <h3 id="listcont4">View All OCI Containers (Running or Not)</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbb385e835807'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker container ls -a <span class='unselectable'>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 250f56d9aced amazonlinux "echo 'Hello World!'" 14 minutes ago Exited (0) 14 minutes ago competent_einstein </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3ba44b0de973'><span class='unselectable'>$ </span>podman container ls -a <span class='unselectable'>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0f8203e9d3b8 docker.io/library/amazonlinux:latest echo Hello world! 36 minutes ago Exited (0) 36 minutes ago beautiful_mestorf 14282ace8978 docker.io/library/amazonlinux:latest echo Hello world! 36 minutes ago Exited (0) 36 minutes ago beautiful_goldwasser 1b9a8db52fb9 docker.io/library/alpine:latest echo Hello World! About an hour ago Exited (0) About an hour ago zealous_easley 6444ee144488 docker.io/library/amazonlinux:latest echo Hello World! 12 minutes ago Exited (0) 12 minutes ago frosty_ritchie 7444122cbc59 docker.io/library/alpine:latest cat /etc/motd About an hour ago Exited (0) About an hour ago elated_sammet aef84973d6ad docker.io/library/amazonlinux:latest echo Hello world! About an hour ago Exited (0) About an hour ago lucid_sinoussi e210f74bc209 docker.io/library/amazonlinux:latest cat /etc/motd About an hour ago Exited (0) About an hour ago jovial_borg </span></pre> <h3 id="listCont3">List Running OCI containers</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idafe83e28d7a9'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker container ps -a <span class='unselectable'>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 250f56d9aced amazonlinux "echo 'Hello World!'" 5 minutes ago Exited (0) 5 minutes ago competent_einstein </span></pre> <p> <code>podman</code> has a problem with the <code>container ps</code> sub-subcommand. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3f4af938affc'><span class='unselectable'>$ </span>podman container ps -a <span class='unselectable'>Error: unrecognized command `podman container ps` Try 'podman container --help' for more information. </span></pre> <h3 id="buildah_push_use"><span class="code">buildah push</span> to Docker Daemon</h3> <div style=""> <picture> <source srcset="/blog/images/buildahPodman/containerShipTugboat.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/containerShipTugboat.png" type="image/png"> <img src="/blog/images/buildahPodman/containerShipTugboat.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide21de0c45d2a'><button class='copyBtn' data-clipboard-target='#ide21de0c45d2a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah push hello:latest docker-daemon:hello:latest <span class='unselectable'>Getting image source signatures Copying blob sha256:72fcdba8cff9f105a61370d930d7f184702eeea634ac986da0105d8422a17028 247.02 MiB / 247.02 MiB [==================================================] 2s Copying blob sha256:e567905cf805891b514af250400cc75db3cb47d61219750e0db047c5308bd916 144.75 MiB / 144.75 MiB [==================================================] 1s Copying config sha256:6d54bef73e638f2e2dd8b7bf1c4dfa26e7ed1188f1113ee787893e23151ff3ff 1.59 KiB / 1.59 KiB [======================================================] 0s Writing manifest to image destination Storing signatures </span> <span class='unselectable'>$ </span>buildah images | head -n2 <span class='unselectable'>REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/hello latest 6d54bef73e63 2 minutes ago 398 MB </span> <span class='unselectable'>$ </span>buildah run -t hello:latest <span class='unselectable'>Hello, world! </span></pre> <h3 id="buildah_rmi">Delete an OCI Image</h3> <div style=""> <picture> <source srcset="/blog/images/buildahPodman/containerSky.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/containerSky.png" type="image/png"> <img src="/blog/images/buildahPodman/containerSky.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <p> Delete an OCI image in Buildah's <code>~/.local/share/container</code> directory with the <code>rmi</code> subcommand: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id46ec91189800'><button class='copyBtn' data-clipboard-target='#id46ec91189800' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah rmi e12ea62c5582 <span class='unselectable'>e12ea62c5582f91a2228e3e284ea957f2df4f1cdb150fd2c189ef8f11d7633ce </span></pre> </editor-fold> Stack Overflow Culture: Zero-Sum, Authoritarian and Hormonally Imbalanced 2021-04-18T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/18/so-culture <p> This blog post describes some problems that significantly impact <a href='https://www.crunchbase.com/organization/stack-overflow' target='_blank' rel='nofollow'>Stack Overflow</a> users, and offers suggestions for improvement. If you are unfamiliar with Stack Overflow, or would like to read a summary of what this blog post is based on, Kevin Workman wrote a terrific summary in March 2019 entitled &ldquo;<a href='https://happycoding.io/blog/stack-overflow-culture-wars' target='_blank' rel='nofollow'>The Stack Overflow Culture Wars</a>&rdquo;. Very little has changed since then. </p> <div style="text-align: center;"> <a href="https://www.stackoverflow.com" target="_blank" ><picture> <source srcset="/blog/images/stackOverflow/stackoverflowLogo.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/stackoverflowLogo.png" type="image/png"> <img src="/blog/images/stackOverflow/stackoverflowLogo.png" class="center halfsize liImg2 rounded shadow" style="padding: 20px;" /> </picture></a> </div> <fold-article intro> <h2 id="intro">Introduction</h2> <p> <a href='https://www.stackoverflow.com' target='_blank' rel='nofollow'>Stack Overflow</a> is the premier website world-wide for programmers to help each other by asking and answering questions. It has a defined protocol for this type of interaction, however new user on-boarding is often ineffective, so newcomers are not properly informed, and old-timers often do not exhibit appropriate people skills. The <a href='https://games.greggman.com/game/done-with-stackoverflow/' target='_blank' rel='nofollow'>protocol is somewhat misguided</a> and appropriate tools for better interaction are not provided. </p> <p> As a result, this website has developed a well-documented reputation for being “<a href='https://codeblog.jonskeet.uk/2018/03/17/stack-overflow-culture/' target='_blank' rel='nofollow'>a valuable resource, but a scary place to contribute due to potential hostility.</a>” </p> <div class="pullQuote"> Sometimes, loving something means caring enough to admit that it has a problem.<br><br> &nbsp; &ndash; <a href='https://stackoverflow.blog/2018/04/26/stack-overflow-isnt-very-welcoming-its-time-for-that-to-change/' target='_blank' rel='nofollow'>Jay Hanlon</a>, writing about Stack Overflow when he was EVP of Culture and Experience. </div> </fold-article> <fold-article problem> <div class="clear quote"> Stack Overflow suffers from militant moderators who close and delete reasonable submissions and answers due to Draconian rules.<br><br> &nbsp; &ndash; <code>sleavey</code> commenting on a Hacker News thread entitled <a href='https://news.ycombinator.com/item?id=16610353' target='_blank' rel='nofollow'>Stack Overflow Culture</a>. </div> </fold-article> <fold-article top> <h2 id="topdown">Change Starts At the Top</h2> <h3 id="Chandrasekar">Prashanth Chandrasekar, CEO</h3> <p> Prashanth Chandrasekar became CEO of Stack Overflow in September, 2019. <a href='https://www.intercom.com/blog/podcasts/prashanth-chandrasekar-on-writing-the-script-of-the-future/' target='_blank' rel='nofollow'>Inside Intercom interviewed Mr. Chandrasekar</a> in August 2020. This puff piece made no mention of any cultural problems. The focus was on the brilliance of Stack Overflow&rsquo;s technology. If Mr. Chandrasekar has a vision for how to guide social change, he did not mention it. Instead, in the article and his publications since then he only speaks publicly about <a href='https://stackoverflow.blog/author/pchandrasekar/' target='_blank' rel='nofollow'>transitioning Stack Overflow to a product-led SaaS company</a>. </p> <p> Shortly after becoming CEO, Mr. Chandrasekar published &ldquo;<a href='https://stackoverflow.blog/2020/01/21/scripting-the-future-of-stack-2020-plans-vision/' target='_blank' rel='nofollow'>Scripting the Future of Stack Overflow</a>, in which he wrote: </p> <p class="quote"> We learned that we needed much better channels to listen to our moderators and community members. We have not evolved the existing channels of engagement for power users in our community, like Meta, or articulated how we intended to make improvements going forward. This has caused friction as our user base and business have rapidly grown. We acknowledge these issues, apologize for our mistakes, and have plans for improving in the future. </p> <p> Later in the article, he mentioned improvements to the code of conduct, a survey and establishing a moderator council. More than 2 years later, none of this has made the slightest difference in Stack Overflow's culture. </p> <h3 id="pathak">Mihir Pathak — EVP, Strategy & Transformation</h3> <p> Prior to his employment at Stack Overflow, Mr. Pathak was a derivatives strategist McKinsey &amp; Company, a <a href='https://www.mckinsey.com/business-functions/organization/our-insights/the-four-building-blocks--of-change#' target='_blank' rel='nofollow'>world-renouned change management company</a>. That is to say, although Mr. Pathak worked at McKinsey, he was not there to assist other companies make structural changes; instead, he was responsible for pricing methodologies and hedging techniques underlying financial derivative products and options trading strategies &ndash; a heads-down money manager. </p> <p> <a href='https://stackoverflow.com/company/leadership/mihir-pathak' target='_blank' rel='nofollow'>Mr. Pathak&rsquo;s page at StackOverflow</a> is no longer available. <a href='https://webcache.googleusercontent.com/search?q=cache:1eKtOpLW2VoJ:https://stackoverflow.com/company/leadership/mihir-pathak' target='_blank' rel='nofollow'>Google cached it</a>. No announcement has been made about his departure or if there will be a replacement. </p> <p> Mr. Pathak's job description is still visible at <a href='https://www.themuse.com/profiles/stackoverflow' target='_blank' rel='nofollow'><code>themuse.com</code></a>: </p> <p class="quote"> Mihir is responsible for the long-term business strategy of Stack Overflow, which includes forming partnerships with like-minded organizations and understanding how to best serve the needs of future developers. </p> <p> Mr. Pathak was employed from September 2016 at Stack Overflow as a business development executive, not a change management champion. </p> <h3 id="dietrich">Teresa Dietrich, Chief Product and Technology Officer</h3> <p> In January 2020 <a href='https://www.crunchbase.com/person/teresa-dietrich' target='_blank' rel='nofollow'>Teresa Dietrich</a> was made Chief Product and Technology Officer. She also came from McKinsey &amp; Company, where she was Global Head of Product &amp; Engineering. Two months after she took the job she wrote &ldquo;<a href='https://stackoverflow.blog/2020/02/25/sharing-our-first-quarter-2020-community-roadmap/' target='_blank' rel='nofollow'>Sharing our first quarter 2020 community roadmap</a>&rdquo;. That rather bland article did not seem to indicate any recognition of serious problems within the Stack Overflow culture, and I could not find any publicly available results of the studies that were mentioned. </p> <p> 16 months after Ms. Dietrich started her job, I do not see any cultural change. Has Ms. Dietrich been tasked with leading such a change? If so: </p> <ul> <li>Does she have board-level support?</li> <li>Does she have what she needs to make significant cultural changes?</li> <li> Why has she not made any public acknowledgement of a cultural problem? One possible answer is that her boss, Mr. Chandrasekar, has not done so. </li> <li> Is Ms. Dietrich the right person for the job? This is not a purely technical problem, it is a social problem. I believe that women in general possess more highly developed social skills, but the skills necessary to climb to the top are not the skills required to make this type of cultural shift. </li> </ul> <div class="pullQuote"> Are Mr. Chandrasekar and Ms. Dietrich part of the cultural problem, or part of the solution? </div> </fold-article> <fold-article women> </fold-article> <fold-article me> <h2 id="me">Where Am I Coming From?</h2> <p> I have no agenda, no investment in the status quo, nothing to prove, no contacts at the company, I am not an undercover journalist, and I am not a competitor or investor. I am just Joe User... and I am not shy when I believe I have something that I think needs to be heard. </p> <p> If I mispeak, please tell me. If I missed something, again please tell me. If there is a bigger picture I would like to learn about it. </p> <p> I have used Stack Overflow and its <a href='https://stackexchange.com/sites' target='_blank' rel='nofollow'>sibling websites</a> for more than 10 years. Until a few weeks ago, I contributed a few answers here and there, and otherwise did not spend much time on the sites. All contributors are volunteers, so the only reasons to contribute are for prestige, social bonding and altruism. </p> <p> For almost 30 days, in my spare time, I helped people who asked questions on Stack Overflow. I am writing this blog post because although I enjoy helping others, the enjoyment I experienced while doing so within the Stack Overflow environment was overshadowed by the regressive behavior tolerated and even enforced by other contributors. I should also say that I did encounter certain other contributors with whom interaction was very pleasant. However, interactions with alpha contributors with the highest scores seemed more often than not to be quite unpleasant. Since I first published this blog post, I have mostly not interacted on the site. I remain open to contributing to improvements in the culture. </p> <p> The next image is provided so readers know that I am able to effectively participate in the current Stack Overflow website. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/stackOverflow/stackOverflow.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/stackOverflow.png" type="image/png"> <img src="/blog/images/stackOverflow/stackOverflow.png" title="The volume of accepted and upvoted answers put me in the top 0.14% of Stack Overflow answerers in under 30 days." class="center halfsize liImg2 rounded shadow" alt="The volume of accepted and upvoted answers put me in the top 0.14% of Stack Overflow answerers in under 30 days." /> </picture> <figcaption class="halfsize" style="width: 100%; text-align: center;"> The volume of accepted and upvoted answers put me in the top 0.14% of Stack Overflow answerers in under 30 days. </figcaption> </figure> </div> <p> Over 100 of the answers I offered in a 30-day period were accepted as the preferred answer. In fact, most of my answers were selected as the preferred answer. That means many more alternative answers were not accepted. </p> <p> After losing, sometimes a contributor will delete their post. I have done it myself, when the winning post was significantly better by all measures. </p> <p> Some of the other contributors who had provided alternative answers that were not selected clearly felt they had lost a competition. This structure, and others that I describe below, define a system designed to generate hostility. Stack Overflow, as currently implemented, <a href='https://en.wikipedia.org/wiki/Dominance_hierarchy' target='_blank' rel='nofollow'>promotes dominance behavior</a>, which for most primates (other than <a href='https://www.scientificamerican.com/article/bonobo-sex-and-society-2006-06/' target='_blank' rel='nofollow'>bonobos</a>) is patriarchal. </p> </fold-article> <fold-article gamify> <div style="text-align: right;"> <a href="http://localhost:4001/blog/2021/04/18/so-culture.html" target="_blank" ><picture> <source srcset="/blog/images/stackOverflow/codinghorror-app-icon.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/codinghorror-app-icon.png" type="image/png"> <img src="/blog/images/stackOverflow/codinghorror-app-icon.png" class="right liImg2 rounded shadow" /> </picture></a> </div> <h2 id="gamification">Gamification</h2> <p> Stack Overflow's success has be in part due to its successful <a href='https://en.wikipedia.org/wiki/Gamification' target='_blank' rel='nofollow'>gamification</a> of the interaction between questioners and answerers. Gamification is powerful and addictive. Unfortunately, the model chosen resembles FPS (<a href='https://en.wikipedia.org/wiki/First-person_shooter' target='_blank' rel='nofollow'>first-person shooter</a>) games, instead of co-operative games. </p> <p> Jeff Atwood, one of the two original authors of Stack Overflow, wrote an article in 2011 entitled <a href='https://blog.codinghorror.com/the-gamification/' target='_blank' rel='nofollow'>The Gamification</a>, in which he writes: </p> <div class="quote"> Gaming elements are not tacked on to the Stack Exchange Q&A engine, but a natural and essential element of the design. </div> <p> Just below that sentence, Mr. Atwood shows a screenshot from an FPS video game: </p> <div style="text-align: center;"> <a href="https://blog.codinghorror.com/the-gamification/" target="_blank" ><picture> <source srcset="/blog/images/stackOverflow/fps.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/fps.png" type="image/png"> <img src="/blog/images/stackOverflow/fps.png" title="FPS Game screenshot from 'The Gamification'." class="center halfsize liImg2 rounded shadow" alt="FPS Game screenshot from 'The Gamification'." /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://blog.codinghorror.com/the-gamification/" target="_blank" > FPS Game screenshot from 'The Gamification'. </a> </figcaption> </figure> </div> <div class="quote"> I haven't ever quite come out and said it this way, but … I played a lot of Counter-Strike from 1998 to 2001, and Stack Overflow is in many ways my personal Counter-Strike.<br><br> &nbsp; &ndash; Jeff Atwood, from &ldquo;The Gamification&rdquo; </div> <p> FPS games are structured so the player only wins by killing others. This is an entirely different motivational structure than a scenario where a person only wins if others succeed. </p> </fold-article> <fold-article terms> <h2 id="terms">Terminology</h2> <p> I use a few non-standard terms in this blog post: </p> <dl> <dt>Questioner</dt> <dd>One who provides a question</dd> <dt>Answerer</dt> <dd>One who provides an answer</dd> <dt>Downvoter</dt> <dd>One who downvotes another person's contribution</dd> <dt>Downvotee</dt> <dd>One who has their contribution downvoted</dd> </dl> </fold-article> <fold-article downvote> <h2 id="dialog">Dialog Improves Most Questions</h2> <p> A small percentage of questions asked on Stack Overflow are unambiguous, contain all the necessary information, and are phrased well enough to be understood. For these questions, answers can be posted without any interaction between questioner and potential answerers. </p> <p> However, most questions involve a dialog between potential answerers and the questioner. In the dialog, the question is refined, and the questioner's code and any other relevant data is elicited and provided. The tools provided for such a dialog are unfortunately problematic. </p> <p> The only two mechanisms for interaction between questioner and potential answerers are comments and answers. Comments have severe limitations that greatly reduce their effectiveness for eliciting information from a questioner: </p> <ul> <li>Comments must be very short</li> <li>Comments cannot be formatted properly</li> <li>Comments cannot be edited for more than a few minutes</li> </ul> <p> This means that answerers who are trying to explain something to the questioner to elicit more information, or guide the questioner towards understanding their problem better, must resort to posting an incomplete answer. Posting an incomplete answer is risky; other potential answerers can attack the answer by downvoting it. </p> <div class="pullQuote"> Downvotes typically last forever </div> </fold-article> <fold-article incentives> <h2 class="clear" id="broken">Some Incentives Promote Hostility</h2> <p> Many long-time users have completely objectified other users, and act as if Stack Overflow is a video game. Points are accumulated, and at any given time there are a finite number of questions to answer. A person's reputation on Stack Overflow is represented by a single number, which is the number of points they have accumulated. This single number is the structural source of many problems. A more nuanced reputation score would be a giant step forward. </p> <p> Many of these long-term answerers have come to view their participation on Stack Overflow as a zero-sum competition; they can only win (that is, have their answer accepted) if everyone else loses (that is, no-one else provides an answer that is accepted). </p> <p> Some answerers employ intimidation order to suppress competition. Downvotes are often used in the same way against other answerers as a <a href='https://en.wikipedia.org/wiki/Brushback_pitch' target='_blank' rel='nofollow'>brush back pitch</a> in baseball. </p> <div class="pullQuote"> Downvoting has no negative consequences for the downvoter </div> <p> Standard operating procedure for competitively-minded answerers is to downvote answers from others at every opportunity. There is no risk in downvoting: </p> <ul> <li>Downvotes are untraceable; there is no public record of who downvoted or when a downvote was cast.</li> <li>Downvotes are free to downvoters; this encourages liberal downvoting.</li> </ul> <p> This scenario incentivizes competitively-minded answerers to strafe the competition with downvotes at every opportunity. </p> <div class="pullQuote"> If downvotes cost the downvoter the same number of points as they penalize the downvotee, then downvotes would become much rarer. </div> <p> Downvotes typically last forever. Yes, a downvoter could theoretically reverse a downvote, but it is awkward for them to find their old downvotes, and there is no incentive to do so. </p> <p> Questions from newcomers are also frequently downvoted, without discussion, or along with hostile remarks. That leaves a permanent impression, and tends to select for new members who are comfortable with dominance-based hostility. This is a self-perpetuating, and highly toxic, social order. </p> <div style="text-align: center;"> <a href="https://www.focusforhealth.org/how-toxic-masculinity-harms-men-and-society-as-a-whole/" target="_blank" ><picture> <source srcset="/blog/images/stackOverflow/toxicMasulinity.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/toxicMasulinity.png" type="image/png"> <img src="/blog/images/stackOverflow/toxicMasulinity.png" title="Toxic Masculinity Harms Men and Society As A Whole, from Focus for Health" class="center halfsize liImg2 rounded shadow" alt="Toxic Masculinity Harms Men and Society As A Whole, from Focus for Health" /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://www.focusforhealth.org/how-toxic-masculinity-harms-men-and-society-as-a-whole/" target="_blank" > Toxic Masculinity Harms Men and Society As A Whole, from Focus for Health </a> </figcaption> </figure> </div> </fold-article> <fold-article suggest_onboard> <div style="text-align: right;"> <picture> <source srcset="/blog/images/stackOverflow/stackOverflowHelp.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/stackOverflowHelp.png" type="image/png"> <img src="/blog/images/stackOverflow/stackOverflowHelp.png" class="right quartersize liImg2 rounded shadow" /> </picture> </div> <h2 id="onboard">Suggestion: Gamified Onboarding</h2> <p> The only information for new users that is directly accessible from the Stack Overflow front page, is the Help Center, under the question mark icon. It is obvious from the often very polite and tentative inquiries made by new users when they ask their first question that they never noticed that information, or if they did it did not seem relevant. ... and then those new users are mercilessly slammed. </p> <p> New users should be presented with a <a href='https://www.appcues.com/blog/the-5-best-user-onboarding-experiences' target='_blank' rel='nofollow'>short instructional question and answer-style introduction</a>, where information is provided on how to be a good Stack Overflow user. This should happen before they get the opportunity to post their first question. Different levels of users should be explained. Although Stack Overflow is all-English, the onboarding should be multilingual. </p> </fold-article> <fold-article suggest_multilingual> <h2 id="multling">Suggestion: Multilingual Support</h2> <p> A high percentage of users do not speak English very well. They really struggle, and tolerance is low on Stack Overflow for bad English. Other sites, for example Facebook and LinkedIn, have a translation facility built right in. I think Facebook did a particularly good job. Why not do something similar on StackOverflow.com? English would be the official language, but everyone world-wide would be able to interact much more effectively. </p> <div class="pullQuote"> This site is multilingual.<br/> It is not that hard to do! </div> <p> Machine translation is really quite good. I have it on this site. What to view this site in one of over 100 languages? Just <a href='#body'>go to the top of this page</a> and click on this pull-down menu labeled Select Language: </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/stackOverflow/selectLanguage.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/selectLanguage.png" type="image/png"> <img src="/blog/images/stackOverflow/selectLanguage.png" class="center halfsize " /> </picture> </div> <p> You will then see the list of languages that you can view this website in: </p> <div style=""> <picture> <source srcset="/blog/images/stackOverflow/selectLanguages.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/selectLanguages.png" type="image/png"> <img src="/blog/images/stackOverflow/selectLanguages.png" class=" liImg2 rounded shadow" /> </picture> </div> <p> Morally speaking, not to provide a multilingual site discriminates against non-English-speaking people. </p> </fold-article> <fold-article suggest_rep> <h2 id="elicitation">Suggestion: More Nuanced Reputation Score</h2> <p> Instead of a single metric, answerers should be rated along several dimensions. Economists and psychologists both use multidimensional diagrams. The following diagram represents data in 4 dimensions. More dimensions can easily be shown in this type of diagram. </p> <div style=""> <picture> <source srcset="/blog/images/stackOverflow/multiDimentionalPlot.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/multiDimentionalPlot.png" type="image/png"> <img src="/blog/images/stackOverflow/multiDimentionalPlot.png" title="Multi-dimensional data can easily be visualized by outlined shapes" class=" fullsize liImg2 rounded shadow" alt="Multi-dimensional data can easily be visualized by outlined shapes" /> </picture> <figcaption class="fullsize" style="width: 100%; text-align: center;"> Multi-dimensional data can easily be visualized by outlined shapes </figcaption> </figure> </div> <p> Instead of "bigger is better" (a single number indicating status, with the high score indicating alpha status), more information would allow for more detail, so a fuzzy diagram would show little interaction, while a highly detailed and intricate design would indicate a lot of participation. Some answerers might be stronger regarding some metrics, while being weaker on other metrics. </p> <p> Instead of displaying a person's score, the shape of their participation would be shown. Some people prefer to be seen as well-rounded, others prefer to be the best on selected aspects and ignore the others. One size does not fit all. </p> <p> People would start to give pet names for various shapes. Jokes would be made about the shapes. </p> <p> HR personnel would start to hire teams based on how well these shapes meshed together. Money will be made by timely entrepreneurs because these shapes will quickly be adopted industry-wide. Some will aspire to change their shape. </p> <p> An entire industry will spring up servicing those who wish to modify their shape. </p> </fold-article> <fold-article suggest_active> <h2 id="restrict_down">Suggestion: Encourage and Highlight Elicitation</h2> <p> <a href='https://en.wikipedia.org/wiki/Elicitation_technique' target='_blank' rel='nofollow'>Elicitation</a> is a difficult skill to master. The current high-scorers have no incentives to employ elicitation, and they act as if on a campaign to eradicate it. </p> <div class="pullQuote"> Introduce an Elicitation Phase </div> <p> A button should be introduced that lets everyone who visits the question page that a potential answerer would like to elicit information. While at least one such button is enabled, no downvotes are possible relating to the question, and the question cannot be closed or moved to another forum by anyone, regardless of their privilege level. This elicitation mode times out, but not suddenly or unexpectedly. Instead, it backs off, rather like the Ethernet back-off algorithm for collision resolution used in random access MAC protocols. Both the potential answerer and the questioner are given cues that they have a question or a response waiting, and after a period of inactivity the special status ends. I leave the details of the timing undefined for others to discuss. </p> </fold-article> <fold-article suggest_crowd> <h2 class="restrict_down" id="crowd">Suggestion: Restrict Downvoting</h2> <p> Downvoting needs to incorporate: </p> <ul> <li>Accountability (no more anonymous downvoting)</li> <li>Cost (no more drive-by shootings without consequences)</li> <li>Time window (vote after the dust settles, not during the elicitation phase)</li> <li>Public displays of user profiles should prominently display that user's downvotes and upvotes, with links</li> <li>Aggregate statistics on user profiles of their percentage downvotes and upvotes, trends (absolute and relative), etc.</li> </ul> </fold-article> <fold-article suggest_crowd> </fold-article> Serverless E-Commerce 2021-04-14T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/14/serverless-ecommerce <style> h2.numbered:before { color: darkgreen; content: "Option " counter(h2counter) ":\A0\A0\A0"; } </style> <editor-fold intro> <p> As readers of this blog know, I have been <a href='/django/index.html'>chronicling my adventure into Python-powered e-commerce</a> for several months. I have been focusing on Django in general, and <a href='https://django-oscar.readthedocs.io' target='_blank' rel='nofollow'>Django-Oscar</a> in particular. Webapps made with this technology are almost exclusively run on dedicated real or virtual machines. <a href='https://www.cloudflare.com/learning/serverless/what-is-serverless/' target='_blank' rel='nofollow'>Serverless</a> computing is a method of providing backend services on an as-used basis. AWS Lambda is the best-known example of serverless computing, and it combines nicely with a CDN like AWS CloudFront. </p> <p> This blog post discusses 3 goals for an e-commerce system. Two goals are provided by the technology behind <a href='https://martinfowler.com/articles/serverless.html' target='_blank' rel='nofollow'>serverless webapps</a>: </p> <ol> <li>Enormous and instantaneous scalability.</li> <li>Pay-as-you-go without an up-front cost commitment.</li> </ol> <p> I have one more goal: very low latency for online shoppers. </p> </editor-fold> <editor-fold big_picture> <h2 id="big">The Big Picture</h2> <p class="quote"> AWS Lambda consists of two main parts: the Lambda service which manages the execution requests, and the Amazon Linux micro virtual machines provisioned using AWS Firecracker, which actually runs the code. <br><br> A Firecracker VM is started the first time a given Lambda function receives an execution request (the so-called “Cold Start”), and as soon as the VM starts, it begins to poll the Lambda service for messages. When the VM receives a message, it runs your function code handler, passing the received message JSON to the function as the event object. <br><br> Thus every time the Lambda service receives a Lambda execution request, it checks if there is a Firecracker microVM available to manage the execution request. If so, it delivers the message to the VM to be executed. <br><br> In contrast, if no available Firecracker VM is found, it starts a new VM to manage the message. Each VM executes one message at a time, so if a lot of concurrent requests are sent to the Lambda service, for example due to a traffic spike received by an API gateway, several new Firecracker VMs will be started to manage the requests and the average latency of the requests will be higher since each VM takes roughly a second to start. <br><br> &nbsp; &ndash; From <a href='https://www.proud2becloud.com/how-to-run-any-programming-language-on-aws-lambda-custom-runtimes/' target='_blank' rel='nofollow'>How to run any programming language on AWS Lambda: Custom Runtimes</a> by Matteo Moroni. </p> </editor-fold> <editor-fold lambdalimits> <h2 id="lambdalimits">AWS Lambda Limits</h2> <p> AWS Lambda programs have access to considerable resources, enough for most e-commerce stores. The AWS Lambda runtime environment has the following limitations, some of which can be improved upon with some work: </p> <ul> <li>The disk space (ephemeral) is limited to 512 MB.</li> <li>The default deployment package size is 50 MB.</li> <li>The memory range is from 128 to 3008 MB.</li> <li>The maximum execution timeout for a function is 15 minutes.</li> <li>Request and response (synchronous calls) body payload size can be up to 6 MB.</li> <li>Event request (asynchronous calls) body can be up to 128 KB.</li> </ul> </editor-fold> <editor-fold cf> <div style="text-align: right;"> <picture> <source srcset="/blog/images/django/clouds.webp" type="image/webp"> <source srcset="/blog/images/django/clouds.png" type="image/png"> <img src="/blog/images/django/clouds.png" class="right quartersize liImg2 rounded shadow" /> </picture> </div> <h2 id="cf">CloudFront</h2> <p> <a href='https://forum.djangoproject.com/u/briancaffey/summary' target='_blank' rel='nofollow'>Brian Caffey</a> wrote <a href='https://forum.djangoproject.com/t/building-a-django-application-on-aws-with-cloud-development-kit-cdk/2830' target='_blank' rel='nofollow'>Building a Django application on AWS with Cloud Development Kit (CDK)</a>. The website Mr. Caffey's article discusses does not use Lambda, instead his website is always running. So, this option is quite informative and well-thought-out, but it is AWS-specific and does not discuss serverless architecture. </p> <p> For me, the most interesting part about Mr. Caffey's article is it mentions using 3 origins with AWS CloudFront: (1) an origin for ALB (for hosting the Django API), (2) an origin for the S3 website (static Vue.js site), and (3) an S3 origin for Django assets. Mr. Caffey does not say why he used 3 origins, but feeding one CloudFront distribution from multiple origins would mean that all of their content would appear on the same Internet subdomain. </p> <p> This means that the <a href='https://www.w3.org/TR/2020/SPSD-cors-20200602/#resource-preflight-requests' target='_blank' rel='nofollow'>extra HTTP handshaking required for certain CORS</a> (cross-origin HTTP requests) requests between subdomains would be avoided; specifically, there would be no need for pre-flight requests. This would make the website seem noticeably faster if users did lots of content editing and/or transactions with the website. My own pet project has users creating and modifying content, and purchasing product, so taking the requirement for the CORS handshakes away would be a win, plus the end user's web browser could reuse the origin HTTP connection, speeding up even non-cacheable requests. </p> <p> Tamás Sallai wrote <a href='https://advancedweb.hu/how-to-route-to-multiple-origins-with-cloudfront/' target='_blank' rel='nofollow'>How to route to multiple origins with CloudFront &mdash; Set up path-based routing with Terraform</a>. Mr. Sallai <a href='https://advancedweb.hu/' target='_blank' rel='nofollow'>is a prolific writer</a>! </p> </editor-fold> <editor-fold edge> <div style="text-align: right;"> <picture> <source srcset="/blog/images/django/lifeOnTheEdge.webp" type="image/webp"> <source srcset="/blog/images/django/lifeOnTheEdge.png" type="image/png"> <img src="/blog/images/django/lifeOnTheEdge.png" class="right quartersize liImg2 rounded shadow" /> </picture> </div> <h2 id="edge">Edge Computing</h2> <p> Performing computations and serving assets from a <a href='https://aws.amazon.com/cloudfront/features/' target='_blank' rel='nofollow'>nearby point of presence</a> minimizes latency for end users. E-commerce customers much prefer online stores that respond quickly. Edge computing can deliver that experience world-wide, and developers can deploy their work from wherever they are. </p> <p> <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-at-the-edge.html' target='_blank' rel='nofollow'>AWS Lambda@Edge</a> (<a href='https://aws.amazon.com/lambda/edge/' target='_blank' rel='nofollow'>console</a>) runs the Lambda computation in one of 13 regional AWS points of presence, one hop removed from the CloudFront edge locations, or at least in the same availability zone at the CloudFront point of presence. Distributed database issues would need to be addressed before significant benefits would accrue from this implementing this decentralized architecture. Unfortunately, Lambda@Edge has some significant restrictions that prevent it from running nontrivial Django apps. </p> <h3>Lambda@Edge Restrictions</h3> <p>From <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-requirements-limits.html' target='_blank' rel='nofollow'>requirements and restrictions on using Lambda functions with CloudFront</a>, it is apparent that it is not possible to run non-trivial Django apps securely at the edge with good performance.</p> <ul> <li>You can add triggers only for functions in the US East (N. Virginia) Region.</li> <li>You can’t configure your Lambda function to access resources inside your VPC.</li> <li>AWS Lambda environment variables are not supported.</li> <li>Lambda functions with AWS Lambda layers are not supported.</li> <li>Using AWS X-Ray is not supported.</li> <li>AWS Lambda reserved concurrency and provisioned concurrency are not supported.</li> <li>Lambda functions defined as container images are not supported.</li> </ul> <p> Until such time as Lambda@Edge removes the above restrictions, Django webapps will continue to be deployed as centralized webapps, which means that ultra-low latency is not possible world-wide. </p> <h3 id="cf_fns">CloudFront Functions</h3> <p> <a href='https://aws.amazon.com/blogs/aws/introducing-cloudfront-functions-run-your-code-at-the-edge-with-low-latency-at-any-scale/' target='_blank' rel='nofollow'>CloudFront Functions</a> are closer to the user, but have even more restrictions than Lambda@Edge. Alas, CloudFront Functions do not seem likely to be able to support significant computation any time soon. </p> <div style=""> <a href="https://aws.amazon.com/blogs/aws/introducing-cloudfront-functions-run-your-code-at-the-edge-with-low-latency-at-any-scale/" target="_blank" ><picture> <source srcset="/blog/images/serverlessEcommerce/cloudfront-functions-only-lambda-egde-1024x413.webp" type="image/webp"> <source srcset="/blog/images/serverlessEcommerce/cloudfront-functions-only-lambda-egde-1024x413.png" type="image/png"> <img src="/blog/images/serverlessEcommerce/cloudfront-functions-only-lambda-egde-1024x413.png" title="From &ldquo;Introducing CloudFront Functions – Run Your Code at the Edge with Low Latency at Any Scale&rdquo;" class=" liImg2 rounded shadow" alt="From &ldquo;Introducing CloudFront Functions – Run Your Code at the Edge with Low Latency at Any Scale&rdquo;" /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://aws.amazon.com/blogs/aws/introducing-cloudfront-functions-run-your-code-at-the-edge-with-low-latency-at-any-scale/" target="_blank" > From &ldquo;Introducing CloudFront Functions – Run Your Code at the Edge with Low Latency at Any Scale&rdquo; </a> </figcaption> </figure> </div> </editor-fold> <editor-fold iac> <h2 id="vendor">Infrastructure as Code (IaC)</h2> <div class="quote"> For anything bigger than a toy cloud application, Infrastructure as Code (IaC) is table stakes. You’d be hard-pressed to find someone managing anything of scale who thinks letting folks point and click in the console is the optimal route. <br><br>&nbsp; &ndash; From <a href='https://acloudguru.com/blog/engineering/cloudformation-terraform-or-cdk-guide-to-iac-on-aws' target='_blank' rel='nofollow'>CloudFormation, Terraform, or CDK? A guide to IaC on AWS</a> by Jared Short, published by <code>acloudguru.com</code>. </div> <div class="quote"> <a href='https://www.hashicorp.com/products/terraform' target='_blank' rel='nofollow'>Terraform</a>, AWS CloudFormation, Packer, Pulumi, and GeoEngineer are the most popular tools in the category "Infrastructure Build Tools". <br> &nbsp; &ndash; <a href='https://stackshare.io/infrastructure-build-tools' target='_blank' rel='nofollow'>from Stackshare.io</a> </div> </editor-fold> <editor-fold infographic> <h2 id="infographic">Infographic: Lambda Framework Comparison</h2> <p> Yan Cui at Lumigo.io made <a href='https://lumigo.io/aws-lambda-deployment/' target='_blank' rel='nofollow'>this terrific infographic</a>, which compares 9 serverless application frameworks and infrastructure management tools according to opinionatedness and customizability. This article discusses some of those technologies. </p> <div style=""> <a href="https://lumigo.io/aws-lambda-deployment/" target="_blank" ><picture> <source srcset="/blog/images/django/lumigoComparison.webp" type="image/webp"> <source srcset="/blog/images/django/lumigoComparison.png" type="image/png"> <img src="/blog/images/django/lumigoComparison.png" title="From 'AWS Lambda Deployment Frameworks', by Yan Cui at lumigo.io" class=" fullsize liImg2 rounded shadow" alt="From 'AWS Lambda Deployment Frameworks', by Yan Cui at lumigo.io" /> </picture></a> <figcaption class="fullsize" style="width: 100%; text-align: center;"> <a href="https://lumigo.io/aws-lambda-deployment/" target="_blank" > From 'AWS Lambda Deployment Frameworks', by Yan Cui at lumigo.io </a> </figcaption> </figure> </div> <p> The trade-off between customizability and opinionatedness is that highly customizable frameworks require more code to do things that opinionated frameworks do more succinctly. On the other hand, very opinionated frameworks are more limited in their abilities. A classic example of an opinionated framework is Ruby on Rails, which is specifically designed for master/detail applications. Other types of applications should use a different framework, or no framework at all. </p> <p> Two of the technologies on the above infographic are Zappa and Terraform, both of which I discuss in this blog post. Zappa is rather opinionated, while Terraform is very customizable. </p> </editor-fold> <editor-fold cdk> <h2 class="numbered" id="cdk">AWS Cloud Development Kit (CDK)</h2> <p> AWS CDK provides a programmatic interface for modeling and provisioning cloud resources. Languages supported include Java, JavaScript, .NET, Node.js, Python and Typescript. </p> <p> Even if AWS is not directly the service provider, awareness of the <a href='https://aws.amazon.com/cdk/' target='_blank' rel='nofollow'>AWS CDK</a> is important because some other options, for example the <a href='https://aws.amazon.com/blogs/developer/introducing-the-cloud-development-kit-for-terraform-preview/' target='_blank' rel='nofollow'>Cloud Development Kit for Terraform</a> (cdktf), are based on <a href='https://github.com/aws/aws-cdk' target='_blank' rel='nofollow'>AWS CDK</a>. </p> </editor-fold> <editor-fold chalice> <h2 class="numbered" id="chalice">Chalice &ndash; Serverless Django on AWS</h2> <p> <a href='https://aws.github.io/chalice/' target='_blank' rel='nofollow'>Chalice</a> is an AWS open-source project that has good traction. This Python serverless microframework for AWS allows applications that use Amazon API Gateway and AWS Lambda to be quickly created and deployed. </p> <p> The name and logo of this project are suggestive of the Holy Grail. I found the thinly veiled references to Christianity to be off-putting. Religious references have no place in a professional environment. Programmers who work with this project have religious icons, words and phrases continuously presented to them, and they must write words that are strongly identified with Christian doctrine for them to write software. This is forced <a href='https://www.vocabulary.com/dictionary/indoctrination' target='_blank' rel='nofollow'>indoctrination</a>. </p> </editor-fold> <editor-fold zappa> <h2 class="numbered" id="zappa">Django w/ Zappa &amp; AWS Lambda</h2> <div style="text-align: right;"> <picture> <source srcset="/blog/images/django/zappa_400x400.webp" type="image/webp"> <source srcset="/blog/images/django/zappa_400x400.png" type="image/png"> <img src="/blog/images/django/zappa_400x400.png" title="Don't eat the yellow snow" class="right quartersize liImg2 rounded shadow" alt="Don't eat the yellow snow" /> </picture> </div> <p> <a href='https://github.com/zappa/Zappa' target='_blank' rel='nofollow'>Zappa</a> is a popular library for serverless web hosting of Python webapps. Zappa allows Python WSGI webapps like Django to run on AWS Lambda instead of from within a container like AWS EC2. I am particularly interested in using Zappa to package and run <code>django-oscar</code> for AWS Lambda and CloudFront. </p> <p> Zappa can perform two primary functions: </p> <ol> <li> <b>Packaging</b> &ndash; Zappa can build a Django webapp into an AWS Lambda package. The package can be delivered via other mechanisms, for example mechanisms that are not even Python aware. </li> <li> <b>Deploying</b> &ndash; Zappa can deploy and Django webapp to AWS Lambda, and configure several AWS services to feed events to the Django webapp. </li> </ol> <div class="quote"> Zappa does not provide a means to define additional resources as part of the overall infrastructure. It is also somewhat rigid in how it defines certain resources which can lead to friction when incorporating Zappa within organizations with more rigid requirements on cloud resource management. With Zappa, you are better off allowing it to manage all the pieces needed for your web application on its own and manage other resources with a separate tool such as stacker or Terraform. <br><br> &hellip; or use Zappa's <code>package</code> command to create an archive that is ready for upload to lambda and utilize the other helpful functions the project provides for use after code is deployed. <br><br> &nbsp; &ndash; from <a href='https://www.jbssolutions.com/resources/blog/evolution-maintainable-lambda-development-pt-2/' target='_blank' rel='nofollow'>The Evolution of Maintainable Lambda Development Pt 2</a> by JBS Custom Software Solutions. </div> <p> The Zappa documentation is excellent. The project has some rough edges, but the new regime coming on board seem competent and fired up. They have some work ahead to set things straight, but the technical path seems clear. </p> <p> I think this project deserves special attention. Lots of moldy issues and PRs need to be processed, which a small team could get done fairly quickly. The project might also benefit from someone to hone the messaging. I opened an <a href='https://github.com/zappa/Zappa/issues/968' target='_blank' rel='nofollow'>issue on the Zappa GitHub microsite</a> to discuss this. </p> <p> This seminal project has been around several years, and other well-known projects that have been developed since Zappa was first released have acknowledged that Zappa provided inspiration. Time to brush it up and set it straight again; its best days lie ahead! </p> <p> Edgar Roman wrote this helpful document: <a href='https://romandc.com/zappa-django-guide/' target='_blank' rel='nofollow'>Guide to using Django with Zappa</a>. </p> <p> I've messing around with Zappa, will report back. </p> <h3 id="videos">Videos</h3> <p> <a href='https://www.google.com/search?client=firefox-b-d&q=aws+zappa+django+video' target='_blank' rel='nofollow'>Videos of Zappa exist</a>. </p> <ul> <li> This video has got all the right technologies mixed together for me: <a href='https://www.youtube.com/watch?v=Gf0vpJQZeBI' target='_blank' rel='nofollow'>Serverless Deployment of a Django Project with AWS Lambda, Zappa, S3 and PostgreSQL</a>. </li> </ul> </editor-fold> <editor-fold option_djambda> <h2 class="numbered" id="djambda">Djambda / AWS Lambda / Terraform</h2> <p> Terraform does not impose a runtime dependency unless the realtime orchestration features are used. </p> <p> <a href='https://github.com/netsome/djambda' target='_blank' rel='nofollow'>Djambda</a> is an example project setting up Django application in AWS Lambda managed by Terraform. I intend to play with it and write up my experience right here Real Soon Now. </p> <p> This project uses GitHub Actions to create environments for the master branch and pull requests. I wonder if this project can be used without GitHub actions? </p> <div class="quote"> [Terraform] does not provide an abstraction layer for the AWS, Azure, or Google Cloud. It does that deliberately, as you should embrace all aspects when using cloud - not extract a common denominator from the services delivered by the cloud provider. <br><br> &nbsp; &ndash; From <a href='https://awsmaniac.com/aws-cdk-why-not-terraform/' target='_blank' rel='nofollow'>AWS CDK? Why not Terraform?</a> by Wojciech Gawroński. </div> </editor-fold> <editor-fold option_serverless> <h2 class="numbered" id="serverless">Serverless Framework with WSGI</h2> <p> The <a href='https://www.serverless.com/plugins/serverless-wsgi' target='_blank' rel='nofollow'>docs</a> describe Serverless WSGI as: </p> <div class="quote"> Serverless plugin to deploy WSGI applications (Flask/Django/Pyramid etc.) and bundle Python packages. </div> <p> I am concerned that the Serverless architecture requires an <a href='https://www.serverless.com/pricing/fair-use-policy/' target='_blank' rel='nofollow'>ongoing runtime dependency</a> on the viability and good will of Serverless, Inc. Any hiccup on their part will immediately be felt by all their users. It would make me nervous to base daily operational infrastructure on this. </p> </editor-fold> <editor-fold poor-trade> <h3 id="bintray">Bintray and JCenter Went <i>Poof!</i></h3> <p> I do not want to rely upon online services from a software tool vendor to run my builds. The Scala community is still recovering from Bintray and JCenter shutting down. I had dozens of Scala libraries on Bintray. I do not plan to migrate them, they are gone from public access. </p> <div class="quote"> On February 3, 2021, JFrog announced that they will be shutting down Bintray and JCenter. A complete shutdown is planned for February 2022. <br><br> &ndash; <a href='https://blog.gradle.org/jcenter-shutdown' target='_blank' rel='nofollow'>JCenter shutdown impact on Gradle builds</a> </div> <h3 id="free">Trading Autonomy for Minimal Convenience is a Poor Trade</h3> <p> Remember that free products are usually subject to change or termination without notice. Examples abound of many companies whose free (and non-free) products suddenly ceased. There is no need to assume this type of vulnerability, so I block my metaphoric ears to the siren sound that tempts trusting souls into assuming unnecessary dependencies, and I chose tooling that is completely under my control. </p> <div class="quote"> <h2>What happens if I exceed the fair use policy?</h2> <p> <i>From the <a href='https://www.serverless.com/pricing/' target='_blank' rel='nofollow'>Serverless Pricing and Terms page</a>.</i> </p> We want to offer a lot of value for free so you can get your idea off the ground, before introducing any infrastructure cost. The intent of the fair use policy is to ensure that we can provide a high quality of service without incurring significant infrastructure costs. The great majority of users will fall well within range of the typical usage guidelines. While we reserve the right to throttle services if usage exceeds the fair use policy, we do not intend to do so as long as we can deliver a high quality of service without significant infrastructure costs. <br><br> If you anticipate your project will exceed these guidelines, please contact our support team. We’ll work with you on a plan which scales well. </div> </editor-fold> <h2 id="apprunner">AWS AppRunner</h2> <p> AWS just announced <a href='https://aws.amazon.com/apprunner/' target='_blank' rel='nofollow'>AppRunner</a>. I wonder how suitable it is... </p> Merging a Remote File with a Local File 2021-04-12T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/12/merging-remote-file <p> Today I am once again re-installing WSL2 on one of my laptops. Seems that a Windows 10 installation&rsquo;s half-life is measured in months, after which time a reset is required. The reset preserves data, but not installed programs and not the WSL setup. </p> <p> When I set up an OS I often use a pre-existing system&rsquo;s files as templates for the new system&rsquo;s files. </p> <h2 id="meld">Meld</h2> <p> <a href='https://meldmerge.org/' target='_blank' rel='nofollow'>Meld</a> is a fantastic, F/OSS file and directory merge tool. 2-way and 3-way merges are supported. Meld uses X Windows for its user interface. <a href='https://opticos.github.io/gwsl/' target='_blank' rel='nofollow'>GWSL</a> makes it easy to run X apps on WSL and WSL2. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id10c892091d02'><button class='copyBtn' data-clipboard-target='#id10c892091d02' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install meld</pre> <p> Merging a remote file with a local file using Meld is easy once you know how. Unless the remote file system is mounted locally, Meld cannot be used to modify <i>remote</i> files and directories, just <i>local</i> files and directories. </p> <p> Following is the incantation I used to display my local <code>.profile</code> and interactively merge it with my profile on an Ubuntu Linux machine called <code>gojira</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd14d63988f9b'><button class='copyBtn' data-clipboard-target='#idd14d63988f9b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>meld ~/.profile &lt;(ssh mslinn@gojira cat .profile)&</pre> <p> The above runs <code>ssh</code> in a subshell, logs in as <code>mslinn</code> to the machine called <code>gojira</code> and then displays the contents of <code>.profile</code> on <code>gojira</code>. Meld compares the output of <code>cat</code> with the local copy of <code>~/.profile</code>, and displays the differences: </p> <div style=""> <picture> <source srcset="/blog/images/mergeRemote/meld.webp" type="image/webp"> <source srcset="/blog/images/mergeRemote/meld.png" type="image/png"> <img src="/blog/images/mergeRemote/meld.png" class=" liImg2 rounded shadow" /> </picture> </div> <div class="right" style="font-size: 3em;">&#128513;</div> <p> Meld makes it easy to reconcile file versions. </p> Visual Studio Code Workspace Settings 2021-04-11T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/11/svcode-workspace-settings <p> For me, the killer feature that Visual Studio Code is how it integrates the Windows user interface with working on WSL and WSL2. Programs residing on the active WSL OS image execute natively on that OS, while VSCode continues to run as a native Windows application. This is possible because VSCode installs a proxy on the target OS. The proxy does the bidding of the Windows executable. </p> <p> Getting a project to execute on the target OS instead of the host OS can be tricky. I have found that using a workspace to hold a collection of VSCode projects is very helpful, because the definition of the collection also defines how they are handled. </p> <p> WSL projects have different types of VSCode workspace entries than Windows entries do. They are easy to recognize and change once you know what to look for. The two possibile types of VSCode workspace project entries in a <code>.workspace</code> file are: </p> <ul> <li><b>WSL Project</b> &mdash; <code>"uri": "vscode-remote://wsl+ubuntu/path/to/vscode/project"</code></li> <li><b>Windows Project</b> &mdash; <code>"path": "C:\\path\\to\\vscode\\project"</code></li> </ul> <p> The following VSCode workspace file has both types of entries. For me, this is an error; I only want WSL projects. My task is to change the yellow highlighted Windows project and make it look like the other WSL projects. </p> <div class='codeLabel unselectable' data-lt-active='false'>aw.workspace.code_workspace</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id44ff9e34ffd5'>{ "folders": [ { "uri": "vscode-remote://wsl+ubuntu/var/sitesUbuntu/www.ancientwarmth.com" }, { "uri": "vscode-remote://wsl+ubuntu/var/work/django/django" }, { "uri": "vscode-remote://wsl+ubuntu/var/work/django/oscar" }, { "uri": "vscode-remote://wsl+ubuntu/var/work/ancientWarmth/ancientWarmth" }, { <span class="bg_yellow">"path": "../../var/work/django/main"</span> } ], "remoteAuthority": "wsl+Ubuntu", "settings": { "liveServer.settings.multiRootWorkspaceName": "www.mslinn.com", "python.pythonPath": "/var/work/django/oscar/bin/python", "git.ignoreLimitWarning": true, "sqltools.connections": [ { "previewLimit": 50, "server": "localhost", "port": 5432, "driver": "PostgreSQL", "name": "Ancient Warmth on Camille", "database": "ancient_warmth", "username": "postgres", "password": "hithere" } ] } }</pre> <p> All I need to do is change this entry: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd18b121e08f2'><span class="bg_yellow">"path": "../..</span>/var/work/django/main"</pre> <p>To:</p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf4ac98fe1e6e'><span class="bg_yellow">"uri": "vscode-remote://wsl+ubuntu</span>/var/var/work/django/main"</pre> <div class="right" style="font-size: 3em;">&#128513;</div> <p> The modified entry will cause VSCode to launch the project from WSL, instead of Windows. </p> Replicating a Git Directory Tree 2021-04-10T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/10/git-tree <p> Whenever I set up an operating system for one of my computers one of the tedious tasks that must be performed is to replicate the git repositories. </p> <p> It is a bad idea to attempt to copy an entire git repository between computers, because the <code>.git</code> directories within them can quite large. So large, in fact, that it might much more time to copy than re-cloning. I think the reason is that copying the entire git repo actually means copying the same information twice: first the <code>.git</code> hidden directory, complete with all the history for the project, and then again for the files in the currently checked out branch. Git repos store the entire development history of the project in their <code>.git</code> directories, so they are often much larger than the actual code that is checked out at any given time. </p> <p> I have several trees of git repositories, grouped into subdirectories. Here is a sanitized depiction of one of my git directory trees: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0bbc3af5e610'>├── cadenzaHome │   ├── cadenzaAssets │   ├── cadenzaCode │   │   ├── cadenzaClient │   │   ├── cadenzaCourseCode │   │   ├── cadenzaDependencies │   │   ├── cadenzaLibs │   │   ├── cadenzaServer │   │   ├── cadenzaServerNext │   │   └── cadenzaSupport │   ├── cadenzaCreative │   │   └── cadenzaCreativeTemplates │   ├── cadenzaCreativeBackup │   └── cadenzaCurriculum ├── django │   ├── django │   ├── django-oscar │   ├── frobshop │   ├── main │   └── oscar ├── jekyll │   ├── jekyllTemplate │   └── jekyll-flexible-include-plugin</pre> <p> Some git repos are forks, and I defined <code>upstream</code> git remotes for them, in addition to the usual <code>origin</code> remote. </p> <p> This morning I found myself facing the boring task of doing this manually once again. Instead, I wrote this script, which scans a git directory tree and writes out a script that clones the repos in the tree, and adds <code>upstream</code> remotes as required. Directories containing a file called <code>.ignore</code> are ignored. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,gitUrls' download='gitUrls' title='Click on the file name to download the file'>gitUrls</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idd96c4365f6b1">#!/bin/bash function help &#123; printf "$1Replicates tree of git repos " exit 1 &#125; function doOne &#123; cd "$CLONE_DIR" > /dev/null PROJECT_DIR="$( basename "$CLONE_DIR" )" # Might have been renamed after cloning # echo "CLONE_DIR: $CLONE_DIR" # echo "PROJECT_DIR: $PROJECT_DIR" ORIGIN_URL="$( git config --local remote.origin.url )" CLONE_DIR_PARENT="$( realpath "$CLONE_DIR/.." )" echo "mkdir -p '$CLONE_DIR_PARENT'" echo "pushd '$CLONE_DIR_PARENT' > /dev/null" echo "git clone '$ORIGIN_URL'" UPSTREAM_URL="$( git config --local remote.upstream.url )" if [ "$UPSTREAM_URL" ]; then if [ "$ORIGIN_URL" != "no_push" ]; then echo "cd \"$PROJECT_DIR\"" echo "git remote add upstream '$UPSTREAM_URL'" fi fi echo "popd > /dev/null" GIT_DIR_NAME="$( basename "$PWD" )" if [ "$GIT_DIR_NAME" != "$PROJECT_DIR" ]; then echo "# Git project directory was renamed, renaming this copy to match original directory structure" echo "mv \"$GIT_DIR_NAME\" \"$PROJECT_DIR\"" fi echo &#125; if [ -z "$1" ]; then help "Error: Please specify the subdirectory to traverse.\n\n"; fi BASE="$1" DIRS="$( find $BASE -type d \( -execdir test -e &#123;&#125;/.ignore \; -prune \) -o \( -execdir test -d &#123;&#125;/.git \; -prune -print \) )" for DIR in $DIRS; do export CLONE_DIR="$( realpath "$DIR" )" doOne done </pre> <p> Here is the output generated for the above directory tree: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id58320f8f8657'><span class='unselectable'>$ </span>gitUrls $work <span class='unselectable'>mkdir -p '/var/work/cadenzaHome/cadenzaCreative' pushd '/var/work/cadenzaHome/cadenzaCreative' > /dev/null git clone 'git@github.com:mslinn/cadenzaCreativeTemplates.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome' pushd '/var/work/cadenzaHome' > /dev/null git clone 'git@github.com:mslinn/cadenzaAssets.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaSupport' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaSupport' > /dev/null git clone 'git@github.com:mslinn/dottyTemplate.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' > /dev/null git clone 'git@github.com:mslinn/scalacourses-play-utils.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' > /dev/null git clone 'git@github.com:mslinn/scalacourses-utils.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' > /dev/null git clone 'git@github.com:mslinn/scalacourses-slick-utils.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode' pushd '/var/work/cadenzaHome/cadenzaCode' > /dev/null git clone 'git@bitbucket.org:mslinn/cadenzaserver.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaCourseCode/ScalaCourses.com/group_scalaCore' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaCourseCode/ScalaCourses.com/group_scalaCore' > /dev/null git clone 'ssh://git@bitbucket.org/mslinn/course_scala_intro_code.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaCourseCode/ScalaCourses.com/group_scalaCore' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaCourseCode/ScalaCourses.com/group_scalaCore' > /dev/null git clone 'git@bitbucket.org:mslinn/course_scala_intermediate_code.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode' pushd '/var/work/cadenzaHome/cadenzaCode' > /dev/null git clone 'git@github.com:mslinn/cadenzaClient.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome' pushd '/var/work/cadenzaHome' > /dev/null git clone 'git@github.com:mslinn/cadenzaCurriculum.git' popd > /dev/null mkdir -p '/var/work' pushd '/var/work' > /dev/null git clone 'git@github.com:mslinn/jekyllTemplate.git' popd > /dev/null mkdir -p '/var/work/django' pushd '/var/work/django' > /dev/null git clone 'git@github.com:mslinn/django-oscar.git' cd "django-oscar" git remote add upstream 'git@github.com:django-oscar/django-oscar.git' popd > /dev/null mkdir -p '/var/work/django' pushd '/var/work/django' > /dev/null git clone 'git@github.com:mslinn/frobshop.git' popd > /dev/null mkdir -p '/var/work/django' pushd '/var/work/django' > /dev/null git clone 'git@github.com:mslinn/django.git' cd "django" git remote add upstream 'git@github.com:django/django.git' popd > /dev/null mkdir -p '/var/work/jekyll' pushd '/var/work/jekyll' > /dev/null git clone 'git@github.com:mslinn/jekyll-flexible-include-plugin.git' cd "jekyll-flexible-include-plugin" git remote add upstream 'https://idiomdrottning.org/jekyll-include-absolute-plugin' popd > /dev/null mkdir -p '/var/work/jekyll' pushd '/var/work/jekyll' > /dev/null git clone 'git@github.com:mslinn/jekyllTemplate.git' popd > /dev/null </span></pre> <div class="right" style="font-size: 3em;">&#128513;</div> <p> Now all I had to do was paste the above bash commands into a terminal on the new system, and a short time later the git repositories were set up the way I needed them. </p> A Python Virtual Environment For Every Project 2021-04-09T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/09/python-venvs <p> Python virtual environments are cheap to make and use. I have adopted the habit of making a Python virtual environment (<i>venv</i>) for each significant Python project, plus a default venv for trivial Python work. </p> <p> Dedicating a venv for each Python project means that dependencies for any given Python project do not impact the dependencies for any other Python projects. Things just work better. </p> <editor-fold free> <h2 id="manual">Manually Creating a VEnv</h2> <p> Install <a href='https://virtualenv.pypa.io' target='_blank' rel='nofollow'><code>virtualenv</code></a>, a tool to create isolated Python environments. For Ubuntu and other Debian distros, the incantation is: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfa8066c09676'><button class='copyBtn' data-clipboard-target='#idfa8066c09676' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install virtualenv</pre> <p> Create a new virtual python environment in the <code>~/venv/aw/</code> directory like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id91215f09ce43'><button class='copyBtn' data-clipboard-target='#id91215f09ce43' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>virtualenv ~/venv/aw/</pre> <h2 id="free">VEnvs are Nearly Free</h2> <p> The cost of a venv is virtually free. This is because by default, the executable images are linked, so they do not require much storage space. The <code>ls</code> command below shows that the <code>python</code> program in the <code>aw</code> venv is linked to <code>/usr/bin/python3.8</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb6dec34ff982'><button class='copyBtn' data-clipboard-target='#idb6dec34ff982' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls -go ~/venv/aw/bin/python <span class='unselectable'>lrwxrwxrwx 1 18 Apr 9 06:01 <span class="bg_yellow">/home/mslinn/venv/aw/bin/python -> /usr/bin/python3.8</span> </span></pre> </editor-fold> <h2 id="standard">Create a VEnv for Every Python Project</h2> <p> I name each venv the same as the python project that it is dedicated to. My projects are stored under the directory pointed to by <code>$work</code>. </p> <p> My standard procedure when making a Python project called <code>$work/blah</code> is to also create a venv for it at <code>~/venv/blah</code>. </p> <p> A bash alias could be defined called <code>blah</code> that activates the venv and <code>cd</code>s into the project directory: </p> <div class='codeLabel unselectable' data-lt-active='false'>~/.bash_aliases</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1b5ecc81c565'><button class='copyBtn' data-clipboard-target='#id1b5ecc81c565' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>alias blah="source ~/venv/blah/bin/activate; cd $work/blah"</pre> <div class="right" style="font-size: 3em;">&#128513;</div> <p> Now you could type <code>blah</code> at a shell prompt and you would be working on that project. Boom! </p> </editor-fold> <editor-fold create_script> <h2 id="create_script" class="clear">Script For Creating a VEnv</h2> <p> Here is a bash script that creates the venv and changes <code>~/.bashrc</code> and <code>~/.bash_aliases</code> for you. It assumes that you keep your projects under <code>$work</code>. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,newVenv' download='newVenv' title='Click on the file name to download the file'>newVenv</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idac4e13f03230">#!/bin/bash function help &#123; echo -e "$1$(basename $0) - Create a Python virtual environment with a given name. Usage: $(basename $0) venv_name The new virtual environment will be created under ~/venv/. If a project directory called \$work/venv_name exists before this script runs, then a bash alias is created named after the venv." exit 1 &#125; if [ -z `which virtualenv` ]; then sudo apt install virtualenv; fi if [ -z "$1" ]; then help "Please specify a name for the virtual environment.\n\n"; fi if [ "$1" == -h ]; then help; fi VENV="$1" shift mkdir -p "$HOME/venv" cd "$HOME/venv" virtualenv "$VENV" DIR="$HOME/venv/$VENV" echo "source $DIR/bin/activate" >> $HOME/.bashrc echo echo "Activation for "$VENV" in future shells was appended to $HOME/.bashrc" echo "To activate the "$VENV" venv in this shell right now, type: source ~/venv/$VENV/bin/activate" if [ "$work" ] &amp;&amp; [ -d "$work/$VENV" ]; then echo "alias $VENV='source $DIR/bin/activate; cd $work/$VENV'" >> $HOME/.bash_aliases echo "An alias called $VENV for future shells was appended to $HOME/.bash_aliases" echo "To define the alias in this shell right now, type: alias $VENV='source $DIR/bin/activate; cd $work/$VENV'" else echo "To define an alias, type something like this: alias $VENV=\"source $DIR/bin/activate; cd $work/$VENV\"" fi </pre> <p> This is the help message for the script: </p> <div class='codeLabel unselectable' data-lt-active='false'>newVenv help message</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbc033e372154'><button class='copyBtn' data-clipboard-target='#idbc033e372154' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>newVenv -h <span class='unselectable'>newVenv - Create a Python virtual environment with a given name. Usage: newVenv venv_name The new virtual environment will be created under ~/venv/. If a project directory called $work/venv_name exists before this script runs, then a bash alias is created named after the venv. </span></pre> <p> Let's use the script to create a venv called <code>aw</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>newVenv help message</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id095fe0fd42d1'><button class='copyBtn' data-clipboard-target='#id095fe0fd42d1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>newVenv aw <span class='unselectable'>created virtual environment CPython3.8.6.final.0-64 in 528ms creator CPython3Posix(dest=/home/mslinn/venv/aw, clear=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/mslinn/.local/share/virtualenv) added seed packages: pip==20.1.1, pkg_resources==0.0.0, setuptools==44.0.0, wheel==0.34.2 activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator </span></pre> </editor-fold> <editor-fold use> <h2 id="use">Script for Using a VEnv</h2> <p> Here is a script that can display the available Python virtual environments, and optionally activates one them. It does not use bash aliases. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,use' download='use' title='Click on the file name to download the file'>use</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id1a6658a8a785">#!/bin/bash function help &#123; echo "Usage:" for f in $HOME/venv/*; do if [ -d "$f" ]; then echo " . $(basename $0) $(basename $f)"; fi done return 2 &#125; unset PV if [ "$1" == -h ]; then help elif [ "$1" ]; then PV="$1" else PV="default" fi if [ "$PV" ]; then DIR="$HOME/venv/$PV" if [ ! -d "$DIR" ]; then echo "Error: $DIR does not exist." return 1 fi if [ ! -f "$DIR/bin/python" ]; then echo "Error: No Python virtual environment is installed in $DIR" return 1 fi echo "Setting Python virtual environment to $DIR" source "$DIR/bin/activate" fi </pre> <p> Here are examples of using the script to change virtual environments: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id80cf0b36faba'><span class='unselectable'>$ </span>. use -h <span class='unselectable'>Usage: . bash aw . bash default </span> <span class='unselectable'>$ </span>. use <span class='unselectable'>Setting Python virtual environment to /home/mslinn/venv/default </span> <span class='unselectable'>(default) $ </span>. use aw <span class='unselectable'>Setting Python virtual environment to /home/mslinn/venv/aw </span> <span class='unselectable'>(aw) $ </span></pre> <p> Notice that the last command above changed the shell prompt, in that <code>(aw)</code> was prepended to the normal prompt. To cause all future shells to use this virtual environment by default, the script adds a line to <code>~/.bashrc</code> that looks like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id57d98be90738'><button class='copyBtn' data-clipboard-target='#id57d98be90738' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo "source ~/venv/aw/bin/activate" &gt;&gt; ~/.bashrc</pre> <p> At this point the virtual environment just contained executable images for Python. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id60e85eac49f0'><span class='unselectable'>$ </span>ls ~/venv/aw/** <span class='unselectable'>~/venv/aw/pyvenv.cfg ~/venv/aw/bin: activate activate.ps1 chardetect distro easy_install pip pip3.8 python3.8 wheel3 activate.csh activate.xsh chardetect-3.8 distro-3.8 easy_install-3.8 pip-3.8 python wheel activate.fish activate_this.py chardetect3 distro3 easy_install3 pip3 python3 wheel-3.8 ~/venv/aw/lib: python3.8 </span></pre> </editor-fold> <editor-fold deactivate> <h2 id="deactivate">Deactivate a VEnv</h2> <p> Stop using venvs with `deactivate`: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id32c490e21fd5'><span class='unselectable'>(aw) $ </span>deactivate <span class='unselectable'>$ </span></pre> </editor-fold> <editor-fold alias2> <h2 id="alias2">Activate With an Alias</h2> <p> Once again we can use a bash alias, this time to invoke the <code>use</code> script. We can call the alias <code>use</code>, because bash aliases have precedence over bash scripts. This alias removes the need to type <code>.</code> or <code>source</code> before the script name (which you know is <code>use</code>, if you have been following along). </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0f834cdf1a38'><span class='unselectable'>$ </span>alias use="source use" <span class='unselectable'>$ </span>use <span class='unselectable'>(default) $ </span>use aw <span class='unselectable'>(aw) $ </span></pre> <p> You can add the alias to <code>bash_aliases</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id99ee5cfdf882'><span class='unselectable'>$ </span>echo 'alias use="source use"' >> ~/.bash_aliases</pre> <div class="right" style="font-size: 3em;">&#128513;</div> </editor-fold> <editor-fold virt> <h2 id="virt">Directory-Locked Python Virtualization</h2> <p> After setting up a Python virtual environment, a quick examination of the <code>pip</code> script shows that it is hard-coded to the directory that it was made for: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id30deffde447f'><button class='copyBtn' data-clipboard-target='#id30deffde447f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>head -n 1 ~/venv/aw/bin/pip <span class='unselectable'><span class="bg_yellow">#!/home/mslinn/venv/aw/bin/python</span> </span></pre> <p> For virtualized environments, such as Docker, this means that a Python virtual environment created without Docker can only be used within a Docker image if the path to it is the same from within the Docker image as when it was created. </p> </editor-fold> <h2 id="updating">Updating Python</h2> <p> To update the version of Python in a venv, just run the same command that you used to create the venv in the first place. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd30f4c7758cc'><button class='copyBtn' data-clipboard-target='#idd30f4c7758cc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>virtualenv ~/venv/aw</pre> <h2 id="further">For Further Reading</h2> <p> <a href='https://mitelman.engineering/blog/python-best-practice/automating-python-best-practices-for-a-new-project/' target='_blank' rel='nofollow'>Python Best Practices for a New Project in 2021</a> </p> <editor-fold summary> <h2 id="summary">Summary</h2> <ul> <li>Demonstrated how to make an alias for working with Python virtual environments (<i>venvs</i>) that are coupled with Python projects.</li> <li>The <code>newVenv</code> bash script was demonstrated for making new Python virtual environments.</li> <li>The <code>use</code> bash source script was demonstrated for activating a venv.</li> <li>Deactivating the current venv was demonstrated using the <code>deactivate</code> command, provided with every venv.</li> <li>The <code>use</code> alias for <code>source use</code> was demonstrated for more conveniently selecting a venv.</li> <li>Locked directories mean that Python virtual environments should normally only be created in the same environment they are intended to be used.</li> </ul> </editor-fold> Escaping HTML on Clipboard From a Windows Hot Key via WSL 2021-04-03T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/03/escape-html-clipboard <p> I frequently show HTML source code when I write. That HTML must be escaped prior to displaying it on a web page. </p> <h2 id="bash_script">Script to Apply HTML Escape to Clipboard</h2> <p> This bash script applies an HTML escape conversion to the contents of the system clipboard. If you have WSL on your machine, you could store it on the WSL file system, for example in <code>~/.local/bin/escapeHtml</code>. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,escapeHtml' download='escapeHtml' title='Click on the file name to download the file'>escapeHtml</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idd5f125e2cc11">#!/bin/bash # SPDX-License-Identifier: Apache-2.0 function help &#123; echo "$(basename $0) - Escapes HTML with entities. Reads from STDIN or pipe, or converts the clipboard. Result is copied to the clipboard. " exit 1 &#125; function filesAreLinked &#123; "$1" -ef "$2" &#125; function checkDependencies &#123; if [ -z `which recode` ]; then yes | sudo apt install recode; fi # See https://github.com/sindresorhus/clipboard-cli if [ -z `which clipboard` ]; then if [ "$( filesAreLinked /bin/npm /usr/local/lib/node_modules/npm/bin/npm-cli.js )" ]; then # No nodejs venv sudo -H npm install --global clipboard-cli else # nodejs venv npm install --global clipboard-cli fi fi &#125; if [ "$1" == -h ]; then help; fi checkDependencies if [ -t 0 ]; then # Not reading from a terminal HTML="$( clipboard )" else # Reading from stdin or pipe # See https://stackoverflow.com/a/32365596/553865 HTML=$(cat; echo x) HTML=$&#123;HTML%x&#125; # Strip the trailing x fi RESULT="$( recode utf8..html &lt;&lt;&lt; "$HTML" )" echo "$RESULT" | sed "s/&amp;#13;//g" | sed "s/'/\&amp;#39;/g" | clipboard echo "$( echo "$RESULT" | wc -l ) lines have been placed on the clipboard." </pre> <h2 id="use">Using the Script</h2> <ol> <li>Select some text in any document or anywhere that text can be selected.</li> <li>Run <code>escapeHtml</code> on the same machine. If you have Windows with WSL you can run the script there, or run it in native Windows, does not matter.</li> <li>Paste escaped HTML into your target document.</li> </ol> <h2 id="trigger">Hot Key Trigger</h2> <p> Trigger the script with a hot key via your OS's facilities. This section just discusses how native Windows hot keys can be used to trigger this script running in WSL. </p> <ol> <li>Right-click in a folder</li> <li>Select <b>New</b> / <b>Shortcut</b></li> <li> For <b>Type the location of the item</b>, type:<br><br> <div class='codeLabel unselectable' data-lt-active='false'>Type the location of item</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id624a0dcceb6a'><button class='copyBtn' data-clipboard-target='#id624a0dcceb6a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>%windir%\System32\wsl.exe ~/.local/bin/escapeHtml/escapeHtml</pre> </li> <li>Click <b>Change icon</b> and select a retro icon for this shortcut.</li> <li>Click on <b>Apply</b>.</li> <li>Click on <b>Next</b>.</li> <li> When prompted for <b>Type a name for this shortcut</b>, save as <code>HTML Escape Clip to Clip</code>. </li> <li>Click on <b>Finish</b>.</li> <li>Right-click on the new shortcut and click in <b>Shortcut key</b></li> <li>I used <kbd>Ctrl</kbd>+<kbd>Alt</kbd>+<kbd>A</kbd>.</li> <li>Click on <b>OK</b>.</li> </ol> <div class="right" style="font-size: 3em;">&#128513;</div> <p> You can now quickly copy HTML from any source to the clipboard, apply an HTML escape conversion to the clipboard contents, and then paste the escaped HTML into an editor. </p> Microsoft Visual Studio Code Notes 2021-03-22T00:00:00-04:00 https://mslinn.github.io/blog/2021/03/22/vscode-notes <h2 id="keys">Useful Default Key Bindings</h2> <dl> <dt><kbd>Ctrl</kbd>+<kbd>K</kbd> &nbsp; <kbd>Ctrl</kbd>+<kbd>S</kbd></dt> <dd> <b>Display / edit the <a href='https://code.visualstudio.com/docs/getstarted/keybindings' target='_blank' rel='nofollow'>Keyboard Shortcuts</a> definitions.</b><br> You can filter the keybindings by pressing <kbd>Alt</kbd>+<kbd>K</kbd> or clicking the icon of the little keyboard at the top right of the Keyboard Shortcut page, then press the keys that you want to see the key binding for. The keyboard icon starts keystroke recording mode. Recording mode is sticky; each time you revisit the Keyboard Shortcuts tab you can just press the keys you are interested in to see their bindings. Step by step: <ol> <li> When you press the <kbd>Ctrl</kbd> key you will see <code>"ctrl"</code> displayed, and recording mode continues to listen to what you type. Don't do this right now, but FYI, if you toggle keystroke recording mode now, and then remove the quotes around <code>"ctrl"</code>, you will see a sorted list of all the key chords bound to <kbd>Ctrl</kbd>. </li> <li> Next, when you add the <kbd>Shift</kbd> key to the key chord, you then see <code>"ctrl+shift"</code> displayed. Don't do this right now, but FYI, if you toggle keystroke recording mode now, and then remove the quotes, you will see a sorted list of all the key chords bound to <kbd>Ctrl</kbd>+<kbd>Shift</kbd>. </li> <li>Finally, adding the <kbd>=</kbd> key to the key chord shows all the commands bound to <kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>=</kbd>.</li> </ol> </dd> <dt><kbd>Ctrl</kbd>+<kbd>K</kbd>+<kbd>0</kbd> (zero)</dt> <dd>Completely fold the active editor contents.</dd> <dt><kbd>Ctrl</kbd>+<kbd>K</kbd>+<kbd>1</kbd> (one)</dt> <dd>Fold level 1 the active editor contents.</dd> <dt><kbd>Ctrl</kbd>+<kbd>K</kbd>+<kbd>2</kbd></dt> <dd>Fold level 2 the active editor contents.</dd> <dt><kbd>Ctrl</kbd>+<kbd>B</kbd></dt> <dd>Toggle side bar visibility.</dd> <dt><kbd>Ctrl</kbd>+<kbd>P</kbd></dt> <dd> Show names of recently opened tabs, which might contain files to edit, or might be VSCode settings, or VSCode key bindings, etc. Click on a tab name to open it. This key binding is bound to <b>Go to File</b>, which is slightly logical but a not a good descriptive name. </dd> <dt><kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>P</kbd></dt> <dd>Open the <a href='https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette' target='_blank' rel='nofollow'>Command Palette</a>.</dd> <dt><kbd>Ctrl</kbd>+<kbd>L</kbd> &nbsp; <kbd>G</kbd></dt> <dd> Open the active (currently edited) file on GitHub in the default web browser. Requires the <a href='https://marketplace.visualstudio.com/items?itemName=sysoev.vscode-open-in-github' target='_blank' rel='nofollow'>Open in GitHub</a> extension. </dd> <dt><kbd>Ctrl</kbd>+<kbd>,</kbd> (comma)</dt> <dd>Open the <a href='https://code.visualstudio.com/docs/getstarted/settings' target='_blank' rel='nofollow'>Settings</a> tab.</dd> <dt><kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>T</kbd></dt> <dd>Reopen the most recently closed editor tab.</dd> <dt><kbd>Alt</kbd>+<kbd>Shift</kbd>+<kbd>F</kbd></dt> <dd>Format the current file.</dd> </dl> <h2 id="reveal">Annoying Side Bar Auto Reveal</h2> <h3>Problem</h3> <p> When editing a file, if you <kbd>Ctrl</kbd>+<kbd>click</kbd> on a function, method or class name defined in a dependency, the dependency's folder will be expanded in the side bar. Some dependencies are deeply nested, which means that the side bar expands quite a lot. In order to close the folders in the side bar it is necessary to go all the way back to the top of that folder, which is annoying and wastes time. </p> <h3>Solution</h3> <p> In settings, look for <b>Explorer: Auto Reveal</b>, which controls whether the explorer should automatically reveal and select files when opening them. </p> <p> To do that, bring up settings with <kbd>Ctrl</kbd>+<kbd>,</kbd> (comma), and then type <code>reveal ex</code> into the filter. </p> <p> The default value is <code>true</code>. Change the value to <code>false</code>. </p> <h3>Bonus: Reveal Active File in Side Bar</h3> <p> To scroll to a file that you are editing in the list of files in the side bar, right-click on the file's tab, then select <b>Reveal in side bar</b>. </p> <p> Even better, define a keyboard shortcut to do this: </p> <ol> <li> Bring up the <b>Keyboard Shortcuts</b> definitions by typing <kbd>Ctrl</kbd>+<kbd>K</kbd> &nbsp; <kbd>Ctrl</kbd>+<kbd>S</kbd>. </li> <li>Type <code>reveal side</code> into the search bar.</li> <li>Double-click on <b>File: Reveal Active File in Side Bar</b>.</li> <li> For my desired key shortcut, I pressed <kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>Alt</kbd>+<kbd>R</kbd>, then pressed <kbd>Enter</kbd>. </li> </ol> <h2 id="plugins">Plugins</h2> <ul> <li><a href='https://mitelman.engineering/blog/python-best-practice/automating-python-best-practices-for-a-new-project/#code-analysis-with-flake8-linter' target='_blank' rel='nofollow'>Flake 8</a></li> </ul> <h2 id="settings">Settings</h2> <p> <b>User settings</b> are stored in <code>%AppData%\Code\User\</code> on Windows, in files called <code>settings.json</code>, <code>keybindings.json</code>, <code>syncLocalSettings.json</code> and <code>tasks.json</code>. </p> <p> <b>Project settings</b> are stored in <code>.vscode/</code> on all platforms, in files called <code>launch.json</code>, <code>settings.json</code> and <code>tasks.json</code>. </p> <p> <b>Workspace settings</b> are stored in files with a <code>.code-workspace</code> filetype. They can be painful to work with because paths in them are OS-dependent. </p> <h2 id="ext">Extensions</h2> <p> Extensions are stored in <code>~/.vscode/extensions/</code>. </p> Command-Line AWS Utilities 2021-03-22T00:00:00-04:00 https://mslinn.github.io/blog/2021/03/22/command-line-aws-utilities <editor-fold intro> <p> Here are some command-line utilities I have written for AWS. They are dependent on <a href='https://aws.amazon.com/cli/' target='_blank' rel='nofollow'>aws cli</a>. You can <a href='/mslinn_aws.tar'>download all of these utilities</a> in tar format. Extract them into the current directory like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd44cd26aee20'><button class='copyBtn' data-clipboard-target='#idd44cd26aee20' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>tar xf mslinn_aws.tar</pre> </editor-fold> <editor-fold awsCfInvalidate> <h2 id="awsCfInvalidate"><span class="code">awsCfInvalidate</span></h2> <p> Given a CloudFront distribution ID, invalidate the distribution. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,awsCfInvalidate' download='awsCfInvalidate' title='Click on the file name to download the file'>awsCfInvalidate</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id8d5d8cbb7c43">#!/bin/bash function help &#123; printf "$1$(basename $0) - Invalidate the CloudFront distribution for the given ID. If no distribution with the given ID exists, the empty string is returned and the return code is 2. A message is printed asynchronously to the console when the invalidation completes. Syntax: $(basename $0) distId Syntax: awsCfS3Dist www.mslinn.com | $(basename $0) " exit 1 &#125; function waitForInvalidation &#123; echo "Waiting for invalidation $2 to complete." aws cloudfront wait invalidation-completed \ --distribution-id "$1" \ --id "$2" echo "Invalidation $2 has completed." &#125; if [ "$1" == -h ]; then help; fi if [ "$1" ]; then DIST_ID="$1" shift elif [ ! -t 0 ]; then read -r DIST_ID fi if [ -z "$DIST_ID" ]; then help 'Error: No CloudFront distribution ID was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi JSON="$( aws cloudfront create-invalidation \ --distribution-id "$DIST_ID" \ --paths "/*" )" INVALIDATION_ID="$( jq -r .Invalidation.Id &lt;&lt;&lt; "$JSON" )" waitForInvalidation "$DIST_ID" "$INVALIDATION_ID" &amp; </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddb665e0afc4a'><button class='copyBtn' data-clipboard-target='#iddb665e0afc4a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfInvalidate E2P5S6OYKQNB6B <span class='unselectable'>Waiting for invalidation IFOPKECU4YYHD to complete. </span> <span class='unselectable'><i>... do other things ...</i> </span> <span class='unselectable'>$ </span><span class='unselectable'>Invalidation IFOPKECU4YYHD has completed. </span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida58a39346282'><button class='copyBtn' data-clipboard-target='#ida58a39346282' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfS3Dist www.mslinn.com | awsCfInvalidate <span class='unselectable'>Waiting for invalidation IFOPKECU4YYHD to complete. </span> <span class='unselectable'><i>... do other things ...</i> </span> <span class='unselectable'>$ </span><span class='unselectable'>Invalidation IFOPKECU4YYHD has completed. </span></pre> </editor-fold> <editor-fold awsCfS3Dist> <h2 id="awsCfS3Dist"><span class="code">awsCfS3Dist</span></h2> <p> Given an S3 bucket name, return the CloudFront distribution JSON. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,awsCfS3Dist' download='awsCfS3Dist' title='Click on the file name to download the file'>awsCfS3Dist</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id8327388d10bf">#!/bin/bash function help &#123; printf "$1$(basename $0) - Obtain the CloudFront distribution JSON for an S3 bucket. If no S3 bucket with the given name exists, the empty string is returned and the return code is 2. Syntax: $(basename $0) bucketName Syntax: echo bucketName | $(basename $0) " exit 1 &#125; if [ "$1" == -h ]; then help; fi if [ "$1" ]; then BUCKET_NAME="$1" shift elif [ ! -t 0 ]; then read -r BUCKET_NAME fi if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi if [ "$( aws s3api head-bucket --bucket $BUCKET_NAME 2> >(grep -i 'Not Found') )" ]; then >&amp;2 echo "Error: Bucket $BUCKET_NAME does not exist." exit 2 fi DIST_ID="$( awsCfS3DistId "$BUCKET_NAME" )" if [ -z "$DIST_ID" ]; then exit 2; fi aws cloudfront get-distribution-config --id "$DIST_ID" </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id231a8c5c1c3d'><button class='copyBtn' data-clipboard-target='#id231a8c5c1c3d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfS3Dist www.mslinn.com <span class='unselectable'>{ "ETag": "E1DIZUSLMOLXKP", "DistributionConfig": { "CallerReference": "1454487160038", "Aliases": { "Quantity": 2, "Items": [ "www.mslinn.com", "mslinn.com" ] }, "DefaultRootObject": "index.html", "Origins": { "Quantity": 1, "Items": [ { "Id": "S3-www.mslinn.com", "DomainName": "www.mslinn.com.s3-website-us-east-1.amazonaws.com", "OriginPath": "", "CustomHeaders": { "Quantity": 0 }, "CustomOriginConfig": { "HTTPPort": 80, "HTTPSPort": 443, "OriginProtocolPolicy": "http-only", "OriginSslProtocols": { "Quantity": 3, "Items": [ "TLSv1", "TLSv1.1", "TLSv1.2" ] }, "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5 }, "ConnectionAttempts": 3, "ConnectionTimeout": 10 } ] }, "OriginGroups": { "Quantity": 0 }, "DefaultCacheBehavior": { "TargetOriginId": "S3-www.mslinn.com", "TrustedSigners": { "Enabled": false, "Quantity": 0 }, "ViewerProtocolPolicy": "redirect-to-https", "AllowedMethods": { "Quantity": 2, "Items": [ "HEAD", "GET" ], "CachedMethods": { "Quantity": 2, "Items": [ "HEAD", "GET" ] } }, "SmoothStreaming": false, "Compress": true, "LambdaFunctionAssociations": { "Quantity": 0 }, "FieldLevelEncryptionId": "", "CachePolicyId": "658327ea-f89d-4fab-a63d-7e88639e58f6" }, "CacheBehaviors": { "Quantity": 0 }, "CustomErrorResponses": { "Quantity": 2, "Items": [ { "ErrorCode": 403, "ResponsePagePath": "", "ResponseCode": "", "ErrorCachingMinTTL": 60 }, { "ErrorCode": 404, "ResponsePagePath": "", "ResponseCode": "", "ErrorCachingMinTTL": 60 } ] }, "Comment": "", "Logging": { "Enabled": false, "IncludeCookies": false, "Bucket": "", "Prefix": "" }, "PriceClass": "PriceClass_All", "Enabled": true, "ViewerCertificate": { "ACMCertificateArn": "arn:aws:acm:us-east-1:031372724784:certificate/2be42926-829c-4db9-be7d-a72e951256d4", "SSLSupportMethod": "sni-only", "MinimumProtocolVersion": "TLSv1", "Certificate": "arn:aws:acm:us-east-1:031372724784:certificate/2be42926-829c-4db9-be7d-a72e951256d4", "CertificateSource": "acm" }, "Restrictions": { "GeoRestriction": { "RestrictionType": "none", "Quantity": 0 } }, "WebACLId": "", "HttpVersion": "http1.1", "IsIPV6Enabled": false } } </span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8a1119f83781'><button class='copyBtn' data-clipboard-target='#id8a1119f83781' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo www.mslinn.com | awsCfS3Dist <span class='unselectable'>{ "ETag": "E1DIZUSLMOLXKP", "DistributionConfig": { "CallerReference": "1454487160038", "Aliases": { "Quantity": 2, "Items": [ "www.mslinn.com", "mslinn.com" ] }, "DefaultRootObject": "index.html", "Origins": { "Quantity": 1, "Items": [ { "Id": "S3-www.mslinn.com", "DomainName": "www.mslinn.com.s3-website-us-east-1.amazonaws.com", "OriginPath": "", "CustomHeaders": { "Quantity": 0 }, "CustomOriginConfig": { "HTTPPort": 80, "HTTPSPort": 443, "OriginProtocolPolicy": "http-only", "OriginSslProtocols": { "Quantity": 3, "Items": [ "TLSv1", "TLSv1.1", "TLSv1.2" ] }, "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5 }, "ConnectionAttempts": 3, "ConnectionTimeout": 10 } ] }, "OriginGroups": { "Quantity": 0 }, "DefaultCacheBehavior": { "TargetOriginId": "S3-www.mslinn.com", "TrustedSigners": { "Enabled": false, "Quantity": 0 }, "ViewerProtocolPolicy": "redirect-to-https", "AllowedMethods": { "Quantity": 2, "Items": [ "HEAD", "GET" ], "CachedMethods": { "Quantity": 2, "Items": [ "HEAD", "GET" ] } }, "SmoothStreaming": false, "Compress": true, "LambdaFunctionAssociations": { "Quantity": 0 }, "FieldLevelEncryptionId": "", "CachePolicyId": "658327ea-f89d-4fab-a63d-7e88639e58f6" }, "CacheBehaviors": { "Quantity": 0 }, "CustomErrorResponses": { "Quantity": 2, "Items": [ { "ErrorCode": 403, "ResponsePagePath": "", "ResponseCode": "", "ErrorCachingMinTTL": 60 }, { "ErrorCode": 404, "ResponsePagePath": "", "ResponseCode": "", "ErrorCachingMinTTL": 60 } ] }, "Comment": "", "Logging": { "Enabled": false, "IncludeCookies": false, "Bucket": "", "Prefix": "" }, "PriceClass": "PriceClass_All", "Enabled": true, "ViewerCertificate": { "ACMCertificateArn": "arn:aws:acm:us-east-1:031372724784:certificate/2be42926-829c-4db9-be7d-a72e951256d4", "SSLSupportMethod": "sni-only", "MinimumProtocolVersion": "TLSv1", "Certificate": "arn:aws:acm:us-east-1:031372724784:certificate/2be42926-829c-4db9-be7d-a72e951256d4", "CertificateSource": "acm" }, "Restrictions": { "GeoRestriction": { "RestrictionType": "none", "Quantity": 0 } }, "WebACLId": "", "HttpVersion": "http1.1", "IsIPV6Enabled": false } } </span></pre> </editor-fold> <editor-fold awsCfS3DistId> <h2 id="awsCfS3DistId"><span class="code">awsCfS3DistId</span></h2> <p> Given an S3 bucket name, return the CloudFront distribution ID. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,awsCfS3DistId' download='awsCfS3DistId' title='Click on the file name to download the file'>awsCfS3DistId</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idc5d420214d31">#!/bin/bash function help &#123; printf "$1$(basename $0) - Obtain the CloudFront distribution ID for an S3 bucket. If no S3 bucket with the given name exists, the empty string is returned and the return code is 2. Syntax: $(basename $0) bucketName Syntax: echo bucketName | $(basename $0) " exit 1 &#125; if [ "$1" == -h ]; then help; fi if [ "$1" ]; then BUCKET_NAME="$1" shift elif [ ! -t 0 ]; then read -r BUCKET_NAME fi if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi if [ "$( aws s3api head-bucket --bucket $BUCKET_NAME 2> >(grep -i 'Not Found') )" ]; then >&amp;2 echo "Error: Bucket $BUCKET_NAME does not exist." exit 2 fi DIST_ID="$( aws cloudfront list-distributions \ --query "DistributionList.Items[*].&#123;id:Id,origin:Origins.Items[0].Id&#125;[?origin=='S3-$BUCKET_NAME'].id" \ --output text )" if [ -z "$DIST_ID" ]; then exit 2; fi echo "$DIST_ID" </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbefd6cf8bd12'><button class='copyBtn' data-clipboard-target='#idbefd6cf8bd12' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfS3DistId www.mslinn.com <span class='unselectable'>E2P5S6OYKQNB6B </span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide3c868769aac'><button class='copyBtn' data-clipboard-target='#ide3c868769aac' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo www.mslinn.com | awsCfS3DistId <span class='unselectable'>E2P5S6OYKQNB6B </span></pre> </editor-fold> <editor-fold awsCfS3MakeDist> <h2 id="awsCfS3MakeDist"><span class="code">awsCfS3MakeDist</span></h2> <p> Creates a CloudFront distribution for the given bucket name. Returns the new distribution's ID. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,awsCfS3MakeDist' download='awsCfS3MakeDist' title='Click on the file name to download the file'>awsCfS3MakeDist</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id6366c031caaa">#!/bin/bash function help &#123; printf "$1$(basename $0) - Make a new CloudFront distribution for the given S3 bucket name. Returns the new distribution's ID. Syntax: $(basename $0) bucketName Syntax: echo bucketName | $(basename $0) " exit 1 &#125; function doesDistributionExist &#123; DIST_ID="$( awsCfS3Dist "$BUCKET_NAME" )" if [ "$DIST_ID" ]; then echo true; fi &#125; function createDist &#123; read -r -d '' NEW_DIST_JSON &lt;&lt;EOF &#123; "CallerReference": "$BUCKET_NAME", "Aliases": &#123; "Quantity": 0 &#125;, "DefaultRootObject": "index.html", "Origins": &#123; "Quantity": 1, "Items": [ &#123; "Id": "$BUCKET_NAME", "DomainName": "$BUCKET_NAME.s3.amazonaws.com", "S3OriginConfig": &#123; "OriginAccessIdentity": "" &#125; &#125; ] &#125;, "DefaultCacheBehavior": &#123; "TargetOriginId": "$BUCKET_NAME", "ForwardedValues": &#123; "QueryString": true, "Cookies": &#123; "Forward": "none" &#125; &#125;, "TrustedSigners": &#123; "Enabled": false, "Quantity": 0 &#125;, "ViewerProtocolPolicy": "redirect-to-https", "MinTTL": 3600 &#125;, "CacheBehaviors": &#123; "Quantity": 0 &#125;, "Comment": "", "Logging": &#123; "Enabled": false, "IncludeCookies": true, "Bucket": "", "Prefix": "" &#125;, "PriceClass": "PriceClass_All", "Enabled": true &#125; EOF NEW_DIST_RESULT_JSON = "$( aws cloudfront create-distribution --distribution-config "$NEW_DIST_JSON" )" DISTRIBUTION_ID="$( jq -r '.Distribution.Id' &lt;&lt;&lt; "$NEW_DIST_RESULT_JSON" )" echo "$DISTRIBUTION_ID" &#125; if [ "$1" == -h ]; then help; fi if [ -t 0 ]; then if [ -z "$1" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi BUCKET_NAME="$1" shift else read -r BUCKET_NAME fi if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi if [ "$( aws s3api head-bucket --bucket $BUCKET_NAME 2> >(grep -i 'Not Found') )" ]; then >&amp;2 echo "Error: Bucket $BUCKET_NAME does not exist." exit 2 fi if [ "$(doesDistributionExist)" ]; then >&amp;2 echo "Error: a CloudFront distibution already exists for S3 bucket $BUCKET_NAME" exit 3 fi createDist </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id43f58d22584c'><button class='copyBtn' data-clipboard-target='#id43f58d22584c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfS3MakeDist my_bucket <span class='unselectable'>E2P5S6OYKQNB6B </span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id19ebdb9e4697'><button class='copyBtn' data-clipboard-target='#id19ebdb9e4697' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo my_bucket | awsCfS3MakeDist <span class='unselectable'>E2P5S6OYKQNB6B </span></pre> </editor-fold> <editor-fold awsS3Mb> <h2 id="awsS3Mb"><span class="code">awsS3Mb</span></h2> <p> Make a new S3 bucket with the given name in the default AWS region. If the <code>--public-read</code> option is provided, set the ACL to <code>public-read</code> </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,awsS3Mb' download='awsS3Mb' title='Click on the file name to download the file'>awsS3Mb</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id12f8044718e4">#!/bin/bash function help &#123; printf "$1$(basename $0) - Make a new S3 bucket with the given name in the default AWS region. Syntax: $(basename $0) bucketName [OPTIONS] Syntax: echo bucketName | $(basename $0) [OPTIONS] Options are: --public-read Set bucket ACL to public-read " exit 1 &#125; if [ "$1" == -h ]; then help; fi if [ "$1" == "--public-read" ]; then ACL="public-read" shift fi if [ "$1" ]; then BUCKET_NAME="$1" shift elif [ ! -t 0 ]; then read -r BUCKET_NAME fi if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi aws s3 mb s3://$BUCKET_NAME if [ "$ACL" ]; then aws s3api put-bucket-acl --bucket $BUCKET_NAME --acl $ACL fi </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id25724bbfd779'><button class='copyBtn' data-clipboard-target='#id25724bbfd779' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsS3Mb my_bucket</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9026bff7fca1'><button class='copyBtn' data-clipboard-target='#id9026bff7fca1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsS3Mb my_bucket --public-read</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf5b009a27f74'><button class='copyBtn' data-clipboard-target='#idf5b009a27f74' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo my_bucket | awsS3Mb --public-read</pre> </editor-fold> <editor-fold awsS3Website> <h2 id="awsS3Website"><span class="code">awsS3Website</span></h2> <p> Enable an S3 bucket to be a website. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,awsS3Website' download='awsS3Website' title='Click on the file name to download the file'>awsS3Website</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id8894abb0b384">#!/bin/bash function help &#123; printf "$1$(basename $0) - Enable an S3 bucket to be a website. Syntax: $(basename $0) bucketName Syntax: echo bucketName | $(basename $0) " exit 1 &#125; if [ "$1" == -h ]; then help; fi if [ "$1" ]; then BUCKET_NAME="$1" shift elif [ ! -t 0 ]; then read -r BUCKET_NAME fi if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi aws s3 website s3://$BUCKET_NAME \ --index-document index.html \ --error-document 404.html </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb601d2d75eee'><button class='copyBtn' data-clipboard-target='#idb601d2d75eee' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsS3Website my_bucket</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id50052760707a'><button class='copyBtn' data-clipboard-target='#id50052760707a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo my_bucket | awsS3Website</pre> </editor-fold> CORS on AWS S3 and Cloudfront 2021-03-21T00:00:00-04:00 https://mslinn.github.io/blog/2021/03/21/cors-aws <editor-fold intro> <p> This post shows how to enable CORS on an AWS S3 bucket with AWS CLI, then modify the bucket&rsquo;s CloudFront distribution. In preparing this blog post, I found that the <a href='https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html' target='_blank' rel='nofollow'>AWS S3 CORS documentation</a> needs to be read in conjunction with how <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/header-caching.html#header-caching-web-cors' target='_blank' rel='nofollow'>AWS CloudFront can be configured to handle CORS</a>. </p> <p> I used one origin for testing. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id43f8efc04622'><button class='copyBtn' data-clipboard-target='#id43f8efc04622' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ORIGIN=ancientwarmth.com <span class='unselectable'>$ </span>JSON_FILE=cors.json</pre> <p> The CORS configuration for the AWS S3 bucket will be stored in the file pointed to by <code>JSON_FILE</code>. </p> </editor-fold> <editor-fold defS3Cors> <h2 id="defS3Cors">Define the AWS S3 Bucket CORS Configuration</h2> <p> This configuration (in JSON format) contains 1 rule: </p> <ol> <li>Allow <code>GET</code> HTTP methods from anywhere.</li> </ol> <div class='codeLabel unselectable' data-lt-active='false'>cors.json</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2273dba51533'><button class='copyBtn' data-clipboard-target='#id2273dba51533' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{ "CORSRules": [ { "AllowedHeaders": [], "AllowedMethods": [ "GET" ], "AllowedOrigins": [ "*" ], "ExposeHeaders": [] } ] }</pre> <p> You can read about CORS configuration in the <a href='https://docs.aws.amazon.com/AmazonS3/latest/userguide/ManageCorsUsing.html#cors-example-1' target='_blank' rel='nofollow'>AWS documentation</a>. </p> </editor-fold> <editor-fold setS3Cors> <h2 id="setS3Cors">Set the AWS S3 Bucket CORS Configuration</h2> <p> It is easy to set the CORS configuration for an AWS S3 bucket using AWS CLI&rsquo;s <a href='https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-cors.html' target='_blank' rel='nofollow'><code>aws s3api put-bucket-cors</code> subcommand</a>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idaec1969533cb'><button class='copyBtn' data-clipboard-target='#idaec1969533cb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>BUCKET=assets.ancientwarmth.com <span class='unselectable'>$ </span>aws s3api put-bucket-cors \ --bucket $BUCKET \ --cors-configuration "file://$JSON_FILE"</pre> </editor-fold> <editor-fold testS3Cors> <h2 id="testS3Cors">Test the AWS S3 Bucket CORS Configuration</h2> <p> Now it is time to test the S3 bucket&rsquo;s CORS configuration using <code>curl</code>. I defined a bash function to peform the test to save typing. You can use it by first copy/pasting the code below into a shell prompt, then calling the function with the proper arguments, as shown. The function requires 3 arguments: the request origin, the URL of an asset in an AWS S3 bucket, and an HTTP method (which must be in UPPPER CASE). </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id105b625e81d2'><button class='copyBtn' data-clipboard-target='#id105b625e81d2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>function testCors { if [ -z "$1" ]; then echo "Error: No origin was provided"; exit 1; fi if [ -z "$2" ]; then echo "Error: No URL to test was provided"; exit 1; fi if [ "$3" ]; then METHOD="$3"; else METHOD=GET; fi curl -I -X OPTIONS \ --no-progress-meter \ -H "Origin: $1" \ -H "Access-Control-Request-Method: $METHOD" \ "$2" 2>&1 | \ grep --color=never 'Access-Control' }</pre> <p> The JSON file for testing CORS was <code><a href='https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/index.html#path-argument-type' target='_blank' rel='nofollow'>s3://</a>$BUCKET/testCors.json</code>: </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,testCors.json' download='testCors.json' title='Click on the file name to download the file'>testCors.json</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id4801de7477a2">&#123; "key1": "value1", "key2": "value2" &#125; </pre> <p> We will know if CORS is set up properly by receiving a header containing <code>Access-Control-Allow-Origin: *</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id490eab60e3e9'><button class='copyBtn' data-clipboard-target='#id490eab60e3e9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>URL="https://s3.amazonaws.com/$BUCKET/testCors.json" <span class='unselectable'>$ </span>testCors $ORIGIN $URL GET <span class='unselectable'><span class="bg_yellow">Access-Control-Allow-Origin: *</span> Access-Control-Allow-Methods: GET Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method </span></pre> <p> The origin worked when the bucket is accessed via a <code>GET</code> method sent to its <code>s3.amazonaws.com</code> DNS alias (yay!). </p> <p> CORScanner (<a href="/blog/2021/03/20/cors.html#corscanner">discussed in a previous blog post</a>) reported no issues: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idabadb5dab84f'><button class='copyBtn' data-clipboard-target='#idabadb5dab84f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cors -u s3.amazonaws.com/assets.ancientwarmth.com/testCors.json <span class='unselectable'>Starting CORS scan... Finished CORS scanning... </span></pre> </editor-fold> <editor-fold cf> <h2 id="cf">CloudFront</h2> <p> I have not worked through the process of using AWS CLI to obtain a JSON object describing the distribution, and then changing some properties and writing it back. So until that happy day comes, here are 2 screen shots of the <a href='https://console.aws.amazon.com/cloudfront/home' target='_blank' rel='nofollow'>AWS CloudFront web console</a> showing the settings. The first screen shot shows the <b>Behaviors</b> tab of the top-level details of the <code>assets.ancientwarmth.com</code> CloudFront distribution. </p> <div style=""> <picture> <source srcset="/blog/images/aws/cfBehaviorCors0.webp" type="image/webp"> <source srcset="/blog/images/aws/cfBehaviorCors0.png" type="image/png"> <img src="/blog/images/aws/cfBehaviorCors0.png" title="CloudFront / Edit Distribution / Behaviors <br> About to click on <b>Edit</b> (default behavior)" class=" liImg2 rounded shadow" alt="CloudFront / Edit Distribution / Behaviors <br> About to click on <b>Edit</b> (default behavior)" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> CloudFront / Edit Distribution / Behaviors <br> About to click on <b>Edit</b> (default behavior) </figcaption> </figure> </div> <p> My application does not require users to upload anything, so everything in the S3 bucket is truly static. Thus I have no need to <code>PUT</code>, <code>POST</code> or <code>DELETE</code> HTTP methods for the AWS S3 content. I have not seen a good explanation of why enabling <code>OPTIONS</code> HTTP methods is necessary, but every person on Stack Overflow who got CORS to work with AWS S3 says this was necessary. With that in mind, I set the following for the next screen shot: </p> <ul> <li><b>Viewer Protocol Policy:</b> <code>Redirect HTTP to HTTPS</code></li> <li><b>Allowed HTTP Methods:</b> <code>GET, HEAD, OPTIONS</code></li> <li><b>Cached HTTP Methods:</b> Enable <code>OPTIONS</code></li> <li><b>Use a cache policy and origin request policy:</b> (default is Use legacy cache settings, which is usually undesirable)</li> <li><b>Cache Policy:</b> <code>Managed-CachingOptimized</code></li> <li><b>Origin Request Policy:</b> <code>Managed-CORS-S3Origin</code></li> </ul> <div style=""> <picture> <source srcset="/blog/images/aws/cfBehaviorCors1.webp" type="image/webp"> <source srcset="/blog/images/aws/cfBehaviorCors1.png" type="image/png"> <img src="/blog/images/aws/cfBehaviorCors1.png" title="Editing default CloudFront distribution behavior" class=" liImg2 rounded shadow" alt="Editing default CloudFront distribution behavior" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> Editing default CloudFront distribution behavior </figcaption> </figure> </div> <h3 id="cfManagedCorsS3OriginPolicy">Managed CORS S3 Origin Poligy</h3> <p> AWS CloudFront's <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/using-managed-origin-request-policies.html' target='_blank' rel='nofollow'>managed origin request policy</a> called <code>Managed-CORS-S3Origin</code> includes the headers that enable cross-origin resource sharing (CORS) requests when the origin is an Amazon S3 bucket. This policy's settings are: </p> <ul> <li><b>Query strings included in origin requests</b>: None</li> <li><b>Headers included in origin requests</b>: <ul> <li><code>Origin</code></li> <li><code>Access-Control-Request-Headers</code></li> <li><code>Access-Control-Request-Method</code></li> </ul> </li> <li><b>Cookies included in origin requests</b>: None</li> </ul> <div style=""> <picture> <source srcset="/blog/images/aws/cfManagedCorsS3OriginPolicy.webp" type="image/webp"> <source srcset="/blog/images/aws/cfManagedCorsS3OriginPolicy.png" type="image/png"> <img src="/blog/images/aws/cfManagedCorsS3OriginPolicy.png" class=" liImg2 rounded shadow" /> </picture> </div> </editor-fold> <editor-fold wait> <h2 id="wait">Wait or Invalidate</h2> <p> Whenever you make a configuration change to a CloudFront distribution, or the contents change, the distributed assets will not reflect those changes until the next CloudFront invalidation. Automatic invalidations take 20 minutes. You can invalidate manually for near-instant gratification. I use my <a href='/blog/2021/03/22/command-line-aws-utilities.html#awsCfInvalidate'>AWS command-line utilities</a> to invalidate manually: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfd453e399656'><button class='copyBtn' data-clipboard-target='#idfd453e399656' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfS3DistId $BUCKET | awsCfInvalidate</pre> <p> Now the grand finale: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1a14894f2f47'><button class='copyBtn' data-clipboard-target='#id1a14894f2f47' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>testCors $ORIGIN $URL GET <span class='unselectable'>Access-Control-Allow-Origin: * Access-Control-Allow-Methods: GET Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method </span></pre> <div class="right" style="font-size: 3em;">&#128513;</div> <p> The presence of the <code>Access-Control-Allow-Origin</code> header indicates that CORS allowed the data file to be transferred from the content server (AWS S3/CloudFront) to the origin server (the command line). </p> </editor-fold> Cross-Origin Resource Sharing (CORS) 2021-03-20T00:00:00-04:00 https://mslinn.github.io/blog/2021/03/20/cors <editor-fold intro> <p> Many have tried to explain CORS, but most have not provided a clear explanation. I am going to try, then I will refer to explanations by others, who also provide examples. </p> </editor-fold> <editor-fold origin> <h2 id="origin">Origin and Origin Server</h2> <p> A website is delivered to web browsers from an <i>origin server</i>, or <i>origin</i> for short. The origin server is principally responsible for generating web pages. </p> <p> An origin is a combination of 3 things: </p> <ol> <li>A scheme (<code>http</code>, <code>https</code>, etc.)</li> <li>A (sub)domain, for example <code>localhost</code>, <code>blah.com</code> or <code>assets.blah.com</code>.</li> <li>A port, for example 80, 443, 8000, etc.</li> </ol> <p> All three things must match in order for two URLs to be considered to be from the same origin. For example: </p> <table class="table"> <tr> <th>URL 1</th> <th>URL 2</th> <th>Same Origin?</th> </tr> <tr> <td><code>http://blah.com</code></td> <td><code>http<span class="bg_yellow">s</span>://blah.com</code></td> <th>No</th> </tr> <tr> <td><code>https://blah.com</code></td> <td><code>https://<span class="bg_yellow">assets.</span>blah.com</code></td> <th>No</th> </tr> <tr> <td><code>https://blah.com</code></td> <td><code>https://blah.com<span class="bg_yellow">/path/page.html</span></code></td> <th>Yes</th> </tr> </table> </editor-fold> <editor-fold contentServer> <h2 id="contentServer">Content Server</h2> <p> In this article, I use the term <i>content server</i> to refer to sources of online information other than the origin server. Resources referenced by a web page, such as images, JavaScript, CSS, and data might be provided by the origin server, or they might come from a content server. </p> <p> Because every server has by definition a different origin, content servers always have a different origin than the origin server. Static resources (resources that do not change) are often served by <i>content delivery networks</i> (CDNs), which are also content servers. </p> <p> The Cross-Origin Resource Sharing (CORS) standard controls if a web page can load resources from content servers. Content servers are in charge of their content; they decide which origin servers they wish to co-operate with. When CORS support is properly configured, content servers include HTTP headers into their responses that tell a web browser if those resources may be read by the web page being constructed. </p> <p> Data is a special type of resource. CORS restricts how data is exchanged between the web page delivered to the web browser from the origin server and content servers. In particular, JSON and XML data communicated to and from content servers requires CORS authorization. Furthermore, requests (from the web browser) that send JSON, XML and other data formats to content servers also require CORS authorization. </p> <div class="pullQuote">Content servers are in charge of their content; they decide which origin servers they wish to co-operate with.</div> </editor-fold> <editor-fold ctype> <h2 id="ctype">Content-Type Header</h2> <p> The <a href='https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type' target='_blank' rel='nofollow'><code>Content-Type</code> header</a> is used to indicate the <a href='https://developer.mozilla.org/en-US/docs/Glossary/MIME_type' target='_blank' rel='nofollow'><code>media type</code></a> of the resource. The old name <i>MIME type</i> has been replaced by <i>media type</i>. <a href='https://www.iana.org/assignments/media-types/media-types.xhtml' target='_blank' rel='nofollow'>Here is a list of media types.</a> </p> <p> Media types with names that start with <code>application</code> require CORS authentication if they are delivered from content servers, for example <code>application/json</code> and <code>application/javascript</code>. </p> <p> As well, a few media types with names that start with <code>text</code> require CORS authentication if they are delivered from content servers, for example <code>text/xml</code> and <code>text/xml-external-parsed-entity</code>. </p> </editor-fold> <editor-fold resources> <h2 id="resources">Further Reading</h2> <h3 id="Kosaka">Mariko Kosaka</h3> <p> Mariko Kosaka has written an easy-to-understand article describing CORS, and provides a simple but effective working Express website for demonstration. </p> <div class="quote"> The same-origin policy tells the browser to block cross-origin requests. When you want to get a public resource from a different origin, the resource-providing server needs to tell the browser &lsquo;This origin where the request is coming from can access my resource&rsquo;. The browser remembers that and allows cross-origin resource sharing. <br><br> <span style="font-style: normal"> &nbsp; &ndash; <a href='https://web.dev/cross-origin-resource-sharing/' target='_blank' rel='nofollow'>Mariko Kosaka</a></span> </div> </editor-fold> <editor-fold gilling> <h3 id="Gilling">Derric Gilling and MDN</h3> <p> Derric Gilling has written a more in-depth yet very approachable article describing CORS. I've paraphrased his quoting of the Mozilla Developer Network documentation into the following: </p> <div class="quote"> CORS is a security mechanism that allows a web page from one domain or Origin to access a resource with a different domain (a cross-domain request). CORS is a relaxation of the same-origin policy implemented in modern browsers. Without features like CORS, websites are restricted to accessing resources from the same origin through what is known as same-origin policy. <br><br> Any CORS request has to be preflighted if:<br> <ul> <li>It uses methods other than <code>GET</code>, <code>HEAD</code> or <code>POST</code>.</li> <li> If POST is used to send request data with a <code>Content-Type</code> other than <code>application/x-www-form-urlencoded</code>, <code>multipart/form-data</code>, or <code>text/plain</code>. Examples: <ul> <li> A <code>POST</code> request sends an XML payload to the server; this requires the <code>Content-Type</code> header is set either to <code>application/xml</code> or <code>text/xml</code>. </li> <li> A website makes an AJAX call that <code>POST</code>s JSON data to a REST API, this requires the <code>Content-Type</code> header is set to <code>application/json</code>. </li> </ul> </li> </ul> <span style="font-style: normal">&nbsp; &ndash; <a href='https://www.moesif.com/blog/technical/cors/Authoritative-Guide-to-CORS-Cross-Origin-Resource-Sharing-for-REST-APIs/#how-cors-works/' target='_blank' rel='nofollow'>Derric Gilling</a> <br> &nbsp; &ndash; <a href='https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#preflighted_requests' target='_blank' rel='nofollow'>Mozilla Developer Network</a> </span> </div> </editor-fold> <editor-fold preflight> <h3 id="preflight">Preflight Requests</h3> <p> CORS preflight requests effectively double the latency of user requests for <a href='https://developer.mozilla.org/en-US/docs/Glossary/CRUD' target='_blank' rel='nofollow'>CRUD actions</a>. Client-side and server-side caching can help reduce this overhead for many circumstances. In <a href='https://www.mslinn.com/blog/2021/04/14/serverless-ecommerce.html#cf' target='_blank' rel='nofollow'>another blog post</a> I discuss how to use a CDN with multiple origin servers to completely eliminate the need for preflight requests. </p> <p> For additional background, please see: </p> <ul> <li><a href='https://www.rehanvdm.com/serverless/cloudfront-reverse-proxy-api-gateway-to-prevent-cors/index.html' target='_blank' rel='nofollow'>CloudFront reverse proxy API Gateway to prevent CORS</a> by Rehan van der Merwe</li> <li><a href='https://httptoolkit.tech/blog/cache-your-cors/' target='_blank' rel='nofollow'>Cache your CORS, for performance & profit</a> by Tim Perry</li> </ul> </editor-fold> <editor-fold keycdn> <h3 id="KeyCDN">KeyCDN</h3> <p> KeyCDN has an even more in-depth yet still very approachable <a href='https://www.keycdn.com/support/cors' target='_blank' rel='nofollow'>article describing CORS</a>. </p> </editor-fold> <editor-fold corscanner> <h2 id="corscanner">CORScanner</h2> <p> <a href='https://github.com/chenjj/CORScanner' target='_blank' rel='nofollow'>CORScanner</a> is a popular tool for detecting CORS misconfiguration. It is a Python module that can be executed as a shell command. Install CORScanner like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc2fe6f7f59f7'><button class='copyBtn' data-clipboard-target='#idc2fe6f7f59f7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pip install cors</pre> <p> The above adds a new executable called <code>cors</code> in the same directory where your <code>python</code> command resides. <p> <p> The <code>cors</code> documentation <a href='https://www.merriam-webster.com/thesaurus/conflate#verb' target='_blank' rel='nofollow'>conflates</a> the words URL and origin. Everywhere the word <code>URL</code> appears in the documentation, the word <code>origin</code> should be assumed. </p> </editor-fold> <editor-fold scannEx> <h3 id="scannEx">Example: Check Domain</h3> <p>Use the <code>-u</code> option to specify an origin to test:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0a4d4c6af0f8'><button class='copyBtn' data-clipboard-target='#id0a4d4c6af0f8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cors -u api.github.com <span class='unselectable'>Starting CORS scan... Finished CORS scanning... </span></pre> <p> To enable more debug info, use the <code>-v</code> option more than once. We can see that specifying <code>https</code> restricts testing to that <code>scheme</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4cbb1d54e304'><button class='copyBtn' data-clipboard-target='#id4cbb1d54e304' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cors -vv -u https://api.github.com <span class='unselectable'>Starting CORS scan... 2021-03-21 09:55:58 INFO Start checking reflect_origin for https://api.github.com 2021-03-21 09:55:58 INFO nothing found for {url: https://api.github.com, origin: https://evil.com, type: reflect_origin} 2021-03-21 09:55:58 INFO Start checking prefix_match for https://api.github.com 2021-03-21 09:55:58 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com.evil.com, type: prefix_match} 2021-03-21 09:55:58 INFO Start checking suffix_match for https://api.github.com 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://evilgithub.com, type: suffix_match} 2021-03-21 09:55:59 INFO Start checking trust_null for https://api.github.com 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: null, type: trust_null} 2021-03-21 09:55:59 INFO Start checking include_match for https://api.github.com 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://ithub.com, type: include_match} 2021-03-21 09:55:59 INFO Start checking not_escape_dot for https://api.github.com 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://api.githubacom, type: not_escape_dot} 2021-03-21 09:55:59 INFO Start checking custom_third_parties for https://api.github.com 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://whatever.github.io, type: custom_third_parties} 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: http://jsbin.com, type: custom_third_parties} 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://codepen.io, type: custom_third_parties} 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://jsfiddle.net, type: custom_third_parties} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: http://www.webdevout.net, type: custom_third_parties} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://repl.it, type: custom_third_parties} 2021-03-21 09:56:00 INFO Start checking special_characters_bypass for https://api.github.com 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com_.evil.com, type: special_characters_bypass} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com-.evil.com, type: special_characters_bypass} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&quot;.evil.com, type: special_characters_bypass} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com{.evil.com, type: special_characters_bypass} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com}.evil.com, type: special_characters_bypass} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com^.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%60.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com!.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com~.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com`.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com;.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com|.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&apos;.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com(.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com).evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com*.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com,.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com$.evil.com, type: special_characters_bypass} 2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com=.evil.com, type: special_characters_bypass} 2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%0b.evil.com, type: special_characters_bypass} 2021-03-21 09:56:03 INFO Start checking trust_any_subdomain for https://api.github.com 2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: https://evil.api.github.com, type: trust_any_subdomain} 2021-03-21 09:56:03 INFO Start checking https_trust_http for https://api.github.com 2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: http://api.github.com, type: https_trust_http} Finished CORS scanning... </span></pre> </editor-fold> <editor-fold scannEx2> <h3 id="scannEx2">Example: Check Origin</h3> <p> To check CORS misconfigurations of an origin: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id22ac039e608f'><button class='copyBtn' data-clipboard-target='#id22ac039e608f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cors -vvu https://api.github.com/users/mslinn/repos <span class='unselectable'>Starting CORS scan... 2021-03-21 10:08:49 INFO Start checking reflect_origin for https://api.github.com 2021-03-21 10:08:49 INFO nothing found for {url: https://api.github.com, origin: https://evil.com, type: reflect_origin} 2021-03-21 10:08:49 INFO Start checking prefix_match for https://api.github.com 2021-03-21 10:08:49 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com.evil.com, type: prefix_match} 2021-03-21 10:08:49 INFO Start checking suffix_match for https://api.github.com 2021-03-21 10:08:49 INFO nothing found for {url: https://api.github.com, origin: https://evilgithub.com, type: suffix_match} 2021-03-21 10:08:49 INFO Start checking trust_null for https://api.github.com 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: null, type: trust_null} 2021-03-21 10:08:50 INFO Start checking include_match for https://api.github.com 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://ithub.com, type: include_match} 2021-03-21 10:08:50 INFO Start checking not_escape_dot for https://api.github.com 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://api.githubacom, type: not_escape_dot} 2021-03-21 10:08:50 INFO Start checking custom_third_parties for https://api.github.com 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://whatever.github.io, type: custom_third_parties} 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: http://jsbin.com, type: custom_third_parties} 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://codepen.io, type: custom_third_parties} 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://jsfiddle.net, type: custom_third_parties} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: http://www.webdevout.net, type: custom_third_parties} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://repl.it, type: custom_third_parties} 2021-03-21 10:08:51 INFO Start checking special_characters_bypass for https://api.github.com 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com_.evil.com, type: special_characters_bypass} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com-.evil.com, type: special_characters_bypass} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&quot;.evil.com, type: special_characters_bypass} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com{.evil.com, type: special_characters_bypass} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com}.evil.com, type: special_characters_bypass} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com^.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%60.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com!.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com~.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com`.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com;.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com|.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&apos;.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com(.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com).evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com*.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com,.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com$.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com=.evil.com, type: special_characters_bypass} 2021-03-21 10:08:54 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:08:54 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%0b.evil.com, type: special_characters_bypass} 2021-03-21 10:08:54 INFO Start checking trust_any_subdomain for https://api.github.com 2021-03-21 10:08:54 INFO nothing found for {url: https://api.github.com, origin: https://evil.api.github.com, type: trust_any_subdomain} 2021-03-21 10:08:54 INFO Start checking https_trust_http for https://api.github.com 2021-03-21 10:08:54 INFO nothing found for {url: https://api.github.com, origin: http://api.github.com, type: https_trust_http} Finished CORS scanning... </span></pre> <p> If a <code>scheme</code> is not specified, then both <code>http</code> and <code>https</code> are tested: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf79875019a82'><button class='copyBtn' data-clipboard-target='#idf79875019a82' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cors -vvu api.github.com/users/mslinn/repos <span class='unselectable'>Starting CORS scan... 2021-03-21 10:03:30 INFO Start checking reflect_origin for http://api.github.com 2021-03-21 10:03:30 INFO Start checking reflect_origin for https://api.github.com 2021-03-21 10:03:30 INFO nothing found for {url: https://api.github.com, origin: https://evil.com, type: reflect_origin}2021-03-21 10:03:30 INFO Start checking prefix_match for https://api.github.com 2021-03-21 10:03:30 INFO nothing found for {url: http://api.github.com, origin: http://evil.com, type: reflect_origin} 2021-03-21 10:03:30 INFO Start checking prefix_match for http://api.github.com 2021-03-21 10:03:30 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com.evil.com, type: prefix_match} 2021-03-21 10:03:30 INFO Start checking suffix_match for https://api.github.com 2021-03-21 10:03:30 INFO nothing found for {url: https://api.github.com, origin: https://evilgithub.com, type: suffix_match} 2021-03-21 10:03:30 INFO Start checking trust_null for https://api.github.com 2021-03-21 10:03:30 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com.evil.com, type: prefix_match} 2021-03-21 10:03:30 INFO Start checking suffix_match for http://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: null, type: trust_null} 2021-03-21 10:03:31 INFO Start checking include_match for https://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: http://api.github.com, origin: http://evilgithub.com, type: suffix_match} 2021-03-21 10:03:31 INFO Start checking trust_null for http://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://ithub.com, type: include_match}2021-03-21 10:03:31 INFO Start checking not_escape_dot for https://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://api.githubacom, type: not_escape_dot} 2021-03-21 10:03:31 INFO Start checking custom_third_parties for https://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: http://api.github.com, origin: null, type: trust_null} 2021-03-21 10:03:31 INFO Start checking include_match for http://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://whatever.github.io, type: custom_third_parties} 2021-03-21 10:03:31 INFO nothing found for {url: http://api.github.com, origin: http://ithub.com, type: include_match} 2021-03-21 10:03:31 INFO Start checking not_escape_dot for http://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: http://jsbin.com, type: custom_third_parties} 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://codepen.io, type: custom_third_parties} 2021-03-21 10:03:31 INFO nothing found for {url: http://api.github.com, origin: http://api.githubacom, type: not_escape_dot} 2021-03-21 10:03:31 INFO Start checking custom_third_parties for http://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://jsfiddle.net, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: https://whatever.github.io, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: http://www.webdevout.net, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://repl.it, type: custom_third_parties} 2021-03-21 10:03:32 INFO Start checking special_characters_bypass for https://api.github.com 2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: http://jsbin.com, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com_.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com-.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: https://codepen.io, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&quot;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com{.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: https://jsfiddle.net, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com}.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: http://www.webdevout.net, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com^.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: https://repl.it, type: custom_third_parties} 2021-03-21 10:03:33 INFO Start checking special_characters_bypass for http://api.github.com 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%60.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com!.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com_.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com~.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com`.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com-.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com&quot;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com|.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com{.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&qpos;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com(.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com}.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com).evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com*.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com,.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com^.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com$.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com=.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com%60.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com!.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%0b.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO Start checking trust_any_subdomain for https://api.github.com 2021-03-21 10:03:35 INFO nothing found for {url: https://api.github.com, origin: https://evil.api.github.com, type: trust_any_subdomain} 2021-03-21 10:03:35 INFO Start checking https_trust_http for https://api.github.com 2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com~.evil.com, type: special_characters_bypass} 2021-03-21 10:03:35 INFO nothing found for {url: https://api.github.com, origin: http://api.github.com, type: https_trust_http} 2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com`.evil.com, type: special_characters_bypass} 2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com|.evil.com, type: special_characters_bypass} 2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com&.evil.com, type: special_characters_bypass} 2021-03-21 10:03:36 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com&apos;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:36 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com(.evil.com, type: special_characters_bypass} 2021-03-21 10:03:36 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com).evil.com, type: special_characters_bypass} 2021-03-21 10:03:36 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com*.evil.com, type: special_characters_bypass} 2021-03-21 10:03:37 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com,.evil.com, type: special_characters_bypass} 2021-03-21 10:03:38 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com$.evil.com, type: special_characters_bypass} 2021-03-21 10:03:38 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com=.evil.com, type: special_characters_bypass} 2021-03-21 10:03:38 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:03:38 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com%0b.evil.com, type: special_characters_bypass} 2021-03-21 10:03:38 INFO Start checking trust_any_subdomain for http://api.github.com 2021-03-21 10:03:39 INFO nothing found for {url: http://api.github.com, origin: http://evil.api.github.com, type: trust_any_subdomain} Finished CORS scanning... </span></pre> </editor-fold> AWS S3 and CloudFront SSL 2021-03-19T00:00:00-04:00 https://mslinn.github.io/blog/2021/03/19/aws-ssl <style> .sslScenario { border: thin solid grey; border-radius: 4px; padding: 8px; } </style> <p> SSL certificates need to match the domain they are served from. </p> <p> AWS uses one of several SSL certificates, depending on the <a href='https://docs.aws.amazon.com/AmazonS3/latest/dev-retired/UsingBucket.html#access-bucket-intro' target='_blank' rel='nofollow'>domain that an asset is requested from</a>. </p> <ul> <li> AWS S3 applies an SSL certificate for <code>https</code> requests. The SSL certificate chosen depends on the bucket endpoint used: <code>s3.amazonaws.com</code>, <code>*.s3.amazonaws.com</code>, or <code>s3.<i>region</i>.amazonaws.com</code>. </li> <li> AWS CloudFront will apply your custom SSL certificate (for example, a wildcard certificate such as <code>*.ancientwarmth.com</code>) for <code>https</code> requests to the CNAME for that distribution, otherwise it will apply the wildcard SSL certificate for <code>*.cloudfront.net</code>. </li> </ul> <h2 id="examples">Example <span class="code">SSL</span> URLs</h2> <p> My AWS S3 bucket called <code>assets.ancientwarmth.com</code> is served via a CloudFront distribution with URL <code>d1bci9l8cjf24o.cloudfront.net</code> that applies a wildcard SSL certificate for <code>*.ancientwarmth.com</code> that I created using <a href='https://aws.amazon.com/certificate-manager/' target='_blank' rel='nofollow'>AWS Certificate Manager</a>. I defined a CNAME called <code>assets.ancientwarmth.com</code> for that same CloudFront distribution using Route 53. </p> <p> All of the following URLs can be used to access my content, providing the SSL certificate matches the requested domain. </p> <p class="sslScenario"> <b>URL:</b> <code>https://d1bci9l8cjf24o.cloudfront.net</code><br> <b>Origin Type:</b> CloudFront distribution<br> <b>SSL certificate origin:</b> <code>*.cloudfront.net</code><br> <b>Valid SSL certificate?</b> Yes. </p> <p class="sslScenario"> <b>URL:</b> <code>https://assets.ancientwarmth.com</code><br> <b>Origin Type:</b> CloudFront distribution<br> <b>SSL certificate origin:</b> <code>*.ancientwarmth.com</code><br> <b>Valid SSL certificate?</b> Yes. (I created this wildcard certificate using Route 53.) </p> <p class="sslScenario"> <b>S3 path-style URL:</b> <code>https://s3.us-east-1.amazonaws.com/assets.ancientwarmth.com</code><br> <b>Origin Type:</b> S3 bucket<br> <b>SSL certificate origin:</b> <code>s3.us-east-1.amazonaws.com</code><br> <b>Valid SSL certificate?</b> Yes. </p> <p class="sslScenario"> <b>S3 dot URL:</b> <code>https://assets.ancientwarmth.com.s3.amazonaws.com</code><br> <b>Origin Type:</b> S3 bucket<br> <b>SSL certificate origin:</b> <code>*.s3.amazonaws.com</code><br> <b>Valid SSL certificate?</b> No, does not match URL (wildcards only match one subdomain). </p> <p class="sslScenario"> <b>S3 dot Region URL:</b> <code>https://assets.ancientwarmth.com.s3.us-east-1.amazonaws.com</code><br> <b>Origin Type:</b> S3 bucket<br> <b>SSL certificate origin:</b> <code>s3.amazonaws.com</code><br> <b>Valid SSL certificate?</b> No, does not match URL. </p> <h2 id="curl">Testing with <span class="code">curl</span></h2> <p> <code>Curl</code> is often used to test SSL requests. In the following <code>curl</code> commands, the <a href='https://curl.se/docs/manpage.html#-I' target='_blank' rel='nofollow'><code>-I</code> option</a> just fetches the headers, and the <a href='https://curl.se/docs/manpage.html#-v' target='_blank' rel='nofollow'><code>-v</code> option</a> provides verbose output. You can see the SSL certificate negotation. </p> <p> Fetching an asset from a CloudFront distribution using the AWS <code>*.cloudfront.net</code> wildcard SSL certificate: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7e5266acf709'><button class='copyBtn' data-clipboard-target='#id7e5266acf709' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -Iv \ https://d1bci9l8cjf24o.cloudfront.net/js/jquery.modal.min.js <span class='unselectable'>* Trying 52.85.149.22:443... * TCP_NODELAY set * Connected to d1bci9l8cjf24o.cloudfront.net (52.85.149.22) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=*.cloudfront.net * start date: Feb 22 00:00:00 2021 GMT * expire date: Feb 21 23:59:59 2022 GMT * subjectAltName: host "d1bci9l8cjf24o.cloudfront.net" matched cert's "*.cloudfront.net" * issuer: C=US; O=DigiCert Inc; CN=DigiCert Global CA G2 * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x5585dcfaf7e0) > HEAD /js/jquery.modal.min.js HTTP/2 > Host: d1bci9l8cjf24o.cloudfront.net > user-agent: curl/7.68.0 > accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS == 128)! < HTTP/2 200 HTTP/2 200 < content-type: application/javascript content-type: application/javascript < content-length: 4953 content-length: 4953 < date: Sat, 20 Mar 2021 14:11:34 GMT date: Sat, 20 Mar 2021 14:11:34 GMT < last-modified: Sat, 20 Mar 2021 03:14:08 GMT last-modified: Sat, 20 Mar 2021 03:14:08 GMT < etag: "c8f50397e0560719c62a35318f413e16" etag: "c8f50397e0560719c62a35318f413e16" < accept-ranges: bytes accept-ranges: bytes < server: AmazonS3 server: AmazonS3 < x-cache: Miss from cloudfront x-cache: Miss from cloudfront < via: 1.1 0712e4ad4264127dfcb76a114b130495.cloudfront.net (CloudFront) via: 1.1 0712e4ad4264127dfcb76a114b130495.cloudfront.net (CloudFront) < x-amz-cf-pop: IAD89-C3 x-amz-cf-pop: IAD89-C3 < x-amz-cf-id: hWrjwajqqkI9-rJnK1BSQqkX9DPXIlZJLfa28UaIeze7taBP5kqMNg== x-amz-cf-id: hWrjwajqqkI9-rJnK1BSQqkX9DPXIlZJLfa28UaIeze7taBP5kqMNg== < * Connection #0 to host d1bci9l8cjf24o.cloudfront.net left intact </span></pre> <p> Fetching an asset from a CloudFront distribution using my <code>*.ancientwarmth.com</code> wildcard SSL certificate: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id83f924a98349'><button class='copyBtn' data-clipboard-target='#id83f924a98349' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -Iv \ https://assets.ancientwarmth.com/js/jquery.modal.min.js <span class='unselectable'>modal.min.js> https://assets.ancientwarmth.com/js/jquery.modal.min.js * Trying 13.226.36.16:443... * TCP_NODELAY set * Connected to assets.ancientwarmth.com (13.226.36.16) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=*.ancientwarmth.com * start date: Mar 14 00:00:00 2021 GMT * expire date: Apr 12 23:59:59 2022 GMT * subjectAltName: host "assets.ancientwarmth.com" matched cert's "*.ancientwarmth.com" * issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x5599720157e0) > GET /js/jquery.modal.min.js HTTP/2 > Host: assets.ancientwarmth.com > user-agent: curl/7.68.0 > accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS == 128)! < HTTP/2 200 < content-type: application/javascript < content-length: 4953 < date: Sat, 20 Mar 2021 12:34:25 GMT < last-modified: Sat, 20 Mar 2021 03:14:08 GMT < etag: "c8f50397e0560719c62a35318f413e16" < accept-ranges: bytes < server: AmazonS3 < x-cache: Hit from cloudfront < via: 1.1 4e3df844337032b56b8434990b0f76ca.cloudfront.net (CloudFront) < x-amz-cf-pop: EWR53-C2 < x-amz-cf-id: 17Dxn6QqtK6JfkJwFnESVYsG-Cbzu6H-sOTWcGDpznGcpjIZbhJDRA== < age: 5195 < * Connection #0 to host assets.ancientwarmth.com left intact </span></pre> <p> Fetching an asset from an S3 bucket using an AWS SSL certificate for all S3 buckets in region <code>us-east-1</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfd0abdd97c61'><button class='copyBtn' data-clipboard-target='#idfd0abdd97c61' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -Iv \ https://s3.us-east-1.amazonaws.com/assets.ancientwarmth.com/js/jquery.modal.min.js <span class='unselectable'>* Trying 52.216.24.46:443... * TCP_NODELAY set * Connected to s3.us-east-1.amazonaws.com (52.216.24.46) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=s3.amazonaws.com * start date: Aug 4 00:00:00 2020 GMT * expire date: Aug 9 12:00:00 2021 GMT * subjectAltName: host "s3.us-east-1.amazonaws.com" matched cert's "s3.us-east-1.amazonaws.com" * issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2 * SSL certificate verify ok. > GET /assets.ancientwarmth.com/js/jquery.modal.min.js HTTP/1.1 > Host: s3.us-east-1.amazonaws.com > User-Agent: curl/7.68.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < x-amz-id-2: xIXUHy7YBpjZaF+cpGoSAwNvC5+NrmM5pmJM8nInI6weEkbht350xSPC9+yOBJrGs9GY0hn2V7Y= < x-amz-request-id: JM2K8HR109JNMMB1 < Date: Sat, 20 Mar 2021 14:03:56 GMT < Last-Modified: Sat, 20 Mar 2021 03:14:08 GMT < ETag: "c8f50397e0560719c62a35318f413e16" < Accept-Ranges: bytes < Content-Type: application/javascript < Content-Length: 4953 < Server: AmazonS3 < * Connection #0 to host s3.us-east-1.amazonaws.com left intact </span></pre> Pretty JSON Reduces Errors and Fatigue 2021-02-23T00:00:00-05:00 https://mslinn.github.io/blog/2021/02/23/pretty-json <p> I've been using <a href='https://stedolan.github.io/jq/' target='_blank' rel='nofollow'>jq</a> to format my JSON for years. It is easy to format a JSON document, just pass it through <code>jq</code> without any options or arguments. Notice, however, that a lot of vertical space is wasted using this formatting: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfb8380214e8d'><button class='copyBtn' data-clipboard-target='#idfb8380214e8d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ jq < blog/colors.json { "colors": [ { "color": "black", "hex": "#000", "rgb": [ 0, 0, 0 ] }, { "color": "red", "hex": "#f00", "rgb": [ 255, 0, 0 ] }, { "color": "yellow", "hex": "#ff0", "rgb": [ 255, 255, 0 ] }, { "color": "green", "hex": "#0f0", "rgb": [ 0, 255, 0 ] }, { "color": "cyan", "hex": "#0ff", "rgb": [ 0, 255, 255 ] }, { "color": "blue", "hex": "#00f", "rgb": [ 0, 0, 255 ] }, { "color": "magenta", "hex": "#f0f", "rgb": [ 255, 0, 255 ] }, { "color": "white", "hex": "#fff", "rgb": [ 255, 255, 255 ] } ] }</pre> <p> After reading <a href='http://www.ohler.com/dev/pretty.html' target='_blank' rel='nofollow'>The Pretty JSON Revolution</a> I decided to try the program the article mentioned, <code>oj</code>. <code>oj</code> is a Go program. Here is how I compiled it on Ubuntu: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id55737a21e9ed'><button class='copyBtn' data-clipboard-target='#id55737a21e9ed' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install golang-go <span class='unselectable'>$ </span>go get github.com/ohler55/ojg <span class='unselectable'>$ </span>go get github.com/ohler55/ojg/cmd/oj</pre> <p> By default, compiled go projects are placed in the <code>~/go/bin/</code> directory. Here is how I added that directory the the <code>PATH</code>, and made an alias for invoking the program with the proper options for maximum prettiness: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4645829b1f03'><button class='copyBtn' data-clipboard-target='#id4645829b1f03' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo "$HOME/go/bin/:$PATH" >> ~/.bashrc <span class='unselectable'>$ </span>echo "alias pprint='oj -i 2 -s -p 80.3'" >> ~/.bash_aliases <span class='unselectable'>$ </span>source ~/.bashrc</pre> <p> Pretty-printing the JSON in <code>colors.json</code> with <code>oj</code> is easy: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7dbfc765ea00'><button class='copyBtn' data-clipboard-target='#id7dbfc765ea00' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pprint colors.json <span class='unselectable'>{ "colors": [ {"color": "black", "hex": "#000", "rgb": [0, 0, 0]}, {"color": "red", "hex": "#f00", "rgb": [255, 0, 0]}, {"color": "yellow", "hex": "#ff0", "rgb": [255, 255, 0]}, {"color": "green", "hex": "#0f0", "rgb": [0, 255, 0]}, {"color": "cyan", "hex": "#0ff", "rgb": [0, 255, 255]}, {"color": "blue", "hex": "#00f", "rgb": [0, 0, 255]}, {"color": "magenta", "hex": "#f0f", "rgb": [255, 0, 255]}, {"color": "white", "hex": "#fff", "rgb": [255, 255, 255]} ] } </span></pre> <p> I like it! </p> <style>.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }</style><div class='embed-container'> <iframe title="YouTube video player" width="640" height="390" src="//www.youtube.com/embed/34wJt3pRY0w" frameborder="0" allowfullscreen></iframe></div> <p style="text-align: center"> <i>Give it to Mikey. He won't eat it. He hates everything!</i> </p> <p> I will continue to use <a href='https://stedolan.github.io/jq/manual/' target='_blank' rel='nofollow'><code>jq</code> for queries</a>, but I'll use <code>oj</code> for pretty-printing from now on. </p> JavaScript Named Arguments and Class Constructors 2021-02-11T00:00:00-05:00 https://mslinn.github.io/blog/2021/02/11/javascript-named-arguments <p> Named arguments make a program safe from errors caused by changes to method arguments. JavaScript named arguments can appear in any order. Default values for parameters allow an API to evolve gracefully without runtime errors. </p> <p> Building on the article entitled <a href='https://afontcu.medium.com/cool-javascript-9-named-arguments-functions-that-get-and-return-objects-337b6f8cfa07' target='_blank' rel='nofollow'>Cool JavaScript 9: Named arguments — Functions that get and return Objects</a>, this article shows how JavaScript class constructors can use named arguments, optionally define default values for parameters, and conveniently inflate new class instances from JSON. </p> <p> In this article I use Node.js for convenience, however the code shown will run in all modern web browsers. </p> <h2 id="stdArgs">JavaScript Class Definition Encapsulating Properties</h2> <p> Let&rsquo;s quickly review how to define a JavaScript class and instantiate an instance. Here is a simple JavaScript / ECMAScript 6 class that encapsulates two properties: <code>id</code> and <code>parts</code>. The constructor merely lists the names of the parameters, which happen to be the same as the names of the class properties. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcb3760bd668d'><button class='copyBtn' data-clipboard-target='#idcb3760bd668d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span> js <span class='unselectable'>Welcome to Node.js v12.18.2. Type ".help" for more information. > </span>class Ingredient { <span class='unselectable'>... </span> constructor(id, parts) { <span class='unselectable'>..... </span> this.id = id; <span class='unselectable'>..... </span> this.parts = parts; <span class='unselectable'>..... </span> } <span class='unselectable'>... </span>} <span class='unselectable'>undefined </span></pre> <p> New <code>Ingredient</code> instances can be created using this familiar syntax: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id954d66bab8d7'><button class='copyBtn' data-clipboard-target='#id954d66bab8d7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredient = new Ingredient("123", 10); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredient <span class='unselectable'>Ingredient { id: '123', parts: 10 } </span></pre> <h2 id="lits">Object Literals</h2> <p> Object literals look like JSON objects, but without quotes around property names. For example, the following defines an object literal called <code>lit</code> with 2 properties, called <code>id</code> and <code>parts</code>, with values <code>"123"</code> and <code>10</code>, respectively. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8afe4044abb1'><button class='copyBtn' data-clipboard-target='#id8afe4044abb1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span> var lit = {id: "123", parts: 10}; <span class='unselectable'>undefined </span> <span class='unselectable'>$ </span>lit <span class='unselectable'>{ id: '123', parts: 10 } </span> <span class='unselectable'>> </span>lit.id <span class='unselectable'>'123' </span> <span class='unselectable'>> </span>lit.parts <span class='unselectable'>10 </span></pre> <h2 id="jsonArgs">Use Object Literals to Define Arguments</h2> <p> We can define a class similar to <code>Ingredient</code>, but with the arguments replaced by a something that looks like an object literal without values. For want of a better term I call this an <i>object name literal</i>. The following class definition encapsulates the same two properties as before as an object name literal. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfcc6da5ecd5a'><button class='copyBtn' data-clipboard-target='#idfcc6da5ecd5a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>class IngredientX { <span class='unselectable'>... </span> constructor(<span class="bg_yellow">{id, parts}</span>) { <span class='unselectable'>..... </span> this.id = id; <span class='unselectable'>..... </span> this.parts = parts; <span class='unselectable'>..... </span> } <span class='unselectable'>... </span>} <span class='unselectable'>undefined </span></pre> <p> New <code>IngredientX</code> instances can be created from an object literal: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id119646c08f93'><button class='copyBtn' data-clipboard-target='#id119646c08f93' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredientX1 = new IngredientX(<span class="bg_yellow">{id: "123", parts: 10 }</span>); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientX1 <span class='unselectable'>IngredientX { id: '123', parts: 10 } </span></pre> <p> Because the <code>IngredientX</code> class definition requires an object name literal (or a JSON object, more on that later) to provide constructor arguments, constructor invocations must specify the names of each parameter being passed to the constructor arguments. This has the benefit of making your software more robust in the face of changing method signatures. </p> <p> Caution: new <code>IngredientX</code> instances cannot be created from scalar arguments. JavaScript gives no error or warning if you do not: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd8c361011707'><button class='copyBtn' data-clipboard-target='#idd8c361011707' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredientX2 = new IngredientX("123", 10); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientX2 <span class='unselectable'>IngredientX { id: <span class="bg_yellow">undefined</span>, parts: <span class="bg_yellow">undefined</span> } </span></pre> <h2 id="jsonArgs">JSON Object Can Be Supplied Instead of Object Literals</h2> <p> JSON objects can be provided as arguments instead of object literals. This is extremely handy. Replacing several arguments with a JSON object would possibly be the most significant improvement in robustness that could be made to a JavaScript project. The number of runtime errors encountered as a code base evolves would be greatly reduced. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfc048c98ff66'><button class='copyBtn' data-clipboard-target='#idfc048c98ff66' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredientX3 = new IngredientX({ <span class='unselectable'>... </span> <span class="bg_yellow">"id"</span>: "123", <span class='unselectable'>... </span> <span class="bg_yellow">"parts"</span>: 10 <span class='unselectable'>... </span> }); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientX3 <span class='unselectable'>IngredientX { id: '123', parts: 10 } </span></pre> <h2 id="jsonArgs">Arguments and Parameters Can Be Provided In Any Order</h2> <p> This definition of <code>ingredientX4</code> is identical to the definition of <code>ingredientX3</code>, even though the order of the arguments has been reversed: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9a336f2de8f8'><button class='copyBtn' data-clipboard-target='#id9a336f2de8f8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredientX4 = new IngredientX({ <span class='unselectable'>... </span> <span class="bg_yellow">"parts"</span>: 10, <span class='unselectable'>... </span> <span class="bg_yellow">"id"</span>: "123" <span class='unselectable'>... </span> }); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientX4 <span class='unselectable'>IngredientX { id: '123', parts: 10 } </span></pre> <p> The parameters in the function or method declaration are also insensitive to ordering: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida57b3d222bdf'><button class='copyBtn' data-clipboard-target='#ida57b3d222bdf' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>class IngredientXReordered { <span class='unselectable'>... </span> constructor(<span class="bg_yellow">{parts, id}</span>) { <span class='unselectable'>..... </span> this.parts = parts; <span class='unselectable'>..... </span> this.id = id; <span class='unselectable'>..... </span> } <span class='unselectable'>... </span>} <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>var ingredientX5 = new IngredientXReordered({ <span class='unselectable'>... </span> <span class="bg_yellow">"parts"</span>: 10, <span class='unselectable'>... </span> <span class="bg_yellow">"id"</span>: "123" <span class='unselectable'>... </span> }); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientX5 <span class='unselectable'>IngredientXReordered { id: '123', parts: 10 } </span></pre> <h2 id="litArgs">Object Literals Can Be Used With Any Method</h2> <p> Object literals / named arguments can be used to define the signature of any function or method, not just class constructors. For example: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0e296779001d'><button class='copyBtn' data-clipboard-target='#id0e296779001d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>class IngredientY { <span class='unselectable'>... </span> constructor({id, parts}) { <span class='unselectable'>..... </span> this.id = id; <span class='unselectable'>..... </span> this.parts = parts; <span class='unselectable'>..... </span> } <span class='unselectable'>... </span> <span class='unselectable'>... </span> mix(<span class="bg_yellow">{duration, intensity}</span>) { <span class='unselectable'>... </span> console.log(`Shake for ${duration} hours at intensity ${intensity}.`); <span class='unselectable'>... </span> } <span class='unselectable'>... </span> } <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>var ingredientY = new IngredientY({id: "123", parts: 10 }); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientY.mix(<span class="bg_yellow">{duration: 2.5, intensity: 2}</span>); <span class='unselectable'>Shake for 2.5 hours at intensity 2. </span> <span class='unselectable'>undefined </span></pre> <h2 id="jsonArgs">Default Values for Named Arguments</h2> <p> To make this example more interesting, the default value for <code>id</code> will be generated as a GUID. <a href='https://stackoverflow.com/questions/105034/how-to-create-a-guid-uuid' target='_blank' rel='nofollow'>Here are some other GUID implementations</a>, but the best implementations have dependencies and that would just make the article more complex than necessary. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8f63c9bf75ec'><button class='copyBtn' data-clipboard-target='#id8f63c9bf75ec' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>function uuidv4() { <span class='unselectable'>... </span> return 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, <span class='unselectable'>... </span> function(c) { <span class='unselectable'>..... </span> var r = Math.random() * 16 | 0, v = c == 'x' ? r : (r & 0x3 | 0x8); <span class='unselectable'>..... </span> return v.toString(16); <span class='unselectable'>..... </span> }); <span class='unselectable'>... </span>} <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>uuidv4() <span class='unselectable'>'b13137c1-1598-42ca-9498-c1502e5405ed' </span></pre> <p> A JavaScript object literal or JSON object must be passed to a method whose parameters were defined by object literal names. If a name/value pair is not provided in the argument then the default parameter value is used. Some examples should help demonstrate how this works: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8caf9b61e9ec'><button class='copyBtn' data-clipboard-target='#id8caf9b61e9ec' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>class IngredientZ { <span class='unselectable'>... </span> constructor({id<span class="bg_yellow">=uuidv4()</span>, parts<span class="bg_yellow">=10</span>}) { <span class='unselectable'>..... </span> this.id = id; <span class='unselectable'>..... </span> this.parts = parts; <span class='unselectable'>..... </span> } <span class='unselectable'>... </span> <span class='unselectable'>... </span> mix({duration<span class="bg_yellow">=1.2</span>, intensity<span class="bg_yellow">=6</span>}) { <span class='unselectable'>... </span> console.log(`Shake for ${duration} hours at intensity ${intensity}.`); <span class='unselectable'>... </span> } <span class='unselectable'>... </span> } <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>var ingredientZ1 = new IngredientZ(<span class="bg_yellow">{parts: 4}</span>); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientZ1 <span class='unselectable'>IngredientZ { id: <span class="bg_yellow">'4290dc1a-4f4c-4579-9e27-39b68085ad97'</span>, parts: <span class="bg_yellow">4</span> } </span> <span class='unselectable'>undefined </span></pre> <p> Empty objects are allowed as arguments. All this means is that default values are used for all parameters of the object name literal. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc64a88c85024'><button class='copyBtn' data-clipboard-target='#idc64a88c85024' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredientZ2 = new IngredientZ(<span class="bg_yellow">{}</span>); <span class='unselectable'>undefined </span> > ingredientZ2 <span class='unselectable'>IngredientZ { id: '9e70dc12-1f4c-3579-6a17-49a68385bf73', parts: 10 } </span> <span class='unselectable'>> </span>ingredientZ2.mix(<span class="bg_yellow">{}</span>); <span class='unselectable'>Shake for 2.5 hours at intensity 2. </span></pre> <p> Missing objects result in a syntax error. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfcfa853806f1'><button class='copyBtn' data-clipboard-target='#idfcfa853806f1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>ingredientZ2.mix<span class="bg_yellow">()</span>; <span class='unselectable'>Uncaught TypeError: Cannot r