Mike Slinn's Blog 2021-08-27T19:16:56-04:00 https://mslinn.github.io/blog Mike Slinn mslinn@gmail.com Disappointing Scala 3 Installation Experience 2021-05-19T00:00:00-04:00 https://mslinn.github.io/blog/2021/05/19/installing-scala-3.0 <p> I run <a href='https://scalacourses.com' target='_blank' rel='nofollow'>ScalaCourses.com</a>, an online Scala training web site. After years of hype, Scala 3 is now available. </p> <h2 id="prime">Scala 3: Not Yet Ready for Production</h2> <p> <a href='https://www.infoq.com/news/2021/03/scala3/' target='_blank' rel='nofollow'>Scala 3</a> was eight years in the making. You would never know that from the horrible installation process and the disappointing installation instructions. </p> <p> It is going to take quite a while before <a href='https://scalatimes.com/d374aea433' target='_blank' rel='nofollow'>Scala 3</a>, which was known as Dotty before it was released, can be trusted in production. According to the <a href='https://github.com/lampepfl/dotty' target='_blank' rel='nofollow'>Dotty GitHub project</a>, the only published future milestone is <a href='https://github.com/lampepfl/dotty/milestones' target='_blank' rel='nofollow'>v3.1.0, which has no due date</a>. Given that Scala 3 uses an entirely new build process, and an entirely new (and nonstandard) installation process, and that the internals of the Scala compiler were almost completely replaced, I doubt that version will be stable enough for use on production projects. </p> <h2 id="choices">Installation Choices</h2> <p> Installation cholices include: </p> <ul> <li>Command-line Scala has limited use cases, but is nice to have around for occassional experimentation.</li> <li> <a href='https://www.scala-lang.org/blog/2021/04/08/scala-3-in-sbt.html' target='_blank' rel='nofollow'>SBT</a> (which features an enhanced Scala REPL) is very helpful for interactively developing code, as well as for building and testing. </li> <li> <a href='https://www.jetbrains.com/help/idea/discover-intellij-idea-for-scala.html' target='_blank' rel='nofollow'>IntelliJ</a> provides the best Scala coding productivity and code quality. </li> <li> <a href='https://shunsvineyard.info/2020/11/20/setting-up-vs-code-for-scala-development-on-wsl/' target='_blank' rel='nofollow'>VSCode</a> has been playing catch-up but is not full-featured yet. </li> </ul> <h2 id="install">Installation Transcript</h2> <p> The following is the transcript of how I installed command-line Scala on Ubuntu 20.10 running under WSL2. </p> <h3 class="numbered" id="remove_scala2">Remove Scala 2</h3> <p> This step is not required. Scala 2 and Scala 3 can easily co-exist on the same system because their names are different. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc00557b8e573'><button class='copyBtn' data-clipboard-target='#idc00557b8e573' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt remove scala <span class='unselectable'>Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be REMOVED: scala 0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded. After this operation, 666 MB disk space will be freed. Do you want to continue? [Y/n] (Reading database ... 236024 files and directories currently installed.) Removing scala (2.13.4-400) ... Processing triggers for man-db (2.9.3-2) ... </span></pre> <h3 class="numbered" id="cs">Install Coursier</h3> <p> Coursier is a multithreaded downloader for project dependencies, and it now also downloads Scala 3. SBT uses coursier internally. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id664a9a7bc756'><button class='copyBtn' data-clipboard-target='#id664a9a7bc756' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -fLo cs https://git.io/coursier-cli-"$(uname | tr LD ld)" <span class='unselectable'>&nbsp; % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 144 100 144 0 0 285 0 --:--:-- --:--:-- --:--:-- 6000 100 57.1M 100 57.1M 0 0 3656k 0 0:00:15 0:00:15 --:--:-- 4092k </span> <span class='unselectable'>$ </span>mv cs ~/.local/bin/ <span class='unselectable'>$ </span>chmod a+x ~/.local/bin/cs <span class='unselectable'>$ </span>cs install cs <span class='unselectable'>https://repo1.maven.org/maven2/io/get-coursier/apps/maven-metadata.xml 100.0% [##########] 1.8 KiB (8.5 KiB / s) https://repo1.maven.org/maven2/io/get-coursier/coursier-cli_2.12/maven-metadata.xml No new update since 2021-03-23 14:35:16 Wrote cs Warning: /home/mslinn/.local/share/coursier/bin is not in your PATH To fix that, add the following line to ~/.bashrc export PATH="$PATH:/home/mslinn/.local/share/coursier/bin" </span> <span class='unselectable'>$ </span> export PATH="$PATH:/home/mslinn/.local/share/coursier/bin"</pre> <p> Now let's test Coursier: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8b61e72d0709'><button class='copyBtn' data-clipboard-target='#id8b61e72d0709' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cs <span class='unselectable'>Coursier 2.0.16 Usage: cs [options] [command] [command-options] Available commands: bootstrap, channel, complete, fetch, get, install, java, java-home, launch, list, publish, resolve, setup, uninstall, update, search Type cs command --help for help on an individual command </span></pre> <p> This is the Coursier help message for the <code>install</code> subcommand: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8f6f13ef15e6'><button class='copyBtn' data-clipboard-target='#id8f6f13ef15e6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cs install --help <span class='unselectable'>Command: install Usage: cs install --cache &lt;string?&gt; Cache directory (defaults to environment variable COURSIER_CACHE, or ~/.cache/coursier/v1 on Linux and ~/Library/Caches/Coursier/v1 on Mac) --mode | -m &lt;offline|update-changing|update|missing|force&gt; Download mode (default: missing, that is fetch things missing from cache) --ttl | -l &lt;duration&gt; TTL duration (e.g. &quot;24 hours&quot;) --parallel | -n &lt;int&gt; Maximum number of parallel downloads (default: 6) --checksum &lt;checksum1,checksum2,...&gt; Checksum types to check - end with none to allow for no checksum validation if no checksum is available, example: SHA-256,SHA-1,none --retry-count &lt;int&gt; Retry limit for Checksum error when fetching a file --cache-file-artifacts | --cfa &lt;bool&gt; Flag that specifies if a local artifact should be cached. --follow-http-to-https-redirect &lt;bool&gt; Whether to follow http to https redirections --credentials &lt;host(realm) user:pass|host user:pass&gt; Credentials to be used when fetching metadata or artifacts. Specify multiple times to pass multiple credentials. Alternatively, use the COURSIER_CREDENTIALS environment variable --credential-file &lt;string*&gt; Path to credential files to read credentials from --use-env-credentials &lt;bool&gt; Whether to read credentials from COURSIER_CREDENTIALS (env) or coursier.credentials (Java property), along those passed with --credentials and --credential-file --quiet | -q &lt;counter&gt; Quiet output --verbose | -v &lt;counter&gt; Increase verbosity (specify several times to increase more) --progress | -P &lt;bool&gt; Force display of progress bars --log-changing &lt;bool&gt; Log changing artifacts --log-channel-version | --log-index-version | --log-jvm-index-version &lt;bool&gt; Log app channel or JVM index version --graalvm-home &lt;string?&gt; --graalvm-option &lt;string*&gt; --graalvm-default-version &lt;string?&gt; --install-dir | --dir &lt;string?&gt; --install-platform &lt;string?&gt; Platform for prebuilt binaries (e.g. &quot;x86_64-pc-linux&quot;, &quot;x86_64-apple-darwin&quot;, &quot;x86_64-pc-win32&quot;) --install-prefer-prebuilt &lt;bool&gt; --only-prebuilt &lt;bool&gt; Require prebuilt artifacts for native applications, don&#39;t try to build native executable ourselves --repository | -r &lt;maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|ivy:$pattern&gt; Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots) --default-repositories &lt;bool&gt; --proguarded &lt;bool?&gt; --channel &lt;org:name&gt; Channel for apps --default-channels &lt;bool&gt; Add default channels --contrib &lt;bool&gt; Add contrib channel --file-channels &lt;bool&gt; Add channels read from the configuration directory --jvm &lt;string?&gt; --jvm-dir &lt;string?&gt; --system-jvm &lt;bool?&gt; --local-only &lt;bool&gt; --update &lt;bool&gt; --jvm-index &lt;string?&gt; --repository | -r &lt;maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|scala-integration|scala-nightlies|ivy:$pattern|jitpack|clojars|jcenter|apache:$repo&gt; Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots) --no-default &lt;bool&gt; Do not add default repositories (~/.ivy2/local, and Central) --sbt-plugin-hack &lt;bool&gt; Modify names in Maven repository paths for sbt plugins --drop-info-attr &lt;bool&gt; Drop module attributes starting with &#39;info.&#39; - these are sometimes used by projects built with sbt --channel &lt;org:name&gt; Channel for apps --default-channels &lt;bool&gt; Add default channels --contrib &lt;bool&gt; Add contrib channel --file-channels &lt;bool&gt; Add channels read from the configuration directory --repository | -r &lt;maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|scala-integration|scala-nightlies|ivy:$pattern|jitpack|clojars|jcenter|apache:$repo&gt; Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots) --no-default &lt;bool&gt; Do not add default repositories (~/.ivy2/local, and Central) --sbt-plugin-hack &lt;bool&gt; Modify names in Maven repository paths for sbt plugins --drop-info-attr &lt;bool&gt; Drop module attributes starting with &#39;info.&#39; - these are sometimes used by projects built with sbt --channel &lt;org:name&gt; Channel for apps --default-channels &lt;bool&gt; Add default channels --contrib &lt;bool&gt; Add contrib channel --file-channels &lt;bool&gt; Add channels read from the configuration directory --env &lt;bool&gt; --disable-env | --disable &lt;bool&gt; --setup &lt;bool&gt; --user-home &lt;string?&gt; --add-channel &lt;string*&gt; (deprecated) --force | -f &lt;bool&gt; </span></pre> <h3 class="numbered" id="s3">Install Scala 3</h3> <p> The Scala 3 compiler and REPL are separate programs: <code>scala3-compiler</code> and <code>scala3-repl</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3c9ce75030a5'><button class='copyBtn' data-clipboard-target='#id3c9ce75030a5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cs install scala3-compiler <span class='unselectable'>https://repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.0.0/scala3-compiler_3-3.0.0.pom 100.0% [##########] 4.8 KiB (79.7 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.0.0/tasty-core_3-3.0.0.pom 100.0% [##########] 3.5 KiB (69.5 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.0.0/scala3-library_3-3.0.0.pom 100.0% [##########] 3.6 KiB (53.9 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.0.0/scala3-interfaces-3.0.0.pom 100.0% [##########] 3.4 KiB (65.9 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.0.0/scala3-interfaces-3.0.0.jar 100.0% [##########] 3.4 KiB (113.9 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.0.0/tasty-core_3-3.0.0.jar 100.0% [##########] 71.9 KiB (192.7 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.0.0/scala3-library_3-3.0.0.jar 100.0% [##########] 1.1 MiB (1.8 MiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.0.0/scala3-compiler_3-3.0.0.jar 100.0% [##########] 14.7 MiB (3.7 MiB / s) Wrote scala3-compiler </span> <span class='unselectable'>$ </span>cs install scala3-repl <span class='unselectable'>https://repo1.maven.org/maven2/io/get-coursier/apps/maven-metadata.xml No new update since 2021-05-14 04:42:19 Wrote scala3-repl </span></pre> <h3 class="numbered" id="repl3">Run Scala 3 REPL</h3> <p> The part is easy! </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide1a82cdf6370'><button class='copyBtn' data-clipboard-target='#ide1a82cdf6370' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>scala3-repl --version <span class='unselectable'>Scala code runner version 3.0.0 -- Copyright 2002-2021, LAMP/EPFL </span> <span class='unselectable'>$ </span>scala3-repl <span class='unselectable'>scala&gt; </span></pre> <h2 id="sbt">Easily Run Scala REPL With SBT</h2> <p> If you do not mind directories called <code>project/</code> and <code>target/</code> being created in your current directory, and you have already <a href='https://www.scala-sbt.org/download.html' target='_blank' rel='nofollow'>installed sbt</a>, you can get a REPL powered by Scala 3 like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3696c2597c3a'><button class='copyBtn' data-clipboard-target='#id3696c2597c3a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sbt "-Dsbt.version=1.5.2" ++3.0.0! console <span class='unselectable'>[info] welcome to sbt 1.5.2 (Ubuntu Java 11.0.11) [info] loading global plugins from /home/mslinn/.sbt/1.0/plugins [info] loading project definition from /var/work/ancientWarmth/ancientWarmth/project [info] set current project to ancientwarmth (in build file:/var/work/ancientWarmth/ancientWarmth/) [info] Forcing Scala version to 3.0.0 on all projects. [info] Reapplying settings... [info] set current project to ancientwarmth (in build file:/var/work/ancientWarmth/ancientWarmth/) [info] Updating [info] Resolved dependencies [info] Updating https://repo1.maven.org/maven2/org/scala-lang/scaladoc_3/3.0.0/scaladoc_3-3.0.0.pom 100.0% [##########] 6.1 KiB (82.6 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-tasty-inspector_3/3.0.0/scala3-tasty-inspector_3-3.0.0.pom 100.0% [##########] 3.6 KiB (80.8 KiB / s) [info] Resolved dependencies [info] Fetching artifacts of [info] Fetched artifacts of [info] Fetching artifacts of https://repo1.maven.org/maven2/org/scala-lang/scala3-tasty-inspector_3/3.0.0/scala3-tasty-inspector_3-3.0.0.jar 100.0% [##########] 16.6 KiB (338.1 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scaladoc_3/3.0.0/scaladoc_3-3.0.0.jar 100.0% [##########] 1.5 MiB (3.1 MiB / s) [info] Fetched artifacts of scala&gt; </span></pre> <p> Thanks to <a href='https://twitter.com/renghenKornel/status/1395684928440791040' target='_blank' rel='nofollow'>@renghen</a> for this tip. </p> <h2 id="sc">ScalaCourses</h2> <p> If you want to learn how to work effectively with Scala for functional and object-oriented programming, <a href='https://scalacourses.com' target='_blank' rel='nofollow'>ScalaCourses.com</a> is your best option. The course material is suitable for Scala 2 and Scala 3. Visit ScalaCourses.com to learn how to become a proficient Scala programmer. </p> OCI / Docker / AWS Lambda / Django / Buildah / podman 2021-04-29T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/29/buildah-podman-python-lambda <style> body { counter-reset: pcounter; } p.count:before { counter-increment: pcounter; content: counter(pcounter) ")\A0"; } </style> <editor-fold goal> <p> This blog post is a work in progress. Some of it may be incorrect, and some thoughts might lead nowhere. I am publicly posting it in this state so I can discuss it with others. This post will be improved as information becomes available. </p> <h2 id="goal">Goal</h2> <div style="text-align: right;"> <a href="https://podman.io" target="_blank" ><picture> <source srcset="/blog/images/podman-logo-crop.webp" type="image/webp"> <source srcset="/blog/images/podman-logo-crop.png" type="image/png"> <img src="/blog/images/podman-logo-crop.png" class="right liImg2 rounded shadow" style="padding: 1em; height: 191px; width: auto;" /> </picture></a> </div> <div style="text-align: right;"> <a href="https://buildah.io/" target="_blank" ><picture> <source srcset="/blog/images/buildah-logo-crop.webp" type="image/webp"> <source srcset="/blog/images/buildah-logo-crop.png" type="image/png"> <img src="/blog/images/buildah-logo-crop.png" class="right liImg2 rounded shadow" style="padding: 0.73em; height: 191px; width: auto;" /> </picture></a> </div> <p> <a href='/blog/2021/04/28/buildah-podman.html'>As previously discussed</a>, Buildah is a drop-in replacement for using <code>docker build</code> and a <code>Dockerfile</code>. Buildah’s <code>build-using-dockerfile</code>, or <code>bud</code> argument makes it behave just like <code>docker build</code> does. </p> <p> The goal of this blog post is to use Buildah / <code>podman</code> to create an Open Container Initiative (OCI) container image with a Django app, including the Python 3.8 runtime installed. The Django app will start when the container is created. The code for the Django app will be stored on the local machine where its source code can be edited, and it will be mapped into the container from the host system. Changes made to the code from the host system will be immediately visible inside the container. </p> <h2 id="todo">TODO</h2> <p class="count"> Background: AWS publishes <a href='https://docs.aws.amazon.com/lambda/latest/dg/python-image.html' target='_blank' rel='nofollow'>Deploying Python with an AWS base image</a>, but that does not discuss running or testing. <a href='https://docs.aws.amazon.com/lambda/latest/dg/getting-started-create-function.html' target='_blank' rel='nofollow'>Create a Lambda function with the console</a> is a more complete article, but is focused on using the web browser console, using Docker, and Node.js. So many differences from the desired goal make the articles difficult to translate to AWS CLI, Buildah / <code>podman</code> and Python. </p> <p class="count"> Talk about the <a href='https://github.com/aws/aws-lambda-runtime-interface-emulator' target='_blank' rel='nofollow'>AWS Lambda Runtime Interface Emulator</a>, compare and contrast with the <a href='https://pypi.org/project/awslambdaric/' target='_blank' rel='nofollow'>AWS Lambda Python Runtime Interface Client</a>. </p> <p class="count"> Compare these <a href='https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html' target='_blank' rel='nofollow'>AWS Lambda Runtimes</a> with other, equivalant runtimes. </p> <p class="count"> OCI images are swapped in when AWS Lambda is invoked. Do larger images cost more to use? If so, discuss. </p> </editor-fold> <editor-fold main> <h2 id="main">Deploy Python Lambda function with Container Image</h2> <p> Consider this <code>Dockerfile</code>, which launches a Python 3.8 command-line application in a manner compatible with AWS Lambda: </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/docker/podman/Dockerfile" download="Dockerfile" title="Click on the file name to download the file">Dockerfile</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="ided7ccad68d59"><button class='copyBtn' data-clipboard-target='#ided7ccad68d59' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>FROM public.ecr.aws/lambda/python:3.8 COPY app.py ./ CMD ["app.handler"] </pre> <p> Following is a small Python app called <code>app.py</code>, which will be launched by the <code>Dockerfile</code>. The Python app can be run as an AWS Lambda program because it implements the <code>handler</code> entry point. </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/docker/podman/app.py" download="app.py" title="Click on the file name to download the file">app.py</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idfc34bb7f6744"><button class='copyBtn' data-clipboard-target='#idfc34bb7f6744' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>import sys def handler(event, context): return f"Hello from AWS Lambda using Python &#123;sys.version&#125;!" </pre> </editor-fold> <editor-fold build_hello> <h2 id="build">Build image</h2> <p> Buildah builds the image, just the same way that Docker would: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9f306b6ca2f7'><button class='copyBtn' data-clipboard-target='#id9f306b6ca2f7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah bud -t hello . <span class='unselectable'>STEP 1: FROM public.ecr.aws/lambda/python:3.8 Getting image source signatures Copying blob 03ac043af787 skipped: already exists Copying blob 420e64b38334 done Copying blob ff259f25b075 done Copying blob 3ff716981d54 done Copying blob 6b6e623a48a8 done Copying blob 9aa8f1e66d54 done Copying config 67dc3a2a54 done Writing manifest to image destination Storing signatures STEP 2: COPY app.py ./ STEP 3: CMD ["app.handler"] STEP 4: COMMIT hello Getting image source signatures Copying blob 683073d39306 skipped: already exists Copying blob 658871a69e1f skipped: already exists Copying blob 6fa16f35d11e skipped: already exists Copying blob d6fa53d6caa6 skipped: already exists Copying blob 61c062506436 skipped: already exists Copying blob 1c1d66a5fd95 skipped: already exists Copying blob 33af9dc6463a done Copying config 98862dfd20 done Writing manifest to image destination Storing signatures --&gt; 98862dfd208 98862dfd2087152ee821553d6cb1c033e735af06e5f11c814bcc9300fb65584e </span></pre> </editor-fold> <editor-fold deploy_local> <h2 id="deploy_local">Test Lambda function Locally</h2> <p> Before calling the Lambda API from a local container, first run the container. Containers default to running in the foreground, but the <code>-d</code> option causes a container to be run as a background process. This container is given the name <code>hello</code>, the external HTTP endpoint at 9000 is mapped to internal port 8080, and the latest version of the <code>hello</code> lambda function is run in the container. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb8e04cc3e790'><button class='copyBtn' data-clipboard-target='#idb8e04cc3e790' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman run \ -d \ --name hello \ -p 9000:8080 \ hello:latest <span class='unselectable'>d4d296e4c91d01c98d312e3f79599dca53990d95218e94bbdfbbac6a43cde9e8 </span></pre> <p> Call the local version of the Lambda API: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idca3f3be3dfa6'><button class='copyBtn' data-clipboard-target='#idca3f3be3dfa6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl \ -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" \ -d '{}' <span class='unselectable'>"Hello from AWS Lambda using Python 3.8.9 (default, Apr 20 2021, 13:58:54) \n[GCC 7.3.1 20180712 (Red Hat 7.3.1-12)]!" </span></pre> <p> Stop the container called <code>hello</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id07d3428bddbe'><button class='copyBtn' data-clipboard-target='#id07d3428bddbe' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman stop hello <span class='unselectable'>96cc1b1ed92368a1165d6a6ad0b1e5544d4ac751b64e94df33bf2322e6d7b30c </span></pre> </editor-fold> <editor-fold create_repo> <h2 id="podman_tag">Create AWS ECR Repository</h2> <p> AWS provides a registry for OCI-compatible image repositories called the <a href='https://aws.amazon.com/ecr/' target='_blank' rel='nofollow'>AWS Elastic Container Registry (ECR)</a>. </p> <editor-fold create_repo_help> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id681dc0fb6796'><button class='copyBtn' data-clipboard-target='#id681dc0fb6796' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ecr create-repository help CREATE-REPOSITORY() CREATE-REPOSITORY()<br/> NAME create-repository -<br/> DESCRIPTION Creates a repository. For more information, see Amazon ECR Repositories in the Amazon Elastic Container Registry User Guide .<br/> See also: AWS API Documentation<br/> See &#39;aws help&#39; for descriptions of global parameters.<br/> SYNOPSIS create-repository --repository-name &lt;value&gt; [--tags &lt;value&gt;] [--image-tag-mutability &lt;value&gt;] [--image-scanning-configuration &lt;value&gt;] [--cli-input-json &lt;value&gt;] [--generate-cli-skeleton &lt;value&gt;]<br/> OPTIONS --repository-name (string) The name to use for the repository. The repository name may be spec- ified on its own (such as nginx-web-app ) or it can be prepended with a namespace to group the repository into a category (such as project-a/nginx-web-app ).<br/> --tags (list) The metadata that you apply to the repository to help you categorize and organize them. Each tag consists of a key and an optional value, both of which you define. Tag keys can have a maximum character length of 128 characters, and tag values can have a maximum length of 256 characters.<br/> (structure) The metadata that you apply to a resource to help you categorize and organize them. Each tag consists of a key and an optional value, both of which you define. Tag keys can have a maximum character length of 128 characters, and tag values can have a maximum length of 256 characters.<br/> Key -&gt; (string) One part of a key-value pair that make up a tag. A key is a general label that acts like a category for more specific tag values.<br/> Value -&gt; (string) The optional part of a key-value pair that make up a tag. A value acts as a descriptor within a tag category (key).<br/> Shorthand Syntax:<br/> Key=string,Value=string ...<br/> JSON Syntax:<br/> [ { &quot;Key&quot;: &quot;string&quot;, &quot;Value&quot;: &quot;string&quot; } ... ]<br/> --image-tag-mutability (string) The tag mutability setting for the repository. If this parameter is omitted, the default setting of MUTABLE will be used which will al- low image tags to be overwritten. If IMMUTABLE is specified, all im- age tags within the repository will be immutable which will prevent them from being overwritten.<br/> Possible values:<br/> o MUTABLE<br/> o IMMUTABLE<br/> --image-scanning-configuration (structure) The image scanning configuration for the repository. This setting determines whether images are scanned for known vulnerabilities af- ter being pushed to the repository.<br/> scanOnPush -&gt; (boolean) The setting that determines whether images are scanned after be- ing pushed to a repository. If set to true , images will be scanned after being pushed. If this parameter is not specified, it will default to false and images will not be scanned unless a scan is manually started with the StartImageScan API.<br/> Shorthand Syntax:<br/> scanOnPush=boolean<br/> JSON Syntax:<br/> { &quot;scanOnPush&quot;: true|false }<br/> --cli-input-json (string) Performs service operation based on the JSON string provided. The JSON string follows the format provided by --gen- erate-cli-skeleton. If other arguments are provided on the command line, the CLI values will override the JSON-provided values. It is not possible to pass arbitrary binary values using a JSON-provided value as the string will be taken literally.<br/> --generate-cli-skeleton (string) Prints a JSON skeleton to standard output without sending an API request. If provided with no value or the value input, prints a sample input JSON that can be used as an argument for --cli-input-json. If provided with the value output, it validates the command inputs and returns a sample output JSON for that command.<br/> See &#39;aws help&#39; for descriptions of global parameters.<br/> EXAMPLES Example 1: To create a repository<br/> The following create-repository example creates a repository inside the specified namespace in the default registry for an account.<br/> aws ecr create-repository \ --repository-name project-a/nginx-web-app<br/> Output:<br/> { &quot;repository&quot;: { &quot;registryId&quot;: &quot;123456789012&quot;, &quot;repositoryName&quot;: &quot;sample-repo&quot;, &quot;repositoryArn&quot;: &quot;arn:aws:ecr:us-west-2:123456789012:repository/project-a/nginx-web-app&quot; } }<br/> For more information, see Creating a Repository in the Amazon ECR User Guide.<br/> Example 2: To create a repository configured with image tag immutabil- ity<br/> The following create-repository example creates a repository configured for tag immutability in the default registry for an account.<br/> aws ecr create-repository \ --repository-name sample-repo \ --image-tag-mutability IMMUTABLE<br/> Output:<br/> { &quot;repository&quot;: { &quot;registryId&quot;: &quot;123456789012&quot;, &quot;repositoryName&quot;: &quot;sample-repo&quot;, &quot;repositoryArn&quot;: &quot;arn:aws:ecr:us-west-2:123456789012:repository/sample-repo&quot;, &quot;imageTagMutability&quot;: &quot;IMMUTABLE&quot; } }<br/> For more information, see Image Tag Mutability in the Amazon ECR User Guide.<br/> Example 3: To create a repository configured with a scanning configura- tion<br/> The following create-repository example creates a repository configured to perform a vulnerability scan on image push in the default registry for an account.<br/> aws ecr create-repository \ --repository-name sample-repo \ --image-scanning-configuration scanOnPush=true<br/> Output:<br/> { &quot;repository&quot;: { &quot;registryId&quot;: &quot;123456789012&quot;, &quot;repositoryName&quot;: &quot;sample-repo&quot;, &quot;repositoryArn&quot;: &quot;arn:aws:ecr:us-west-2:123456789012:repository/sample-repo&quot;, &quot;imageScanningConfiguration&quot;: { &quot;scanOnPush&quot;: true } } }<br/> For more information, see Image Scanning in the Amazon ECR User Guide.<br/> OUTPUT repository -&gt; (structure) The repository that was created.<br/> repositoryArn -&gt; (string) The Amazon Resource Name (ARN) that identifies the repository. The ARN contains the arn:aws:ecr namespace, followed by the re- gion of the repository, AWS account ID of the repository owner, repository namespace, and repository name. For example, arn:aws:ecr:region:012345678910:repository/test .<br/> registryId -&gt; (string) The AWS account ID associated with the registry that contains the repository.<br/> repositoryName -&gt; (string) The name of the repository.<br/> repositoryUri -&gt; (string) The URI for the repository. You can use this URI for Docker push or pull operations.<br/> createdAt -&gt; (timestamp) The date and time, in JavaScript date format, when the reposi- tory was created.<br/> imageTagMutability -&gt; (string) The tag mutability setting for the repository.<br/> imageScanningConfiguration -&gt; (structure) The image scanning configuration for a repository.<br/> scanOnPush -&gt; (boolean) The setting that determines whether images are scanned after being pushed to a repository. If set to true , images will be scanned after being pushed. If this parameter is not speci- fied, it will default to false and images will not be scanned unless a scan is manually started with the StartImageScan API.<br/> <br/> CREATE-REPOSITORY()</pre> </editor-fold> <p> The following creates an AWS ECR image repository in called <code>hello</code> within the <code>test</code> namespace. <a href='https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-scanning.html' target='_blank' rel='nofollow'>Images are scanned</a> for known vulnerabilities after they are pushed to the repository. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide1d65818870f'><button class='copyBtn' data-clipboard-target='#ide1d65818870f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ecr create-repository \ --repository-name test/hello \ --image-scanning-configuration scanOnPush=true <span class='unselectable'>{ "repository": { "repositoryArn": "arn:aws:ecr:us-east-1:031372724784:repository/test/hello", "registryId": "031372724784", "repositoryName": "test/hello", "repositoryUri": "031372724784.dkr.ecr.us-east-1.amazonaws.com/test/hello", "createdAt": 1620232146.0, "imageTagMutability": "MUTABLE", "imageScanningConfiguration": { "scanOnPush": true } } } </span></pre> </editor-fold> <editor-fold podman_tag> <h2 id="podman_tag">Tag Image</h2> <p class="quote"> <b><code>podman tag</code></b> &ndash; Assigns a new image name to an existing image. A full name refers to the entire image name, including the optional tag after the <code>:</code>. If there is no tag provided, then podman will default to latest for both the image and the target-name. &nbsp; &ndash; From <code>man podman-tag</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id20a3e3040768'><button class='copyBtn' data-clipboard-target='#id20a3e3040768' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>IMAGE_NAME=hello <span class='unselectable'>$ </span>IMAGE_VERSION=0.1 <span class='unselectable'>$ </span>podman tag $IMAGE_NAME:$IMAGE_VERSION \ $REGISTRY/$IMAGE_NAME:$IMAGE_VERSION <span class='unselectable'>$ </span>podman images <span class='unselectable'>REPOSITORY TAG IMAGE ID CREATED SIZE localhost/hello 0.1 98862dfd2087 39 minutes ago 622 MB 752246127823.dkr.ecr.us-east-1.amazonaws.com/hello latest 98862dfd2087 39 minutes ago 622 MB public.ecr.aws/lambda/python 3.8 67dc3a2a54fb 25 hours ago 622 MB 752246127823.dkr.ecr.us-east-1.amazonaws.com/ancientwarmth latest 5d18ea34fc30 28 hours ago 2.03 GB localhost/ancientwarmth latest 5d18ea34fc30 28 hours ago 2.03 GB &lt;none&gt; &lt;none&gt; 40ef32b39cf4 5 days ago 622 MB docker.io/library/amazonlinux latest 53ef897d731f 5 days ago 170 MB docker.io/amazon/aws-lambda-python 3.8 e12ea62c5582 9 days ago 622 MB docker.io/library/alpine latest 6dbb9cc54074 2 weeks ago 5.88 MB docker.io/lambci/lambda build-python3.8 714c659c9f6f 3 months ago 2.03 GB </span></pre> </editor-fold> <editor-fold push_ecr> <h2 id="push">Push Image to ECR</h2> <p> <code>Podman</code> will use the IAM credentials for the <code>dev</code> <a href='https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html' target='_blank' rel='nofollow'>profile</a> in <code>~/.aws/credentials</code> to log into that AWS account: </p> <div class='codeLabel unselectable' data-lt-active='false'>~/.aws/credentials</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4b422ebeaeac'><button class='copyBtn' data-clipboard-target='#id4b422ebeaeac' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[default] aws_access_key_id = ******************** aws_secret_access_key = **************************************** region = us-east-1<br> [dev] aws_access_key_id = ******************** aws_secret_access_key = **************************************** region = us-east-1</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb7a084a07d93'><button class='copyBtn' data-clipboard-target='#idb7a084a07d93' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>export AWS_PROFILE=dev <span class='unselectable'>$ </span>AWS_ACCOUNT="$( aws sts get-caller-identity \ --query Account \ --output text )" <span class='unselectable'>$ </span>AWS_REGION="$( aws configure get region )" <span class='unselectable'>$ </span>REGISTRY="$AWS_ACCOUNT.dkr.ecr.$AWS_REGION.amazonaws.com" <span class='unselectable'>$ </span>aws ecr get-login-password \ --region "$AWS_REGION" | \ podman login \ --password-stdin \ --username AWS \ "$REGISTRY" <span class='unselectable'>Login Succeeded! </span></pre> <p> Now that <code>podman</code> is logged into AWS, use <code>podman</code> push the image to AWS ECR: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id07afdd01786e'><button class='copyBtn' data-clipboard-target='#id07afdd01786e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman push test/$IMAGE_NAME \ $REGISTRY/$IMAGE_NAME:$IMAGE_VERSION <span class='unselectable'>Getting image source signatures Copying blob 692590faf2d1 [--------------------------------------] 8.0b / 8.2MiB Copying blob 397718cff58d [--------------------------------------] 8.0b / 206.2MiB Copying blob 9ca787b1c91c [--------------------------------------] 8.0b / 93.1MiB Copying blob ef26f5221b79 [--------------------------------------] 8.0b / 196.7MiB Copying blob 0a3f69c27a89 [--------------------------------------] 8.0b / 316.4MiB Copying blob 5b3cbb76df75 [--------------------------------------] 8.0b / 1.1GiB Copying blob e9cad39831b0 [--------------------------------------] 8.0b / 3.5KiB Error: Error copying image to the remote destination: Error writing blob: Error initiating layer upload to /v2/ancientwarmth/blobs/uploads/ in 752246127823.dkr.ecr.us-east-1.amazonaws.com: name unknown: The repository with name 'hello' does not exist in the registry with id '752246127823' </span></pre> <p> The results of an <a href='https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ecr/describe-image-scan-findings.html' target='_blank' rel='nofollow'>image scan</a> for the new repository can be retrieved as follows: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd9c40b6395a2'><button class='copyBtn' data-clipboard-target='#idd9c40b6395a2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ecr describe-image-scan-findings \ --repository-name test/hello \ --image-id imageTag=tag_name</pre> </editor-fold> <editor-fold aw> </editor-fold> <editor-fold buildah_python> <h2 id="buildah_python">Deploy Python Lambda function with Container Image</h2> <p> <code>Podman</code> can invoke the app using an OCI container with Amazon Linux 2 and Python 3.8: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id744d651abe38'><button class='copyBtn' data-clipboard-target='#id744d651abe38' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman container run -ti \ public.ecr.aws/lambda/python:3.8 \ blog/docker/podman/app.py <span class='unselectable'>Trying to pull public.ecr.aws/lambda/python:3.8... Getting image source signatures Copying blob 1de4740de1c2 done Copying blob 03ac043af787 done Copying blob 2e2bb77ae2dc done Copying blob 842c9dce67e8 done Copying blob df513d38f4d9 done Copying blob 031c6369fb2b done Copying config e12ea62c55 done Writing manifest to image destination Storing signatures time="2021-05-02T23:38:30.971" level=info msg="exec '/var/runtime/bootstrap' (cwd=/var/task, handler=)" </span></pre> </editor-fold> Docker, OCI Images, Buildah and podman 2021-04-28T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/28/buildah-podman <editor-fold intro> <p> There are many ways to create and run Docker-compatible images. Docker is probably the worst option, mostly because it runs as a daemon, and all *nix daemons run with <code>root</code> privileges. Also, the <code>docker-ce</code> package lists <code>iptables</code> as a dependency, which needs <code>systemd</code> to be running normally, and WSL2 only partially supports <code>systemd</code>. </p> <p> <a href='https://www.capitalone.com/tech/cloud/container-runtime/' target='_blank' rel='nofollow'>A Comprehensive Container Runtime Comparison</a> provides helpful background information and an interesting historical viewpoint. </p> <h2 id="oci">Open Container Initiative (OCI)</h2> <div style=""> <a href="https://opencontainers.org/" target="_blank" ><picture> <source srcset="/blog/images/oci_logo.webp" type="image/webp"> <source srcset="/blog/images/oci_logo.png" type="image/png"> <img src="/blog/images/oci_logo.png" class=" fullsize liImg2 rounded shadow" /> </picture></a> </div> <p> The latest evolution of Docker-compatible images, <a href='https://github.com/opencontainers/image-spec' target='_blank' rel='nofollow'>OCI image format</a> (not to be confused with <a href='https://www.oracle.com/ca-en/cloud/' target='_blank' rel='nofollow'>Oracle Cloud Infrastructure</a>), is compatible with: </p> <ul> <li><a href='https://aws.amazon.com/lambda/' target='_blank' rel='nofollow'>AWS Lambda</a></li> <li><a href='https://azure.microsoft.com/en-us/services/functions/' target='_blank' rel='nofollow'>Azure Functions</a></li> <li><a href='https://azure.microsoft.com/en-us/services/kubernetes-service/' target='_blank' rel='nofollow'>Azure Kubernetes Service</a></li> <li><a href='https://buildah.io/' target='_blank' rel='nofollow'>Buildah</a></li> <li><a href='https://buildpacks.io/' target='_blank' rel='nofollow'>Cloud Native Buildpacks</a></li> <li><a href='https://circleci.com/' target='_blank' rel='nofollow'>CircleCI</a></li> <li><a href='https://www.docker.com/' target='_blank' rel='nofollow'>Docker</a></li> <li><a href='https://dokku.com/' target='_blank' rel='nofollow'>Dokku</a></li> <li><a href='https://gitlab.com' target='_blank' rel='nofollow'>GitLab</a></li> <li><a href='https://cloud.google.com/container-registry/docs/image-formats' target='_blank' rel='nofollow'>Google Cloud</a></li> <li><a href='https://heroku.com' target='_blank' rel='nofollow'>Heroku</a></li> <li><a href='https://containerjournal.com/topics/container-management/what-is-knative-and-what-can-it-do-for-you/' target='_blank' rel='nofollow'>Knative</a></li> <li><a href='https://kubernetes.io/' target='_blank' rel='nofollow'>Kubernetes</a></li> <li><a href='https://podman.io/' target='_blank' rel='nofollow'><code>podman</code></a></li> <li><a href='https://github.com/containers/skopeo' target='_blank' rel='nofollow'><code>skopeo</code></a></li> <li><a href='https://spring.io/guides/topicals/spring-boot-docker/' target='_blank' rel='nofollow'>Spring Boot</a></li> <li><a href='https://cloud.google.com/tekton' target='_blank' rel='nofollow'>Tekton</a></li> </ul> <p> Supported OCI formats include: </p> <ul> <li>Docker containers schema 1</li> <li>Docker containers schema 2</li> <li>Pods (groups of containers)</li> <li>Images</li> <li>Volumes</li> </ul> <h2 id="three">Buildah, podman and skopeo</h2> <p> This blog post discusses 3 related open source projects from RedHat / IBM that provide an alternative to Docker: Buildah, <code>podman</code> and <code>skopeo</code>. These 3 projects share a common source code base, and are daemonless tools for managing Open Container Initiative (OCI) images. </p> <p> Paraphrasing the reasons expressed in <a href='https://developers.redhat.com/blog/2019/02/21/podman-and-buildah-for-docker-users/' target='_blank' rel='nofollow'>Podman and Buildah for Docker Users</a> for using <code>podman</code> instead of Docker, wherever <code>podman</code> is mentioned, read &ldquo;<code>podman</code>, Buildah and <code>skopeo</code>&rdquo;: </p> <p class="quoteCite" cite="From &ldquo;Podman and Buildah for Docker Users&rdquo;"> The Podman approach is simply to directly interact with the image registry, with the container and image storage, and with the Linux kernel through the <code>runC</code> container runtime process (not a daemon).<br><br> Running Podman as a normal user means that Podman will, by default, store images and containers in the user’s home directory. Podman uses a repository in the user’s home directory: <code>~/.local/share/containers</code> (instead of <code>/var/lib/docker</code>).<br><br> Despite the new locations for the local repositories, the images created by Docker and Podman are compatible with the OCI standard. Podman can push to and pull from popular container registries like Quay.io and Docker hub, as well as private registries. </p> <h2 id="buildah_vs_podman">Buildah vs. podman</h2> <p> <code>Podman</code> can build OCI containers interactively or in batch mode. You can either build using a <code>Dockerfile</code> using <code>podman build</code> (batch mode), or you can interactively run a container, make changes to the running image, and then <code>podman commit</code> those changes to a new image tag. </p> <p> Buildah was written before <code>podman</code>. Some of Buildah's source code for creating and managing container images was ported to <code>podman</code>. The <code>podman build</code> command is a subset of Buildah&rsquo;s functionality. </p> <p> <p> However, apparently the differences between the two programs are important: </p> <p class="quote"> Buildah builds OCI images. Confusingly, <code>podman build</code> can also be used to build Docker images also, but it’s incredibly slow and used up a lot of disk space by using the <code>vfs</code> storage driver by default. <code>buildah bud</code> (‘build using Dockerfile’) was much faster for me, and uses the overlay storage driver. <br><br> &nbsp; &ndash; From <a href='https://zwischenzugs.com/page/3/' target='_blank' rel='nofollow'>Goodbye Docker: Purging is Such Sweet Sorrow</a> by Ian Miell. </p> </editor-fold> <editor-fold podman> <h2 id="podman">podman</h2> <div style=""> <a href="https://podman.io" target="_blank" ><picture> <source srcset="/blog/images/podman-logo.webp" type="image/webp"> <source srcset="/blog/images/podman-logo.png" type="image/png"> <img src="/blog/images/podman-logo.png" class=" liImg2 rounded shadow" style="padding: 1em" /> </picture></a> </div> <p> <code>Podman</code> supports developing, managing, and running OCI Containers on Linux systems, including WSL, without requiring <code>root</code> privilege. </p> <div class='codeLabel unselectable' data-lt-active='false'>shell Installation on Ubuntu</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5b85324ffe3c'><button class='copyBtn' data-clipboard-target='#id5b85324ffe3c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install buildah podman skopeo</pre> <div class="pullQuote"> Podman commands are very nearly the same as Docker’s. </div> <p> Because <code>podman</code> is a drop-in replacement for <code>docker</code>, the following alias enables the <code>docker</code> command to invoke <code>podman</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>~/.bash_aliases</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida3b4d5d8c8ab'><button class='copyBtn' data-clipboard-target='#ida3b4d5d8c8ab' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>alias docker=podman</pre> <p> As described in <a href='https://www.vultr.com/docs/how-to-install-and-use-podman-on-ubuntu-20-04' target='_blank' rel='nofollow'>How to Install and Use Podman on Ubuntu 20.04</a>, I added <code>'registry.access.redhat.com'</code> to the list of <code>registries</code> in <code>/etc/containers/registries.conf</code>. I also added <a href='https://gallery.ecr.aws/' target='_blank' rel='nofollow'><code>'gallery.ecr.aws'</code></a> and <a href='https://cloud.google.com/container-registry/docs/pushing-and-pulling#add-registry' target='_blank' rel='nofollow'><code>'gcr.io'</code></a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/containers/registries.conf</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddc29ae990870'><button class='copyBtn' data-clipboard-target='#iddc29ae990870' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[registries.search] registries = ['docker.io', 'gallery.ecr.aws', 'gcr.io', 'quay.io', 'registry.access.redhat.com']</pre> </editor-fold> <editor-fold help> <h3 id="podmanHelp"><span class="code">podman</span> Help</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8d7c349864ef'><button class='copyBtn' data-clipboard-target='#id8d7c349864ef' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman --help <span class='unselectable'>Manage pods, containers and images<br/> Usage: podman [flags] podman [command]<br/> Available Commands: attach Attach to a running container auto-update Auto update containers according to their auto-update policy build Build an image using instructions from Containerfiles commit Create new image based on the changed container container Manage containers cp Copy files/folders between a container and the local filesystem create Create but do not start a container diff Display the changes to the object&#39;s file system events Show podman events exec Run a process in a running container export Export container&#39;s filesystem contents as a tar archive generate Generate structured data based on containers and pods. healthcheck Manage health checks on containers help Help about any command history Show history of a specified image image Manage images images List images in local storage import Import a tarball to create a filesystem image info Display podman system information init Initialize one or more containers inspect Display the configuration of object denoted by ID kill Kill one or more running containers with a specific signal load Load an image from container archive login Login to a container registry logout Logout of a container registry logs Fetch the logs of one or more containers manifest Manipulate manifest lists and image indexes mount Mount a working container&#39;s root filesystem network Manage networks pause Pause all the processes in one or more containers play Play a pod and its containers from a structured file. pod Manage pods port List port mappings or a specific mapping for the container ps List containers pull Pull an image from a registry push Push an image to a specified destination restart Restart one or more containers rm Remove one or more containers rmi Removes one or more images from local storage run Run a command in a new container save Save image to an archive search Search registry for image start Start one or more containers stats Display a live stream of container resource usage statistics stop Stop one or more containers system Manage podman tag Add an additional name to a local image top Display the running processes of a container unmount Unmounts working container&#39;s root filesystem unpause Unpause the processes in one or more containers unshare Run a command in a modified user namespace untag Remove a name from a local image version Display the Podman Version Information volume Manage volumes wait Block on one or more containers<br/> Flags: --cgroup-manager string Cgroup manager to use (&quot;cgroupfs&quot;|&quot;systemd&quot;) (default &quot;cgroupfs&quot;) --cni-config-dir string Path of the configuration directory for CNI networks --conmon string Path of the conmon binary -c, --connection string Connection to use for remote Podman service --events-backend string Events backend to use (&quot;file&quot;|&quot;journald&quot;|&quot;none&quot;) (default &quot;file&quot;) --help Help for podman --hooks-dir strings Set the OCI hooks directory path (may be set multiple times) (default [/usr/share/containers/oci/hooks.d]) --identity string path to SSH identity file, (CONTAINER_SSHKEY) --log-level string Log messages above specified level (debug, info, warn, error, fatal, panic) (default &quot;error&quot;) --namespace string Set the libpod namespace, used to create separate views of the containers and pods on the system --network-cmd-path string Path to the command for configuring the network -r, --remote Access remote Podman service (default false) --root string Path to the root directory in which data, including images, is stored --runroot string Path to the &#39;run directory&#39; where all state information is stored --runtime string Path to the OCI-compatible binary used to run containers, default is /usr/bin/runc --storage-driver string Select which storage driver is used to manage storage of images and containers (default is overlay) --storage-opt stringArray Used to pass an option to the storage driver --syslog Output logging information to syslog as well as the console (default false) --tmpdir string Path to the tmp directory for libpod state content. Note: use the environment variable &#39;TMPDIR&#39; to change the temporary storage location for container images, &#39;/var/tmp&#39;. --url string URL to access Podman service (CONTAINER_HOST) (default &quot;unix:/home/mslinn/.docker/run/podman/podman.sock&quot;) -v, --version Version of Podman<br/> Use &quot;podman [command] --help&quot; for more information about a command. </span></pre> </editor-fold> <editor-fold padman_info> <h3 id="padman_info"><span class="code">podman info</span></h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb0a9fe0e4cb8'><button class='copyBtn' data-clipboard-target='#idb0a9fe0e4cb8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman info <span class='unselectable'>host: arch: amd64 buildahVersion: 1.15.2 cgroupVersion: v1 conmon: package: &#39;conmon: /usr/libexec/podman/conmon&#39; path: /usr/libexec/podman/conmon version: &#39;conmon version 2.0.20, commit: unknown&#39; cpus: 8 distribution: distribution: ubuntu version: &quot;20.10&quot; eventLogger: file hostname: Bear idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 kernel: 5.4.72-microsoft-standard-WSL2 linkmode: dynamic memFree: 897724416 memTotal: 6231638016 ociRuntime: name: runc package: &#39;containerd.io: /usr/bin/runc&#39; path: /usr/bin/runc version: |- runc version 1.0.0-rc93 commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec spec: 1.0.2-dev go: go1.13.15 libseccomp: 2.5.1 os: linux remoteSocket: path: /home/mslinn/.docker/run/podman/podman.sock rootless: true slirp4netns: executable: /bin/slirp4netns package: Unknown version: |- slirp4netns version 1.0.1 commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4 libslirp: 4.3.1 swapFree: 0 swapTotal: 0 uptime: 306h 23m 6.37s (Approximately 12.75 days) registries: search: - quay.io - docker.io - gallery.ecr.aws - registry.access.redhat.com store: configFile: /home/mslinn/.config/containers/storage.conf containerStore: number: 8 paused: 0 running: 0 stopped: 8 graphDriverName: overlay graphOptions: overlay.mount_program: Executable: /bin/fuse-overlayfs Package: Unknown Version: |- fusermount3 version: 3.9.3 fuse-overlayfs: version 1.0.0 FUSE library version 3.9.3 using FUSE kernel interface version 7.31 graphRoot: /home/mslinn/.local/share/containers/storage graphStatus: Backing Filesystem: extfs Native Overlay Diff: &quot;false&quot; Supports d_type: &quot;true&quot; Using metacopy: &quot;false&quot; imageStore: number: 4 runRoot: /home/mslinn/.docker/run/containers volumePath: /home/mslinn/.local/share/containers/storage/volumes version: APIVersion: 1 Built: 0 BuiltTime: Wed Dec 31 19:00:00 1969 GitCommit: &quot;&quot; GoVersion: go1.14.7 OsArch: linux/amd64 Version: 2.0.6 </span></pre> </editor-fold> <editor-fold padman_container_help> <h3 id="padman_container_help"><span class="code">podman container</span> Help</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id155af033161d'><button class='copyBtn' data-clipboard-target='#id155af033161d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>man podman-container <span class='unselectable'>podman-container(1) General Commands Manual podman-container(1)<br/> NAME podman-container - Manage containers<br/> SYNOPSIS podman container subcommand<br/> DESCRIPTION The container command allows you to manage containers<br/> COMMANDS &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9516;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9516;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488; &#9474;Command &#9474; Man Page &#9474; Description &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;attach &#9474; podman-attach(1) &#9474; Attach to a running container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;checkpoint &#9474; podman-container-checkpoint(1) &#9474; Checkpoints one or more running &#9474; &#9474; &#9474; &#9474; containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;cleanup &#9474; podman-container-cleanup(1) &#9474; Cleanup the container&#39;s network &#9474; &#9474; &#9474; &#9474; and mountpoints. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;commit &#9474; podman-commit(1) &#9474; Create new image based on the &#9474; &#9474; &#9474; &#9474; changed container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;cp &#9474; podman-cp(1) &#9474; Copy files/folders between a &#9474; &#9474; &#9474; &#9474; container and the local &#9474; &#9474; &#9474; &#9474; filesystem. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;create &#9474; podman-create(1) &#9474; Create a new container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;diff &#9474; podman-diff(1) &#9474; Inspect changes on a container or &#9474; &#9474; &#9474; &#9474; image&#39;s filesystem. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;exec &#9474; podman-exec(1) &#9474; Execute a command in a running &#9474; &#9474; &#9474; &#9474; container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;exists &#9474; podman-container-exists(1) &#9474; Check if a container exists in &#9474; &#9474; &#9474; &#9474; local storage &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;export &#9474; podman-export(1) &#9474; Export a container&#39;s filesystem &#9474; &#9474; &#9474; &#9474; contents as a tar archive. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;init &#9474; podman-init(1) &#9474; Initialize a container &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;inspect &#9474; podman-inspect(1) &#9474; Display a container or image&#39;s &#9474; &#9474; &#9474; &#9474; configuration. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;kill &#9474; podman-kill(1) &#9474; Kill the main process in one or &#9474; &#9474; &#9474; &#9474; more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;list &#9474; podman-ps(1) &#9474; List the containers on the &#9474; &#9474; &#9474; &#9474; system.(alias ls) &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;logs &#9474; podman-logs(1) &#9474; Display the logs of a container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;mount &#9474; podman-mount(1) &#9474; Mount a working container&#39;s root &#9474; &#9474; &#9474; &#9474; filesystem. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;pause &#9474; podman-pause(1) &#9474; Pause one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;port &#9474; podman-port(1) &#9474; List port mappings for the &#9474; &#9474; &#9474; &#9474; container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;prune &#9474; podman-container-prune(1) &#9474; Remove all stopped containers &#9474; &#9474; &#9474; &#9474; from local storage. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;restart &#9474; podman-restart(1) &#9474; Restart one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;restore &#9474; podman-container-restore(1) &#9474; Restores one or more containers &#9474; &#9474; &#9474; &#9474; from a checkpoint. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;rm &#9474; podman-rm(1) &#9474; Remove one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;run &#9474; podman-run(1) &#9474; Run a command in a container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;runlabel &#9474; podman-container-runlabel(1) &#9474; Executes a command as described &#9474; &#9474; &#9474; &#9474; by a container image label. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;start &#9474; podman-start(1) &#9474; Starts one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;stats &#9474; podman-stats(1) &#9474; Display a live stream of one or &#9474; &#9474; &#9474; &#9474; more container&#39;s resource usage &#9474; &#9474; &#9474; &#9474; statistics. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;stop &#9474; podman-stop(1) &#9474; Stop one or more running &#9474; &#9474; &#9474; &#9474; containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;top &#9474; podman-top(1) &#9474; Display the running processes of &#9474; &#9474; &#9474; &#9474; a container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;unmount &#9474; podman-unmount(1) &#9474; Unmount a working container&#39;s &#9474; &#9474; &#9474; &#9474; root filesystem.(Alias unmount) &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;unpause &#9474; podman-unpause(1) &#9474; Unpause one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;wait &#9474; podman-wait(1) &#9474; Wait on one or more containers to &#9474; &#9474; &#9474; &#9474; stop and print their exit codes. &#9474; &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9524;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9524;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;<br/> SEE ALSO podman, podman-exec, podman-run<br/> podman-container(1) </span></pre> </editor-fold> <editor-fold podman_run_help> <h3 id="podman_help">Podman Run Help</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idef8e66135488'><button class='copyBtn' data-clipboard-target='#idef8e66135488' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman container run --help <span class='unselectable'>Run a command in a new container<br/> Description: Runs a command in a new container from the given image<br/> Usage: podman container run [flags] IMAGE [COMMAND [ARG...]]<br/> Examples: podman container run imageID ls -alF /etc podman container run --network=host imageID dnf -y install java podman container run --volume /var/hostdir:/var/ctrdir -i -t fedora /bin/bash<br/> Flags: --add-host strings Add a custom host-to-IP mapping (host:ip) (default []) --annotation strings Add annotations to container (key:value) -a, --attach strings Attach to STDIN, STDOUT or STDERR --authfile string Path of the authentication file. Use REGISTRY_AUTH_FILE environment variable to override --blkio-weight string Block IO weight (relative weight) accepts a weight value between 10 and 1000. --blkio-weight-device DEVICE_NAME:WEIGHT Block IO weight (relative device weight, format: DEVICE_NAME:WEIGHT) --cap-add strings Add capabilities to the container --cap-drop strings Drop capabilities from the container --cgroup-parent string Optional parent cgroup for the container --cgroupns string cgroup namespace to use --cgroups string control container cgroup configuration (&quot;enabled&quot;|&quot;disabled&quot;|&quot;no-conmon&quot;) (default &quot;enabled&quot;) --cidfile string Write the container ID to the file --conmon-pidfile string Path to the file that will receive the PID of conmon --cpu-period uint Limit the CPU CFS (Completely Fair Scheduler) period --cpu-quota int Limit the CPU CFS (Completely Fair Scheduler) quota --cpu-rt-period uint Limit the CPU real-time period in microseconds --cpu-rt-runtime int Limit the CPU real-time runtime in microseconds --cpu-shares uint CPU shares (relative weight) --cpus float Number of CPUs. The default is 0.000 which means no limit --cpuset-cpus string CPUs in which to allow execution (0-3, 0,1) --cpuset-mems string Memory nodes (MEMs) in which to allow execution (0-3, 0,1). Only effective on NUMA systems. -d, --detach Run container in background and print container ID --detach-keys [a-Z] Override the key sequence for detaching a container. Format is a single character [a-Z] or a comma separated sequence of `ctrl-&lt;value&gt;`, where `&lt;value&gt;` is one of: `a-cf`, `@`, `^`, `[`, `\`, `]`, `^` or `_` (default &quot;ctrl-p,ctrl-q&quot;) --device strings Add a host device to the container --device-cgroup-rule strings Add a rule to the cgroup allowed devices list --device-read-bps strings Limit read rate (bytes per second) from a device (e.g. --device-read-bps=/dev/sda:1mb) --device-read-iops strings Limit read rate (IO per second) from a device (e.g. --device-read-iops=/dev/sda:1000) --device-write-bps strings Limit write rate (bytes per second) to a device (e.g. --device-write-bps=/dev/sda:1mb) --device-write-iops strings Limit write rate (IO per second) to a device (e.g. --device-write-iops=/dev/sda:1000) --disable-content-trust This is a Docker specific option and is a NOOP --dns strings Set custom DNS servers --dns-opt strings Set custom DNS options --dns-search strings Set custom DNS search domains --entrypoint string Overwrite the default ENTRYPOINT of the image -e, --env stringArray Set environment variables in container (default [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,TERM=xterm]) --env-file strings Read in a file of environment variables --env-host Use all current host environment variables in container --expose strings Expose a port or a range of ports --gidmap strings GID map to use for the user namespace --group-add strings Add additional groups to join --health-cmd string set a healthcheck command for the container (&#39;none&#39; disables the existing healthcheck) --health-interval string set an interval for the healthchecks (a value of disable results in no automatic timer setup) (default &quot;30s&quot;) --health-retries uint the number of retries allowed before a healthcheck is considered to be unhealthy (default 3) --health-start-period string the initialization time needed for a container to bootstrap (default &quot;0s&quot;) --health-timeout string the maximum time allowed to complete the healthcheck before an interval is considered failed (default &quot;30s&quot;) -h, --hostname string Set container hostname --http-proxy Set proxy environment variables in the container based on the host proxy vars (default true) --image-volume string Tells podman how to handle the builtin image volumes (&quot;bind&quot;|&quot;tmpfs&quot;|&quot;ignore&quot;) (default &quot;bind&quot;) --init Run an init binary inside the container that forwards signals and reaps processes --init-path string Path to the container-init binary -i, --interactive Keep STDIN open even if not attached --ip string Specify a static IPv4 address for the container --ipc string IPC namespace to use --kernel-memory &lt;number&gt;[&lt;unit&gt;] Kernel memory limit (format: &lt;number&gt;[&lt;unit&gt;], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes)) -l, --label stringArray Set metadata on container --label-file strings Read in a line delimited file of labels --log-driver string Logging driver for the container --log-opt strings Logging driver options --mac-address string Container MAC address (e.g. 92:d0:c6:0a:29:33) -m, --memory &lt;number&gt;[&lt;unit&gt;] Memory limit (format: &lt;number&gt;[&lt;unit&gt;], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes)) --memory-reservation &lt;number&gt;[&lt;unit&gt;] Memory soft limit (format: &lt;number&gt;[&lt;unit&gt;], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes)) --memory-swap string Swap limit equal to memory plus swap: &#39;-1&#39; to enable unlimited swap --memory-swappiness int Tune container memory swappiness (0 to 100, or -1 for system default) (default -1) --mount stringArray Attach a filesystem mount to the container --name string Assign a name to the container --network string Connect a container to a network (default &quot;slirp4netns&quot;) --no-healthcheck Disable healthchecks on container --no-hosts Do not create /etc/hosts within the container, instead use the version from the image --oom-kill-disable Disable OOM Killer --oom-score-adj int Tune the host&#39;s OOM preferences (-1000 to 1000) --pid string PID namespace to use --pids-limit int Tune container pids limit (set 0 for unlimited, -1 for server defaults) --pod string Run container in an existing pod --pod-id-file string Read the pod ID from the file --privileged Give extended privileges to container -p, --publish strings Publish a container&#39;s port, or a range of ports, to the host (default []) -P, --publish-all Publish all exposed ports to random ports on the host interface --pull string Pull image before creating (&quot;always&quot;|&quot;missing&quot;|&quot;never&quot;) (default &quot;missing&quot;) -q, --quiet Suppress output information when pulling images --read-only Make containers root filesystem read-only --read-only-tmpfs When running containers in read-only mode mount a read-write tmpfs on /run, /tmp and /var/tmp (default true) --replace If a container with the same name exists, replace it --restart string Restart policy to apply when a container exits (&quot;always&quot;|&quot;no&quot;|&quot;on-failure&quot;) --rm Remove container (and pod if created) after exit --rmi Remove container image unless used by other containers --rootfs The first argument is not an image but the rootfs to the exploded container --seccomp-policy string Policy for selecting a seccomp profile (experimental) (default &quot;default&quot;) --security-opt stringArray Security Options --shm-size &lt;number&gt;[&lt;unit&gt;] Size of /dev/shm (format: &lt;number&gt;[&lt;unit&gt;], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes)) (default &quot;65536k&quot;) --sig-proxy Proxy received signals to the process (default true) --stop-signal string Signal to stop a container. Default is SIGTERM --stop-timeout uint Timeout (in seconds) to stop a container. Default is 10 (default 10) --subgidname string Name of range listed in /etc/subgid for use in user namespace --subuidname string Name of range listed in /etc/subuid for use in user namespace --sysctl strings Sysctl options --systemd string Run container in systemd mode (&quot;true&quot;|&quot;false&quot;|&quot;always&quot;) (default &quot;true&quot;) --tmpfs tmpfs Mount a temporary filesystem (tmpfs) into a container -t, --tty Allocate a pseudo-TTY for container --uidmap strings UID map to use for the user namespace --ulimit strings Ulimit options -u, --user string Username or UID (format: &lt;name|uid&gt;[:&lt;group|gid&gt;]) --userns string User namespace to use --uts string UTS namespace to use -v, --volume stringArray Bind mount a volume into the container --volumes-from strings Mount volumes from the specified container(s) -w, --workdir string Working directory inside the container </span></pre> </editor-fold> <editor-fold podman_run> <h2 id="podman_run">podman run</h2> <p> From <a href='https://chariotsolutions.com/blog/post/building-and-deploying-lambdas-from-a-docker-container/' target='_blank' rel='nofollow'>Building and Deploying Lambdas from a Docker Container</a> by Keith Gregory: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id84457828dac5'><button class='copyBtn' data-clipboard-target='#id84457828dac5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman run \ -it \ --entrypoint /bin/bash \ --rm \ -v /tmp:/mnt \ amazon/aws-lambda-python:3.8 <span class='unselectable'>Trying to pull quay.io/amazon/aws-lambda-python:3.8... Requesting bear token: invalid status code from registry 405 (Method Not Allowed) Trying to pull docker.io/amazon/aws-lambda-python:3.8... Getting image source signatures Copying blob df513d38f4d9 skipped: already exists Copying blob 2e2bb77ae2dc skipped: already exists Copying blob 031c6369fb2b skipped: already exists Copying blob 03ac043af787 skipped: already exists Copying blob 842c9dce67e8 skipped: already exists Copying blob 1de4740de1c2 [--------------------------------------] 0.0b / 0.0b Copying config e12ea62c55 done Writing manifest to image destination Storing signatures bash-4.2# </span>pwd <span class='unselectable'>/var/task </span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8d230093c1d7'><button class='copyBtn' data-clipboard-target='#id8d230093c1d7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah run \ --entrypoint /var/lang/bin/pip \ --rm \ --user "$(id -u):$(id -g)" \ -v "$(pwd):/mnt" \ amazon/aws-lambda-python:3.8 \ install --target /mnt/build --upgrade psycopg2-binary</pre> </editor-fold> <editor-fold buildah> <h2 id="builah">Buildah</h2> <div style=""> <a href="https://buildah.io/" target="_blank" ><picture> <source srcset="/blog/images/buildah-logo.webp" type="image/webp"> <source srcset="/blog/images/buildah-logo.png" type="image/png"> <img src="/blog/images/buildah-logo.png" class=" liImg2 rounded shadow" style="padding: 1em" /> </picture></a> </div> <p> <a href='https://buildah.io/' target='_blank' rel='nofollow'>Buildah</a> is a drop-in replacement for using <code>docker build</code> and a <code>Dockerfile</code>. </p> <div class="quote"> Where Buildah really shines is in its native commands, which you can use to interact with container builds. Rather than using <code>build-using-dockerfile/bud</code> for each build, Buildah has commands to actually interact with the temporary container created during the build process. (Docker uses temporary, or intermediate containers, too, but you don’t really interact with them while the image is being built.) <br><br> Unlike <code>docker build</code>, Buildah doesn’t commit changes to a layer automatically for every instruction in the <code>Dockerfile</code> &ndash; it builds everything from top to bottom, every time. On the positive side, this means non-cached builds (for example, those you would do with automation or build pipelines) end up being somewhat faster than their Docker build counterparts, especially if there are many instructions. <br><br> &nbsp; &ndash; From <a href='https://opensource.com/article/18/6/getting-started-buildah' target='_blank' rel='nofollow'>Getting started with Buildah.</a>, published by <code>opensource.com</code> </div> <p> Some key Buildah subcommands: </p> <dl> <dt class="code">buildah bud</dt> <dd>Buildah’s <code>build-using-dockerfile</code>, or <code>bud</code> argument makes it behave just like <code>docker build</code> does.</dd> <dt class="code">buildah from</dt> <dd>Build up a container root filesystem from an image or from scratch.</dd> <dt class="code">buildah config</dt> <dd>Adjust defaults in the image's configuration blob.</dd> <dt class="code">buildah run</dt> <dd> <code>buildah run</code> is for running commands that build a container image. This is similar to <code>RUN</code> in a <code>Dockerfile</code>, and unlike <code>docker run</code>. </dd> <dt class="code">buildah commit</dt> <dd>Commit changes to the container to a new image.</dd> <dt class="code">buildah push</dt> <dd>Push images to registries (such a Quay) or a local <code>dockerd</code> instance.</dd> <dt class="code"></dt> <dd></dd> <dt class="code"></dt> <dd></dd> <dt class="code"></dt> <dd></dd> </dl> <h3 id="buildahHelp">Buildah Help</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbe5d9d456f41'><button class='copyBtn' data-clipboard-target='#idbe5d9d456f41' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah -h <span class='unselectable'>A tool that facilitates building OCI images<br/> Usage: buildah [flags] buildah [command]<br/> Available Commands: add Add content to the container build-using-dockerfile Build an image using instructions in a Dockerfile commit Create an image from a working container config Update image configuration settings containers List working containers and their base images copy Copy content into the container from Create a working container based on an image help Help about any command images List images in local storage info Display Buildah system information inspect Inspect the configuration of a container or image login Login to a container registry logout Logout of a container registry manifest Manipulate manifest lists and image indexes mount Mount a working container&#39;s root filesystem pull Pull an image from the specified location push Push an image to a specified destination rename Rename a container rm Remove one or more working containers rmi Remove one or more images from local storage run Run a command inside of the container tag Add an additional name to a local image umount Unmount the root file system of the specified working containers unshare Run a command in a modified user namespace version Display the Buildah version information<br/> Flags: -h, --help help for buildah --log-level string The log level to be used. Either &quot;debug&quot;, &quot;info&quot;, &quot;warn&quot; or &quot;error&quot;. (default &quot;error&quot;) --registries-conf string path to registries.conf file (not usually used) --registries-conf-dir string path to registries.conf.d directory (not usually used) --root string storage root dir (default &quot;/var/lib/containers/storage&quot;) --runroot string storage state dir (default &quot;/var/run/containers/storage&quot;) --storage-driver string storage-driver --storage-opt strings storage driver option --userns-gid-map ctrID:hostID:length default ctrID:hostID:length GID mapping to use --userns-uid-map ctrID:hostID:length default ctrID:hostID:length UID mapping to use -v, --version version for buildah<br/> Use &quot;buildah [command] --help&quot; for more information about a command. </span></pre> </editor-fold> <editor-fold buildah_use> <h3 id="buildahUse">Buildah / <span class="code">Dockerfile</span> Compatibility</h3> <div style=""> <picture> <source srcset="/blog/images/whales.webp" type="image/webp"> <source srcset="/blog/images/whales.png" type="image/png"> <img src="/blog/images/whales.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <p> <a href='https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/building_running_and_managing_containers/building-container-images-with-buildah_porting-containers-to-systemd-using-podman' target='_blank' rel='nofollow'>Buildah</a> can create an image from a Dockerfile by typing: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddd81f148b53e'><button class='copyBtn' data-clipboard-target='#iddd81f148b53e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah bud -t hello .</pre> <p> &hellip;instead of: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ideab382c59771'><button class='copyBtn' data-clipboard-target='#ideab382c59771' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker build -t hello .</pre> <p> Buildah can create an image called <code>hello</code> from the <code>Dockerfile</code> and the Python app by typing: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id48fe32d1e1fc'><button class='copyBtn' data-clipboard-target='#id48fe32d1e1fc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah bud -t hello . <span class='unselectable'>STEP 1: FROM public.ecr.aws/lambda/python:3.8 Getting image source signatures Copying blob 1de4740de1c2 done Copying blob 2e2bb77ae2dc done Copying blob df513d38f4d9 done Copying blob 03ac043af787 done Copying blob 031c6369fb2b done Copying blob 842c9dce67e8 done Copying config e12ea62c55 done Writing manifest to image destination Storing signatures STEP 2: COPY app.py ./ STEP 3: CMD [&quot;app.handler&quot;] STEP 4: COMMIT hello Getting image source signatures Copying blob 109f575f8e6a skipped: already exists Copying blob ff64b4f854ad skipped: already exists Copying blob dd66ad8702f4 skipped: already exists Copying blob d6fa53d6caa6 skipped: already exists Copying blob 80166c3283e5 skipped: already exists Copying blob 61f74564c3aa skipped: already exists Copying blob d95ebdc79761 done Copying config 40ef32b39c done Writing manifest to image destination Storing signatures --&gt; 40ef32b39cf 40ef32b39cf4ffd3d2e4e3426bec4a5ea168524f7f3fcfe863a378abd9794270 </span></pre> <p> Once the build is complete, the new image can be displayed with the <code>buildah images</code> command: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3043cc6b0f38'><button class='copyBtn' data-clipboard-target='#id3043cc6b0f38' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah images <span class='unselectable'>REPOSITORY TAG IMAGE ID CREATED SIZE localhost/hello latest 40ef32b39cf4 56 seconds ago 622 MB </span></pre> <p> The new image, tagged <code>hello:latest</code>, can be pushed to a remote image registry. This is easily accomplished with the <code>buildah push</code> command. </p> </editor-fold> <editor-fold buildah_push> <h3 id="buildah_push"><span class="code">buildah push</span> Help</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3e9ff05c17a9'><button class='copyBtn' data-clipboard-target='#id3e9ff05c17a9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>man buildah-push <span class='unselectable'>buildah-push(1) General Commands Manual buildah-push(1)<br/> NAME buildah-push - Push an image from local storage to elsewhere.<br/> SYNOPSIS buildah push [options] image [destination]<br/> DESCRIPTION Pushes an image from local storage to a specified destination, decompressing and recompessing layers as needed.<br/> imageID Image stored in local container/storage<br/> DESTINATION The DESTINATION is a location to store container images. If omitted, the source image parameter will be reused as destination.<br/> The Image &quot;DESTINATION&quot; uses a &quot;transport&quot;:&quot;details&quot; format. Multiple transports are supported:<br/> dir:path An existing local directory path storing the manifest, layer tarballs and signatures as individual files. This is a non-standardized format, primarily useful for debugging or noninvasive container inspection.<br/> docker://docker-reference An image in a registry implementing the &quot;Docker Registry HTTP API V2&quot;. By default, uses the authorization state in $XDG\_RUNTIME\_DIR/containers/auth.json, which is set using (buildah login). If the authorization state is not found there, $HOME/.docker/config.json is checked, which is set using (docker login). If docker-reference does not include a registry name, the image will be pushed to a registry running on local&#8208; host.<br/> docker-archive:path[:docker-reference] An image is stored in the docker save formatted file. docker-reference is only used when creating such a file, and it must not contain a digest.<br/> docker-daemon:docker-reference An image _dockerreference stored in the docker daemon internal storage. If _dockerreference does not begin with a valid registry name (a domain name containing &quot;.&quot; or the reserved name &quot;localhost&quot;) then the default registry name &quot;docker.io&quot; will be prepended. _dockerreference must contain either a tag or a digest. Alternatively, when reading images, the format can also be docker-daemon:algo:digest (an image ID).<br/> oci:path:tag An image tag in a directory compliant with &quot;Open Container Image Layout Specification&quot; at path.<br/> oci-archive:path:tag An image tag in a tar archive compliant with &quot;Open Container Image Layout Specification&quot; at path.<br/> If the transport part of DESTINATION is omitted, &quot;docker://&quot; is assumed.<br/> OPTIONS --authfile path<br/> Path of the authentication file. Default is ${XDG_RUNTIME_DIR}/containers/auth.json, which is set using buildah lo&#8208; gin. If the authorization state is not found there, $HOME/.docker/config.json is checked, which is set using docker login.<br/> --cert-dir path<br/> Use certificates at path (*.crt, *.cert, *.key) to connect to the registry. Default certificates directory is /etc/containers/certs.d.<br/> --creds creds<br/> The [username[:password]] to use to authenticate with the registry if required. If one or both values are not sup&#8208; plied, a command line prompt will appear and the value can be entered. The password is entered without echo.<br/> --digestfile Digestfile<br/> After copying the image, write the digest of the resulting image to the file.<br/> --disable-compression, -D<br/> Don&#39;t compress copies of filesystem layers which will be pushed.<br/> --encryption-key key<br/> The [protocol:keyfile] specifies the encryption protocol, which can be JWE (RFC7516), PGP (RFC4880), and PKCS7 (RFC2315) and the key material required for image encryption. For instance, jwe:/path/to/key.pem or pgp:admin@exam&#8208; ple.com or pkcs7:/path/to/x509-file.<br/> --format, -f<br/> Manifest Type (oci, v2s1, or v2s2) to use when saving image to directory using the &#39;dir:&#39; transport (default is manifest type of source)<br/> --quiet, -q<br/> When writing the output image, suppress progress output.<br/> --remove-signatures<br/> Don&#39;t copy signatures when pushing images.<br/> --sign-by fingerprint<br/> Sign the pushed image using the GPG key that matches the specified fingerprint.<br/> --tls-verify bool-value<br/> Require HTTPS and verify certificates when talking to container registries (defaults to true)<br/> EXAMPLE This example pushes the image specified by the imageID to a local directory in docker format.<br/> # buildah push imageID dir:/path/to/image<br/> This example pushes the image specified by the imageID to a local directory in oci format.<br/> # buildah push imageID oci:/path/to/layout:image:tag<br/> This example pushes the image specified by the imageID to a tar archive in oci format.<br/> # buildah push imageID oci-archive:/path/to/archive:image:tag<br/> This example pushes the image specified by the imageID to a container registry named registry.example.com.<br/> # buildah push imageID docker://registry.example.com/repository:tag<br/> This example pushes the image specified by the imageID to a container registry named registry.example.com and saves the digest in the specified digestfile.<br/> # buildah push --digestfile=/tmp/mydigest imageID docker://registry.example.com/repository:tag<br/> This example works like docker push, assuming registry.example.com/my_image is a local image.<br/> # buildah push registry.example.com/my_image<br/> This example pushes the image specified by the imageID to a private container registry named registry.example.com with authentication from /tmp/auths/myauths.json.<br/> # buildah push --authfile /tmp/auths/myauths.json imageID docker://registry.example.com/repository:tag<br/> This example pushes the image specified by the imageID and puts into the local docker container store.<br/> # buildah push imageID docker-daemon:image:tag<br/> This example pushes the image specified by the imageID and puts it into the registry on the localhost while turning off tls verification. # buildah push --tls-verify=false imageID docker://localhost:5000/my-imageID<br/> This example pushes the image specified by the imageID and puts it into the registry on the localhost using creden&#8208; tials and certificates for authentication. # buildah push --cert-dir /auth --tls-verify=true --creds=username:password imageID docker://local&#8208; host:5000/my-imageID<br/> ENVIRONMENT BUILD_REGISTRY_SOURCES<br/> BUILD_REGISTRY_SOURCES, if set, is treated as a JSON object which contains lists of registry names under the keys insecureRegistries, blockedRegistries, and allowedRegistries.<br/> When pushing an image to a registry, if the portion of the destination image name that corresponds to a registry is compared to the items in the blockedRegistries list, and if it matches any of them, the push attempt is denied. If there are registries in the allowedRegistries list, and the portion of the name that corresponds to the registry is not in the list, the push attempt is denied.<br/> TMPDIR The TMPDIR environment variable allows the user to specify where temporary files are stored while pulling and pushing images. Defaults to &#39;/var/tmp&#39;.<br/> FILES registries.conf (/etc/containers/registries.conf)<br/> registries.conf is the configuration file which specifies which container registries should be consulted when com&#8208; pleting image names which do not include a registry or domain portion.<br/> policy.json (/etc/containers/policy.json)<br/> Signature policy file. This defines the trust policy for container images. Controls which container registries can be used for image, and whether or not the tool should trust the images.<br/> SEE ALSO buildah(1), buildah-login(1), containers-policy.json(5), docker-login(1), containers-registries.conf(5)<br/> buildah June 2017 buildah-push(1) </span></pre> </editor-fold> <editor-fold how_to> <div style="text-align: right;"> <a href="https://shop.scholastic.com/parent-ecommerce/books/how-to-speak-dolphin-9780545676076.html" target="_blank" ><picture> <source srcset="/blog/images/howToSpeakDolphin.webp" type="image/webp"> <source srcset="/blog/images/howToSpeakDolphin.png" type="image/png"> <img src="/blog/images/howToSpeakDolphin.png" class="right liImg2 rounded shadow" style="width: 25%; height: auto;" /> </picture></a> </div> <h2 id="howto">How To</h2> <p> The following was inspired by <a href='https://github.com/groda/big_data/blob/master/docker_for_beginners.md#recap-images-and-containers' target='_blank' rel='nofollow'>Recap: images and containers</a> from <b>Docker for beginners</b>. The equivalent commands for Docker alternatives are shown. </p> <h3 class="clear" id="ver">Check software version</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd7a35b11029f'><span class='unselectable'>$ </span>docker -v<br><span class='unselectable'>Docker version 20.10.2, build 20.10.2-0ubuntu1~20.10.1 </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4a0721cb1b56'><span class='unselectable'>$ </span>buildah -v<br><span class='unselectable'>buildah version 1.15.2 (image-spec 1.0.1, runtime-spec 1.0.2-dev) </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8b600825d654'><span class='unselectable'>$ </span>podman -v<br><span class='unselectable'>podman version 2.0.6 </span></pre> <div style="text-align: right;"> <picture> <source srcset="/blog/images/aws_linux.webp" type="image/webp"> <source srcset="/blog/images/aws_linux.png" type="image/png"> <img src="/blog/images/aws_linux.png" class="right " style="width: 25%; height: auto; margin-bottom: 1em;" /> </picture> </div> <h3 id="dlimg">Download the Amazon Linux 2 image</h3> <p> AWS Lambda functions run under Amazon Linux. </p> <p> Each of these 3 commands does a very similar task, downloading a specific image. Docker uses different subdirectories for images than Buildah and <code>podman</code> do. </p> <div class="clear"> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idea998a160748'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker pull amazonlinux <span class='unselectable'>Using default tag: latest latest: Pulling from library/amazonlinux 3c2c91c7c431: Pull complete Digest: sha256:06b9e2433e4e563e1d75bc8c71d32b76dc49a2841e9253746eefc8ca40b80b5e Status: Downloaded newer image for amazonlinux:latest docker.io/library/amazonlinux:latest </span></pre> </div> <p> Buildah works without complaint. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7347e490a7ab'><span class='unselectable'>$ </span>buildah pull amazonlinux <span class='unselectable'>53ef897d731f9a5673c083d0e86d7911f85d6e63bb2be2346b17bdbacdc58637 </span></pre> <p> <code>podman</code> seems to hiccup and then complete successfully. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id64c039abd8e9'><span class='unselectable'>$ </span>podman pull amazonlinux <span class='unselectable'>Trying to pull quay.io/amazonlinux... error parsing HTTP 404 response body: invalid character &#39;&lt;&#39; looking for beginning of value: &quot;&lt;!DOCTYPE HTML PUBLIC \&quot;-//W3C//DTD HTML 3.2 Final//EN\&quot;&gt;\n&lt;title&gt;404 Not Found&lt;/title&gt;\n&lt;h1&gt;Not Found&lt;/h1&gt;\n&lt;p&gt;The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.&lt;/p&gt;\n&quot; Trying to pull docker.io/library/amazonlinux... Getting image source signatures Copying blob 3c2c91c7c431 [--------------------------------------] 0.0b / 0.0b Copying config 53ef897d73 done Writing manifest to image destination Storing signatures 53ef897d731f9a5673c083d0e86d7911f85d6e63bb2be2346b17bdbacdc58637 </span></pre> <div style="text-align: right;"> <picture> <source srcset="/blog/images/bash-logo.webp" type="image/webp"> <source srcset="/blog/images/bash-logo.png" type="image/png"> <img src="/blog/images/bash-logo.png" class="right liImg2 rounded shadow" style="width: 35%; height: auto; margin-bottom: 1em;" /> </picture> </div> <h3 id="echo">Run a Bash Command in an OCI Container</h3> <p> Again, <code>Docker</code> must be run as root for this operation, this represents an unnecessary security risk. </p> <div class="clear"> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7c10bd90d19c'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker container run amazonlinux echo 'Hello World!' <span class='unselectable'>Hello World! </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3271868a00fe'><span class='unselectable'>$ </span>podman container run amazonlinux <a href='https://opensource.com/article/18/6/linux-version#how-to-find-the-linux-kernel-version' target='_blank' rel='nofollow'>cat /etc/os-release</a> <span class='unselectable'>VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" </span></pre> </div> <div style="text-align: right;"> <picture> <source srcset="/blog/images/kodak-carousel-projector.webp" type="image/webp"> <source srcset="/blog/images/kodak-carousel-projector.png" type="image/png"> <img src="/blog/images/kodak-carousel-projector.png" class="right liImg2 rounded shadow" style="width: 35%; height: auto; margin-bottom: 1em;" /> </picture> </div> <h3 id="localImages">Show All Locally Available Images</h3> <p> Again, <code>Docker</code> must be run as root for this operation, this represents an unnecessary security risk. </p> <div class="clear"><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd806a8a717fd'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker images <span class='unselectable'>REPOSITORY TAG IMAGE ID CREATED SIZE amazonlinux latest 53ef897d731f 21 hours ago 163MB </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id92080ef43364'><span class='unselectable'>$ </span>podman images <span class='unselectable'>REPOSITORY TAG IMAGE ID CREATED SIZE localhost/hello latest 40ef32b39cf4 5 hours ago 622 MB docker.io/library/amazonlinux latest 53ef897d731f 21 hours ago 170 MB </span></pre> </div> <div style="text-align: right;"> <picture> <source srcset="/blog/images/containerShip.webp" type="image/webp"> <source srcset="/blog/images/containerShip.png" type="image/png"> <img src="/blog/images/containerShip.png" class="right liImg2 rounded shadow" style="width: 35%; height: auto; margin-bottom: 1em;" /> </picture> </div> <h3 id="listCont">List OCI Containers</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfdd5447067a6'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker container ls<br><span class='unselectable'>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id48f0df98fbd4'><span class='unselectable'>$ </span>podman container ls<br><span class='unselectable'>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES </span></pre> <h3 id="listcont4">View All OCI Containers (Running or Not)</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0fc6fc303c76'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker container ls -a <span class='unselectable'>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 250f56d9aced amazonlinux "echo 'Hello World!'" 14 minutes ago Exited (0) 14 minutes ago competent_einstein </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd0781dd6606b'><span class='unselectable'>$ </span>podman container ls -a <span class='unselectable'>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0f8203e9d3b8 docker.io/library/amazonlinux:latest echo Hello world! 36 minutes ago Exited (0) 36 minutes ago beautiful_mestorf 14282ace8978 docker.io/library/amazonlinux:latest echo Hello world! 36 minutes ago Exited (0) 36 minutes ago beautiful_goldwasser 1b9a8db52fb9 docker.io/library/alpine:latest echo Hello World! About an hour ago Exited (0) About an hour ago zealous_easley 6444ee144488 docker.io/library/amazonlinux:latest echo Hello World! 12 minutes ago Exited (0) 12 minutes ago frosty_ritchie 7444122cbc59 docker.io/library/alpine:latest cat /etc/motd About an hour ago Exited (0) About an hour ago elated_sammet aef84973d6ad docker.io/library/amazonlinux:latest echo Hello world! About an hour ago Exited (0) About an hour ago lucid_sinoussi e210f74bc209 docker.io/library/amazonlinux:latest cat /etc/motd About an hour ago Exited (0) About an hour ago jovial_borg </span></pre> <h3 id="listCont3">List Running OCI containers</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd232ea1c6cf7'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker container ps -a <span class='unselectable'>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 250f56d9aced amazonlinux "echo 'Hello World!'" 5 minutes ago Exited (0) 5 minutes ago competent_einstein </span></pre> <p> <code>podman</code> has a problem with the <code>container ps</code> sub-subcommand. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6d649b374f46'><span class='unselectable'>$ </span>podman container ps -a <span class='unselectable'>Error: unrecognized command `podman container ps` Try 'podman container --help' for more information. </span></pre> <h3 id="buildah_push_use"><span class="code">buildah push</span> to Docker Daemon</h3> <div style=""> <picture> <source srcset="/blog/images/containerShipTugboat.webp" type="image/webp"> <source srcset="/blog/images/containerShipTugboat.png" type="image/png"> <img src="/blog/images/containerShipTugboat.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id38448256b3dd'><button class='copyBtn' data-clipboard-target='#id38448256b3dd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah push hello:latest docker-daemon:hello:latest <span class='unselectable'>Getting image source signatures Copying blob sha256:72fcdba8cff9f105a61370d930d7f184702eeea634ac986da0105d8422a17028 247.02 MiB / 247.02 MiB [==================================================] 2s Copying blob sha256:e567905cf805891b514af250400cc75db3cb47d61219750e0db047c5308bd916 144.75 MiB / 144.75 MiB [==================================================] 1s Copying config sha256:6d54bef73e638f2e2dd8b7bf1c4dfa26e7ed1188f1113ee787893e23151ff3ff 1.59 KiB / 1.59 KiB [======================================================] 0s Writing manifest to image destination Storing signatures </span> <span class='unselectable'>$ </span>buildah images | head -n2 <span class='unselectable'>REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/hello latest 6d54bef73e63 2 minutes ago 398 MB </span> <span class='unselectable'>$ </span>buildah run -t hello:latest <span class='unselectable'>Hello, world! </span></pre> <h3 id="buildah_rmi">Delete an OCI Image</h3> <div style=""> <picture> <source srcset="/blog/images/containerSky.webp" type="image/webp"> <source srcset="/blog/images/containerSky.png" type="image/png"> <img src="/blog/images/containerSky.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <p> Delete an OCI image in Buildah's <code>~/.local/share/container</code> directory with the <code>rmi</code> subcommand: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id79c1b068ad84'><button class='copyBtn' data-clipboard-target='#id79c1b068ad84' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah rmi e12ea62c5582 <span class='unselectable'>e12ea62c5582f91a2228e3e284ea957f2df4f1cdb150fd2c189ef8f11d7633ce </span></pre> </editor-fold> Stack Overflow Culture: Zero-Sum, Authoritarian and Hormonally Imbalanced 2021-04-18T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/18/so-culture <p> This blog post describes some problems that significantly impact <a href='https://www.crunchbase.com/organization/stack-overflow' target='_blank' rel='nofollow'>Stack Overflow</a> users, and offers suggestions for improvement. If you are unfamiliar with Stack Overflow, or would like to read a summary of what this blog post is based on, Kevin Workman wrote a terrific summary in March 2019 entitled &ldquo;<a href='https://happycoding.io/blog/stack-overflow-culture-wars' target='_blank' rel='nofollow'>The Stack Overflow Culture Wars</a>&rdquo;. Very little has changed since then. </p> <div style="text-align: center;"> <a href="https://www.stackoverflow.com" target="_blank" ><picture> <source srcset="/blog/images/stackoverflowLogo.webp" type="image/webp"> <source srcset="/blog/images/stackoverflowLogo.png" type="image/png"> <img src="/blog/images/stackoverflowLogo.png" class="center halfsize liImg2 rounded shadow" style="padding: 20px;" /> </picture></a> </div> <fold-article intro> <h2 id="intro">Introduction</h2> <p> <a href='https://www.stackoverflow.com' target='_blank' rel='nofollow'>Stack Overflow</a> is the premier website world-wide for programmers to help each other by asking and answering questions. It has a defined protocol for this type of interaction, however new user on-boarding is often ineffective, so newcomers are not properly informed, and old-timers often do not exhibit appropriate people skills. The <a href='https://games.greggman.com/game/done-with-stackoverflow/' target='_blank' rel='nofollow'>protocol is somewhat misguided</a> and appropriate tools for better interaction are not provided. </p> <p> As a result, this website has developed a well-documented reputation for being “<a href='https://codeblog.jonskeet.uk/2018/03/17/stack-overflow-culture/' target='_blank' rel='nofollow'>a valuable resource, but a scary place to contribute due to potential hostility.</a>” </p> <div class="pullQuote"> Sometimes, loving something means caring enough to admit that it has a problem.<br><br> &nbsp; &ndash; <a href='https://stackoverflow.blog/2018/04/26/stack-overflow-isnt-very-welcoming-its-time-for-that-to-change/' target='_blank' rel='nofollow'>Jay Hanlon</a>, writing about Stack Overflow when he was EVP of Culture and Experience. </div> </fold-article> <fold-article problem> <div class="clear quote"> Stack Overflow suffers from militant moderators who close and delete reasonable submissions and answers due to Draconian rules.<br><br> &nbsp; &ndash; <code>sleavey</code> commenting on a Hacker News thread entitled <a href='https://news.ycombinator.com/item?id=16610353' target='_blank' rel='nofollow'>Stack Overflow Culture</a>. </div> </fold-article> <fold-article top> <h2 id="topdown">Change Starts At the Top</h2> <h3 id="Chandrasekar">Prashanth Chandrasekar, CEO</h3> <p> Prashanth Chandrasekar became CEO of Stack Overflow in September, 2019. <a href='https://www.intercom.com/blog/podcasts/prashanth-chandrasekar-on-writing-the-script-of-the-future/' target='_blank' rel='nofollow'>Inside Intercom interviewed Mr. Chandrasekar</a> in August 2020. This puff piece made no mention of any cultural problems. The focus was on the brilliance of Stack Overflow&rsquo;s technology. If Mr. Chandrasekar has a vision for how to guide social change, he did not mention it. Instead, in the article and his publications since then he only speaks publicly about <a href='https://stackoverflow.blog/author/pchandrasekar/' target='_blank' rel='nofollow'>transitioning Stack Overflow to a product-led SaaS company</a>. </p> <p> Shortly after becoming CEO, Mr. Chandrasekar published &ldquo;<a href='https://stackoverflow.blog/2020/01/21/scripting-the-future-of-stack-2020-plans-vision/' target='_blank' rel='nofollow'>Scripting the Future of Stack Overflow</a>, in which he wrote: </p> <p class="quote"> We learned that we needed much better channels to listen to our moderators and community members. We have not evolved the existing channels of engagement for power users in our community, like Meta, or articulated how we intended to make improvements going forward. This has caused friction as our user base and business have rapidly grown. We acknowledge these issues, apologize for our mistakes, and have plans for improving in the future. </p> <p> Later in the article, he mentioned improvements to the code of conduct, a survey and establishing a moderator council. More than 2 years later, none of this has made the slightest difference in Stack Overflow's culture. </p> <h3 id="pathak">Mihir Pathak — EVP, Strategy & Transformation</h3> <p> Prior to his employment at Stack Overflow, Mr. Pathak was a derivatives strategist McKinsey &amp; Company, a <a href='https://www.mckinsey.com/business-functions/organization/our-insights/the-four-building-blocks--of-change#' target='_blank' rel='nofollow'>world-renouned change management company</a>. That is to say, although Mr. Pathak worked at McKinsey, he was not there to assist other companies make structural changes; instead, he was responsible for pricing methodologies and hedging techniques underlying financial derivative products and options trading strategies &ndash; a heads-down money manager. </p> <p> <a href='https://stackoverflow.com/company/leadership/mihir-pathak' target='_blank' rel='nofollow'>Mr. Pathak&rsquo;s page at StackOverflow</a> is no longer available. <a href='https://webcache.googleusercontent.com/search?q=cache:1eKtOpLW2VoJ:https://stackoverflow.com/company/leadership/mihir-pathak' target='_blank' rel='nofollow'>Google cached it</a>. No announcement has been made about his departure or if there will be a replacement. </p> <p> Mr. Pathak's job description is still visible at <a href='https://www.themuse.com/profiles/stackoverflow' target='_blank' rel='nofollow'><code>themuse.com</code></a>: </p> <p class="quote"> Mihir is responsible for the long-term business strategy of Stack Overflow, which includes forming partnerships with like-minded organizations and understanding how to best serve the needs of future developers. </p> <p> Mr. Pathak was employed from September 2016 at Stack Overflow as a business development executive, not a change management champion. </p> <h3 id="dietrich">Teresa Dietrich, Chief Product and Technology Officer</h3> <p> In January 2020 <a href='https://www.crunchbase.com/person/teresa-dietrich' target='_blank' rel='nofollow'>Teresa Dietrich</a> was made Chief Product and Technology Officer. She also came from McKinsey &amp; Company, where she was Global Head of Product &amp; Engineering. Two months after she took the job she wrote &ldquo;<a href='https://stackoverflow.blog/2020/02/25/sharing-our-first-quarter-2020-community-roadmap/' target='_blank' rel='nofollow'>Sharing our first quarter 2020 community roadmap</a>&rdquo;. That rather bland article did not seem to indicate any recognition of serious problems within the Stack Overflow culture, and I could not find any publicly available results of the studies that were mentioned. </p> <p> 16 months after Ms. Dietrich started her job, I do not see any cultural change. Has Ms. Dietrich been tasked with leading such a change? If so: </p> <ul> <li>Does she have board-level support?</li> <li>Does she have what she needs to make significant cultural changes?</li> <li> Why has she not made any public acknowledgement of a cultural problem? One possible answer is that her boss, Mr. Chandrasekar, has not done so. </li> <li> Is Ms. Dietrich the right person for the job? This is not a purely technical problem, it is a social problem. I believe that women in general possess more highly developed social skills, but the skills necessary to climb to the top are not the skills required to make this type of cultural shift. </li> </ul> <div class="pullQuote"> Are Mr. Chandrasekar and Ms. Dietrich part of the cultural problem, or part of the solution? </div> </fold-article> <fold-article women> </fold-article> <fold-article me> <h2 id="me">Where Am I Coming From?</h2> <p> I have no agenda, no investment in the status quo, nothing to prove, no contacts at the company, I am not an undercover journalist, and I am not a competitor or investor. I am just Joe User... and I am not shy when I believe I have something that I think needs to be heard. </p> <p> If I mispeak, please tell me. If I missed something, again please tell me. If there is a bigger picture I would like to learn about it. </p> <p> I have used Stack Overflow and its <a href='https://stackexchange.com/sites' target='_blank' rel='nofollow'>sibling websites</a> for more than 10 years. Until a few weeks ago, I contributed a few answers here and there, and otherwise did not spend much time on the sites. All contributors are volunteers, so the only reasons to contribute are for prestige, social bonding and altruism. </p> <p> For almost 30 days, in my spare time, I helped people who asked questions on Stack Overflow. I am writing this blog post because although I enjoy helping others, the enjoyment I experienced while doing so within the Stack Overflow environment was overshadowed by the regressive behavior tolerated and even enforced by other contributors. I should also say that I did encounter certain other contributors with whom interaction was very pleasant. However, interactions with alpha contributors with the highest scores seemed more often than not to be quite unpleasant. Since I first published this blog post, I have mostly not interacted on the site. I remain open to contributing to improvements in the culture. </p> <p> The next image is provided so readers know that I am able to effectively participate in the current Stack Overflow website. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/stackOverflow.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow.png" type="image/png"> <img src="/blog/images/stackOverflow.png" title="The volume of accepted and upvoted answers put me in the top 0.14% of Stack Overflow answerers in under 30 days." class="center halfsize liImg2 rounded shadow" alt="The volume of accepted and upvoted answers put me in the top 0.14% of Stack Overflow answerers in under 30 days." /> </picture> <figcaption class="halfsize" style="width: 100%; text-align: center;"> The volume of accepted and upvoted answers put me in the top 0.14% of Stack Overflow answerers in under 30 days. </figcaption> </figure> </div> <p> Over 100 of the answers I offered in a 30-day period were accepted as the preferred answer. In fact, most of my answers were selected as the preferred answer. That means many more alternative answers were not accepted. </p> <p> After losing, sometimes a contributor will delete their post. I have done it myself, when the winning post was significantly better by all measures. </p> <p> Some of the other contributors who had provided alternative answers that were not selected clearly felt they had lost a competition. This structure, and others that I describe below, define a system designed to generate hostility. Stack Overflow, as currently implemented, <a href='https://en.wikipedia.org/wiki/Dominance_hierarchy' target='_blank' rel='nofollow'>promotes dominance behavior</a>, which for most primates (other than <a href='https://www.scientificamerican.com/article/bonobo-sex-and-society-2006-06/' target='_blank' rel='nofollow'>bonobos</a>) is patriarchal. </p> </fold-article> <fold-article gamify> <div style="text-align: right;"> <a href="http://localhost:4001/blog/2021/04/18/so-culture.html" target="_blank" ><picture> <source srcset="/blog/images/codinghorror-app-icon.webp" type="image/webp"> <source srcset="/blog/images/codinghorror-app-icon.png" type="image/png"> <img src="/blog/images/codinghorror-app-icon.png" class="right liImg2 rounded shadow" /> </picture></a> </div> <h2 id="gamification">Gamification</h2> <p> Stack Overflow's success has be in part due to its successful <a href='https://en.wikipedia.org/wiki/Gamification' target='_blank' rel='nofollow'>gamification</a> of the interaction between questioners and answerers. Gamification is powerful and addictive. Unfortunately, the model chosen resembles FPS (<a href='https://en.wikipedia.org/wiki/First-person_shooter' target='_blank' rel='nofollow'>first-person shooter</a>) games, instead of co-operative games. </p> <p> Jeff Atwood, one of the two original authors of Stack Overflow, wrote an article in 2011 entitled <a href='https://blog.codinghorror.com/the-gamification/' target='_blank' rel='nofollow'>The Gamification</a>, in which he writes: </p> <div class="quote"> Gaming elements are not tacked on to the Stack Exchange Q&A engine, but a natural and essential element of the design. </div> <p> Just below that sentence, Mr. Atwood shows a screenshot from an FPS video game: </p> <div style="text-align: center;"> <a href="https://blog.codinghorror.com/the-gamification/" target="_blank" ><picture> <source srcset="/blog/images/fps.webp" type="image/webp"> <source srcset="/blog/images/fps.png" type="image/png"> <img src="/blog/images/fps.png" title="FPS Game screenshot from 'The Gamification'." class="center halfsize liImg2 rounded shadow" alt="FPS Game screenshot from 'The Gamification'." /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://blog.codinghorror.com/the-gamification/" target="_blank" > FPS Game screenshot from 'The Gamification'. </a> </figcaption> </figure> </div> <div class="quote"> I haven't ever quite come out and said it this way, but … I played a lot of Counter-Strike from 1998 to 2001, and Stack Overflow is in many ways my personal Counter-Strike.<br><br> &nbsp; &ndash; Jeff Atwood, from &ldquo;The Gamification&rdquo; </div> <p> FPS games are structured so the player only wins by killing others. This is an entirely different motivational structure than a scenario where a person only wins if others succeed. </p> </fold-article> <fold-article terms> <h2 id="terms">Terminology</h2> <p> I use a few non-standard terms in this blog post: </p> <dl> <dt>Questioner</dt> <dd>One who provides a question</dd> <dt>Answerer</dt> <dd>One who provides an answer</dd> <dt>Downvoter</dt> <dd>One who downvotes another person's contribution</dd> <dt>Downvotee</dt> <dd>One who has their contribution downvoted</dd> </dl> </fold-article> <fold-article downvote> <h2 id="dialog">Dialog Improves Most Questions</h2> <p> A small percentage of questions asked on Stack Overflow are unambiguous, contain all the necessary information, and are phrased well enough to be understood. For these questions, answers can be posted without any interaction between questioner and potential answerers. </p> <p> However, most questions involve a dialog between potential answerers and the questioner. In the dialog, the question is refined, and the questioner's code and any other relevant data is elicited and provided. The tools provided for such a dialog are unfortunately problematic. </p> <p> The only two mechanisms for interaction between questioner and potential answerers are comments and answers. Comments have severe limitations that greatly reduce their effectiveness for eliciting information from a questioner: </p> <ul> <li>Comments must be very short</li> <li>Comments cannot be formatted properly</li> <li>Comments cannot be edited for more than a few minutes</li> </ul> <p> This means that answerers who are trying to explain something to the questioner to elicit more information, or guide the questioner towards understanding their problem better, must resort to posting an incomplete answer. Posting an incomplete answer is risky; other potential answerers can attack the answer by downvoting it. </p> <div class="pullQuote"> Downvotes typically last forever </div> </fold-article> <fold-article incentives> <h2 class="clear" id="broken">Some Incentives Promote Hostility</h2> <p> Many long-time users have completely objectified other users, and act as if Stack Overflow is a video game. Points are accumulated, and at any given time there are a finite number of questions to answer. A person's reputation on Stack Overflow is represented by a single number, which is the number of points they have accumulated. This single number is the structural source of many problems. A more nuanced reputation score would be a giant step forward. </p> <p> Many of these long-term answerers have come to view their participation on Stack Overflow as a zero-sum competition; they can only win (that is, have their answer accepted) if everyone else loses (that is, no-one else provides an answer that is accepted). </p> <p> Some answerers employ intimidation order to suppress competition. Downvotes are often used in the same way against other answerers as a <a href='https://en.wikipedia.org/wiki/Brushback_pitch' target='_blank' rel='nofollow'>brush back pitch</a> in baseball. </p> <div class="pullQuote"> Downvoting has no negative consequences for the downvoter </div> <p> Standard operating procedure for competitively-minded answerers is to downvote answers from others at every opportunity. There is no risk in downvoting: </p> <ul> <li>Downvotes are untraceable; there is no public record of who downvoted or when a downvote was cast.</li> <li>Downvotes are free to downvoters; this encourages liberal downvoting.</li> </ul> <p> This scenario incentivizes competitively-minded answerers to strafe the competition with downvotes at every opportunity. </p> <div class="pullQuote"> If downvotes cost the downvoter the same number of points as they penalize the downvotee, then downvotes would become much rarer. </div> <p> Downvotes typically last forever. Yes, a downvoter could theoretically reverse a downvote, but it is awkward for them to find their old downvotes, and there is no incentive to do so. </p> <p> Questions from newcomers are also frequently downvoted, without discussion, or along with hostile remarks. That leaves a permanent impression, and tends to select for new members who are comfortable with dominance-based hostility. This is a self-perpetuating, and highly toxic, social order. </p> <div style="text-align: center;"> <a href="https://www.focusforhealth.org/how-toxic-masculinity-harms-men-and-society-as-a-whole/" target="_blank" ><picture> <source srcset="/blog/images/toxicMasulinity.webp" type="image/webp"> <source srcset="/blog/images/toxicMasulinity.png" type="image/png"> <img src="/blog/images/toxicMasulinity.png" title="Toxic Masculinity Harms Men and Society As A Whole, from Focus for Health" class="center halfsize liImg2 rounded shadow" alt="Toxic Masculinity Harms Men and Society As A Whole, from Focus for Health" /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://www.focusforhealth.org/how-toxic-masculinity-harms-men-and-society-as-a-whole/" target="_blank" > Toxic Masculinity Harms Men and Society As A Whole, from Focus for Health </a> </figcaption> </figure> </div> </fold-article> <fold-article suggest_onboard> <div style="text-align: right;"> <picture> <source srcset="/blog/images/stackOverflowHelp.webp" type="image/webp"> <source srcset="/blog/images/stackOverflowHelp.png" type="image/png"> <img src="/blog/images/stackOverflowHelp.png" class="right quartersize liImg2 rounded shadow" /> </picture> </div> <h2 id="onboard">Suggestion: Gamified Onboarding</h2> <p> The only information for new users that is directly accessible from the Stack Overflow front page, is the Help Center, under the question mark icon. It is obvious from the often very polite and tentative inquiries made by new users when they ask their first question that they never noticed that information, or if they did it did not seem relevant. ... and then those new users are mercilessly slammed. </p> <p> New users should be presented with a <a href='https://www.appcues.com/blog/the-5-best-user-onboarding-experiences' target='_blank' rel='nofollow'>short instructional question and answer-style introduction</a>, where information is provided on how to be a good Stack Overflow user. This should happen before they get the opportunity to post their first question. Different levels of users should be explained. Although Stack Overflow is all-English, the onboarding should be multilingual. </p> </fold-article> <fold-article suggest_multilingual> <h2 id="multling">Suggestion: Multilingual Support</h2> <p> A high percentage of users do not speak English very well. They really struggle, and tolerance is low on Stack Overflow for bad English. Other sites, for example Facebook and LinkedIn, have a translation facility built right in. I think Facebook did a particularly good job. Why not do something similar on StackOverflow.com? English would be the official language, but everyone world-wide would be able to interact much more effectively. </p> <div class="pullQuote"> This site is multilingual.<br/> It is not that hard to do! </div> <p> Machine translation is really quite good. I have it on this site. What to view this site in one of over 100 languages? Just <a href='#body'>go to the top of this page</a> and click on this pull-down menu labeled Select Language: </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/selectLanguage.png" type="image/webp"> <source srcset="/blog/images/selectLanguage.png" type="image/png"> <img src="/blog/images/selectLanguage.png" class="center halfsize " /> </picture> </div> <p> You will then see the list of languages that you can view this website in: </p> <div style=""> <picture> <source srcset="/blog/images/selectLanguages.png" type="image/webp"> <source srcset="/blog/images/selectLanguages.png" type="image/png"> <img src="/blog/images/selectLanguages.png" class=" liImg2 rounded shadow" /> </picture> </div> <p> Morally speaking, not to provide a multilingual site discriminates against non-English-speaking people. </p> </fold-article> <fold-article suggest_rep> <h2 id="elicitation">Suggestion: More Nuanced Reputation Score</h2> <p> Instead of a single metric, answerers should be rated along several dimensions. Economists and psychologists both use multidimensional diagrams. The following diagram represents data in 4 dimensions. More dimensions can easily be shown in this type of diagram. </p> <div style=""> <picture> <source srcset="/blog/images/multiDimentionalPlot.webp" type="image/webp"> <source srcset="/blog/images/multiDimentionalPlot.png" type="image/png"> <img src="/blog/images/multiDimentionalPlot.png" title="Multi-dimensional data can easily be visualized by outlined shapes" class=" fullsize liImg2 rounded shadow" alt="Multi-dimensional data can easily be visualized by outlined shapes" /> </picture> <figcaption class="fullsize" style="width: 100%; text-align: center;"> Multi-dimensional data can easily be visualized by outlined shapes </figcaption> </figure> </div> <p> Instead of "bigger is better" (a single number indicating status, with the high score indicating alpha status), more information would allow for more detail, so a fuzzy diagram would show little interaction, while a highly detailed and intricate design would indicate a lot of participation. Some answerers might be stronger regarding some metrics, while being weaker on other metrics. </p> <p> Instead of displaying a person's score, the shape of their participation would be shown. Some people prefer to be seen as well-rounded, others prefer to be the best on selected aspects and ignore the others. One size does not fit all. </p> <p> People would start to give pet names for various shapes. Jokes would be made about the shapes. </p> <p> HR personnel would start to hire teams based on how well these shapes meshed together. Money will be made by timely entrepreneurs because these shapes will quickly be adopted industry-wide. Some will aspire to change their shape. </p> <p> An entire industry will spring up servicing those who wish to modify their shape. </p> </fold-article> <fold-article suggest_active> <h2 id="restrict_down">Suggestion: Encourage and Highlight Elicitation</h2> <p> <a href='https://en.wikipedia.org/wiki/Elicitation_technique' target='_blank' rel='nofollow'>Elicitation</a> is a difficult skill to master. The current high-scorers have no incentives to employ elicitation, and they act as if on a campaign to eradicate it. </p> <div class="pullQuote"> Introduce an Elicitation Phase </div> <p> A button should be introduced that lets everyone who visits the question page that a potential answerer would like to elicit information. While at least one such button is enabled, no downvotes are possible relating to the question, and the question cannot be closed or moved to another forum by anyone, regardless of their privilege level. This elicitation mode times out, but not suddenly or unexpectedly. Instead, it backs off, rather like the Ethernet back-off algorithm for collision resolution used in random access MAC protocols. Both the potential answerer and the questioner are given cues that they have a question or a response waiting, and after a period of inactivity the special status ends. I leave the details of the timing undefined for others to discuss. </p> </fold-article> <fold-article suggest_crowd> <h2 class="restrict_down" id="crowd">Suggestion: Restrict Downvoting</h2> <p> Downvoting needs to incorporate: </p> <ul> <li>Accountability (no more anonymous downvoting)</li> <li>Cost (no more drive-by shootings without consequences)</li> <li>Time window (vote after the dust settles, not during the elicitation phase)</li> <li>Public displays of user profiles should prominently display that user's downvotes and upvotes, with links</li> <li>Aggregate statistics on user profiles of their percentage downvotes and upvotes, trends (absolute and relative), etc.</li> </ul> </fold-article> <fold-article suggest_crowd> </fold-article> Serverless E-Commerce 2021-04-14T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/14/serverless-ecommerce <style> h2.numbered:before { color: darkgreen; content: "Option " counter(h2counter) ":\A0\A0\A0"; } </style> <editor-fold intro> <p> As readers of this blog know, I have been <a href='/django/index.html'>chronicling my adventure into Python-powered e-commerce</a> for several months. I have been focusing on Django in general, and <a href='https://django-oscar.readthedocs.io' target='_blank' rel='nofollow'>Django-Oscar</a> in particular. Webapps made with this technology are almost exclusively run on dedicated real or virtual machines. <a href='https://www.cloudflare.com/learning/serverless/what-is-serverless/' target='_blank' rel='nofollow'>Serverless</a> computing is a method of providing backend services on an as-used basis. AWS Lambda is the best-known example of serverless computing, and it combines nicely with a CDN like AWS CloudFront. </p> <p> This blog post discusses 3 goals for an e-commerce system. Two goals are provided by the technology behind <a href='https://martinfowler.com/articles/serverless.html' target='_blank' rel='nofollow'>serverless webapps</a>: </p> <ol> <li>Enormous and instantaneous scalability.</li> <li>Pay-as-you-go without an up-front cost commitment.</li> </ol> <p> I have one more goal: very low latency for online shoppers. </p> </editor-fold> <editor-fold big_picture> <h2 id="big">The Big Picture</h2> <p class="quote"> AWS Lambda consists of two main parts: the Lambda service which manages the execution requests, and the Amazon Linux micro virtual machines provisioned using AWS Firecracker, which actually runs the code. <br><br> A Firecracker VM is started the first time a given Lambda function receives an execution request (the so-called “Cold Start”), and as soon as the VM starts, it begins to poll the Lambda service for messages. When the VM receives a message, it runs your function code handler, passing the received message JSON to the function as the event object. <br><br> Thus every time the Lambda service receives a Lambda execution request, it checks if there is a Firecracker microVM available to manage the execution request. If so, it delivers the message to the VM to be executed. <br><br> In contrast, if no available Firecracker VM is found, it starts a new VM to manage the message. Each VM executes one message at a time, so if a lot of concurrent requests are sent to the Lambda service, for example due to a traffic spike received by an API gateway, several new Firecracker VMs will be started to manage the requests and the average latency of the requests will be higher since each VM takes roughly a second to start. <br><br> &nbsp; &ndash; From <a href='https://www.proud2becloud.com/how-to-run-any-programming-language-on-aws-lambda-custom-runtimes/' target='_blank' rel='nofollow'>How to run any programming language on AWS Lambda: Custom Runtimes</a> by Matteo Moroni. </p> </editor-fold> <editor-fold lambdalimits> <h2 id="lambdalimits">AWS Lambda Limits</h2> <p> AWS Lambda programs have access to considerable resources, enough for most e-commerce stores. The AWS Lambda runtime environment has the following limitations, some of which can be improved upon with some work: </p> <ul> <li>The disk space (ephemeral) is limited to 512 MB.</li> <li>The default deployment package size is 50 MB.</li> <li>The memory range is from 128 to 3008 MB.</li> <li>The maximum execution timeout for a function is 15 minutes.</li> <li>Request and response (synchronous calls) body payload size can be up to 6 MB.</li> <li>Event request (asynchronous calls) body can be up to 128 KB.</li> </ul> </editor-fold> <editor-fold cf> <div style="text-align: right;"> <picture> <source srcset="/blog/images/django/clouds.webp" type="image/webp"> <source srcset="/blog/images/django/clouds.png" type="image/png"> <img src="/blog/images/django/clouds.png" class="right quartersize liImg2 rounded shadow" /> </picture> </div> <h2 id="cf">CloudFront</h2> <p> <a href='https://forum.djangoproject.com/u/briancaffey/summary' target='_blank' rel='nofollow'>Brian Caffey</a> wrote <a href='https://forum.djangoproject.com/t/building-a-django-application-on-aws-with-cloud-development-kit-cdk/2830' target='_blank' rel='nofollow'>Building a Django application on AWS with Cloud Development Kit (CDK)</a>. The website Mr. Caffey's article discusses does not use Lambda, instead his website is always running. So, this option is quite informative and well-thought-out, but it is AWS-specific and does not discuss serverless architecture. </p> <p> For me, the most interesting part about Mr. Caffey's article is it mentions using 3 origins with AWS CloudFront: (1) an origin for ALB (for hosting the Django API), (2) an origin for the S3 website (static Vue.js site), and (3) an S3 origin for Django assets. Mr. Caffey does not say why he used 3 origins, but feeding one CloudFront distribution from multiple origins would mean that all of their content would appear on the same Internet subdomain. </p> <p> This means that the <a href='https://www.w3.org/TR/2020/SPSD-cors-20200602/#resource-preflight-requests' target='_blank' rel='nofollow'>extra HTTP handshaking required for certain CORS</a> (cross-origin HTTP requests) requests between subdomains would be avoided; specifically, there would be no need for pre-flight requests. This would make the website seem noticeably faster if users did lots of content editing and/or transactions with the website. My own pet project has users creating and modifying content, and purchasing product, so taking the requirement for the CORS handshakes away would be a win, plus the end user's web browser could reuse the origin HTTP connection, speeding up even non-cacheable requests. </p> <p> Tamás Sallai wrote <a href='https://advancedweb.hu/how-to-route-to-multiple-origins-with-cloudfront/' target='_blank' rel='nofollow'>How to route to multiple origins with CloudFront &mdash; Set up path-based routing with Terraform</a>. Mr. Sallai <a href='https://advancedweb.hu/' target='_blank' rel='nofollow'>is a prolific writer</a>! </p> </editor-fold> <editor-fold edge> <div style="text-align: right;"> <picture> <source srcset="/blog/images/django/lifeOnTheEdge.webp" type="image/webp"> <source srcset="/blog/images/django/lifeOnTheEdge.png" type="image/png"> <img src="/blog/images/django/lifeOnTheEdge.png" class="right quartersize liImg2 rounded shadow" /> </picture> </div> <h2 id="edge">Edge Computing</h2> <p> Performing computations and serving assets from a <a href='https://aws.amazon.com/cloudfront/features/' target='_blank' rel='nofollow'>nearby point of presence</a> minimizes latency for end users. E-commerce customers much prefer online stores that respond quickly. Edge computing can deliver that experience world-wide, and developers can deploy their work from wherever they are. </p> <p> <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-at-the-edge.html' target='_blank' rel='nofollow'>AWS Lambda@Edge</a> (<a href='https://aws.amazon.com/lambda/edge/' target='_blank' rel='nofollow'>console</a>) runs the Lambda computation in one of 13 regional AWS points of presence, one hop removed from the CloudFront edge locations, or at least in the same availability zone at the CloudFront point of presence. Distributed database issues would need to be addressed before significant benefits would accrue from this implementing this decentralized architecture. Unfortunately, Lambda@Edge has some significant restrictions that prevent it from running nontrivial Django apps. </p> <h3>Lambda@Edge Restrictions</h3> <p>From <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-requirements-limits.html' target='_blank' rel='nofollow'>requirements and restrictions on using Lambda functions with CloudFront</a>, it is apparent that it is not possible to run non-trivial Django apps securely at the edge with good performance.</p> <ul> <li>You can add triggers only for functions in the US East (N. Virginia) Region.</li> <li>You can’t configure your Lambda function to access resources inside your VPC.</li> <li>AWS Lambda environment variables are not supported.</li> <li>Lambda functions with AWS Lambda layers are not supported.</li> <li>Using AWS X-Ray is not supported.</li> <li>AWS Lambda reserved concurrency and provisioned concurrency are not supported.</li> <li>Lambda functions defined as container images are not supported.</li> </ul> <p> Until such time as Lambda@Edge removes the above restrictions, Django webapps will continue to be deployed as centralized webapps, which means that ultra-low latency is not possible world-wide. </p> <h3 id="cf_fns">CloudFront Functions</h3> <p> <a href='https://aws.amazon.com/blogs/aws/introducing-cloudfront-functions-run-your-code-at-the-edge-with-low-latency-at-any-scale/' target='_blank' rel='nofollow'>CloudFront Functions</a> are closer to the user, but have even more restrictions than Lambda@Edge. Alas, CloudFront Functions do not seem likely to be able to support significant computation any time soon. </p> <div style=""> <a href="https://aws.amazon.com/blogs/aws/introducing-cloudfront-functions-run-your-code-at-the-edge-with-low-latency-at-any-scale/" target="_blank" ><picture> <source srcset="/blog/images/cloudfront-functions-only-lambda-egde-1024x413.webp" type="image/webp"> <source srcset="/blog/images/cloudfront-functions-only-lambda-egde-1024x413.png" type="image/png"> <img src="/blog/images/cloudfront-functions-only-lambda-egde-1024x413.png" title="From &ldquo;Introducing CloudFront Functions – Run Your Code at the Edge with Low Latency at Any Scale&rdquo;" class=" liImg2 rounded shadow" alt="From &ldquo;Introducing CloudFront Functions – Run Your Code at the Edge with Low Latency at Any Scale&rdquo;" /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://aws.amazon.com/blogs/aws/introducing-cloudfront-functions-run-your-code-at-the-edge-with-low-latency-at-any-scale/" target="_blank" > From &ldquo;Introducing CloudFront Functions – Run Your Code at the Edge with Low Latency at Any Scale&rdquo; </a> </figcaption> </figure> </div> </editor-fold> <editor-fold iac> <h2 id="vendor">Infrastructure as Code (IaC)</h2> <div class="quote"> For anything bigger than a toy cloud application, Infrastructure as Code (IaC) is table stakes. You’d be hard-pressed to find someone managing anything of scale who thinks letting folks point and click in the console is the optimal route. <br><br>&nbsp; &ndash; From <a href='https://acloudguru.com/blog/engineering/cloudformation-terraform-or-cdk-guide-to-iac-on-aws' target='_blank' rel='nofollow'>CloudFormation, Terraform, or CDK? A guide to IaC on AWS</a> by Jared Short, published by <code>acloudguru.com</code>. </div> <div class="quote"> <a href='https://www.hashicorp.com/products/terraform' target='_blank' rel='nofollow'>Terraform</a>, AWS CloudFormation, Packer, Pulumi, and GeoEngineer are the most popular tools in the category "Infrastructure Build Tools". <br> &nbsp; &ndash; <a href='https://stackshare.io/infrastructure-build-tools' target='_blank' rel='nofollow'>from Stackshare.io</a> </div> </editor-fold> <editor-fold infographic> <h2 id="infographic">Infographic: Lambda Framework Comparison</h2> <p> Yan Cui at Lumigo.io made <a href='https://lumigo.io/aws-lambda-deployment/' target='_blank' rel='nofollow'>this terrific infographic</a>, which compares 9 serverless application frameworks and infrastructure management tools according to opinionatedness and customizability. This article discusses some of those technologies. </p> <div style=""> <a href="https://lumigo.io/aws-lambda-deployment/" target="_blank" ><picture> <source srcset="/blog/images/django/lumigoComparison.webp" type="image/webp"> <source srcset="/blog/images/django/lumigoComparison.png" type="image/png"> <img src="/blog/images/django/lumigoComparison.png" title="From 'AWS Lambda Deployment Frameworks', by Yan Cui at lumigo.io" class=" fullsize liImg2 rounded shadow" alt="From 'AWS Lambda Deployment Frameworks', by Yan Cui at lumigo.io" /> </picture></a> <figcaption class="fullsize" style="width: 100%; text-align: center;"> <a href="https://lumigo.io/aws-lambda-deployment/" target="_blank" > From 'AWS Lambda Deployment Frameworks', by Yan Cui at lumigo.io </a> </figcaption> </figure> </div> <p> The trade-off between customizability and opinionatedness is that highly customizable frameworks require more code to do things that opinionated frameworks do more succinctly. On the other hand, very opinionated frameworks are more limited in their abilities. A classic example of an opinionated framework is Ruby on Rails, which is specifically designed for master/detail applications. Other types of applications should use a different framework, or no framework at all. </p> <p> Two of the technologies on the above infographic are Zappa and Terraform, both of which I discuss in this blog post. Zappa is rather opinionated, while Terraform is very customizable. </p> </editor-fold> <editor-fold cdk> <h2 class="numbered" id="cdk">AWS Cloud Development Kit (CDK)</h2> <p> AWS CDK provides a programmatic interface for modeling and provisioning cloud resources. Languages supported include Java, JavaScript, .NET, Node.js, Python and Typescript. </p> <p> Even if AWS is not directly the service provider, awareness of the <a href='https://aws.amazon.com/cdk/' target='_blank' rel='nofollow'>AWS CDK</a> is important because some other options, for example the <a href='https://aws.amazon.com/blogs/developer/introducing-the-cloud-development-kit-for-terraform-preview/' target='_blank' rel='nofollow'>Cloud Development Kit for Terraform</a> (cdktf), are based on <a href='https://github.com/aws/aws-cdk' target='_blank' rel='nofollow'>AWS CDK</a>. </p> </editor-fold> <editor-fold chalice> <h2 class="numbered" id="chalice">Chalice &ndash; Serverless Django on AWS</h2> <p> <a href='https://aws.github.io/chalice/' target='_blank' rel='nofollow'>Chalice</a> is an AWS open-source project that has good traction. This Python serverless microframework for AWS allows applications that use Amazon API Gateway and AWS Lambda to be quickly created and deployed. </p> <p> The name and logo of this project are suggestive of the Holy Grail. I found the thinly veiled references to Christianity to be off-putting. Religious references have no place in a professional environment. Programmers who work with this project have religious icons, words and phrases continuously presented to them, and they must write words that are strongly identified with Christian doctrine for them to write software. This is forced <a href='https://www.vocabulary.com/dictionary/indoctrination' target='_blank' rel='nofollow'>indoctrination</a>. </p> </editor-fold> <editor-fold zappa> <h2 class="numbered" id="zappa">Django w/ Zappa &amp; AWS Lambda</h2> <div style="text-align: right;"> <picture> <source srcset="/blog/images/django/zappa_400x400.webp" type="image/webp"> <source srcset="/blog/images/django/zappa_400x400.png" type="image/png"> <img src="/blog/images/django/zappa_400x400.png" title="Don't eat the yellow snow" class="right quartersize liImg2 rounded shadow" alt="Don't eat the yellow snow" /> </picture> </div> <p> <a href='github.com/zappa/Zappa'>Zappa</a> is a popular library for serverless web hosting of Python webapps. Zappa allows Python WSGI webapps like Django to run on AWS Lambda instead of from within a container like AWS EC2. I am particularly interested in using Zappa to package and run <code>django-oscar</code> for AWS Lambda and CloudFront. </p> <p> Zappa can perform two primary functions: </p> <ol> <li> <b>Packaging</b> &ndash; Zappa can build a Django webapp into an AWS Lambda package. The package can be delivered via other mechanisms, for example mechanisms that are not even Python aware. </li> <li> <b>Deploying</b> &ndash; Zappa can deploy and Django webapp to AWS Lambda, and configure several AWS services to feed events to the Django webapp. </li> </ol> <div class="quote"> Zappa does not provide a means to define additional resources as part of the overall infrastructure. It is also somewhat rigid in how it defines certain resources which can lead to friction when incorporating Zappa within organizations with more rigid requirements on cloud resource management. With Zappa, you are better off allowing it to manage all the pieces needed for your web application on its own and manage other resources with a separate tool such as stacker or Terraform. <br><br> &hellip; or use Zappa's <code>package</code> command to create an archive that is ready for upload to lambda and utilize the other helpful functions the project provides for use after code is deployed. <br><br> &nbsp; &ndash; from <a href='https://www.jbssolutions.com/resources/blog/evolution-maintainable-lambda-development-pt-2/' target='_blank' rel='nofollow'>The Evolution of Maintainable Lambda Development Pt 2</a> by JBS Custom Software Solutions. </div> <p> The Zappa documentation is excellent. The project has some rough edges, but the new regime coming on board seem competent and fired up. They have some work ahead to set things straight, but the technical path seems clear. </p> <p> I think this project deserves special attention. Lots of moldy issues and PRs need to be processed, which a small team could get done fairly quickly. The project might also benefit from someone to hone the messaging. I opened an <a href='https://github.com/zappa/Zappa/issues/968' target='_blank' rel='nofollow'>issue on the Zappa GitHub microsite</a> to discuss this. </p> <p> This seminal project has been around several years, and other well-known projects that have been developed since Zappa was first released have acknowledged that Zappa provided inspiration. Time to brush it up and set it straight again; its best days lie ahead! </p> <p> Edgar Roman wrote this helpful document: <a href='https://romandc.com/zappa-django-guide/' target='_blank' rel='nofollow'>Guide to using Django with Zappa</a>. </p> <p> I've messing around with Zappa, will report back. </p> <h3 id="videos">Videos</h3> <p> <a href='https://www.google.com/search?client=firefox-b-d&q=aws+zappa+django+video' target='_blank' rel='nofollow'>Videos of Zappa exist</a>. </p> <ul> <li> This video has got all the right technologies mixed together for me: <a href='https://www.youtube.com/watch?v=Gf0vpJQZeBI' target='_blank' rel='nofollow'>Serverless Deployment of a Django Project with AWS Lambda, Zappa, S3 and PostgreSQL</a>. </li> </ul> </editor-fold> <editor-fold option_djambda> <h2 class="numbered" id="djambda">Djambda / AWS Lambda / Terraform</h2> <p> Terraform does not impose a runtime dependency unless the realtime orchestration features are used. </p> <p> <a href='https://github.com/netsome/djambda' target='_blank' rel='nofollow'>Djambda</a> is an example project setting up Django application in AWS Lambda managed by Terraform. I intend to play with it and write up my experience right here Real Soon Now. </p> <p> This project uses GitHub Actions to create environments for the master branch and pull requests. I wonder if this project can be used without GitHub actions? </p> <div class="quote"> [Terraform] does not provide an abstraction layer for the AWS, Azure, or Google Cloud. It does that deliberately, as you should embrace all aspects when using cloud - not extract a common denominator from the services delivered by the cloud provider. <br><br> &nbsp; &ndash; From <a href='https://awsmaniac.com/aws-cdk-why-not-terraform/' target='_blank' rel='nofollow'>AWS CDK? Why not Terraform?</a> by Wojciech Gawroński. </div> </editor-fold> <editor-fold option_serverless> <h2 class="numbered" id="serverless">Serverless Framework with WSGI</h2> <p> The <a href='https://www.serverless.com/plugins/serverless-wsgi' target='_blank' rel='nofollow'>docs</a> describe Serverless WSGI as: </p> <div class="quote"> Serverless plugin to deploy WSGI applications (Flask/Django/Pyramid etc.) and bundle Python packages. </div> <p> I am concerned that the Serverless architecture requires an <a href='https://www.serverless.com/pricing/fair-use-policy/' target='_blank' rel='nofollow'>ongoing runtime dependency</a> on the viability and good will of Serverless, Inc. Any hiccup on their part will immediately be felt by all their users. It would make me nervous to base daily operational infrastructure on this. </p> </editor-fold> <editor-fold poor-trade> <h3 id="bintray">Bintray and JCenter Went <i>Poof!</i></h3> <p> I do not want to rely upon online services from a software tool vendor to run my builds. The Scala community is still recovering from Bintray and JCenter shutting down. I had dozens of Scala libraries on Bintray. I do not plan to migrate them, they are gone from public access. </p> <div class="quote"> On February 3, 2021, JFrog announced that they will be shutting down Bintray and JCenter. A complete shutdown is planned for February 2022. <br><br> &ndash; <a href='https://blog.gradle.org/jcenter-shutdown' target='_blank' rel='nofollow'>JCenter shutdown impact on Gradle builds</a> </div> <h3 id="free">Trading Autonomy for Minimal Convenience is a Poor Trade</h3> <p> Remember that free products are usually subject to change or termination without notice. Examples abound of many companies whose free (and non-free) products suddenly ceased. There is no need to assume this type of vulnerability, so I block my metaphoric ears to the siren sound that tempts trusting souls into assuming unnecessary dependencies, and I chose tooling that is completely under my control. </p> <div class="quote"> <h2>What happens if I exceed the fair use policy?</h2> <p> <i>From the <a href='https://www.serverless.com/pricing/' target='_blank' rel='nofollow'>Serverless Pricing and Terms page</a>.</i> </p> We want to offer a lot of value for free so you can get your idea off the ground, before introducing any infrastructure cost. The intent of the fair use policy is to ensure that we can provide a high quality of service without incurring significant infrastructure costs. The great majority of users will fall well within range of the typical usage guidelines. While we reserve the right to throttle services if usage exceeds the fair use policy, we do not intend to do so as long as we can deliver a high quality of service without significant infrastructure costs. <br><br> If you anticipate your project will exceed these guidelines, please contact our support team. We’ll work with you on a plan which scales well. </div> </editor-fold> <h2 id="apprunner">AWS AppRunner</h2> <p> AWS just announced <a href='https://aws.amazon.com/apprunner/' target='_blank' rel='nofollow'>AppRunner</a>. I wonder how suitable it is... </p> Merging a Remote File with a Local File 2021-04-12T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/12/merging-remote-file <p> Today I am once again re-installing WSL2 on one of my laptops. Seems that a Windows 10 installation&rsquo;s half-life is measured in months, after which time a reset is required. The reset preserves data, but not installed programs and not the WSL setup. </p> <p> When I set up an OS I often use a pre-existing system&rsquo;s files as templates for the new system&rsquo;s files. </p> <h2 id="meld">Meld</h2> <p> <a href='https://meldmerge.org/' target='_blank' rel='nofollow'>Meld</a> is a fantastic, F/OSS file and directory merge tool. 2-way and 3-way merges are supported. Meld uses X Windows for its user interface. <a href='https://opticos.github.io/gwsl/' target='_blank' rel='nofollow'>GWSL</a> makes it easy to run X apps on WSL and WSL2. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id812ac76ecd7f'><button class='copyBtn' data-clipboard-target='#id812ac76ecd7f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install meld</pre> <p> Merging a remote file with a local file using Meld is easy once you know how. Unless the remote file system is mounted locally, Meld cannot be used to modify <i>remote</i> files and directories, just <i>local</i> files and directories. </p> <p> Following is the incantation I used to display my local <code>.profile</code> and interactively merge it with my profile on an Ubuntu Linux machine called <code>gojira</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idecefdc601d6d'><button class='copyBtn' data-clipboard-target='#idecefdc601d6d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>meld ~/.profile &lt;(ssh mslinn@gojira cat .profile)&</pre> <p> The above runs <code>ssh</code> in a subshell, logs in as <code>mslinn</code> to the machine called <code>gojira</code> and then displays the contents of <code>.profile</code> on <code>gojira</code>. Meld compares the output of <code>cat</code> with the local copy of <code>~/.profile</code>, and displays the differences: </p> <div style=""> <picture> <source srcset="/blog/images/meld.webp" type="image/webp"> <source srcset="/blog/images/meld.png" type="image/png"> <img src="/blog/images/meld.png" class=" liImg2 rounded shadow" /> </picture> </div> <p> Meld makes it easy to reconcile file versions. <span style="font-size: 3em;">&#128513;</span> </p> Visual Studio Code Workspace Settings 2021-04-11T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/11/svcode-workspace-settings <p> For me, the killer feature that Visual Studio Code is how it integrates the Windows user interface with working on WSL and WSL2. Programs residing on the active WSL OS image execute natively on that OS, while VSCode continues to run as a native Windows application. This is possible because VSCode installs a proxy on the target OS. The proxy does the bidding of the Windows executable. </p> <p> Getting a project to execute on the target OS instead of the host OS can be tricky. I have found that using a workspace to hold a collection of VSCode projects is very helpful, because the definition of the collection also defines how they are handled. </p> <p> WSL projects have different types of VSCode workspace entries than Windows entries do. They are easy to recognize and change once you know what to look for. The two possibile types of VSCode workspace project entries in a <code>.workspace</code> file are: </p> <ul> <li><b>WSL Project</b> &mdash; <code>"uri": "vscode-remote://wsl+ubuntu/path/to/vscode/project"</code></li> <li><b>Windows Project</b> &mdash; <code>"path": "C:\\path\\to\\vscode\\project"</code></li> </ul> <p> The following VSCode workspace file has both types of entries. For me, this is an error; I only want WSL projects. My task is to change the yellow highlighted Windows project and make it look like the other WSL projects. </p> <div class='codeLabel unselectable' data-lt-active='false'>aw.workspace.code-workspace</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id50bf3536f87d'>{ "folders": [ { "uri": "vscode-remote://wsl+ubuntu/var/sitesUbuntu/www.ancientwarmth.com" }, { "uri": "vscode-remote://wsl+ubuntu/var/work/django/django" }, { "uri": "vscode-remote://wsl+ubuntu/var/work/django/oscar" }, { "uri": "vscode-remote://wsl+ubuntu/var/work/ancientWarmth/ancientWarmth" }, { <span class="bg_yellow">"path": "../../var/work/django/main"</span> } ], "remoteAuthority": "wsl+Ubuntu", "settings": { "liveServer.settings.multiRootWorkspaceName": "www.mslinn.com", "python.pythonPath": "/var/work/django/oscar/bin/python", "git.ignoreLimitWarning": true, "sqltools.connections": [ { "previewLimit": 50, "server": "localhost", "port": 5432, "driver": "PostgreSQL", "name": "Ancient Warmth on Camille", "database": "ancient_warmth", "username": "postgres", "password": "hithere" } ] } }</pre> <p> All I need to do is change this entry: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id41f113a490d5'><span class="bg_yellow">"path": "../..</span>/var/work/django/main"</pre> <p>To:</p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1f270fad0308'><span class="bg_yellow">"uri": "vscode-remote://wsl+ubuntu</span>/var/var/work/django/main"</pre> <div class="right" style="font-size: 3em;">&#128513;</div> <p> The modified entry will cause VSCode to launch the project from WSL, instead of Windows. </p> Replicating a Git Directory Tree 2021-04-10T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/10/git-tree <p> Whenever I set up an operating system for one of my computers one of the tedious tasks that must be performed is to replicate the git repositories. </p> <p> It is a bad idea to attempt to copy an entire git repository between computers, because the <code>.git</code> directories within them can quite large. So large, in fact, that it might much more time to copy than re-cloning. I think the reason is that copying the entire git repo actually means copying the same information twice: first the <code>.git</code> hidden directory, complete with all the history for the project, and then again for the files in the currently checked out branch. Git repos store the entire development history of the project in their <code>.git</code> directories, so they are often much larger than the actual code that is checked out at any given time. </p> <p> I have several trees of git repositories, grouped into subdirectories. Here is a sanitized depiction of one of my git directory trees: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id171dc0c573be'>├── cadenzaHome │   ├── cadenzaAssets │   ├── cadenzaCode │   │   ├── cadenzaClient │   │   ├── cadenzaCourseCode │   │   ├── cadenzaDependencies │   │   ├── cadenzaLibs │   │   ├── cadenzaServer │   │   ├── cadenzaServerNext │   │   └── cadenzaSupport │   ├── cadenzaCreative │   │   └── cadenzaCreativeTemplates │   ├── cadenzaCreativeBackup │   └── cadenzaCurriculum ├── django │   ├── django │   ├── django-oscar │   ├── frobshop │   ├── main │   └── oscar ├── jekyll │   ├── jekyllTemplate │   └── jekyll-flexible-include-plugin</pre> <p> Some git repos are forks, and I defined <code>upstream</code> git remotes for them, in addition to the usual <code>origin</code> remote. </p> <p> This morning I found myself facing the boring task of doing this manually once again. Instead, I wrote this script, which scans a git directory tree and writes out a script that clones the repos in the tree, and adds <code>upstream</code> remotes as required. Directories containing a file called <code>.ignore</code> are ignored. </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/bin/gitUrls" download="gitUrls" title="Click on the file name to download the file">gitUrls</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id57459d956203"><button class='copyBtn' data-clipboard-target='#id57459d956203' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash function help &#123; printf "$1Replicates tree of git repos " exit 1 &#125; function doOne &#123; cd "$CLONE_DIR" > /dev/null PROJECT_DIR="$( basename "$CLONE_DIR" )" # Might have been renamed after cloning # echo "CLONE_DIR: $CLONE_DIR" # echo "PROJECT_DIR: $PROJECT_DIR" ORIGIN_URL="$( git config --local remote.origin.url )" CLONE_DIR_PARENT="$( realpath "$CLONE_DIR/.." )" echo "mkdir -p '$CLONE_DIR_PARENT'" echo "pushd '$CLONE_DIR_PARENT' > /dev/null" echo "git clone '$ORIGIN_URL'" UPSTREAM_URL="$( git config --local remote.upstream.url )" if [ "$UPSTREAM_URL" ]; then if [ "$ORIGIN_URL" != "no_push" ]; then echo "cd \"$PROJECT_DIR\"" echo "git remote add upstream '$UPSTREAM_URL'" fi fi echo "popd > /dev/null" GIT_DIR_NAME="$( basename "$PWD" )" if [ "$GIT_DIR_NAME" != "$PROJECT_DIR" ]; then echo "# Git project directory was renamed, renaming this copy to match original directory structure" echo "mv \"$GIT_DIR_NAME\" \"$PROJECT_DIR\"" fi echo &#125; if [ -z "$1" ]; then help "Error: Please specify the subdirectory to traverse.\n\n"; fi BASE="$1" DIRS="$( find $BASE -type d \( -execdir test -e &#123;&#125;/.ignore \; -prune \) -o \( -execdir test -d &#123;&#125;/.git \; -prune -print \) )" for DIR in $DIRS; do export CLONE_DIR="$( realpath "$DIR" )" doOne done </pre> <p> Here is the output generated for the above directory tree: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc1f039363092'><span class='unselectable'>$ </span>gitUrls $work <span class='unselectable'>mkdir -p '/var/work/cadenzaHome/cadenzaCreative' pushd '/var/work/cadenzaHome/cadenzaCreative' > /dev/null git clone 'git@github.com:mslinn/cadenzaCreativeTemplates.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome' pushd '/var/work/cadenzaHome' > /dev/null git clone 'git@github.com:mslinn/cadenzaAssets.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaSupport' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaSupport' > /dev/null git clone 'git@github.com:mslinn/dottyTemplate.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' > /dev/null git clone 'git@github.com:mslinn/scalacourses-play-utils.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' > /dev/null git clone 'git@github.com:mslinn/scalacourses-utils.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' > /dev/null git clone 'git@github.com:mslinn/scalacourses-slick-utils.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode' pushd '/var/work/cadenzaHome/cadenzaCode' > /dev/null git clone 'git@bitbucket.org:mslinn/cadenzaserver.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaCourseCode/ScalaCourses.com/group_scalaCore' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaCourseCode/ScalaCourses.com/group_scalaCore' > /dev/null git clone 'ssh://git@bitbucket.org/mslinn/course_scala_intro_code.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaCourseCode/ScalaCourses.com/group_scalaCore' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaCourseCode/ScalaCourses.com/group_scalaCore' > /dev/null git clone 'git@bitbucket.org:mslinn/course_scala_intermediate_code.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode' pushd '/var/work/cadenzaHome/cadenzaCode' > /dev/null git clone 'git@github.com:mslinn/cadenzaClient.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome' pushd '/var/work/cadenzaHome' > /dev/null git clone 'git@github.com:mslinn/cadenzaCurriculum.git' popd > /dev/null mkdir -p '/var/work' pushd '/var/work' > /dev/null git clone 'git@github.com:mslinn/jekyllTemplate.git' popd > /dev/null mkdir -p '/var/work/django' pushd '/var/work/django' > /dev/null git clone 'git@github.com:mslinn/django-oscar.git' cd "django-oscar" git remote add upstream 'git@github.com:django-oscar/django-oscar.git' popd > /dev/null mkdir -p '/var/work/django' pushd '/var/work/django' > /dev/null git clone 'git@github.com:mslinn/frobshop.git' popd > /dev/null mkdir -p '/var/work/django' pushd '/var/work/django' > /dev/null git clone 'git@github.com:mslinn/django.git' cd "django" git remote add upstream 'git@github.com:django/django.git' popd > /dev/null mkdir -p '/var/work/jekyll' pushd '/var/work/jekyll' > /dev/null git clone 'git@github.com:mslinn/jekyll-flexible-include-plugin.git' cd "jekyll-flexible-include-plugin" git remote add upstream 'https://idiomdrottning.org/jekyll-include-absolute-plugin' popd > /dev/null mkdir -p '/var/work/jekyll' pushd '/var/work/jekyll' > /dev/null git clone 'git@github.com:mslinn/jekyllTemplate.git' popd > /dev/null </span></pre> <div class="right" style="font-size: 3em;">&#128513;</div> <p> Now all I had to do was paste the above bash commands into a terminal on the new system, and a short time later the git repositories were set up the way I needed them. </p> A Python Virtual Environment For Every Project 2021-04-09T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/09/python-venvs <p> Python virtual environments are cheap to make and use. I have adopted the habit of making a Python virtual environment (<i>venv</i>) for each significant Python project, plus a default venv for trivial Python work. </p> <p> Dedicating a venv for each Python project means that dependencies for any given Python project do not impact the dependencies for any other Python projects. Things just work better. </p> <editor-fold free> <h2 id="free">VEnvs are Nearly Free</h2> <p> The cost of a venv is virtually free. This is because by default, the executable images are linked, so they do not require much storage space. The <code>ls</code> command below shows that the <code>python</code> program in the <code>aw</code> venv is linked to <code>/usr/bin/python3.8</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id93d04005880a'><button class='copyBtn' data-clipboard-target='#id93d04005880a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls -go ~/venv/aw/bin/python <span class='unselectable'>lrwxrwxrwx 1 18 Apr 9 06:01 <span class="bg_yellow">/home/mslinn/venv/aw/bin/python -> /usr/bin/python3.8</span> </span></pre> </editor-fold> <editor-fold std> <h2 id="standard">Standard Procedure For Creating a VEnv</h2> <p> I name each venv the same as my python project. My projects are stored under the directory pointed to by <code>$work</code>. </p> <p> My standard procedure when making a Python project called <code>$work/blah</code> is to also create a venv for it at <code>~/venv/blah</code>. A bash alias could be defined called <code>blah</code> that activates the venv and <code>cd</code>s into the project directory: </p> <div class='codeLabel unselectable' data-lt-active='false'>~/.bash_aliases</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfde90a9689b5'><button class='copyBtn' data-clipboard-target='#idfde90a9689b5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>alias blah="source ~/venv/blah/bin/activate; cd $work/blah"</pre> <div class="right" style="font-size: 3em;">&#128513;</div> <p> Now you could type <code>blah</code> at a shell prompt and you would be working on that project. Boom! </p> </editor-fold> <editor-fold create_script> <h2 id="create_script" class="clear">Script For Creating a VEnv</h2> <p> Here is a bash script that creates the venv and changes <code>~/.bashrc</code> and <code>~/.bash_aliases</code> for you. It assumes that you keep your projects under <code>$work</code>. </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/bin/newVenv" download="newVenv" title="Click on the file name to download the file">newVenv</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id2a0685d1ab88"><button class='copyBtn' data-clipboard-target='#id2a0685d1ab88' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash function help &#123; echo -e "$1$(basename $0) - Create a Python virtual environment with a given name. Usage: $(basename $0) venv_name The new virtual environment will be created under ~/venv/. If a project directory called \$work/venv_name exists before this script runs, then a bash alias is created named after the venv." exit 1 &#125; if [ -z `which virtualenv` ]; then sudo apt install virtualenv; fi if [ -z "$1" ]; then help "Please specify a name for the virtual environment.\n\n"; fi if [ "$1" == -h ]; then help; fi VENV="$1" shift mkdir -p "$HOME/venv" cd "$HOME/venv" virtualenv "$VENV" DIR="$HOME/venv/$VENV" echo "source $DIR/bin/activate" >> $HOME/.bashrc echo echo "Activation for "$VENV" in future shells was appended to $HOME/.bashrc" echo "To activate the "$VENV" venv in this shell right now, type: source ~/venv/$VENV/bin/activate" if [ "$work" ] && [ -d "$work/$VENV" ]; then echo "alias $VENV='source $DIR/bin/activate; cd $work/$VENV'" >> $HOME/.bash_aliases echo "An alias called $VENV for future shells was appended to $HOME/.bash_aliases" echo "To define the alias in this shell right now, type: alias $VENV='source $DIR/bin/activate; cd $work/$VENV'" else echo "To define an alias, type something like this: alias $VENV=\"source $DIR/bin/activate; cd $work/$VENV\"" fi </pre> <p> This is the help message for the script: </p> <div class='codeLabel unselectable' data-lt-active='false'>newVenv help message</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id87d55e3989c8'><button class='copyBtn' data-clipboard-target='#id87d55e3989c8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>newVenv -h <span class='unselectable'>newVenv - Create a Python virtual environment with a given name. Usage: newVenv venv_name The new virtual environment will be created under ~/venv/. If a project directory called $work/venv_name exists before this script runs, then a bash alias is created named after the venv. </span></pre> <p> Let's use the script to create a venv called <code>aw</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>newVenv help message</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida3cb2441303c'><button class='copyBtn' data-clipboard-target='#ida3cb2441303c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>newVenv aw <span class='unselectable'>created virtual environment CPython3.8.6.final.0-64 in 528ms creator CPython3Posix(dest=/home/mslinn/venv/aw, clear=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/mslinn/.local/share/virtualenv) added seed packages: pip==20.1.1, pkg_resources==0.0.0, setuptools==44.0.0, wheel==0.34.2 activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator </span></pre> </editor-fold> <editor-fold use> <h2 id="use">Script for Using a VEnv</h2> <p> Here is a script that can display the available Python virtual environments, and optionally activates one them. It does not use bash aliases. </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/bin/use" download="use" title="Click on the file name to download the file">use</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id4ddc15d79cea"><button class='copyBtn' data-clipboard-target='#id4ddc15d79cea' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash function help &#123; echo "Usage:" for f in $HOME/venv/*; do if [ -d "$f" ]; then echo " . $(basename $0) $(basename $f)"; fi done return 2 &#125; unset PV if [ "$1" == -h ]; then help elif [ "$1" ]; then PV="$1" else PV="default" fi if [ "$PV" ]; then DIR="$HOME/venv/$PV" if [ ! -d "$DIR" ]; then echo "Error: $DIR does not exist." return 1 fi if [ ! -f "$DIR/bin/python" ]; then echo "Error: No Python virtual environment is installed in $DIR" return 1 fi echo "Setting Python virtual environment to $DIR" source "$DIR/bin/activate" fi </pre> <p> Here are examples of using the script to change virtual environments: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida980212da9da'><span class='unselectable'>$ </span>. use -h <span class='unselectable'>Usage: . bash aw . bash default </span> <span class='unselectable'>$ </span>. use <span class='unselectable'>Setting Python virtual environment to /home/mslinn/venv/default </span> <span class='unselectable'>(default) $ </span>. use aw <span class='unselectable'>Setting Python virtual environment to /home/mslinn/venv/aw </span> <span class='unselectable'>(aw) $ </span></pre> <p> Notice that the last command above changed the shell prompt, in that <code>(aw)</code> was prepended to the normal prompt. To cause all future shells to use this virtual environment by default, the script adds a line to <code>~/.bashrc</code> that looks like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idca1d890e3706'><button class='copyBtn' data-clipboard-target='#idca1d890e3706' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo "source ~/venv/aw/bin/activate" &gt;&gt; ~/.bashrc</pre> <p> At this point the virtual environment just contained executable images for Python. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfc21284bb330'><span class='unselectable'>$ </span>ls ~/venv/aw/** <span class='unselectable'>~/venv/aw/pyvenv.cfg ~/venv/aw/bin: activate activate.ps1 chardetect distro easy_install pip pip3.8 python3.8 wheel3 activate.csh activate.xsh chardetect-3.8 distro-3.8 easy_install-3.8 pip-3.8 python wheel activate.fish activate_this.py chardetect3 distro3 easy_install3 pip3 python3 wheel-3.8 ~/venv/aw/lib: python3.8 </span></pre> </editor-fold> <editor-fold deactivate> <h2 id="deactivate">Deactivate a VEnv</h2> <p> Stop using venvs with `deactivate`: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb24b4ef61f4a'><span class='unselectable'>(aw) $ </span>deactivate <span class='unselectable'>$ </span></pre> </editor-fold> <editor-fold alias2> <h2 id="alias2">Activate With an Alias</h2> <p> Once again we can use a bash alias, this time to invoke the <code>use</code> script. We can call the alias <code>use</code>, because bash aliases have precedence over bash scripts. This alias removes the need to type <code>.</code> or <code>source</code> before the script name (which you know is <code>use</code>, if you have been following along). </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4f70483ccf5e'><span class='unselectable'>$ </span>alias use="source use" <span class='unselectable'>$ </span>use <span class='unselectable'>(default) $ </span>use aw <span class='unselectable'>(aw) $ </span></pre> <p> You can add the alias to <code>bash_aliases</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7860bd5be1db'><span class='unselectable'>$ </span>echo 'alias use="source use"' >> ~/.bash_aliases</pre> <div style="font-size: 3em;">&#128513;</div> </editor-fold> <editor-fold virt> <h2 id="virt">Directory-Locked Python Virtualization</h2> <p> After setting up a Python virtual environment, a quick examination of the <code>pip</code> script shows that it is hard-coded to the directory that it was made for: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id381f3665f603'><button class='copyBtn' data-clipboard-target='#id381f3665f603' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>head -n 1 ~/venv/aw/bin/pip <span class='unselectable'><span class="bg_yellow">#!/home/mslinn/venv/aw/bin/python</span> </span></pre> <p> For virtualized environments, such as Docker, this means that a Python virtual environment created without Docker can only be used within a Docker image if the path to it is the same from within the Docker image as when it was created. </p> </editor-fold> <h2 id="further">For Further Reading</h2> <p> <a href='https://mitelman.engineering/blog/python-best-practice/automating-python-best-practices-for-a-new-project/' target='_blank' rel='nofollow'>Python Best Practices for a New Project in 2021</a> </p> <editor-fold summary> <h2 id="summary">Summary</h2> <ul> <li>Demonstrated how to make an alias for working with Python virtual environments (<i>venvs</i>) that are coupled with Python projects.</li> <li>The <code>newVenv</code> bash script was demonstrated for making new Python virtual environments.</li> <li>The <code>use</code> bash source script was demonstrated for activating a venv.</li> <li>Deactivating the current venv was demonstrated using the <code>deactivate</code> command, provided with every venv.</li> <li>The <code>use</code> alias for <code>source use</code> was demonstrated for more conveniently selecting a venv.</li> <li>Locked directories mean that Python virtual environments should normally only be created in the same environment they are intended to be used.</li> </ul> </editor-fold> Escaping HTML on Clipboard From a Windows Hot Key via WSL 2021-04-03T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/03/escape-html-clipboard <p> I frequently show HTML source code when I write. That HTML must be escaped prior to displaying it on a web page. </p> <h2 id="bash_script">Script to Apply HTML Escape to Clipboard</h2> <p> This bash script applies an HTML escape conversion to the contents of the system clipboard. If you have WSL on your machine, you could store it on the WSL file system, for example in <code>~/.local/bin/escapeHtml</code>. <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/bin/escapeHtml" download="escapeHtml" title="Click on the file name to download the file">escapeHtml</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id5bcdbd2e55d6"><button class='copyBtn' data-clipboard-target='#id5bcdbd2e55d6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash # SPDX-License-Identifier: Apache-2.0 function help &#123; echo "$(basename $0) - Escapes HTML with entities. Reads from STDIN or pipe, or converts the clipboard. Result is copied to the clipboard. " exit 1 &#125; function filesAreLinked &#123; "$1" -ef "$2" &#125; function checkDependencies &#123; if [ -z `which recode` ]; then yes | sudo apt install recode; fi # See https://github.com/sindresorhus/clipboard-cli if [ -z `which clipboard` ]; then if [ "$( filesAreLinked /bin/npm /usr/local/lib/node_modules/npm/bin/npm-cli.js )" ]; then # No nodejs venv sudo -H npm install --global clipboard-cli else # nodejs venv npm install --global clipboard-cli fi fi &#125; if [ "$1" == -h ]; then help; fi checkDependencies if [ -t 0 ]; then # Not reading from a terminal HTML="$( clipboard )" else # Reading from stdin or pipe # See https://stackoverflow.com/a/32365596/553865 HTML=$(cat; echo x) HTML=$&#123;HTML%x&#125; # Strip the trailing x fi RESULT="$( recode utf8..html &lt;&lt;&lt; "$HTML" )" echo "$RESULT" | sed "s/&#13;//g" | sed "s/'/\&#39;/g" | clipboard echo "$( echo "$RESULT" | wc -l ) lines have been placed on the clipboard." </pre> </p> <h2 id="use">Using the Script</h2> <ol> <li>Select some text in any document or anywhere that text can be selected.</li> <li>Run <code>escapeHtml</code> on the same machine. If you have Windows with WSL you can run the script there, or run it in native Windows, does not matter.</li> <li>Paste escaped HTML into your target document.</li> </ol> <h2 id="trigger">Hot Key Trigger</h2> <p> Trigger the script with a hot key via your OS's facilities. This section just discusses how native Windows hot keys can be used to trigger this script running in WSL. </p> <ol> <li>Right-click in a folder</li> <li>Select <b>New</b> / <b>Shortcut</b></li> <li> For <b>Type the location of the item</b>, type:<br><br> <div class='codeLabel unselectable' data-lt-active='false'>Type the location of the item</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb4da34f00443'><button class='copyBtn' data-clipboard-target='#idb4da34f00443' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>%windir%\System32\wsl.exe ~/.local/bin/escapeHtml/escapeHtml</pre> </li> <li>Click <b>Change icon</b> and select a retro icon for this shortcut.</li> <li>Click on <b>Apply</b>.</li> <li>Click on <b>Next</b>.</li> <li> When prompted for <b>Type a name for this shortcut</b>, save as <code>HTML Escape Clip to Clip</code>. </li> <li>Click on <b>Finish</b>.</li> <li>Right-click on the new shortcut and click in <b>Shortcut key</b></li> <li>I used <kbd>Ctrl</kbd>+<kbd>Alt</kbd>+<kbd>A</kbd>.</li> <li>Click on <b>OK</b>.</li> </ol> <p> You can now quickly copy HTML from any source to the clipboard, apply an HTML escape conversion to the clipboard contents, and then paste the escaped HTML into an editor. </p> <div style="font-size: 3em;">&#128513;</div> Microsoft Visual Studio Code Notes 2021-03-22T00:00:00-04:00 https://mslinn.github.io/blog/2021/03/22/vscode-notes <h2 id="keys">Useful Default Key Bindings</h2> <dl> <dt><kbd>Ctrl</kbd>+<kbd>K</kbd> &nbsp; <kbd>Ctrl</kbd>+<kbd>S</kbd></dt> <dd> <b>Display / edit the <a href='https://code.visualstudio.com/docs/getstarted/keybindings' target='_blank' rel='nofollow'>Keyboard Shortcuts</a> definitions.</b><br> You can filter the keybindings by pressing <kbd>Alt</kbd>+<kbd>K</kbd> or clicking the icon of the little keyboard at the top right of the Keyboard Shortcut page, then press the keys that you want to see the key binding for. The keyboard icon starts keystroke recording mode. Recording mode is sticky; each time you revisit the Keyboard Shortcuts tab you can just press the keys you are interested in to see their bindings. Step by step: <ol> <li> When you press the <kbd>Ctrl</kbd> key you will see <code>"ctrl"</code> displayed, and recording mode continues to listen to what you type. Don't do this right now, but FYI, if you toggle keystroke recording mode now, and then remove the quotes around <code>"ctrl"</code>, you will see a sorted list of all the key chords bound to <kbd>Ctrl</kbd>. </li> <li> Next, when you add the <kbd>Shift</kbd> key to the key chord, you then see <code>"ctrl+shift"</code> displayed. Don't do this right now, but FYI, if you toggle keystroke recording mode now, and then remove the quotes, you will see a sorted list of all the key chords bound to <kbd>Ctrl</kbd>+<kbd>Shift</kbd>. </li> <li>Finally, adding the <kbd>=</kbd> key to the key chord shows all the commands bound to <kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>=</kbd>.</li> </ol> </dd> <dt><kbd>Ctrl</kbd>+<kbd>K</kbd>+<kbd>0</kbd> (zero)</dt> <dd>Completely fold the active editor contents.</dd> <dt><kbd>Ctrl</kbd>+<kbd>K</kbd>+<kbd>1</kbd> (one)</dt> <dd>Fold level 1 the active editor contents.</dd> <dt><kbd>Ctrl</kbd>+<kbd>K</kbd>+<kbd>2</kbd></dt> <dd>Fold level 2 the active editor contents.</dd> <dt><kbd>Ctrl</kbd>+<kbd>B</kbd></dt> <dd>Toggle side bar visibility.</dd> <dt><kbd>Ctrl</kbd>+<kbd>P</kbd></dt> <dd> Show names of recently opened tabs, which might contain files to edit, or might be VSCode settings, or VSCode key bindings, etc. Click on a tab name to open it. This key binding is bound to <b>Go to File</b>, which is slightly logical but a not a good descriptive name. </dd> <dt><kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>P</kbd></dt> <dd>Open the <a href='https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette' target='_blank' rel='nofollow'>Command Palette</a>.</dd> <dt><kbd>Ctrl</kbd>+<kbd>L</kbd> &nbsp; <kbd>G</kbd></dt> <dd> Open the active (currently edited) file on GitHub in the default web browser. Requires the <a href='https://marketplace.visualstudio.com/items?itemName=sysoev.vscode-open-in-github' target='_blank' rel='nofollow'>Open in GitHub</a> extension. </dd> <dt><kbd>Ctrl</kbd>+<kbd>,</kbd> (comma)</dt> <dd>Open the <a href='https://code.visualstudio.com/docs/getstarted/settings' target='_blank' rel='nofollow'>Settings</a> tab.</dd> <dt><kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>T</kbd></dt> <dd>Reopen the most recently closed editor tab.</dd> <dt><kbd>Alt</kbd>+<kbd>Shift</kbd>+<kbd>F</kbd></dt> <dd>Format the current file.</dd> </dl> <h2 id="reveal">Annoying Side Bar Auto Reveal</h2> <h3>Problem</h3> <p> When editing a file, if you <kbd>Ctrl</kbd>+<kbd>click</kbd> on a function, method or class name defined in a dependency, the dependency's folder will be expanded in the side bar. Some dependencies are deeply nested, which means that the side bar expands quite a lot. In order to close the folders in the side bar it is necessary to go all the way back to the top of that folder, which is annoying and wastes time. </p> <h3>Solution</h3> <p> In settings, look for <b>Explorer: Auto Reveal</b>, which controls whether the explorer should automatically reveal and select files when opening them. </p> <p> To do that, bring up settings with <kbd>Ctrl</kbd>+<kbd>,</kbd> (comma), and then type <code>reveal ex</code> into the filter. </p> <p> The default value is <code>true</code>. Change the value to <code>false</code>. </p> <h3>Bonus: Reveal Active File in Side Bar</h3> <p> To scroll to a file that you are editing in the list of files in the side bar, right-click on the file's tab, then select <b>Reveal in side bar</b>. </p> <p> Even better, define a keyboard shortcut to do this: </p> <ol> <li> Bring up the <b>Keyboard Shortcuts</b> definitions by typing <kbd>Ctrl</kbd>+<kbd>K</kbd> &nbsp; <kbd>Ctrl</kbd>+<kbd>S</kbd>. </li> <li>Type <code>reveal side</code> into the search bar.</li> <li>Double-click on <b>File: Reveal Active File in Side Bar</b>.</li> <li> For my desired key shortcut, I pressed <kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>Alt</kbd>+<kbd>R</kbd>, then pressed <kbd>Enter</kbd>. </li> </ol> <h2 id="plugins">Plugins</h2> <ul> <li><a href='https://mitelman.engineering/blog/python-best-practice/automating-python-best-practices-for-a-new-project/#code-analysis-with-flake8-linter' target='_blank' rel='nofollow'>Flake 8</a></li> </ul> Command-Line AWS Utilities 2021-03-22T00:00:00-04:00 https://mslinn.github.io/blog/2021/03/22/command-line-aws-utilities <editor-fold intro> <p> Here are some command-line utilities I have written for AWS. They are dependent on <a href='https://aws.amazon.com/cli/' target='_blank' rel='nofollow'>aws cli</a>. You can <a href='mslinn_aws.tar'>download all of these utilities</a> in tar format. Extract them into the current directory like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id51f2df64ccd1'><button class='copyBtn' data-clipboard-target='#id51f2df64ccd1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>tar xf mslinn_aws.tar</pre> </editor-fold> <editor-fold awsCfInvalidate> <h2 id="awsCfInvalidate"><span class="code">awsCfInvalidate</span></h2> <p> Given a CloudFront distribution ID, invalidate the distribution. </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/bin/awsCfInvalidate" download="awsCfInvalidate" title="Click on the file name to download the file">awsCfInvalidate</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id266322fb717f"><button class='copyBtn' data-clipboard-target='#id266322fb717f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash function help &#123; printf "$1$(basename $0) - Invalidate the CloudFront distribution for the given ID. If no distribution with the given ID exists, the empty string is returned and the return code is 2. A message is printed asynchronously to the console when the invalidation completes. Syntax: $(basename $0) distId Syntax: awsCfS3Dist www.mslinn.com | $(basename $0) " exit 1 &#125; function waitForInvalidation &#123; echo "Waiting for invalidation $2 to complete." aws cloudfront wait invalidation-completed \ --distribution-id "$1" \ --id "$2" echo "Invalidation $2 has completed." &#125; if [ "$1" == -h ]; then help; fi if [ "$1" ]; then DIST_ID="$1" shift elif [ ! -t 0 ]; then read -r DIST_ID fi if [ -z "$DIST_ID" ]; then help 'Error: No CloudFront distribution ID was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi JSON="$( aws cloudfront create-invalidation \ --distribution-id "$DIST_ID" \ --paths "/*" )" INVALIDATION_ID="$( jq -r .Invalidation.Id &lt;&lt;&lt; "$JSON" )" waitForInvalidation "$DIST_ID" "$INVALIDATION_ID" & </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1aa7ea526379'><button class='copyBtn' data-clipboard-target='#id1aa7ea526379' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfInvalidate E2P5S6OYKQNB6B <span class='unselectable'>Waiting for invalidation IFOPKECU4YYHD to complete. </span> <span class='unselectable'><i>... do other things ...</i> </span> <span class='unselectable'>$ </span><span class='unselectable'>Invalidation IFOPKECU4YYHD has completed. </span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3f448dd8f339'><button class='copyBtn' data-clipboard-target='#id3f448dd8f339' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfS3Dist www.mslinn.com | awsCfInvalidate <span class='unselectable'>Waiting for invalidation IFOPKECU4YYHD to complete. </span> <span class='unselectable'><i>... do other things ...</i> </span> <span class='unselectable'>$ </span><span class='unselectable'>Invalidation IFOPKECU4YYHD has completed. </span></pre> </editor-fold> <editor-fold awsCfS3Dist> <h2 id="awsCfS3Dist"><span class="code">awsCfS3Dist</span></h2> <p> Given an S3 bucket name, return the CloudFront distribution JSON. </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/bin/awsCfS3Dist" download="awsCfS3Dist" title="Click on the file name to download the file">awsCfS3Dist</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id54dcc9eaa510"><button class='copyBtn' data-clipboard-target='#id54dcc9eaa510' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash function help &#123; printf "$1$(basename $0) - Obtain the CloudFront distribution JSON for an S3 bucket. If no S3 bucket with the given name exists, the empty string is returned and the return code is 2. Syntax: $(basename $0) bucketName Syntax: echo bucketName | $(basename $0) " exit 1 &#125; if [ "$1" == -h ]; then help; fi if [ "$1" ]; then BUCKET_NAME="$1" shift elif [ ! -t 0 ]; then read -r BUCKET_NAME fi if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi if [ "$( aws s3api head-bucket --bucket $BUCKET_NAME 2> >(grep -i 'Not Found') )" ]; then >&2 echo "Error: Bucket $BUCKET_NAME does not exist." exit 2 fi DIST_ID="$( awsCfS3DistId "$BUCKET_NAME" )" if [ -z "$DIST_ID" ]; then exit 2; fi aws cloudfront get-distribution-config --id "$DIST_ID" </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id51de0bff09af'><button class='copyBtn' data-clipboard-target='#id51de0bff09af' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfS3Dist www.mslinn.com <span class='unselectable'>{ "ETag": "E1DIZUSLMOLXKP", "DistributionConfig": { "CallerReference": "1454487160038", "Aliases": { "Quantity": 2, "Items": [ "www.mslinn.com", "mslinn.com" ] }, "DefaultRootObject": "index.html", "Origins": { "Quantity": 1, "Items": [ { "Id": "S3-www.mslinn.com", "DomainName": "www.mslinn.com.s3-website-us-east-1.amazonaws.com", "OriginPath": "", "CustomHeaders": { "Quantity": 0 }, "CustomOriginConfig": { "HTTPPort": 80, "HTTPSPort": 443, "OriginProtocolPolicy": "http-only", "OriginSslProtocols": { "Quantity": 3, "Items": [ "TLSv1", "TLSv1.1", "TLSv1.2" ] }, "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5 }, "ConnectionAttempts": 3, "ConnectionTimeout": 10 } ] }, "OriginGroups": { "Quantity": 0 }, "DefaultCacheBehavior": { "TargetOriginId": "S3-www.mslinn.com", "TrustedSigners": { "Enabled": false, "Quantity": 0 }, "ViewerProtocolPolicy": "redirect-to-https", "AllowedMethods": { "Quantity": 2, "Items": [ "HEAD", "GET" ], "CachedMethods": { "Quantity": 2, "Items": [ "HEAD", "GET" ] } }, "SmoothStreaming": false, "Compress": true, "LambdaFunctionAssociations": { "Quantity": 0 }, "FieldLevelEncryptionId": "", "CachePolicyId": "658327ea-f89d-4fab-a63d-7e88639e58f6" }, "CacheBehaviors": { "Quantity": 0 }, "CustomErrorResponses": { "Quantity": 2, "Items": [ { "ErrorCode": 403, "ResponsePagePath": "", "ResponseCode": "", "ErrorCachingMinTTL": 60 }, { "ErrorCode": 404, "ResponsePagePath": "", "ResponseCode": "", "ErrorCachingMinTTL": 60 } ] }, "Comment": "", "Logging": { "Enabled": false, "IncludeCookies": false, "Bucket": "", "Prefix": "" }, "PriceClass": "PriceClass_All", "Enabled": true, "ViewerCertificate": { "ACMCertificateArn": "arn:aws:acm:us-east-1:031372724784:certificate/2be42926-829c-4db9-be7d-a72e951256d4", "SSLSupportMethod": "sni-only", "MinimumProtocolVersion": "TLSv1", "Certificate": "arn:aws:acm:us-east-1:031372724784:certificate/2be42926-829c-4db9-be7d-a72e951256d4", "CertificateSource": "acm" }, "Restrictions": { "GeoRestriction": { "RestrictionType": "none", "Quantity": 0 } }, "WebACLId": "", "HttpVersion": "http1.1", "IsIPV6Enabled": false } } </span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc8ab8ecea7ca'><button class='copyBtn' data-clipboard-target='#idc8ab8ecea7ca' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo www.mslinn.com | awsCfS3Dist <span class='unselectable'>{ "ETag": "E1DIZUSLMOLXKP", "DistributionConfig": { "CallerReference": "1454487160038", "Aliases": { "Quantity": 2, "Items": [ "www.mslinn.com", "mslinn.com" ] }, "DefaultRootObject": "index.html", "Origins": { "Quantity": 1, "Items": [ { "Id": "S3-www.mslinn.com", "DomainName": "www.mslinn.com.s3-website-us-east-1.amazonaws.com", "OriginPath": "", "CustomHeaders": { "Quantity": 0 }, "CustomOriginConfig": { "HTTPPort": 80, "HTTPSPort": 443, "OriginProtocolPolicy": "http-only", "OriginSslProtocols": { "Quantity": 3, "Items": [ "TLSv1", "TLSv1.1", "TLSv1.2" ] }, "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5 }, "ConnectionAttempts": 3, "ConnectionTimeout": 10 } ] }, "OriginGroups": { "Quantity": 0 }, "DefaultCacheBehavior": { "TargetOriginId": "S3-www.mslinn.com", "TrustedSigners": { "Enabled": false, "Quantity": 0 }, "ViewerProtocolPolicy": "redirect-to-https", "AllowedMethods": { "Quantity": 2, "Items": [ "HEAD", "GET" ], "CachedMethods": { "Quantity": 2, "Items": [ "HEAD", "GET" ] } }, "SmoothStreaming": false, "Compress": true, "LambdaFunctionAssociations": { "Quantity": 0 }, "FieldLevelEncryptionId": "", "CachePolicyId": "658327ea-f89d-4fab-a63d-7e88639e58f6" }, "CacheBehaviors": { "Quantity": 0 }, "CustomErrorResponses": { "Quantity": 2, "Items": [ { "ErrorCode": 403, "ResponsePagePath": "", "ResponseCode": "", "ErrorCachingMinTTL": 60 }, { "ErrorCode": 404, "ResponsePagePath": "", "ResponseCode": "", "ErrorCachingMinTTL": 60 } ] }, "Comment": "", "Logging": { "Enabled": false, "IncludeCookies": false, "Bucket": "", "Prefix": "" }, "PriceClass": "PriceClass_All", "Enabled": true, "ViewerCertificate": { "ACMCertificateArn": "arn:aws:acm:us-east-1:031372724784:certificate/2be42926-829c-4db9-be7d-a72e951256d4", "SSLSupportMethod": "sni-only", "MinimumProtocolVersion": "TLSv1", "Certificate": "arn:aws:acm:us-east-1:031372724784:certificate/2be42926-829c-4db9-be7d-a72e951256d4", "CertificateSource": "acm" }, "Restrictions": { "GeoRestriction": { "RestrictionType": "none", "Quantity": 0 } }, "WebACLId": "", "HttpVersion": "http1.1", "IsIPV6Enabled": false } } </span></pre> </editor-fold> <editor-fold awsCfS3DistId> <h2 id="awsCfS3DistId"><span class="code">awsCfS3DistId</span></h2> <p> Given an S3 bucket name, return the CloudFront distribution ID. </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/bin/awsCfS3DistId" download="awsCfS3DistId" title="Click on the file name to download the file">awsCfS3DistId</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idc80c98250582"><button class='copyBtn' data-clipboard-target='#idc80c98250582' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash function help &#123; printf "$1$(basename $0) - Obtain the CloudFront distribution ID for an S3 bucket. If no S3 bucket with the given name exists, the empty string is returned and the return code is 2. Syntax: $(basename $0) bucketName Syntax: echo bucketName | $(basename $0) " exit 1 &#125; if [ "$1" == -h ]; then help; fi if [ "$1" ]; then BUCKET_NAME="$1" shift elif [ ! -t 0 ]; then read -r BUCKET_NAME fi if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi if [ "$( aws s3api head-bucket --bucket $BUCKET_NAME 2> >(grep -i 'Not Found') )" ]; then >&2 echo "Error: Bucket $BUCKET_NAME does not exist." exit 2 fi DIST_ID="$( aws cloudfront list-distributions \ --query "DistributionList.Items[*].&#123;id:Id,origin:Origins.Items[0].Id&#125;[?origin=='S3-$BUCKET_NAME'].id" \ --output text )" if [ -z "$DIST_ID" ]; then exit 2; fi echo "$DIST_ID" </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id240f286955bb'><button class='copyBtn' data-clipboard-target='#id240f286955bb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfS3DistId www.mslinn.com <span class='unselectable'>E2P5S6OYKQNB6B </span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd7e82d987787'><button class='copyBtn' data-clipboard-target='#idd7e82d987787' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo www.mslinn.com | awsCfS3DistId <span class='unselectable'>E2P5S6OYKQNB6B </span></pre> </editor-fold> <editor-fold awsCfS3MakeDist> <h2 id="awsCfS3MakeDist"><span class="code">awsCfS3MakeDist</span></h2> <p> Creates a CloudFront distribution for the given bucket name. Returns the new distribution's ID. </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/bin/awsCfS3MakeDist" download="awsCfS3MakeDist" title="Click on the file name to download the file">awsCfS3MakeDist</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id55772c5a5319"><button class='copyBtn' data-clipboard-target='#id55772c5a5319' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash function help &#123; printf "$1$(basename $0) - Make a new CloudFront distribution for the given S3 bucket name. Returns the new distribution's ID. Syntax: $(basename $0) bucketName Syntax: echo bucketName | $(basename $0) " exit 1 &#125; function doesDistributionExist &#123; DIST_ID="$( awsCfS3Dist "$BUCKET_NAME" )" if [ "$DIST_ID" ]; then echo true; fi &#125; function createDist &#123; read -r -d '' NEW_DIST_JSON &lt;&lt;EOF &#123; "CallerReference": "$BUCKET_NAME", "Aliases": &#123; "Quantity": 0 &#125;, "DefaultRootObject": "index.html", "Origins": &#123; "Quantity": 1, "Items": [ &#123; "Id": "$BUCKET_NAME", "DomainName": "$BUCKET_NAME.s3.amazonaws.com", "S3OriginConfig": &#123; "OriginAccessIdentity": "" &#125; &#125; ] &#125;, "DefaultCacheBehavior": &#123; "TargetOriginId": "$BUCKET_NAME", "ForwardedValues": &#123; "QueryString": true, "Cookies": &#123; "Forward": "none" &#125; &#125;, "TrustedSigners": &#123; "Enabled": false, "Quantity": 0 &#125;, "ViewerProtocolPolicy": "redirect-to-https", "MinTTL": 3600 &#125;, "CacheBehaviors": &#123; "Quantity": 0 &#125;, "Comment": "", "Logging": &#123; "Enabled": false, "IncludeCookies": true, "Bucket": "", "Prefix": "" &#125;, "PriceClass": "PriceClass_All", "Enabled": true &#125; EOF NEW_DIST_RESULT_JSON = "$( aws cloudfront create-distribution --distribution-config "$NEW_DIST_JSON" )" DISTRIBUTION_ID="$( jq -r '.Distribution.Id' &lt;&lt;&lt; "$NEW_DIST_RESULT_JSON" )" echo "$DISTRIBUTION_ID" &#125; if [ "$1" == -h ]; then help; fi if [ -t 0 ]; then if [ -z "$1" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi BUCKET_NAME="$1" shift else read -r BUCKET_NAME fi if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi if [ "$( aws s3api head-bucket --bucket $BUCKET_NAME 2> >(grep -i 'Not Found') )" ]; then >&2 echo "Error: Bucket $BUCKET_NAME does not exist." exit 2 fi if [ "$(doesDistributionExist)" ]; then >&2 echo "Error: a CloudFront distibution already exists for S3 bucket $BUCKET_NAME" exit 3 fi createDist </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3bff9d5c004f'><button class='copyBtn' data-clipboard-target='#id3bff9d5c004f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfS3MakeDist my_bucket <span class='unselectable'>E2P5S6OYKQNB6B </span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2fc44fdbe255'><button class='copyBtn' data-clipboard-target='#id2fc44fdbe255' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo my_bucket | awsCfS3MakeDist <span class='unselectable'>E2P5S6OYKQNB6B </span></pre> </editor-fold> <editor-fold awsS3Mb> <h2 id="awsS3Mb"><span class="code">awsS3Mb</span></h2> <p> Make a new S3 bucket with the given name in the default AWS region. If the <code>--public-read</code> option is provided, set the ACL to <code>public-read</code> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/bin/awsS3Mb" download="awsS3Mb" title="Click on the file name to download the file">awsS3Mb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id05e9cc701852"><button class='copyBtn' data-clipboard-target='#id05e9cc701852' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash function help &#123; printf "$1$(basename $0) - Make a new S3 bucket with the given name in the default AWS region. Syntax: $(basename $0) bucketName [OPTIONS] Syntax: echo bucketName | $(basename $0) [OPTIONS] Options are: --public-read Set bucket ACL to public-read " exit 1 &#125; if [ "$1" == -h ]; then help; fi if [ "$1" == "--public-read" ]; then ACL="public-read" shift fi if [ "$1" ]; then BUCKET_NAME="$1" shift elif [ ! -t 0 ]; then read -r BUCKET_NAME fi if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi aws s3 mb s3://$BUCKET_NAME if [ "$ACL" ]; then aws s3api put-bucket-acl --bucket $BUCKET_NAME --acl $ACL fi </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide0044437ae8f'><button class='copyBtn' data-clipboard-target='#ide0044437ae8f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsS3Mb my_bucket</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1700c629a9d9'><button class='copyBtn' data-clipboard-target='#id1700c629a9d9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsS3Mb my_bucket --public-read</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5cbf3501d3f0'><button class='copyBtn' data-clipboard-target='#id5cbf3501d3f0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo my_bucket | awsS3Mb --public-read</pre> </editor-fold> <editor-fold awsS3Website> <h2 id="awsS3Website"><span class="code">awsS3Website</span></h2> <p> Enable an S3 bucket to be a website. </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/bin/awsS3Website" download="awsS3Website" title="Click on the file name to download the file">awsS3Website</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id37e5b620e5b8"><button class='copyBtn' data-clipboard-target='#id37e5b620e5b8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash function help &#123; printf "$1$(basename $0) - Enable an S3 bucket to be a website. Syntax: $(basename $0) bucketName Syntax: echo bucketName | $(basename $0) " exit 1 &#125; if [ "$1" == -h ]; then help; fi if [ "$1" ]; then BUCKET_NAME="$1" shift elif [ ! -t 0 ]; then read -r BUCKET_NAME fi if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi aws s3 website s3://$BUCKET_NAME \ --index-document index.html \ --error-document error.html </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id28a359be7725'><button class='copyBtn' data-clipboard-target='#id28a359be7725' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsS3Website my_bucket</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf3590f2afcf5'><button class='copyBtn' data-clipboard-target='#idf3590f2afcf5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo my_bucket | awsS3Website</pre> </editor-fold> CORS on AWS S3 and Cloudfront 2021-03-21T00:00:00-04:00 https://mslinn.github.io/blog/2021/03/21/cors-aws <editor-fold intro> <p> This post shows how to enable CORS on an AWS S3 bucket with AWS CLI, then modify the bucket&rsquo;s CloudFront distribution. In preparing this blog post, I found that the <a href='https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html' target='_blank' rel='nofollow'>AWS S3 CORS documentation</a> needs to be read in conjunction with how <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/header-caching.html#header-caching-web-cors' target='_blank' rel='nofollow'>AWS CloudFront can be configured to handle CORS</a>. </p> <p> I used one origin for testing. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf6068d97de6b'><button class='copyBtn' data-clipboard-target='#idf6068d97de6b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ORIGIN=ancientwarmth.com <span class='unselectable'>$ </span>JSON_FILE=cors.json</pre> <p> The CORS configuration for the AWS S3 bucket will be stored in the file pointed to by <code>JSON_FILE</code>. </p> </editor-fold> <editor-fold defS3Cors> <h2 id="defS3Cors">Define the AWS S3 Bucket CORS Configuration</h2> <p> This configuration (in JSON format) contains 1 rule: </p> <ol> <li>Allow <code>GET</code> HTTP methods from anywhere.</li> </ol> <div class='codeLabel unselectable' data-lt-active='false'>cors.json</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide1325b3bdeb2'><button class='copyBtn' data-clipboard-target='#ide1325b3bdeb2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{ "CORSRules": [ { "AllowedHeaders": [], "AllowedMethods": [ "GET" ], "AllowedOrigins": [ "*" ], "ExposeHeaders": [] } ] }</pre> <p> You can read about CORS configuration in the <a href='https://docs.aws.amazon.com/AmazonS3/latest/userguide/ManageCorsUsing.html#cors-example-1' target='_blank' rel='nofollow'>AWS documentation</a>. </p> </editor-fold> <editor-fold setS3Cors> <h2 id="setS3Cors">Set the AWS S3 Bucket CORS Configuration</h2> <p> It is easy to set the CORS configuration for an AWS S3 bucket using AWS CLI&rsquo;s <a href='https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-cors.html' target='_blank' rel='nofollow'><code>aws s3api put-bucket-cors</code> subcommand</a>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id244649a54478'><button class='copyBtn' data-clipboard-target='#id244649a54478' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>BUCKET=assets.ancientwarmth.com <span class='unselectable'>$ </span>aws s3api put-bucket-cors \ --bucket $BUCKET \ --cors-configuration "file://$JSON_FILE"</pre> </editor-fold> <editor-fold testS3Cors> <h2 id="testS3Cors">Test the AWS S3 Bucket CORS Configuration</h2> <p> Now it is time to test the S3 bucket&rsquo;s CORS configuration using <code>curl</code>. I defined a bash function to peform the test to save typing. You can use it by first copy/pasting the code below into a shell prompt, then calling the function with the proper arguments, as shown. The function requires 3 arguments: the request origin, the URL of an asset in an AWS S3 bucket, and an HTTP method (which must be in UPPPER CASE). </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id84ffe6520c8f'><button class='copyBtn' data-clipboard-target='#id84ffe6520c8f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>function testCors { if [ -z "$1" ]; then echo "Error: No origin was provided"; exit 1; fi if [ -z "$2" ]; then echo "Error: No URL to test was provided"; exit 1; fi if [ "$3" ]; then METHOD="$3"; else METHOD=GET; fi curl -I -X OPTIONS \ --no-progress-meter \ -H "Origin: $1" \ -H "Access-Control-Request-Method: $METHOD" \ "$2" 2>&1 | \ grep --color=never 'Access-Control' }</pre> <p> The JSON file for testing CORS was <code><a href='https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/index.html#path-argument-type' target='_blank' rel='nofollow'>s3://</a>$BUCKET/testCors.json</code>: </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/testCors.json" download="testCors.json" title="Click on the file name to download the file">testCors.json</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idf28c5256981f"><button class='copyBtn' data-clipboard-target='#idf28c5256981f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>&#123; "key1": "value1", "key2": "value2" &#125; </pre> <p> We will know if CORS is set up properly by receiving a header containing <code>Access-Control-Allow-Origin: *</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbe14d0a866bf'><button class='copyBtn' data-clipboard-target='#idbe14d0a866bf' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>URL="https://s3.amazonaws.com/$BUCKET/testCors.json" <span class='unselectable'>$ </span>testCors $ORIGIN $URL GET <span class='unselectable'><span class="bg_yellow">Access-Control-Allow-Origin: *</span> Access-Control-Allow-Methods: GET Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method </span></pre> <p> The origin worked when the bucket is accessed via a <code>GET</code> method sent to its <code>s3.amazonaws.com</code> DNS alias (yay!). </p> <p> CORScanner (<a href="/blog/2021/03/20/cors.html#corscanner">discussed in a previous blog post</a>) reported no issues: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id623159c1f2a0'><button class='copyBtn' data-clipboard-target='#id623159c1f2a0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cors -u s3.amazonaws.com/assets.ancientwarmth.com/testCors.json <span class='unselectable'>Starting CORS scan... Finished CORS scanning... </span></pre> </editor-fold> <editor-fold cf> <h2 id="cf">CloudFront</h2> <p> I have not worked through the process of using AWS CLI to obtain a JSON object describing the distribution, and then changing some properties and writing it back. So until that happy day comes, here are 2 screen shots of the <a href='https://console.aws.amazon.com/cloudfront/home' target='_blank' rel='nofollow'>AWS CloudFront web console</a> showing the settings. The first screen shot shows the <b>Behaviors</b> tab of the top-level details of the <code>assets.ancientwarmth.com</code> CloudFront distribution. </p> <div style=""> <picture> <source srcset="/blog/images/aws/cfBehaviorCors0.webp" type="image/webp"> <source srcset="/blog/images/aws/cfBehaviorCors0.png" type="image/png"> <img src="/blog/images/aws/cfBehaviorCors0.png" title="CloudFront / Edit Distribution / Behaviors <br> About to click on <b>Edit</b> (default behavior)" class=" liImg2 rounded shadow" alt="CloudFront / Edit Distribution / Behaviors <br> About to click on <b>Edit</b> (default behavior)" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> CloudFront / Edit Distribution / Behaviors <br> About to click on <b>Edit</b> (default behavior) </figcaption> </figure> </div> <p> My application does not require users to upload anything, so everything in the S3 bucket is truly static. Thus I have no need to <code>PUT</code>, <code>POST</code> or <code>DELETE</code> HTTP methods for the AWS S3 content. I have not seen a good explanation of why enabling <code>OPTIONS</code> HTTP methods is necessary, but every person on Stack Overflow who got CORS to work with AWS S3 says this was necessary. With that in mind, I set the following for the next screen shot: </p> <ul> <li><b>Viewer Protocol Policy:</b> <code>Redirect HTTP to HTTPS</code></li> <li><b>Allowed HTTP Methods:</b> <code>GET, HEAD, OPTIONS</code></li> <li><b>Cached HTTP Methods:</b> Enable <code>OPTIONS</code></li> <li><b>Use a cache policy and origin request policy:</b> (default is Use legacy cache settings, which is usually undesirable)</li> <li><b>Cache Policy:</b> <code>Managed-CachingOptimized</code></li> <li><b>Origin Request Policy:</b> <code>Managed-CORS-S3Origin</code></li> </ul> <div style=""> <picture> <source srcset="/blog/images/aws/cfBehaviorCors1.webp" type="image/webp"> <source srcset="/blog/images/aws/cfBehaviorCors1.png" type="image/png"> <img src="/blog/images/aws/cfBehaviorCors1.png" title="Editing default CloudFront distribution behavior" class=" liImg2 rounded shadow" alt="Editing default CloudFront distribution behavior" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> Editing default CloudFront distribution behavior </figcaption> </figure> </div> <h3 id="cfManagedCorsS3OriginPolicy">Managed CORS S3 Origin Poligy</h3> <p> AWS CloudFront's <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/using-managed-origin-request-policies.html' target='_blank' rel='nofollow'>managed origin request policy</a> called <code>Managed-CORS-S3Origin</code> includes the headers that enable cross-origin resource sharing (CORS) requests when the origin is an Amazon S3 bucket. This policy's settings are: </p> <ul> <li><b>Query strings included in origin requests</b>: None</li> <li><b>Headers included in origin requests</b>: <ul> <li><code>Origin</code></li> <li><code>Access-Control-Request-Headers</code></li> <li><code>Access-Control-Request-Method</code></li> </ul> </li> <li><b>Cookies included in origin requests</b>: None</li> </ul> <div style=""> <picture> <source srcset="/blog/images/aws/cfManagedCorsS3OriginPolicy.webp" type="image/webp"> <source srcset="/blog/images/aws/cfManagedCorsS3OriginPolicy.png" type="image/png"> <img src="/blog/images/aws/cfManagedCorsS3OriginPolicy.png" class=" liImg2 rounded shadow" /> </picture> </div> </editor-fold> <editor-fold wait> <h2 id="wait">Wait or Invalidate</h2> <p> Whenever you make a configuration change to a CloudFront distribution, or the contents change, the distributed assets will not reflect those changes until the next CloudFront invalidation. Automatic invalidations take 20 minutes. You can invalidate manually for near-instant gratification. I use my <a href='command-line-aws-utilities.html#awsCfInvalidate#awsCfInvalidate'>AWS command-line utilities</a> to invalidate manually: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id26ad94fe6392'><button class='copyBtn' data-clipboard-target='#id26ad94fe6392' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfS3DistId $BUCKET | awsCfInvalidate</pre> <p> Now the grand finale: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5288c99c0531'><button class='copyBtn' data-clipboard-target='#id5288c99c0531' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>testCors $ORIGIN $URL GET <span class='unselectable'>Access-Control-Allow-Origin: * Access-Control-Allow-Methods: GET Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method </span></pre> <div class="right" style="font-size: 3em;">&#128513;</div> <p> The presence of the <code>Access-Control-Allow-Origin</code> header indicates that CORS allowed the data file to be transferred from the content server (AWS S3/CloudFront) to the origin server (the command line). </p> </editor-fold> Cross-Origin Resource Sharing (CORS) 2021-03-20T00:00:00-04:00 https://mslinn.github.io/blog/2021/03/20/cors <editor-fold intro> <p> Many have tried to explain CORS, but most have not provided a clear explanation. I am going to try, then I will refer to explanations by others, who also provide examples. </p> </editor-fold> <editor-fold origin> <h2 id="origin">Origin and Origin Server</h2> <p> A website is delivered to web browsers from an <i>origin server</i>, or <i>origin</i> for short. The origin server is principally responsible for generating web pages. </p> <p> An origin is a combination of 3 things: </p> <ol> <li>A scheme (<code>http</code>, <code>https</code>, etc.)</li> <li>A (sub)domain, for example <code>localhost</code>, <code>blah.com</code> or <code>assets.blah.com</code>.</li> <li>A port, for example 80, 443, 8000, etc.</li> </ol> <p> All three things must match in order for two URLs to be considered to be from the same origin. For example: </p> <table class="table"> <tr> <th>URL 1</th> <th>URL 2</th> <th>Same Origin?</th> </tr> <tr> <td><code>http://blah.com</code></td> <td><code>http<span class="bg_yellow">s</span>://blah.com</code></td> <th>No</th> </tr> <tr> <td><code>https://blah.com</code></td> <td><code>https://<span class="bg_yellow">assets.</span>blah.com</code></td> <th>No</th> </tr> <tr> <td><code>https://blah.com</code></td> <td><code>https://blah.com<span class="bg_yellow">/path/page.html</span></code></td> <th>Yes</th> </tr> </table> </editor-fold> <editor-fold contentServer> <h2 id="contentServer">Content Server</h2> <p> In this article, I use the term <i>content server</i> to refer to sources of online information other than the origin server. Resources referenced by a web page, such as images, JavaScript, CSS, and data might be provided by the origin server, or they might come from a content server. </p> <p> Because every server has by definition a different origin, content servers always have a different origin than the origin server. Static resources (resources that do not change) are often served by <i>content delivery networks</i> (CDNs), which are also content servers. </p> <p> The Cross-Origin Resource Sharing (CORS) standard controls if a web page can load resources from content servers. Content servers are in charge of their content; they decide which origin servers they wish to co-operate with. When CORS support is properly configured, content servers include HTTP headers into their responses that tell a web browser if those resources may be read by the web page being constructed. </p> <p> Data is a special type of resource. CORS restricts how data is exchanged between the web page delivered to the web browser from the origin server and content servers. In particular, JSON and XML data communicated to and from content servers requires CORS authorization. Furthermore, requests (from the web browser) that send JSON, XML and other data formats to content servers also require CORS authorization. </p> <div class="pullQuote">Content servers are in charge of their content; they decide which origin servers they wish to co-operate with.</div> </editor-fold> <editor-fold ctype> <h2 id="ctype">Content-Type Header</h2> <p> The <a href='https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type' target='_blank' rel='nofollow'><code>Content-Type</code> header</a> is used to indicate the <a href='https://developer.mozilla.org/en-US/docs/Glossary/MIME_type' target='_blank' rel='nofollow'><code>media type</code></a> of the resource. The old name <i>MIME type</i> has been replaced by <i>media type</i>. <a href='https://www.iana.org/assignments/media-types/media-types.xhtml' target='_blank' rel='nofollow'>Here is a list of media types.</a> </p> <p> Media types with names that start with <code>application</code> require CORS authentication if they are delivered from content servers, for example <code>application/json</code> and <code>application/javascript</code>. </p> <p> As well, a few media types with names that start with <code>text</code> require CORS authentication if they are delivered from content servers, for example <code>text/xml</code> and <code>text/xml-external-parsed-entity</code>. </p> </editor-fold> <editor-fold resources> <h2 id="resources">Further Reading</h2> <h3 id="Kosaka">Mariko Kosaka</h3> <p> Mariko Kosaka has written an easy-to-understand article describing CORS, and provides a simple but effective working Express website for demonstration. </p> <div class="quote"> The same-origin policy tells the browser to block cross-origin requests. When you want to get a public resource from a different origin, the resource-providing server needs to tell the browser &lsquo;This origin where the request is coming from can access my resource&rsquo;. The browser remembers that and allows cross-origin resource sharing. <br><br> <span style="font-style: normal"> &nbsp; &ndash; <a href='https://web.dev/cross-origin-resource-sharing/' target='_blank' rel='nofollow'>Mariko Kosaka</a></span> </div> </editor-fold> <editor-fold gilling> <h3 id="Gilling">Derric Gilling and MDN</h3> <p> Derric Gilling has written a more in-depth yet very approachable article describing CORS. I've paraphrased his quoting of the Mozilla Developer Network documentation into the following: </p> <div class="quote"> CORS is a security mechanism that allows a web page from one domain or Origin to access a resource with a different domain (a cross-domain request). CORS is a relaxation of the same-origin policy implemented in modern browsers. Without features like CORS, websites are restricted to accessing resources from the same origin through what is known as same-origin policy. <br><br> Any CORS request has to be preflighted if:<br> <ul> <li>It uses methods other than <code>GET</code>, <code>HEAD</code> or <code>POST</code>.</li> <li> If POST is used to send request data with a <code>Content-Type</code> other than <code>application/x-www-form-urlencoded</code>, <code>multipart/form-data</code>, or <code>text/plain</code>. Examples: <ul> <li> A <code>POST</code> request sends an XML payload to the server; this requires the <code>Content-Type</code> header is set either to <code>application/xml</code> or <code>text/xml</code>. </li> <li> A website makes an AJAX call that <code>POST</code>s JSON data to a REST API, this requires the <code>Content-Type</code> header is set to <code>application/json</code>. </li> </ul> </li> </ul> <span style="font-style: normal">&nbsp; &ndash; <a href='https://www.moesif.com/blog/technical/cors/Authoritative-Guide-to-CORS-Cross-Origin-Resource-Sharing-for-REST-APIs/#how-cors-works/' target='_blank' rel='nofollow'>Derric Gilling</a> <br> &nbsp; &ndash; <a href='https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#preflighted_requests' target='_blank' rel='nofollow'>Mozilla Developer Network</a> </span> </div> </editor-fold> <editor-fold preflight> <h3 id="preflight">Preflight Requests</h3> <p> CORS preflight requests effectively double the latency of user requests for <a href='https://developer.mozilla.org/en-US/docs/Glossary/CRUD' target='_blank' rel='nofollow'>CRUD actions</a>. Client-side and server-side caching can help reduce this overhead for many circumstances. In <a href='https://www.mslinn.com/blog/2021/04/14/serverless-ecommerce.html#cf' target='_blank' rel='nofollow'>another blog post</a> I discuss how to use a CDN with multiple origin servers to completely eliminate the need for preflight requests. </p> <p> For additional background, please see: </p> <ul> <li><a href='https://www.rehanvdm.com/serverless/cloudfront-reverse-proxy-api-gateway-to-prevent-cors/index.html' target='_blank' rel='nofollow'>CloudFront reverse proxy API Gateway to prevent CORS</a> by Rehan van der Merwe</li> <li><a href='https://httptoolkit.tech/blog/cache-your-cors/' target='_blank' rel='nofollow'>Cache your CORS, for performance & profit</a> by Tim Perry</li> </ul> </editor-fold> <editor-fold keycdn> <h3 id="KeyCDN">KeyCDN</h3> <p> KeyCDN has an even more in-depth yet still very approachable <a href='https://www.keycdn.com/support/cors' target='_blank' rel='nofollow'>article describing CORS</a>. </p> </editor-fold> <editor-fold corscanner> <h2 id="corscanner">CORScanner</h2> <p> <a href='https://github.com/chenjj/CORScanner' target='_blank' rel='nofollow'>CORScanner</a> is a popular tool for detecting CORS misconfiguration. It is a Python module that can be executed as a shell command. Install CORScanner like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id066cc416e445'><button class='copyBtn' data-clipboard-target='#id066cc416e445' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pip install cors</pre> <p> The above adds a new executable called <code>cors</code> in the same directory where your <code>python</code> command resides. <p> <p> The <code>cors</code> documentation <a href='https://www.merriam-webster.com/thesaurus/conflate#verb' target='_blank' rel='nofollow'>conflates</a> the words URL and origin. Everywhere the word <code>URL</code> appears in the documentation, the word <code>origin</code> should be assumed. </p> </editor-fold> <editor-fold scannEx> <h3 id="scannEx">Example: Check Domain</h3> <p>Use the <code>-u</code> option to specify an origin to test:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb01c440a62e3'><button class='copyBtn' data-clipboard-target='#idb01c440a62e3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cors -u api.github.com <span class='unselectable'>Starting CORS scan... Finished CORS scanning... </span></pre> <p> To enable more debug info, use the <code>-v</code> option more than once. We can see that specifying <code>https</code> restricts testing to that <code>scheme</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4ede65e4edc6'><button class='copyBtn' data-clipboard-target='#id4ede65e4edc6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cors -vv -u https://api.github.com <span class='unselectable'>Starting CORS scan... 2021-03-21 09:55:58 INFO Start checking reflect_origin for https://api.github.com 2021-03-21 09:55:58 INFO nothing found for {url: https://api.github.com, origin: https://evil.com, type: reflect_origin} 2021-03-21 09:55:58 INFO Start checking prefix_match for https://api.github.com 2021-03-21 09:55:58 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com.evil.com, type: prefix_match} 2021-03-21 09:55:58 INFO Start checking suffix_match for https://api.github.com 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://evilgithub.com, type: suffix_match} 2021-03-21 09:55:59 INFO Start checking trust_null for https://api.github.com 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: null, type: trust_null} 2021-03-21 09:55:59 INFO Start checking include_match for https://api.github.com 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://ithub.com, type: include_match} 2021-03-21 09:55:59 INFO Start checking not_escape_dot for https://api.github.com 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://api.githubacom, type: not_escape_dot} 2021-03-21 09:55:59 INFO Start checking custom_third_parties for https://api.github.com 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://whatever.github.io, type: custom_third_parties} 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: http://jsbin.com, type: custom_third_parties} 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://codepen.io, type: custom_third_parties} 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://jsfiddle.net, type: custom_third_parties} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: http://www.webdevout.net, type: custom_third_parties} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://repl.it, type: custom_third_parties} 2021-03-21 09:56:00 INFO Start checking special_characters_bypass for https://api.github.com 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com_.evil.com, type: special_characters_bypass} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com-.evil.com, type: special_characters_bypass} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&quot;.evil.com, type: special_characters_bypass} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com{.evil.com, type: special_characters_bypass} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com}.evil.com, type: special_characters_bypass} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com^.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%60.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com!.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com~.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com`.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com;.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com|.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&apos;.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com(.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com).evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com*.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com,.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com$.evil.com, type: special_characters_bypass} 2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com=.evil.com, type: special_characters_bypass} 2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%0b.evil.com, type: special_characters_bypass} 2021-03-21 09:56:03 INFO Start checking trust_any_subdomain for https://api.github.com 2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: https://evil.api.github.com, type: trust_any_subdomain} 2021-03-21 09:56:03 INFO Start checking https_trust_http for https://api.github.com 2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: http://api.github.com, type: https_trust_http} Finished CORS scanning... </span></pre> </editor-fold> <editor-fold scannEx2> <h3 id="scannEx2">Example: Check Origin</h3> <p> To check CORS misconfigurations of an origin: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id52059cd0450e'><button class='copyBtn' data-clipboard-target='#id52059cd0450e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cors -vvu https://api.github.com/users/mslinn/repos <span class='unselectable'>Starting CORS scan... 2021-03-21 10:08:49 INFO Start checking reflect_origin for https://api.github.com 2021-03-21 10:08:49 INFO nothing found for {url: https://api.github.com, origin: https://evil.com, type: reflect_origin} 2021-03-21 10:08:49 INFO Start checking prefix_match for https://api.github.com 2021-03-21 10:08:49 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com.evil.com, type: prefix_match} 2021-03-21 10:08:49 INFO Start checking suffix_match for https://api.github.com 2021-03-21 10:08:49 INFO nothing found for {url: https://api.github.com, origin: https://evilgithub.com, type: suffix_match} 2021-03-21 10:08:49 INFO Start checking trust_null for https://api.github.com 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: null, type: trust_null} 2021-03-21 10:08:50 INFO Start checking include_match for https://api.github.com 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://ithub.com, type: include_match} 2021-03-21 10:08:50 INFO Start checking not_escape_dot for https://api.github.com 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://api.githubacom, type: not_escape_dot} 2021-03-21 10:08:50 INFO Start checking custom_third_parties for https://api.github.com 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://whatever.github.io, type: custom_third_parties} 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: http://jsbin.com, type: custom_third_parties} 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://codepen.io, type: custom_third_parties} 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://jsfiddle.net, type: custom_third_parties} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: http://www.webdevout.net, type: custom_third_parties} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://repl.it, type: custom_third_parties} 2021-03-21 10:08:51 INFO Start checking special_characters_bypass for https://api.github.com 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com_.evil.com, type: special_characters_bypass} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com-.evil.com, type: special_characters_bypass} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&quot;.evil.com, type: special_characters_bypass} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com{.evil.com, type: special_characters_bypass} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com}.evil.com, type: special_characters_bypass} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com^.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%60.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com!.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com~.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com`.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com;.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com|.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&apos;.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com(.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com).evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com*.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com,.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com$.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com=.evil.com, type: special_characters_bypass} 2021-03-21 10:08:54 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:08:54 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%0b.evil.com, type: special_characters_bypass} 2021-03-21 10:08:54 INFO Start checking trust_any_subdomain for https://api.github.com 2021-03-21 10:08:54 INFO nothing found for {url: https://api.github.com, origin: https://evil.api.github.com, type: trust_any_subdomain} 2021-03-21 10:08:54 INFO Start checking https_trust_http for https://api.github.com 2021-03-21 10:08:54 INFO nothing found for {url: https://api.github.com, origin: http://api.github.com, type: https_trust_http} Finished CORS scanning... </span></pre> <p> If a <code>scheme</code> is not specified, then both <code>http</code> and <code>https</code> are tested: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1696d274c8d7'><button class='copyBtn' data-clipboard-target='#id1696d274c8d7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cors -vvu api.github.com/users/mslinn/repos <span class='unselectable'>Starting CORS scan... 2021-03-21 10:03:30 INFO Start checking reflect_origin for http://api.github.com 2021-03-21 10:03:30 INFO Start checking reflect_origin for https://api.github.com 2021-03-21 10:03:30 INFO nothing found for {url: https://api.github.com, origin: https://evil.com, type: reflect_origin}2021-03-21 10:03:30 INFO Start checking prefix_match for https://api.github.com 2021-03-21 10:03:30 INFO nothing found for {url: http://api.github.com, origin: http://evil.com, type: reflect_origin} 2021-03-21 10:03:30 INFO Start checking prefix_match for http://api.github.com 2021-03-21 10:03:30 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com.evil.com, type: prefix_match} 2021-03-21 10:03:30 INFO Start checking suffix_match for https://api.github.com 2021-03-21 10:03:30 INFO nothing found for {url: https://api.github.com, origin: https://evilgithub.com, type: suffix_match} 2021-03-21 10:03:30 INFO Start checking trust_null for https://api.github.com 2021-03-21 10:03:30 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com.evil.com, type: prefix_match} 2021-03-21 10:03:30 INFO Start checking suffix_match for http://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: null, type: trust_null} 2021-03-21 10:03:31 INFO Start checking include_match for https://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: http://api.github.com, origin: http://evilgithub.com, type: suffix_match} 2021-03-21 10:03:31 INFO Start checking trust_null for http://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://ithub.com, type: include_match}2021-03-21 10:03:31 INFO Start checking not_escape_dot for https://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://api.githubacom, type: not_escape_dot} 2021-03-21 10:03:31 INFO Start checking custom_third_parties for https://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: http://api.github.com, origin: null, type: trust_null} 2021-03-21 10:03:31 INFO Start checking include_match for http://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://whatever.github.io, type: custom_third_parties} 2021-03-21 10:03:31 INFO nothing found for {url: http://api.github.com, origin: http://ithub.com, type: include_match} 2021-03-21 10:03:31 INFO Start checking not_escape_dot for http://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: http://jsbin.com, type: custom_third_parties} 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://codepen.io, type: custom_third_parties} 2021-03-21 10:03:31 INFO nothing found for {url: http://api.github.com, origin: http://api.githubacom, type: not_escape_dot} 2021-03-21 10:03:31 INFO Start checking custom_third_parties for http://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://jsfiddle.net, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: https://whatever.github.io, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: http://www.webdevout.net, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://repl.it, type: custom_third_parties} 2021-03-21 10:03:32 INFO Start checking special_characters_bypass for https://api.github.com 2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: http://jsbin.com, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com_.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com-.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: https://codepen.io, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&quot;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com{.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: https://jsfiddle.net, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com}.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: http://www.webdevout.net, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com^.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: https://repl.it, type: custom_third_parties} 2021-03-21 10:03:33 INFO Start checking special_characters_bypass for http://api.github.com 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%60.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com!.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com_.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com~.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com`.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com-.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com&quot;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com|.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com{.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&qpos;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com(.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com}.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com).evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com*.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com,.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com^.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com$.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com=.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com%60.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com!.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%0b.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO Start checking trust_any_subdomain for https://api.github.com 2021-03-21 10:03:35 INFO nothing found for {url: https://api.github.com, origin: https://evil.api.github.com, type: trust_any_subdomain} 2021-03-21 10:03:35 INFO Start checking https_trust_http for https://api.github.com 2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com~.evil.com, type: special_characters_bypass} 2021-03-21 10:03:35 INFO nothing found for {url: https://api.github.com, origin: http://api.github.com, type: https_trust_http} 2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com`.evil.com, type: special_characters_bypass} 2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com|.evil.com, type: special_characters_bypass} 2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com&.evil.com, type: special_characters_bypass} 2021-03-21 10:03:36 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com&apos;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:36 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com(.evil.com, type: special_characters_bypass} 2021-03-21 10:03:36 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com).evil.com, type: special_characters_bypass} 2021-03-21 10:03:36 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com*.evil.com, type: special_characters_bypass} 2021-03-21 10:03:37 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com,.evil.com, type: special_characters_bypass} 2021-03-21 10:03:38 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com$.evil.com, type: special_characters_bypass} 2021-03-21 10:03:38 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com=.evil.com, type: special_characters_bypass} 2021-03-21 10:03:38 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:03:38 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com%0b.evil.com, type: special_characters_bypass} 2021-03-21 10:03:38 INFO Start checking trust_any_subdomain for http://api.github.com 2021-03-21 10:03:39 INFO nothing found for {url: http://api.github.com, origin: http://evil.api.github.com, type: trust_any_subdomain} Finished CORS scanning... </span></pre> </editor-fold> AWS S3 and CloudFront SSL 2021-03-19T00:00:00-04:00 https://mslinn.github.io/blog/2021/03/19/aws-ssl <style> .sslScenario { border: thin solid grey; border-radius: 4px; padding: 8px; } </style> <p> SSL certificates need to match the domain they are served from. </p> <p> AWS uses one of several SSL certificates, depending on the <a href='https://docs.aws.amazon.com/AmazonS3/latest/dev-retired/UsingBucket.html#access-bucket-intro' target='_blank' rel='nofollow'>domain that an asset is requested from</a>. </p> <ul> <li> AWS S3 applies an SSL certificate for <code>https</code> requests. The SSL certificate chosen depends on the bucket endpoint used: <code>s3.amazonaws.com</code>, <code>*.s3.amazonaws.com</code>, or <code>s3.<i>region</i>.amazonaws.com</code>. </li> <li> AWS CloudFront will apply your custom SSL certificate (for example, a wildcard certificate such as <code>*.ancientwarmth.com</code>) for <code>https</code> requests to the CNAME for that distribution, otherwise it will apply the wildcard SSL certificate for <code>*.cloudfront.net</code>. </li> </ul> <h2 id="examples">Example <span class="code">SSL</span> URLs</h2> <p> My AWS S3 bucket called <code>assets.ancientwarmth.com</code> is served via a CloudFront distribution with URL <code>d1bci9l8cjf24o.cloudfront.net</code> that applies a wildcard SSL certificate for <code>*.ancientwarmth.com</code> that I created using <a href='https://aws.amazon.com/certificate-manager/' target='_blank' rel='nofollow'>AWS Certificate Manager</a>. I defined a CNAME called <code>assets.ancientwarmth.com</code> for that same CloudFront distribution using Route 53. </p> <p> All of the following URLs can be used to access my content, providing the SSL certificate matches the requested domain. </p> <p class="sslScenario"> <b>URL:</b> <code>https://d1bci9l8cjf24o.cloudfront.net</code><br> <b>Origin Type:</b> CloudFront distribution<br> <b>SSL certificate origin:</b> <code>*.cloudfront.net</code><br> <b>Valid SSL certificate?</b> Yes. </p> <p class="sslScenario"> <b>URL:</b> <code>https://assets.ancientwarmth.com</code><br> <b>Origin Type:</b> CloudFront distribution<br> <b>SSL certificate origin:</b> <code>*.ancientwarmth.com</code><br> <b>Valid SSL certificate?</b> Yes. (I created this wildcard certificate using Route 53.) </p> <p class="sslScenario"> <b>S3 path-style URL:</b> <code>https://s3.us-east-1.amazonaws.com/assets.ancientwarmth.com</code><br> <b>Origin Type:</b> S3 bucket<br> <b>SSL certificate origin:</b> <code>s3.us-east-1.amazonaws.com</code><br> <b>Valid SSL certificate?</b> Yes. </p> <p class="sslScenario"> <b>S3 dot URL:</b> <code>https://assets.ancientwarmth.com.s3.amazonaws.com</code><br> <b>Origin Type:</b> S3 bucket<br> <b>SSL certificate origin:</b> <code>*.s3.amazonaws.com</code><br> <b>Valid SSL certificate?</b> No, does not match URL (wildcards only match one subdomain). </p> <p class="sslScenario"> <b>S3 dot Region URL:</b> <code>https://assets.ancientwarmth.com.s3.us-east-1.amazonaws.com</code><br> <b>Origin Type:</b> S3 bucket<br> <b>SSL certificate origin:</b> <code>s3.amazonaws.com</code><br> <b>Valid SSL certificate?</b> No, does not match URL. </p> <h2 id="curl">Testing with <span class="code">curl</span></h2> <p> <code>Curl</code> is often used to test SSL requests. In the following <code>curl</code> commands, the <a href='https://curl.se/docs/manpage.html#-I' target='_blank' rel='nofollow'><code>-I</code> option</a> just fetches the headers, and the <a href='https://curl.se/docs/manpage.html#-v' target='_blank' rel='nofollow'><code>-v</code> option</a> provides verbose output. You can see the SSL certificate negotation. </p> <p> Fetching an asset from a CloudFront distribution using the AWS <code>*.cloudfront.net</code> wildcard SSL certificate: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id789595ba9b59'><button class='copyBtn' data-clipboard-target='#id789595ba9b59' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -Iv \ https://d1bci9l8cjf24o.cloudfront.net/js/jquery.modal.min.js <span class='unselectable'>* Trying 52.85.149.22:443... * TCP_NODELAY set * Connected to d1bci9l8cjf24o.cloudfront.net (52.85.149.22) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=*.cloudfront.net * start date: Feb 22 00:00:00 2021 GMT * expire date: Feb 21 23:59:59 2022 GMT * subjectAltName: host "d1bci9l8cjf24o.cloudfront.net" matched cert's "*.cloudfront.net" * issuer: C=US; O=DigiCert Inc; CN=DigiCert Global CA G2 * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x5585dcfaf7e0) > HEAD /js/jquery.modal.min.js HTTP/2 > Host: d1bci9l8cjf24o.cloudfront.net > user-agent: curl/7.68.0 > accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS == 128)! < HTTP/2 200 HTTP/2 200 < content-type: application/javascript content-type: application/javascript < content-length: 4953 content-length: 4953 < date: Sat, 20 Mar 2021 14:11:34 GMT date: Sat, 20 Mar 2021 14:11:34 GMT < last-modified: Sat, 20 Mar 2021 03:14:08 GMT last-modified: Sat, 20 Mar 2021 03:14:08 GMT < etag: "c8f50397e0560719c62a35318f413e16" etag: "c8f50397e0560719c62a35318f413e16" < accept-ranges: bytes accept-ranges: bytes < server: AmazonS3 server: AmazonS3 < x-cache: Miss from cloudfront x-cache: Miss from cloudfront < via: 1.1 0712e4ad4264127dfcb76a114b130495.cloudfront.net (CloudFront) via: 1.1 0712e4ad4264127dfcb76a114b130495.cloudfront.net (CloudFront) < x-amz-cf-pop: IAD89-C3 x-amz-cf-pop: IAD89-C3 < x-amz-cf-id: hWrjwajqqkI9-rJnK1BSQqkX9DPXIlZJLfa28UaIeze7taBP5kqMNg== x-amz-cf-id: hWrjwajqqkI9-rJnK1BSQqkX9DPXIlZJLfa28UaIeze7taBP5kqMNg== < * Connection #0 to host d1bci9l8cjf24o.cloudfront.net left intact </span></pre> <p> Fetching an asset from a CloudFront distribution using my <code>*.ancientwarmth.com</code> wildcard SSL certificate: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf77cb2392fc5'><button class='copyBtn' data-clipboard-target='#idf77cb2392fc5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -Iv \ https://assets.ancientwarmth.com/js/jquery.modal.min.js <span class='unselectable'>modal.min.js> https://assets.ancientwarmth.com/js/jquery.modal.min.js * Trying 13.226.36.16:443... * TCP_NODELAY set * Connected to assets.ancientwarmth.com (13.226.36.16) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=*.ancientwarmth.com * start date: Mar 14 00:00:00 2021 GMT * expire date: Apr 12 23:59:59 2022 GMT * subjectAltName: host "assets.ancientwarmth.com" matched cert's "*.ancientwarmth.com" * issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x5599720157e0) > GET /js/jquery.modal.min.js HTTP/2 > Host: assets.ancientwarmth.com > user-agent: curl/7.68.0 > accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS == 128)! < HTTP/2 200 < content-type: application/javascript < content-length: 4953 < date: Sat, 20 Mar 2021 12:34:25 GMT < last-modified: Sat, 20 Mar 2021 03:14:08 GMT < etag: "c8f50397e0560719c62a35318f413e16" < accept-ranges: bytes < server: AmazonS3 < x-cache: Hit from cloudfront < via: 1.1 4e3df844337032b56b8434990b0f76ca.cloudfront.net (CloudFront) < x-amz-cf-pop: EWR53-C2 < x-amz-cf-id: 17Dxn6QqtK6JfkJwFnESVYsG-Cbzu6H-sOTWcGDpznGcpjIZbhJDRA== < age: 5195 < * Connection #0 to host assets.ancientwarmth.com left intact </span></pre> <p> Fetching an asset from an S3 bucket using an AWS SSL certificate for all S3 buckets in region <code>us-east-1</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb64b0d988507'><button class='copyBtn' data-clipboard-target='#idb64b0d988507' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -Iv \ https://s3.us-east-1.amazonaws.com/assets.ancientwarmth.com/js/jquery.modal.min.js <span class='unselectable'>* Trying 52.216.24.46:443... * TCP_NODELAY set * Connected to s3.us-east-1.amazonaws.com (52.216.24.46) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=s3.amazonaws.com * start date: Aug 4 00:00:00 2020 GMT * expire date: Aug 9 12:00:00 2021 GMT * subjectAltName: host "s3.us-east-1.amazonaws.com" matched cert's "s3.us-east-1.amazonaws.com" * issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2 * SSL certificate verify ok. > GET /assets.ancientwarmth.com/js/jquery.modal.min.js HTTP/1.1 > Host: s3.us-east-1.amazonaws.com > User-Agent: curl/7.68.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < x-amz-id-2: xIXUHy7YBpjZaF+cpGoSAwNvC5+NrmM5pmJM8nInI6weEkbht350xSPC9+yOBJrGs9GY0hn2V7Y= < x-amz-request-id: JM2K8HR109JNMMB1 < Date: Sat, 20 Mar 2021 14:03:56 GMT < Last-Modified: Sat, 20 Mar 2021 03:14:08 GMT < ETag: "c8f50397e0560719c62a35318f413e16" < Accept-Ranges: bytes < Content-Type: application/javascript < Content-Length: 4953 < Server: AmazonS3 < * Connection #0 to host s3.us-east-1.amazonaws.com left intact </span></pre> Pretty JSON Reduces Errors and Fatigue 2021-02-23T00:00:00-05:00 https://mslinn.github.io/blog/2021/02/23/pretty-json <p> I've been using <a href='https://stedolan.github.io/jq/' target='_blank' rel='nofollow'>jq</a> to format my JSON for years. It is easy to format a JSON document, just pass it through <code>jq</code> without any options or arguments. Notice, however, that a lot of vertical space is wasted using this formatting: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id70df7cdf5bbb'><button class='copyBtn' data-clipboard-target='#id70df7cdf5bbb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ jq < blog/colors.json { "colors": [ { "color": "black", "hex": "#000", "rgb": [ 0, 0, 0 ] }, { "color": "red", "hex": "#f00", "rgb": [ 255, 0, 0 ] }, { "color": "yellow", "hex": "#ff0", "rgb": [ 255, 255, 0 ] }, { "color": "green", "hex": "#0f0", "rgb": [ 0, 255, 0 ] }, { "color": "cyan", "hex": "#0ff", "rgb": [ 0, 255, 255 ] }, { "color": "blue", "hex": "#00f", "rgb": [ 0, 0, 255 ] }, { "color": "magenta", "hex": "#f0f", "rgb": [ 255, 0, 255 ] }, { "color": "white", "hex": "#fff", "rgb": [ 255, 255, 255 ] } ] }</pre> <p> After reading <a href='http://www.ohler.com/dev/pretty.html' target='_blank' rel='nofollow'>The Pretty JSON Revolution</a> I decided to try the program the article mentioned, <code>oj</code>. <code>oj</code> is a Go program. Here is how I compiled it on Ubuntu: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id25344ed8d089'><button class='copyBtn' data-clipboard-target='#id25344ed8d089' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install golang-go <span class='unselectable'>$ </span>go get github.com/ohler55/ojg <span class='unselectable'>$ </span>go get github.com/ohler55/ojg/cmd/oj</pre> <p> By default, compiled go projects are placed in the <code>~/go/bin/</code> directory. Here is how I added that directory the the <code>PATH</code>, and made an alias for invoking the program with the proper options for maximum prettiness: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd46e75eed910'><button class='copyBtn' data-clipboard-target='#idd46e75eed910' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo "$HOME/go/bin/:$PATH" >> ~/.bashrc <span class='unselectable'>$ </span>echo "alias pprint='oj -i 2 -s -p 80.3'" >> ~/.bash_aliases <span class='unselectable'>$ </span>source ~/.bashrc</pre> <p> Pretty-printing the JSON in <code>colors.json</code> with <code>oj</code> is easy: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3060cb7249c5'><button class='copyBtn' data-clipboard-target='#id3060cb7249c5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pprint colors.json <span class='unselectable'>{ "colors": [ {"color": "black", "hex": "#000", "rgb": [0, 0, 0]}, {"color": "red", "hex": "#f00", "rgb": [255, 0, 0]}, {"color": "yellow", "hex": "#ff0", "rgb": [255, 255, 0]}, {"color": "green", "hex": "#0f0", "rgb": [0, 255, 0]}, {"color": "cyan", "hex": "#0ff", "rgb": [0, 255, 255]}, {"color": "blue", "hex": "#00f", "rgb": [0, 0, 255]}, {"color": "magenta", "hex": "#f0f", "rgb": [255, 0, 255]}, {"color": "white", "hex": "#fff", "rgb": [255, 255, 255]} ] } </span></pre> <p> I like it! </p> <style>.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }</style><div class='embed-container'> <iframe title="YouTube video player" width="640" height="390" src="//www.youtube.com/embed/34wJt3pRY0w" frameborder="0" allowfullscreen></iframe></div> <p style="text-align: center"> <i>Give it to Mikey. He won't eat it. He hates everything!</i> </p> <p> I will continue to use <a href='https://stedolan.github.io/jq/manual/' target='_blank' rel='nofollow'><code>jq</code> for queries</a>, but I'll use <code>oj</code> for pretty-printing from now on. </p> JavaScript Named Arguments and Class Constructors 2021-02-11T00:00:00-05:00 https://mslinn.github.io/blog/2021/02/11/javascript-named-arguments <p> Named arguments make a program safe from errors caused by changes to method arguments. JavaScript named arguments can appear in any order. Default values for parameters allow an API to evolve gracefully without runtime errors. </p> <p> Building on the article entitled <a href='https://afontcu.medium.com/cool-javascript-9-named-arguments-functions-that-get-and-return-objects-337b6f8cfa07' target='_blank' rel='nofollow'>Cool JavaScript 9: Named arguments — Functions that get and return Objects</a>, this article shows how JavaScript class constructors can use named arguments, optionally define default values for parameters, and conveniently inflate new class instances from JSON. </p> <p> In this article I use Node.js for convenience, however the code shown will run in all modern web browsers. </p> <h2 id="stdArgs">JavaScript Class Definition Encapsulating Properties</h2> <p> Let&rsquo;s quickly review how to define a JavaScript class and instantiate an instance. Here is a simple JavaScript / ECMAScript 6 class that encapsulates two properties: <code>id</code> and <code>parts</code>. The constructor merely lists the names of the parameters, which happen to be the same as the names of the class properties. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id959eb7d55022'><button class='copyBtn' data-clipboard-target='#id959eb7d55022' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span> js <span class='unselectable'>Welcome to Node.js v12.18.2. Type ".help" for more information. > </span>class Ingredient { <span class='unselectable'>... </span> constructor(id, parts) { <span class='unselectable'>..... </span> this.id = id; <span class='unselectable'>..... </span> this.parts = parts; <span class='unselectable'>..... </span> } <span class='unselectable'>... </span>} <span class='unselectable'>undefined </span></pre> <p> New <code>Ingredient</code> instances can be created using this familiar syntax: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4b45acfb78a2'><button class='copyBtn' data-clipboard-target='#id4b45acfb78a2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredient = new Ingredient("123", 10); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredient <span class='unselectable'>Ingredient { id: '123', parts: 10 } </span></pre> <h2 id="lits">Object Literals</h2> <p> Object literals look like JSON objects, but without quotes around property names. For example, the following defines an object literal called <code>lit</code> with 2 properties, called <code>id</code> and <code>parts</code>, with values <code>"123"</code> and <code>10</code>, respectively. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd22d4b22bf0d'><button class='copyBtn' data-clipboard-target='#idd22d4b22bf0d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span> var lit = {id: "123", parts: 10}; <span class='unselectable'>undefined </span> <span class='unselectable'>$ </span>lit <span class='unselectable'>{ id: '123', parts: 10 } </span> <span class='unselectable'>> </span>lit.id <span class='unselectable'>'123' </span> <span class='unselectable'>> </span>lit.parts <span class='unselectable'>10 </span></pre> <h2 id="jsonArgs">Use Object Literals to Define Arguments</h2> <p> We can define a class similar to <code>Ingredient</code>, but with the arguments replaced by a something that looks like an object literal without values. For want of a better term I call this an <i>object name literal</i>. The following class definition encapsulates the same two properties as before as an object name literal. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id74c4569b3d42'><button class='copyBtn' data-clipboard-target='#id74c4569b3d42' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>class IngredientX { <span class='unselectable'>... </span> constructor(<span class="bg_yellow">{id, parts}</span>) { <span class='unselectable'>..... </span> this.id = id; <span class='unselectable'>..... </span> this.parts = parts; <span class='unselectable'>..... </span> } <span class='unselectable'>... </span>} <span class='unselectable'>undefined </span></pre> <p> New <code>IngredientX</code> instances can be created from an object literal: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb1360bbadee6'><button class='copyBtn' data-clipboard-target='#idb1360bbadee6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredientX1 = new IngredientX(<span class="bg_yellow">{id: "123", parts: 10 }</span>); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientX1 <span class='unselectable'>IngredientX { id: '123', parts: 10 } </span></pre> <p> Because the <code>IngredientX</code> class definition requires an object name literal (or a JSON object, more on that later) to provide constructor arguments, constructor invocations must specify the names of each parameter being passed to the constructor arguments. This has the benefit of making your software more robust in the face of changing method signatures. </p> <p> Caution: new <code>IngredientX</code> instances cannot be created from scalar arguments. JavaScript gives no error or warning if you do not: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id82ee9ecc319b'><button class='copyBtn' data-clipboard-target='#id82ee9ecc319b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredientX2 = new IngredientX("123", 10); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientX2 <span class='unselectable'>IngredientX { id: <span class="bg_yellow">undefined</span>, parts: <span class="bg_yellow">undefined</span> } </span></pre> <h2 id="jsonArgs">JSON Object Can Be Supplied Instead of Object Literals</h2> <p> JSON objects can be provided as arguments instead of object literals. This is extremely handy. Replacing several arguments with a JSON object would possibly be the most significant improvement in robustness that could be made to a JavaScript project. The number of runtime errors encountered as a code base evolves would be greatly reduced. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfb3c8360a5b4'><button class='copyBtn' data-clipboard-target='#idfb3c8360a5b4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredientX3 = new IngredientX({ <span class='unselectable'>... </span> <span class="bg_yellow">"id"</span>: "123", <span class='unselectable'>... </span> <span class="bg_yellow">"parts"</span>: 10 <span class='unselectable'>... </span> }); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientX3 <span class='unselectable'>IngredientX { id: '123', parts: 10 } </span></pre> <h2 id="jsonArgs">Arguments and Parameters Can Be Provided In Any Order</h2> <p> This definition of <code>ingredientX4</code> is identical to the definition of <code>ingredientX3</code>, even though the order of the arguments has been reversed: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id198567e105b2'><button class='copyBtn' data-clipboard-target='#id198567e105b2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredientX4 = new IngredientX({ <span class='unselectable'>... </span> <span class="bg_yellow">"parts"</span>: 10, <span class='unselectable'>... </span> <span class="bg_yellow">"id"</span>: "123" <span class='unselectable'>... </span> }); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientX4 <span class='unselectable'>IngredientX { id: '123', parts: 10 } </span></pre> <p> The parameters in the function or method declaration are also insensitive to ordering: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8d81fd596ceb'><button class='copyBtn' data-clipboard-target='#id8d81fd596ceb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>class IngredientXReordered { <span class='unselectable'>... </span> constructor(<span class="bg_yellow">{parts, id}</span>) { <span class='unselectable'>..... </span> this.parts = parts; <span class='unselectable'>..... </span> this.id = id; <span class='unselectable'>..... </span> } <span class='unselectable'>... </span>} <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>var ingredientX5 = new IngredientXReordered({ <span class='unselectable'>... </span> <span class="bg_yellow">"parts"</span>: 10, <span class='unselectable'>... </span> <span class="bg_yellow">"id"</span>: "123" <span class='unselectable'>... </span> }); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientX5 <span class='unselectable'>IngredientXReordered { id: '123', parts: 10 } </span></pre> <h2 id="litArgs">Object Literals Can Be Used With Any Method</h2> <p> Object literals / named arguments can be used to define the signature of any function or method, not just class constructors. For example: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id684a7bb66717'><button class='copyBtn' data-clipboard-target='#id684a7bb66717' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>class IngredientY { <span class='unselectable'>... </span> constructor({id, parts}) { <span class='unselectable'>..... </span> this.id = id; <span class='unselectable'>..... </span> this.parts = parts; <span class='unselectable'>..... </span> } <span class='unselectable'>... </span> <span class='unselectable'>... </span> mix(<span class="bg_yellow">{duration, intensity}</span>) { <span class='unselectable'>... </span> console.log(`Shake for ${duration} hours at intensity ${intensity}.`); <span class='unselectable'>... </span> } <span class='unselectable'>... </span> } <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>var ingredientY = new IngredientY({id: "123", parts: 10 }); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientY.mix(<span class="bg_yellow">{duration: 2.5, intensity: 2}</span>); <span class='unselectable'>Shake for 2.5 hours at intensity 2. </span> <span class='unselectable'>undefined </span></pre> <h2 id="jsonArgs">Default Values for Named Arguments</h2> <p> To make this example more interesting, the default value for <code>id</code> will be generated as a GUID. <a href='https://stackoverflow.com/questions/105034/how-to-create-a-guid-uuid' target='_blank' rel='nofollow'>Here are some other GUID implementations</a>, but the best implementations have dependencies and that would just make the article more complex than necessary. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc547e48ef02d'><button class='copyBtn' data-clipboard-target='#idc547e48ef02d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>function uuidv4() { <span class='unselectable'>... </span> return 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, <span class='unselectable'>... </span> function(c) { <span class='unselectable'>..... </span> var r = Math.random() * 16 | 0, v = c == 'x' ? r : (r & 0x3 | 0x8); <span class='unselectable'>..... </span> return v.toString(16); <span class='unselectable'>..... </span> }); <span class='unselectable'>... </span>} <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>uuidv4() <span class='unselectable'>'b13137c1-1598-42ca-9498-c1502e5405ed' </span></pre> <p> A JavaScript object literal or JSON object must be passed to a method whose parameters were defined by object literal names. If a name/value pair is not provided in the argument then the default parameter value is used. Some examples should help demonstrate how this works: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id59b2d6389441'><button class='copyBtn' data-clipboard-target='#id59b2d6389441' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>class IngredientZ { <span class='unselectable'>... </span> constructor({id<span class="bg_yellow">=uuidv4()</span>, parts<span class="bg_yellow">=10</span>}) { <span class='unselectable'>..... </span> this.id = id; <span class='unselectable'>..... </span> this.parts = parts; <span class='unselectable'>..... </span> } <span class='unselectable'>... </span> <span class='unselectable'>... </span> mix({duration<span class="bg_yellow">=1.2</span>, intensity<span class="bg_yellow">=6</span>}) { <span class='unselectable'>... </span> console.log(`Shake for ${duration} hours at intensity ${intensity}.`); <span class='unselectable'>... </span> } <span class='unselectable'>... </span> } <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>var ingredientZ1 = new IngredientZ(<span class="bg_yellow">{parts: 4}</span>); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientZ1 <span class='unselectable'>IngredientZ { id: <span class="bg_yellow">'4290dc1a-4f4c-4579-9e27-39b68085ad97'</span>, parts: <span class="bg_yellow">4</span> } </span> <span class='unselectable'>undefined </span></pre> <p> Empty objects are allowed as arguments. All this means is that default values are used for all parameters of the object name literal. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc4de70b8486f'><button class='copyBtn' data-clipboard-target='#idc4de70b8486f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredientZ2 = new IngredientZ(<span class="bg_yellow">{}</span>); <span class='unselectable'>undefined </span> > ingredientZ2 <span class='unselectable'>IngredientZ { id: '9e70dc12-1f4c-3579-6a17-49a68385bf73', parts: 10 } </span> <span class='unselectable'>> </span>ingredientZ2.mix(<span class="bg_yellow">{}</span>); <span class='unselectable'>Shake for 2.5 hours at intensity 2. </span></pre> <p> Missing objects result in a syntax error. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3245fdcc34a0'><button class='copyBtn' data-clipboard-target='#id3245fdcc34a0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>ingredientZ2.mix<span class="bg_yellow">()</span>; <span class='unselectable'>Uncaught TypeError: Cannot read property 'id' of undefined at new IngredientZ2 (repl:3:17) {% noselect undefined </span></pre> <h2 id="info">For More Information</h2> <p> For more information, please see <a href='https://exploringjs.com/impatient-js/ch_callables.html#named-parameters' target='_blank' rel='nofollow'>JavaScript for impatient programmers (ES2021 edition)</a>. </p> JavaScript Linter Configuration 2021-02-08T00:00:00-05:00 https://mslinn.github.io/blog/2021/02/08/js-linter-config <p> Using a lint tool can really help improve your code in a hurry. I am using <a href='https://jshint.com' target='_blank' rel='nofollow'>JSHint</a> for a project that has a big JavaScript file that needs some love. </p> <h2 id=".jshintrc"><span class="code">.jshintrc</span></h2> <p> All modern web browsers support at least the version of JavaScript that conforms to <a href='https://en.wikipedia.org/wiki/ECMAScript#6th_Edition_%E2%80%93_ECMAScript_2015' target='_blank' rel='nofollow'>ECMAScript 6th Edition</a>, also known as ECMAScript 2015. Neither the documentation for the Atom <a href='https://github.com/AtomLinter/linter-jshint' target='_blank' rel='nofollow'>linter-jshint</a> plugin nor <a href='https://jshint.com/docs/' target='_blank' rel='nofollow'>JSHint</a> itself explicitly state that in order to work with the version of JavaScript supported by all modern web browsers, you need to provide a JSON formatted configuration file that sets the <code>esversion</code> property to 6, like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>.jshintrc</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id103d982ab0cd'><button class='copyBtn' data-clipboard-target='#id103d982ab0cd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{ "esversion": 6 }</pre> <p> If you do not do this, then JSHint will indicate errors if it encounters class definitions, for example. </p> <p> I put <code>.jshintrc</code> in the top-level directory of my project. </p> <p> I created a <a href='https://github.com/mslinn/linter-jshint/pull/1' target='_blank' rel='nofollow'>pull request</a> for the `linter-jshint` GitHub project so this documentation would be included. </p> <h2 id="stdin">Reading from <span class="code">stdin</span></h2> <p> The <a href='https://jshint.com/docs/cli/' target='_blank' rel='nofollow'>JSHint CLI docs</a> say: </p> <div class="quote"> If a file path is a dash (-) then JSHint will read from standard input. </div> <p> I needed to preprocess my JavaScript source files before invoking JSHint. Because JSHint can read from standard input, there is no need to write the preprocessed file contents to a temporary file. </p> <h2 id="stdin">Removing Jekyll Front Matter for JSHint</h2> <p> Jekyll can process any text file, including JavaScript files, if they contain front matter markers. This is useful for invoking Jekyll plugins and/or using Liquid expressions. My big JavaScript file has some information injected into it when Jekyll generates the site. </p> <p> Front matter is marked (delimited by) by two lines at the top of a file, consisting of three dashes, like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id47ced33e2d8a'><button class='copyBtn' data-clipboard-target='#id47ced33e2d8a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>--- ---</pre> <p> Here is how the empty front matter can be stripped from <code>myfile.js</code> so JSHint can inspect the remaining lines: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf07c1587e6bf'><button class='copyBtn' data-clipboard-target='#idf07c1587e6bf' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sed '/---/d' < myfile.js | jshint -</pre> Functional and Non-Functional E-Commerce Requirements 2021-02-02T00:00:00-05:00 https://mslinn.github.io/blog/2021/02/02/ecommerce-requirements <p> In a <a href='/blog/2021/01/30/opencart-postgres.html'>previous blog post</a> I described how I searched for <a href='https://www.google.com/search?q=open+source+shopping+cart' target='_blank' rel='nofollow'>open source shopping cart</a> and the disappointing software options that I found. These search results showed software projects that began 20 years ago and were generally of low quality. The businesses that manage these projects use a failed business model for open source, namely software-as-a-service (SaaS) without much added value. The hosted products have not changed much since they were first established, and they have not kept up with the relentless advances in computer technology. Those businesses derive some additional revenue from customizing the open-source projects. </p> <p> I decided to be more rigorous in my needs analysis for a shopping cart with good coupon support, so that I could find more suitable options. I now know that the cart must have: </p> <ul> <li>Strong discount/coupon support.</li> <li>Support for SKUs defined on-the-fly by customers as they interact with the web site.</li> <li>Integration with a variety of payment processors.</li> </ul> <p> Before I share my reviews of candidate shopping carts with you, let&rsquo;s first discuss the state of the e-commerce market, functional and non-functional requirements, the master/detail pattern, and some technology trends for business software. </p> <h2 id="hot">E-Commerce Is Hot, Hot, Hot</h2> <div style="text-align: center;"> <a href="https://www.digitalcommerce360.com/article/us-ecommerce-sales/" target="_blank" ><picture> <source srcset="/blog/images/django/usEcommerce.webp" type="image/webp"> <source srcset="/blog/images/django/usEcommerce.png" type="image/png"> <img src="/blog/images/django/usEcommerce.png" title="US ecommerce grows 44% in 2020<br />Online spending was $861 billion: 21% of total retail sales" class="center halfsize liImg2 rounded shadow" alt="US ecommerce grows 44% in 2020<br />Online spending was $861 billion: 21% of total retail sales" /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://www.digitalcommerce360.com/article/us-ecommerce-sales/" target="_blank" > US ecommerce grows 44% in 2020<br />Online spending was $861 billion: 21% of total retail sales </a> </figcaption> </figure> </div> <div style=""> <a href="https://www.amazon.com/gp/product/0062292986/ref=as_li_qf_sp_asin_il_tl" target="_blank" ><picture> <source srcset="/blog/images/ecommerce/ecommerceGrowth.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/ecommerceGrowth.png" type="image/png"> <img src="/blog/images/ecommerce/ecommerceGrowth.png" title=" US e-commerce vs. retail sales, 2010-2020<br /> Source: Digital Commerce 360, U.S. Department of Commerce; January 2021 " class=" liImg2 rounded shadow" alt=" US e-commerce vs. retail sales, 2010-2020<br /> Source: Digital Commerce 360, U.S. Department of Commerce; January 2021 " /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://www.amazon.com/gp/product/0062292986/ref=as_li_qf_sp_asin_il_tl" target="_blank" > US e-commerce vs. retail sales, 2010-2020<br /> Source: Digital Commerce 360, U.S. Department of Commerce; January 2021 </a> </figcaption> </figure> </div> <p> There are a huge number of e-commerce sites, and that market is experiencing strong growth in part due to the COVID-19 epidemic. E-commerce has <a href='https://www.amazon.com/gp/product/0062292986/ref=as_li_qf_sp_asin_il_tl' target='_blank' rel='nofollow'>entered Main Street</a>, as per Geoffrey Moore&rsquo;s technology adoption lifecycle. </p> <div style="text-align: center;"> <a href="https://www.amazon.com/gp/product/0062292986/ref=as_li_qf_sp_asin_il_tl" target="_blank" ><picture> <source srcset="/blog/images/ecommerce/crossingTheChasm.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/crossingTheChasm.png" type="image/png"> <img src="/blog/images/ecommerce/crossingTheChasm.png" title="&ldquo;Crossing the Chasm&rdquo;, which introduced the technology adoption lifecycle" class="center quartersize liImg2 rounded shadow" alt="&ldquo;Crossing the Chasm&rdquo;, which introduced the technology adoption lifecycle" /> </picture></a> <figcaption class="quartersize" style="width: 100%; text-align: center;"> <a href="https://www.amazon.com/gp/product/0062292986/ref=as_li_qf_sp_asin_il_tl" target="_blank" > &ldquo;Crossing the Chasm&rdquo;, which introduced the technology adoption lifecycle </a> </figcaption> </figure> </div> <h2 id="nfrs">Functional requirements</h2> <p> QRA Corp has a good definition of functional requirements. Paraphrasing information from <a href='https://qracorp.com/functional-vs-non-functional-requirements/' target='_blank' rel='nofollow'>this page</a>: </p> <div class="quote"> <p> Functional requirements describe what the system does or must not do, and can be thought of in terms of how the system responds to inputs. Functional requirements usually define if/then behaviors and include calculations, data input, and business processes. </p> <p> Functional requirements are features that allow the system to function as it was intended. Put another way, if the functional requirements are not met, the system will not work. Functional requirements are product features and focus on user requirements. </p> </div> <p> Functionally, a shopping cart is well understood, yet in order to be certain that a shopping cart actually works properly, test data and test procedures would need to be designed to verify that all the corner cases were exercised. That would be a significant amount of work to do properly, but at least the work would be rather straightforward. </p> <h2 id="nfrs">Non-functional requirements (NFRs)</h2> <p> Non-functional requirements relate to user expectations, and include security, reliability, performance, maintainability, scalability, and usability. Here are some paraphrased excerpts from the <a href='https://en.wikipedia.org/wiki/Non-functional_requirement' target='_blank' rel='nofollow'>Wikipedia article on NFRs</a>: </p> <div class="quote"> <p> An NFR is a requirement that specifies criteria that can be used to judge the operation of a system, rather than specific behaviors&mldr;<br /><br /> The plan for implementing NFRs is detailed in the system architecture, because they are usually architecturally significant requirements&mldr;<br /><br /> Broadly, functional requirements define what a system is supposed to <i>do</i> and NFRs define how a system is supposed to <i>be</i>&mldr;<br /><br /> NFRs can be divided into two main categories: </p> <ul> <li>Execution qualities, such as safety, security and usability, which are observable during operation (at run time).</li> <li>Evolution qualities, such as testability, maintainability, extensibility and scalability, which are embodied in the static structure of the system.</li> </ul> </div> <p> I need a shopping cart that has been properly tested, is flexible to configure, and is easy to use. These non-functional requirements are more important to me than the mechanics of how the cart was built or the technology that was used, especially for a proof of concept. In my reviews of shopping cart candidates I will focus on how well they fulfill these NFRs. </p> <p> I also want to use technology that is current, properly maintained, full-featured and has a future. </p> <h2 id="master-detail">Master-Detail Structures, Interfaces and Reporting</h2> <p> Shopping carts are a classic example of a master-detail structure. The user interface of a shopping cart must exploit the master-detail paradigm effectively. </p> <div style=""> <a href="https://www.oracle.com/webfolder/ux/middleware/alta/patterns/MasterDetail.html" target="_blank" ><picture> <source srcset="/blog/images/ecommerce/MasterDetail.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/MasterDetail.png" type="image/png"> <img src="/blog/images/ecommerce/MasterDetail.png" title="Image from Oracle Alta UI Patterns: Master-Detail" class=" liImg2 rounded shadow" alt="Image from Oracle Alta UI Patterns: Master-Detail" /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://www.oracle.com/webfolder/ux/middleware/alta/patterns/MasterDetail.html" target="_blank" > Image from Oracle Alta UI Patterns: Master-Detail </a> </figcaption> </figure> </div> <p> <a href='https://en.wikipedia.org/wiki/Master%E2%80%93detail_interface' target='_blank' rel='nofollow'>Wikipedia defines master/detail data models</a> as: </p> <p class="quote"> A master–detail relationship is a one-to-many type relationship. Examples of a master-detail relationship are: a set of purchase orders and a set of line items belonging to each purchase order, an expense report with a set of expense line items or a department with a list of employees belonging to it. An application can use this master-detail relationship to enable users to navigate through the purchase order data and see the detail data for line items only related to the master purchase order selected. </p> <p> In the book <a href='https://www.oreilly.com/library/view/sap-businessobjects-bi/9780071773126/lev1sec120.html' target='_blank'>SAP BusinessObjects BI 4.0 The Complete Reference 3/E</a>, authors Cindi Howson and Elizabeth Newbould provide this rather abstract definition of <i>master/detail reports</i>: </p> <p class="quote"> A master/detail report is a particular kind of report in which a dimension value (master) is used to group data (detail) into separate sections. Master/detail reports allow you to analyze and format data for each unique master data value. </p> <p> Page 13 of <a href='https://books.google.ca/books?id=wgBMWCvTEHQC&pg=PA13&dq=First+normal+form+removes+repetition+by+creating+one-to-many+relationships&hl=en&sa=X&ved=2ahUKEwjB-5_NrMvuAhWCElkFHV0aA-MQ6AEwAHoECAEQAg#v=onepage&q&f=false' target='_blank' rel='nofollow'>Oracle&reg; Performance Tuning for 10gR2, Second Edition</a> by Gavin JT Powell has a more concrete definition: </p> <p class="quote"> <b>First normal form removes repetition by creating one-to-many relationships.</b> Data repeated many times in one entity is removed to a subset entity, which becomes the container for the removed repeating data. Each row in the subset entity will contain a single reference to each row in the original entity. The original entity will then contain only nonduplicated data. This one-to-many relationship is commonly known as a master-detail relationship, where repeating columns are removed to a new entity. The new entity gets a primary key consisting of a composite of the primary key in the master entity and a unique identifier (within each master primary key) on the detail entity. </p> <p> Master-detail is such a common architectural pattern that most business software vendors provide support for it. Other vendors include <a href='https://www.ibm.com/support/knowledgecenter/SSEP7J_11.1.0/com.ibm.swg.ba.cognos.ug_cr_rptstd.doc/t_cr_rptstd_modrep_create_master_detail_relationship.html#cr_rptstd_modrep_create_master_detail_relationship' target='_blank'>IBM</a>, <a href='https://docs.oracle.com/cd/E15586_01/web.1111/b31974/web_masterdetail.htm' target='_blank'>Oracle</a>, <a href='https://help.salesforce.com/articleView?id=sf.overview_of_custom_object_relationships.htm&type=5' target='_blank'>Salesforce</a>, and <a href='https://channel9.msdn.com/Blogs/OneCode/How-to-create-a-master-detail-ListBox-in-universal-Windows-apps' target='_blank'>Microsoft</a>. Apple&rsquo;s iPad is used as a client for master-detail applications so often that the iOS SDK even provides a <a href='https://www.oreilly.com/library/view/beginning-ios-5/9781118144251/ch004-sec007.html' target='_blank'>Master-Detail Application template</a>. The master/detail pattern for Google Android applications <a href='https://github.com/lurbas/MaterialMasterDetail' target='_blank' rel='nofollow'>can be implemented using the Material Design</a> visual language. </p> <h2 id="master-detail-frameworks">Master-Detail Frameworks</h2> <p> Shopping carts must have engaging and effective implementations for master/detail patterns throughout: in the user interface, in the design of reports, in the data structures, in the types of software modules and their interfaces to each other, and in the internal data flow. The best-known web framework explicitly designed to support the master/detail pattern is <a href='https://rubyonrails.org/' target='_blank'>Ruby on Rails</a>. Many other web frameworks can support master/detail, of course. </p> <div style="text-align: center;"> <a href="https://rubyonrails.org/" target="_blank" ><picture> <source srcset="/blog/images/ecommerce/Ruby-on-Rails.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/Ruby-on-Rails.png" type="image/png"> <img src="/blog/images/ecommerce/Ruby-on-Rails.png" title="Ruby on Rails" class="center halfsize liImg2 rounded shadow" alt="Ruby on Rails" /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://rubyonrails.org/" target="_blank" > Ruby on Rails </a> </figcaption> </figure> </div> <h2 id="momentum">Momentum</h2> <h3 id="webserver">Web Servers</h3> <p> Web servers are usually placed in front of e-commerce servers for scalability and security reasons. <a href='https://www.netcraft.com/about/' target='_blank' rel='nofollow'>Netcraft</a> has been reporting on web server deployments since 1995. The <a href='https://news.netcraft.com/archives/2021/01/28/january-2021-web-server-survey.html' target='_blank' rel='nofollow'>Netcraft January 2021 Web Server Survey</a> has some interesting trends. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/ecommerce/wss-active-share.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/wss-active-share.png" type="image/png"> <img src="/blog/images/ecommerce/wss-active-share.png" title="Web server developer market share by server type.<br />Apache <code>httpd</code> is losing ground, as developers move to Microsoft and nginx." class="center liImg2 rounded shadow" alt="Web server developer market share by server type.<br />Apache <code>httpd</code> is losing ground, as developers move to Microsoft and nginx." /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> Web server developer market share by server type.<br />Apache <code>httpd</code> is losing ground, as developers move to Microsoft and nginx. </figcaption> </figure> </div> <div style="text-align: center;"> <picture> <source srcset="/blog/images/ecommerce/wss-top-1m-share.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/wss-top-1m-share.png" type="image/png"> <img src="/blog/images/ecommerce/wss-top-1m-share.png" title="For the busiest 1 million websites, Microsoft has taken the lead." class="center liImg2 rounded shadow" alt="For the busiest 1 million websites, Microsoft has taken the lead." /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> For the busiest 1 million websites, Microsoft has taken the lead. </figcaption> </figure> </div> <p> Microsoft&rsquo;s <a href='https://dotnet.microsoft.com/apps/aspnet' target='_blank' rel='nofollow'>ASP.NET</a> web framework is free, and web hosting is free (to a point), but those enticements fulfill their purpose by locking in customers to Microsoft's software stack and tools. The nopCommerce e-commerce server is free, very capable, and has enjoyed terrific adoption. This technology might be a good business decision for e-commerce sites that have typical requirements, but not if significant customization or integration with non-Microsoft services becomes important. This article was written by an experienced developer: <a href='https://www.freecodecamp.org/news/i-built-a-web-api-with-express-flask-aspnet/' target='_blank' rel='nofollow'>I rebuilt the same web API using Express, Flask, and ASP.NET. Here's what I found</a>. </p> <h3 id="language">Computer Languages</h3> <div style="text-align: center;"> <a href="https://github.com/nasa/fprime" target="_blank" ><picture> <source srcset="/blog/images/ecommerce/mars_drone.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/mars_drone.png" type="image/png"> <img src="/blog/images/ecommerce/mars_drone.png" title="Python runs <a rel='nofollow' href='https://www.nasa.gov/feature/jpl/6-things-to-know-about-nasas-ingenuity-mars-helicopter/' target='_blank'>NASA&rsquo;s Mars drone <i>Ingenuity</i></a>." class="center liImg2 rounded shadow" alt="Python runs <a rel='nofollow' href='https://www.nasa.gov/feature/jpl/6-things-to-know-about-nasas-ingenuity-mars-helicopter/' target='_blank'>NASA&rsquo;s Mars drone <i>Ingenuity</i></a>." /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://github.com/nasa/fprime" target="_blank" > Python runs <a rel='nofollow' href='https://www.nasa.gov/feature/jpl/6-things-to-know-about-nasas-ingenuity-mars-helicopter/' target='_blank'>NASA&rsquo;s Mars drone <i>Ingenuity</i></a>. </a> </figcaption> </figure> </div> <p> Of all computer languages, Python has arguably the most momentum at present. Ruby on Rails is terrific, but its market share has dropped dramatically since 2011, and without that framework the Ruby language would probably be much less important that it is. <a href='https://python.org' target='_blank' rel='nofollow'>Python</a> has been growing in all directions for a very long time. Microsoft&rsquo;s C# language, which requires a .NET compatible runtime, is also popular, but never had the momentum that Python has. </p> <p> The following graph compares the popularity of the Python Django library for making web applications against the popularity of the Ruby on Rails framework, for the time period 2009 to the present day. As you can see, in 2011 Ruby on Rails was at its peak popularity; at that time it was 300% more popular than Django. Now the situation is reversed: today Django is 400% more popular than Ruby on Rails. </p> <div style=""> <a href="https://insights.stackoverflow.com/trends?tags=django%2Cruby-on-rails" target="_blank" ><picture> <source srcset="/blog/images/django/django_vs_ror.webp" type="image/webp"> <source srcset="/blog/images/django/django_vs_ror.png" type="image/png"> <img src="/blog/images/django/django_vs_ror.png" title="Stack Overflow Trends: Django vs Ruby on Rails" class=" liImg2 rounded shadow" alt="Stack Overflow Trends: Django vs Ruby on Rails" /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://insights.stackoverflow.com/trends?tags=django%2Cruby-on-rails" target="_blank" > Stack Overflow Trends: Django vs Ruby on Rails </a> </figcaption> </figure> </div> <p> The one thing about Python that I like better than all other languages is the <a href='https://www.python.org/dev/peps/pep-0206/#id3' target='_blank' rel='nofollow'>&ldquo;batteries included&rdquo;</a> feature-driven approach. This means that Python projects can aspire to more ambitious goals for a given amount of programmer effort, as compared to implementing with other languages. That was a significant factor in my decision to focus on Python-powered semi-custom shopping carts, instead of investigating shopping carts programmed with other computer languages. Ruby on Rails would doubtless provide an excellent foundation for shopping carts, but I think Python is a better strategic choice for me at the present time in the world of open source. </p> <p> This is not a decision I would have made in years past. Python's runtime expends a lot of electrical power to run Python programs. As available computing power continues to grow year-on-year within devices everywhere, the &ldquo;Python runtime tax&rdquo; has become much easier to bear. </p> <div style="text-align: center;"> <a href="https://python.org/" target="_blank" ><picture> <source srcset="/blog/images/python.webp" type="image/webp"> <source srcset="/blog/images/python.png" type="image/png"> <img src="/blog/images/python.png" title="The Python Language" class="center halfsize liImg2 rounded shadow" alt="The Python Language" /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://python.org/" target="_blank" > The Python Language </a> </figcaption> </figure> </div> <h2 id="open">Beyond Open Source</h2> <p> A lot of open source software suffers from limited or no funding. This seriously impacts long-term viabilty. I am happy to consider all appropriate technology, and I am not fixated on open source options. Two major technical components need to be considered: </p> <ul> <li>E-commerce framework &ndash; this dictates the computer language and runtime library.</li> <li>Database &ndash; the e-commerce framework often dictates the choice of database.</li> </ul> <h2 id="arch">Monolithic vs Serverless Architectures</h2> <h3 id="serverless">Modular Yet Monolithic Architectures</h3> <div style="text-align: center;"> <picture> <source srcset="/blog/images/ecommerce/easter_island.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/easter_island.png" type="image/png"> <img src="/blog/images/ecommerce/easter_island.png" title="Monoliths on Easter Island" class="center liImg2 rounded shadow" alt="Monoliths on Easter Island" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> Monoliths on Easter Island </figcaption> </figure> </div> <p> Traditionally, software has been built by first constructing functional modules, then combining the modules into a complete program (the monolith). For e-commerce, entire programs are deployed to production by adding configuration information and integrating with external services, such as payment processors. This works well, however, the result is a centralized computing resource that has a fixed capability and resource requirements. The expense of operating the software does not change much, regardless of the traffic volume. In order to handle periods of heavy traffic, complex mechanisms are required to scale up transactional capacity. Conversely, during periods of light traffic, the minimum cost to maintain the system in an operational state can be significant, especially for a startup company. </p> <h3 id="serverless">Serverless Architecture</h3> <div style="text-align: center;"> <a href="https://martinfowler.com/articles/serverless.html" target="_blank" ><picture> <source srcset="/blog/images/ecommerce/serverless_fowler.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/serverless_fowler.png" type="image/png"> <img src="/blog/images/ecommerce/serverless_fowler.png" title="Martin Fowler on Serverless Architectures" class="center liImg2 rounded shadow" alt="Martin Fowler on Serverless Architectures" /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://martinfowler.com/articles/serverless.html" target="_blank" > Martin Fowler on Serverless Architectures </a> </figcaption> </figure> </div> <p> One of the big architectural advances in recent years has been the introduction of <i>serverless computing</i>. It provides near-infinite scalability without paying for unused resources. <a href='https://www.cloudflare.com/learning/serverless/what-is-serverless/' target='_blank' rel='nofollow'>CloudFlare has a good definition</a>: </p> <p class="quote"> Serverless computing is a method of providing backend services on an as-used basis. A serverless provider allows users to write and deploy code without the hassle of worrying about the underlying infrastructure. A company that gets backend services from a serverless vendor is charged based on their computation and do not have to reserve and pay for a fixed amount of bandwidth or number of servers, as the service is auto-scaling. Note that despite the name serverless, physical servers are still used but developers do not need to be aware of them. </p> <p> Vendors that provide serverless computing platforms include <a href='https://aws.amazon.com/lambda/' target='_blank' rel='nofollow'>AWS Lambda</a>, <a href='https://azure.microsoft.com/services/functions/' target='_blank' rel='nofollow'>Azure Functions</a>, <a href='https://workers.cloudflare.com/' target='_blank' rel='nofollow'>CloudFlare Workers</a>, <a href='https://cloud.google.com/functions' target='_blank' rel='nofollow'>Google Cloud Functions</a> </p> <p> The <a href='https://www.serverless.com' target='_blank' rel='nofollow'>Serverless Framework</a> is a language- and platform- agnostic framework. Languages supported include Node.js, Python, Java, Go, C#, Ruby, Swift, Kotlin, PHP, Scala, & F#. Platforms supported include <a href='https://www.serverless.com/framework/docs/providers/' target='_blank' rel='nofollow'>Alibaba Cloud, AWS, Microsoft Azure, Fn Project, Google Cloud Platform, Apache OpenWhisk, CloudFlare Workers, Knative, Kubeless, Spotinst and Tencent Cloud</a>. The <a href='https://github.com/serverless/serverless' target='_blank' rel='nofollow'>GitHub project</a> has the code. </p> <h2 id="responsive">Responsive Web Pages</h2> <div style="text-align: center;"> <a href="https://medium.com/level-up-web/best-practices-of-responsive-web-design-6da8578f65c4" target="_blank" ><picture> <source srcset="/blog/images/ecommerce/responsive_web.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/responsive_web.png" type="image/png"> <img src="/blog/images/ecommerce/responsive_web.png" title="Image from &lsquo;Best Practices of Responsive Web Design&rsquo; by Bradley Nice" class="center liImg2 rounded shadow" alt="Image from &lsquo;Best Practices of Responsive Web Design&rsquo; by Bradley Nice" /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://medium.com/level-up-web/best-practices-of-responsive-web-design-6da8578f65c4" target="_blank" > Image from &lsquo;Best Practices of Responsive Web Design&rsquo; by Bradley Nice </a> </figcaption> </figure> </div> <p> <a href='https://developer.mozilla.org/en-US/docs/Learn/CSS/CSS_layout/Responsive_Design' target='_blank' rel='nofollow'>Responsive web pages</a> alter their layout and appearance to suit different screen widths and resolutions. Many of the shopping carts I looked at did not support responsive web pages. </p> <h2 id="next">Next: Check Out Django Shopping Carts</h2> <p> OK, so Python, and therefore Django is winning. I admit that I like the Django slogan: &ldquo;Django &ndash; The web framework for perfectionists with deadlines&rdquo;. When an option dominates the competition, you would need a really special reason to consider other options. My functional and non-functional requirements are mainstream, so I'm going to check out the leading option. Today the leading option is Django. Stay tuned&mldr; </p> OpenCart - Postgres - ngnix - Ubuntu 2021-01-30T00:00:00-05:00 https://mslinn.github.io/blog/2021/01/30/opencart-postgres <p> I need a shopping cart that has good coupon/discount support with flexible pricing. My requirements are unique in that each item in the cart might be a custom product, with the price computed according to a formula on our server. There are very few standard SKUs. </p> <p> I started to look into 3 options for obtaining a shopping cart with good coupon support: building my own, or using a commercial product, or customizing an open-source project. This blog post is the story of the &lsquo;customize an open source&rsquo; track. </p> <h2 id="opencart">OpenCart</h2> <p> I wanted to evaluate the leading open-source shopping cart contender by installing it on a development machine and giving it real data. OpenCart is renowned as one of the better open-source shopping carts available today. As with many open-source projects, the company that provides the source code have a conflict of interest: if they make installing and configuring the software effortless then their revenue would be much less than if they had a cadre of interested but frustrated developers. I looked at the hosting options and did not like the price/performance and customization options. </p> <p> OpenCart shows its age by using MySQL and its descendants, like Maria. Long ago I moved on from MySQL to Postgres, and I have been pleased with Postgres. I decided to try to make OpenCart run on Postgres and Ubuntu. I knew before I started that OpenCart&rsquo;s support for Postgres was weak. </p> <div class="pullQuote"> Spoiler alert: OpenCart would not pass a proper source code quality review and any claims of PostgreSQL support are bogus </div> <p> If you find watching videos of gory slow-motion accidents entertaining, please read on. </p> <h2 id="tests">Unit Tests</h2> <p> First thing I look for when evaluating software for possible incorporation into a project is unit tests. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id51499c3f83d8'><button class='copyBtn' data-clipboard-target='#id51499c3f83d8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git grep -Ii test \ ":(exclude)*.css" \ ":(exclude)*.yml" \ ":(exclude)*.js" <span class='unselectable'>$ </span>git grep -IiE 'phptest|Codeception' \ ":(exclude)*.css" \ ":(exclude)*.yml" \ ":(exclude)*.js"</pre> <p> There were no tests and no references to PHPUnit or Codeception. This was a big black mark against OpenCart. </p> <h2 id="deps">Install Dependencies</h2> <p> If you need information about PostgreSQL, <a href='https://www.postgresql.org/' target='_blank' rel='nofollow'>here is the mother ship</a>. This is a good description of <a href='https://www.digitalocean.com/community/tutorials/how-to-install-postgresql-on-ubuntu-20-04-quickstart' target='_blank' rel='nofollow'>how to install Postgres</a>. </p> <p> I use PGAdmin when I want a graphical interface to PostgreSQL. Installation instructions are <a href='https://www.pgadmin.org/download/pgadmin-4-apt/' target='_blank' rel='nofollow'>here</a>. In a nutshell: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id09e61a50b86e'><button class='copyBtn' data-clipboard-target='#id09e61a50b86e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl https://www.pgadmin.org/static/packages_pgadmin_org.pub | \ sudo apt-key add <span class='unselectable'>$ </span>sudo sh -c \ 'echo "deb https://ftp.postgresql.org/pub/pgadmin/pgadmin4/apt/$(lsb_release -cs) pgadmin4 main" > /etc/apt/sources.list.d/pgadmin4.list && apt update' <span class='unselectable'>$ </span>sudo apt install pgadmin4</pre> <p> I installed the rest of the dependencies like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id890a4209d581'><button class='copyBtn' data-clipboard-target='#id890a4209d581' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install \ postgresql postgresql-contrib \ software-properties-common lynx \ php7.4 php-fpm php-gd php-curl php-postgre php-zip \ php5-pgsql</pre> <p> <code>phpenmod</code> is a Debian / Ubuntu command for enabling PHP extensions. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id043e923f0f4d'><button class='copyBtn' data-clipboard-target='#id043e923f0f4d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo phpenmod pgsql</pre> <p> I verified that the desired PHP extensions were installed typing <code>php -m</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3ef6d0ababe8'><button class='copyBtn' data-clipboard-target='#id3ef6d0ababe8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>php -m | grep pg <span class='unselectable'>pdo_pgsql pgsql</span></pre> <h2 id="debug">Figuring Out Problems</h2> <p> I also installed the PHP debugger, when I realized needed to do some debugging. IntelliJ (which is the big brother of PHP Storm) did a terrific job of providing debug capability for command-line PHP. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc29737267262'><button class='copyBtn' data-clipboard-target='#idc29737267262' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install php-xdebug</pre> <p> I opened a new console and continuously viewed the <code>nginx</code>, PHP and PostgreSQL error logs. This was a big help whenever I needed to figure out problems. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8abb0fdd3320'><button class='copyBtn' data-clipboard-target='#id8abb0fdd3320' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>tail -f \ /var/log/nginx/*.log \ /var/log/php*.log \ /var/log/postgresql/postgresql-12*.log</pre> <h2 id="nginx_config">Configure <span class='code'>nginx</span></h2> <p> The only change I made to <code>/etc/nginx/nginx.conf</code> was to change the default MIME type from <code>application/octet-stream</code> to <code>text/html</code>. The change is highlighted in yellow. </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/nginx/nginx.conf</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc1699358322c'><button class='copyBtn' data-clipboard-target='#idc1699358322c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>user www-data; worker_processes auto; pid /run/nginx.pid; include /etc/nginx/modules-enabled/*.conf; events { worker_connections 768; # multi_accept on; } http { ## # Basic Settings ## sendfile on; tcp_nopush on; types_hash_max_size 2048; # server_tokens off; # server_names_hash_bucket_size 64; # server_name_in_redirect off; include /etc/nginx/mime.types; <span style="background-color: yellow">#default_type application/octet-stream;</span> <span style="background-color: yellow">default_type text/html;</span> ## # SSL Settings ## ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE ssl_prefer_server_ciphers on; ## # Logging Settings ## access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log notice; ## # Gzip Settings ## gzip on; # gzip_vary on; # gzip_proxied any; # gzip_comp_level 6; # gzip_buffers 16 8k; # gzip_http_version 1.1; # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript; ## # Virtual Host Configs ## include /etc/nginx/conf.d/*.conf; include /etc/nginx/sites-enabled/*; } #mail { # # See sample authentication script at: # # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript # # # auth_http localhost/auth.php; # # pop3_capabilities "TOP" "USER"; # # imap_capabilities "IMAP4rev1" "UIDPLUS"; # # server { # listen localhost:110; # protocol pop3; # proxy on; # } # # server { # listen localhost:143; # protocol imap; # proxy on; # } #}</pre> <p> I deleted <code>/etc/nginx/sites-enabled/default</code> and replaced it with <code>/etc/nginx/sites-enabled/php</code> so PHP files would be parsed properly, no matter what directory they resided in. </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/nginx/sites-enabled/php</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbcbb4dc2e3dd'><button class='copyBtn' data-clipboard-target='#idbcbb4dc2e3dd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># See https://tecadmin.net/setup-nginx-php-fpm-on-ubuntu-20-04/ server { listen 80; root /var/www/html; index index.php index.html index.htm; server_name example.com; location / { try_files $uri $uri/ =404; } location ~ \.php$ { include snippets/fastcgi-php.conf; fastcgi_pass unix:/var/run/php/php7.4-fpm.sock; } }</pre> <p> Now it was time to restart <code>nginx</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id60920d9097da'><button class='copyBtn' data-clipboard-target='#id60920d9097da' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl restart nginx.service</pre> <h2 id="restart">Verify Services Are Running</h2> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id46a523d9466e'><button class='copyBtn' data-clipboard-target='#id46a523d9466e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl status nginx <span class='unselectable'>● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-01-29 16:58:04 EST; 15h ago Docs: man:nginx(8) Main PID: 1584845 (nginx) Tasks: 9 (limit: 38389) Memory: 10.0M CGroup: /system.slice/nginx.service ├─1584845 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; ├─1584846 nginx: worker process ├─1584847 nginx: worker process ├─1584848 nginx: worker process ├─1584849 nginx: worker process ├─1584850 nginx: worker process ├─1584851 nginx: worker process ├─1584852 nginx: worker process └─1584853 nginx: worker process Jan 29 16:58:04 localhost systemd[1]: Starting A high performance web server and a reverse proxy server... Jan 29 16:58:04 localhost systemd[1]: Started A high performance web server and a reverse proxy server.</span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id40d3a5e8e8f3'><button class='copyBtn' data-clipboard-target='#id40d3a5e8e8f3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl status php7.4-fpm <span class='unselectable'>● php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2021-01-30 08:32:36 EST; 10min ago Docs: man:php-fpm7.4(8) Process: 3430785 ExecStartPost=/usr/lib/php/php-fpm-socket-helper install /run/php/php-fpm.sock /etc/php/7.4/fp> Main PID: 3430782 (php-fpm7.4) Status: "Processes active: 0, idle: 2, Requests: 0, slow: 0, Traffic: 0req/sec" Tasks: 3 (limit: 38389) Memory: 8.1M CGroup: /system.slice/php7.4-fpm.service ├─3430782 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf) ├─3430783 php-fpm: pool www └─3430784 php-fpm: pool www Jan 30 08:32:36 localhost systemd[1]: Starting The PHP 7.4 FastCGI Process Manager... Jan 30 08:32:36 localhost systemd[1]: Started The PHP 7.4 FastCGI Process Manager. </span></pre> <h2 id="restart">Verify PHP Works</h2> <p> I made this file, which is very common when working with PHP. Just ensure that it does not appear in your production site, or hackers will know more about your website than they should. </p> <div class='codeLabel unselectable' data-lt-active='false'>/var/www/html/info.php</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide86aef4dc898'><button class='copyBtn' data-clipboard-target='#ide86aef4dc898' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>&lt;?php phpinfo(); ?></pre> <p> Now I verified that PHP worked by viewed information about the setup. The <code>-I</code> option causes <code>curl</code> to just return HTML headers, not the HTML body. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3f02ac778c61'><button class='copyBtn' data-clipboard-target='#id3f02ac778c61' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -I http://localhost/info.php <span class='unselectable'>HTTP/1.1 200 OK Server: nginx/1.18.0 (Ubuntu) Date: Sat, 30 Jan 2021 18:50:30 GMT Content-Type: text/html; charset=UTF-8 Connection: keep-alive </span></pre> <p> View <code>info.php</code> in a web browser to see the details. Lynx is good for that from a command line: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id464d81e35d03'><button class='copyBtn' data-clipboard-target='#id464d81e35d03' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>lynx http://localhost/info.php</pre> <h2 id="postgres_setup">Set Up ‎PostgreSQL</h2> <p> I only changed the value of <code>listen_addresses</code> in <code>postgresql.conf</code>. Again, this change is highlighted in yellow. </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/postgresql/12/main/postgresql.conf</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2387c4506151'><button class='copyBtn' data-clipboard-target='#id2387c4506151' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># ----------------------------- # PostgreSQL configuration file # ----------------------------- # # This file consists of lines of the form: # # name = value # # (The "=" is optional.) Whitespace may be used. Comments are introduced with # "#" anywhere on a line. The complete list of parameter names and allowed # values can be found in the PostgreSQL documentation. # # The commented-out settings shown in this file represent the default values. # Re-commenting a setting is NOT sufficient to revert it to the default value; # you need to reload the server. # # This file is read on server startup and when the server receives a SIGHUP # signal. If you edit the file on a running system, you have to SIGHUP the # server for the changes to take effect, run "pg_ctl reload", or execute # "SELECT pg_reload_conf()". Some parameters, which are marked below, # require a server shutdown and restart to take effect. # # Any parameter can also be given as a command-line option to the server, e.g., # "postgres -c log_connections=on". Some parameters can be changed at run time # with the "SET" SQL command. # # Memory units: kB = kilobytes Time units: ms = milliseconds # MB = megabytes s = seconds # GB = gigabytes min = minutes # TB = terabytes h = hours # d = days #------------------------------------------------------------------------------ # FILE LOCATIONS #------------------------------------------------------------------------------ # The default values of these variables are driven from the -D command-line # option or PGDATA environment variable, represented here as ConfigDir. data_directory = '/var/lib/postgresql/12/main' # use data in another directory # (change requires restart) hba_file = '/etc/postgresql/12/main/pg_hba.conf' # host-based authentication file # (change requires restart) ident_file = '/etc/postgresql/12/main/pg_ident.conf' # ident configuration file # (change requires restart) # If external_pid_file is not explicitly set, no extra PID file is written. external_pid_file = '/var/run/postgresql/12-main.pid' # write an extra PID file # (change requires restart) #------------------------------------------------------------------------------ # CONNECTIONS AND AUTHENTICATION #------------------------------------------------------------------------------ # - Connection Settings - <span style="background-color: yellow">listen_addresses = '*'</span> <span style="background-color: yellow">#listen_addresses = 'localhost' # what IP address(es) to listen on;</span> # comma-separated list of addresses; # defaults to 'localhost'; use '*' for all # (change requires restart) port = 5432 # (change requires restart) max_connections = 100 # (change requires restart) #superuser_reserved_connections = 3 # (change requires restart) unix_socket_directories = '/var/run/postgresql' # comma-separated list of directories # (change requires restart) #unix_socket_group = '' # (change requires restart) #unix_socket_permissions = 0777 # begin with 0 to use octal notation # (change requires restart) #bonjour = off # advertise server via Bonjour # (change requires restart) #bonjour_name = '' # defaults to the computer name # (change requires restart) # - TCP settings - # see "man 7 tcp" for details #tcp_keepalives_idle = 0 # TCP_KEEPIDLE, in seconds; # 0 selects the system default #tcp_keepalives_interval = 0 # TCP_KEEPINTVL, in seconds; # 0 selects the system default #tcp_keepalives_count = 0 # TCP_KEEPCNT; # 0 selects the system default #tcp_user_timeout = 0 # TCP_USER_TIMEOUT, in milliseconds; # 0 selects the system default # - Authentication - #authentication_timeout = 1min # 1s-600s #password_encryption = md5 # md5 or scram-sha-256 #db_user_namespace = off # GSSAPI using Kerberos #krb_server_keyfile = '' #krb_caseins_users = off # - SSL - #ssl = on #ssl_ca_file = '' ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem' #ssl_crl_file = '' ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key' #ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers #ssl_prefer_server_ciphers = on #ssl_ecdh_curve = 'prime256v1' #ssl_min_protocol_version = 'TLSv1' #ssl_max_protocol_version = '' #ssl_dh_params_file = '' #ssl_passphrase_command = '' #ssl_passphrase_command_supports_reload = off #------------------------------------------------------------------------------ # RESOURCE USAGE (except WAL) #------------------------------------------------------------------------------ # - Memory - shared_buffers = 128MB # min 128kB # (change requires restart) #huge_pages = try # on, off, or try # (change requires restart) #temp_buffers = 8MB # min 800kB #max_prepared_transactions = 0 # zero disables the feature # (change requires restart) # Caution: it is not advisable to set max_prepared_transactions nonzero unless # you actively intend to use prepared transactions. #work_mem = 4MB # min 64kB #maintenance_work_mem = 64MB # min 1MB #autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem #max_stack_depth = 2MB # min 100kB #shared_memory_type = mmap # the default is the first option # supported by the operating system: # mmap # sysv # windows # (change requires restart) dynamic_shared_memory_type = posix # the default is the first option # supported by the operating system: # posix # sysv # windows # mmap # (change requires restart) # - Disk - #temp_file_limit = -1 # limits per-process temp file space # in kB, or -1 for no limit # - Kernel Resources - #max_files_per_process = 1000 # min 25 # (change requires restart) # - Cost-Based Vacuum Delay - #vacuum_cost_delay = 0 # 0-100 milliseconds (0 disables) #vacuum_cost_page_hit = 1 # 0-10000 credits #vacuum_cost_page_miss = 10 # 0-10000 credits #vacuum_cost_page_dirty = 20 # 0-10000 credits #vacuum_cost_limit = 200 # 1-10000 credits # - Background Writer - #bgwriter_delay = 200ms # 10-10000ms between rounds #bgwriter_lru_maxpages = 100 # max buffers written/round, 0 disables #bgwriter_lru_multiplier = 2.0 # 0-10.0 multiplier on buffers scanned/round #bgwriter_flush_after = 512kB # measured in pages, 0 disables # - Asynchronous Behavior - #effective_io_concurrency = 1 # 1-1000; 0 disables prefetching #max_worker_processes = 8 # (change requires restart) #max_parallel_maintenance_workers = 2 # taken from max_parallel_workers #max_parallel_workers_per_gather = 2 # taken from max_parallel_workers #parallel_leader_participation = on #max_parallel_workers = 8 # maximum number of max_worker_processes that # can be used in parallel operations #old_snapshot_threshold = -1 # 1min-60d; -1 disables; 0 is immediate # (change requires restart) #backend_flush_after = 0 # measured in pages, 0 disables #------------------------------------------------------------------------------ # WRITE-AHEAD LOG #------------------------------------------------------------------------------ # - Settings - #wal_level = replica # minimal, replica, or logical # (change requires restart) #fsync = on # flush data to disk for crash safety # (turning this off can cause # unrecoverable data corruption) #synchronous_commit = on # synchronization level; # off, local, remote_write, remote_apply, or on #wal_sync_method = fsync # the default is the first option # supported by the operating system: # open_datasync # fdatasync (default on Linux) # fsync # fsync_writethrough # open_sync #full_page_writes = on # recover from partial page writes #wal_compression = off # enable compression of full-page writes #wal_log_hints = off # also do full page writes of non-critical updates # (change requires restart) #wal_init_zero = on # zero-fill new WAL files #wal_recycle = on # recycle WAL files #wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers # (change requires restart) #wal_writer_delay = 200ms # 1-10000 milliseconds #wal_writer_flush_after = 1MB # measured in pages, 0 disables #commit_delay = 0 # range 0-100000, in microseconds #commit_siblings = 5 # range 1-1000 # - Checkpoints - #checkpoint_timeout = 5min # range 30s-1d max_wal_size = 1GB min_wal_size = 80MB #checkpoint_completion_target = 0.5 # checkpoint target duration, 0.0 - 1.0 #checkpoint_flush_after = 256kB # measured in pages, 0 disables #checkpoint_warning = 30s # 0 disables # - Archiving - #archive_mode = off # enables archiving; off, on, or always # (change requires restart) #archive_command = '' # command to use to archive a logfile segment # placeholders: %p = path of file to archive # %f = file name only # e.g. 'test ! -f /mnt/server/archivedir/%f && cp %p /mnt/server/archivedir/%f' #archive_timeout = 0 # force a logfile segment switch after this # number of seconds; 0 disables # - Archive Recovery - # These are only used in recovery mode. #restore_command = '' # command to use to restore an archived logfile segment # placeholders: %p = path of file to restore # %f = file name only # e.g. 'cp /mnt/server/archivedir/%f %p' # (change requires restart) #archive_cleanup_command = '' # command to execute at every restartpoint #recovery_end_command = '' # command to execute at completion of recovery # - Recovery Target - # Set these only when performing a targeted recovery. #recovery_target = '' # 'immediate' to end recovery as soon as a # consistent state is reached # (change requires restart) #recovery_target_name = '' # the named restore point to which recovery will proceed # (change requires restart) #recovery_target_time = '' # the time stamp up to which recovery will proceed # (change requires restart) #recovery_target_xid = '' # the transaction ID up to which recovery will proceed # (change requires restart) #recovery_target_lsn = '' # the WAL LSN up to which recovery will proceed # (change requires restart) #recovery_target_inclusive = on # Specifies whether to stop: # just after the specified recovery target (on) # just before the recovery target (off) # (change requires restart) #recovery_target_timeline = 'latest' # 'current', 'latest', or timeline ID # (change requires restart) #recovery_target_action = 'pause' # 'pause', 'promote', 'shutdown' # (change requires restart) #------------------------------------------------------------------------------ # REPLICATION #------------------------------------------------------------------------------ # - Sending Servers - # Set these on the master and on any standby that will send replication data. #max_wal_senders = 10 # max number of walsender processes # (change requires restart) #wal_keep_segments = 0 # in logfile segments; 0 disables #wal_sender_timeout = 60s # in milliseconds; 0 disables #max_replication_slots = 10 # max number of replication slots # (change requires restart) #track_commit_timestamp = off # collect timestamp of transaction commit # (change requires restart) # - Master Server - # These settings are ignored on a standby server. #synchronous_standby_names = '' # standby servers that provide sync rep # method to choose sync standbys, number of sync standbys, # and comma-separated list of application_name # from standby(s); '*' = all #vacuum_defer_cleanup_age = 0 # number of xacts by which cleanup is delayed # - Standby Servers - # These settings are ignored on a master server. #primary_conninfo = '' # connection string to sending server # (change requires restart) #primary_slot_name = '' # replication slot on sending server # (change requires restart) #promote_trigger_file = '' # file name whose presence ends recovery #hot_standby = on # "off" disallows queries during recovery # (change requires restart) #max_standby_archive_delay = 30s # max delay before canceling queries # when reading WAL from archive; # -1 allows indefinite delay #max_standby_streaming_delay = 30s # max delay before canceling queries # when reading streaming WAL; # -1 allows indefinite delay #wal_receiver_status_interval = 10s # send replies at least this often # 0 disables #hot_standby_feedback = off # send info from standby to prevent # query conflicts #wal_receiver_timeout = 60s # time that receiver waits for # communication from master # in milliseconds; 0 disables #wal_retrieve_retry_interval = 5s # time to wait before retrying to # retrieve WAL after a failed attempt #recovery_min_apply_delay = 0 # minimum delay for applying changes during recovery # - Subscribers - # These settings are ignored on a publisher. #max_logical_replication_workers = 4 # taken from max_worker_processes # (change requires restart) #max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers #------------------------------------------------------------------------------ # QUERY TUNING #------------------------------------------------------------------------------ # - Planner Method Configuration - #enable_bitmapscan = on #enable_hashagg = on #enable_hashjoin = on #enable_indexscan = on #enable_indexonlyscan = on #enable_material = on #enable_mergejoin = on #enable_nestloop = on #enable_parallel_append = on #enable_seqscan = on #enable_sort = on #enable_tidscan = on #enable_partitionwise_join = off #enable_partitionwise_aggregate = off #enable_parallel_hash = on #enable_partition_pruning = on # - Planner Cost Constants - #seq_page_cost = 1.0 # measured on an arbitrary scale #random_page_cost = 4.0 # same scale as above #cpu_tuple_cost = 0.01 # same scale as above #cpu_index_tuple_cost = 0.005 # same scale as above #cpu_operator_cost = 0.0025 # same scale as above #parallel_tuple_cost = 0.1 # same scale as above #parallel_setup_cost = 1000.0 # same scale as above #jit_above_cost = 100000 # perform JIT compilation if available # and query more expensive than this; # -1 disables #jit_inline_above_cost = 500000 # inline small functions if query is # more expensive than this; -1 disables #jit_optimize_above_cost = 500000 # use expensive JIT optimizations if # query is more expensive than this; # -1 disables #min_parallel_table_scan_size = 8MB #min_parallel_index_scan_size = 512kB #effective_cache_size = 4GB # - Genetic Query Optimizer - #geqo = on #geqo_threshold = 12 #geqo_effort = 5 # range 1-10 #geqo_pool_size = 0 # selects default based on effort #geqo_generations = 0 # selects default based on effort #geqo_selection_bias = 2.0 # range 1.5-2.0 #geqo_seed = 0.0 # range 0.0-1.0 # - Other Planner Options - #default_statistics_target = 100 # range 1-10000 #constraint_exclusion = partition # on, off, or partition #cursor_tuple_fraction = 0.1 # range 0.0-1.0 #from_collapse_limit = 8 #join_collapse_limit = 8 # 1 disables collapsing of explicit # JOIN clauses #force_parallel_mode = off #jit = on # allow JIT compilation #plan_cache_mode = auto # auto, force_generic_plan or # force_custom_plan #------------------------------------------------------------------------------ # REPORTING AND LOGGING #------------------------------------------------------------------------------ # - Where to Log - #log_destination = 'stderr' # Valid values are combinations of # stderr, csvlog, syslog, and eventlog, # depending on platform. csvlog # requires logging_collector to be on. # This is used when logging to stderr: #logging_collector = off # Enable capturing of stderr and csvlog # into log files. Required to be on for # csvlogs. # (change requires restart) # These are only used if logging_collector is on: #log_directory = 'log' # directory where log files are written, # can be absolute or relative to PGDATA #log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # log file name pattern, # can include strftime() escapes #log_file_mode = 0600 # creation mode for log files, # begin with 0 to use octal notation #log_truncate_on_rotation = off # If on, an existing log file with the # same name as the new log file will be # truncated rather than appended to. # But such truncation only occurs on # time-driven rotation, not on restarts # or size-driven rotation. Default is # off, meaning append to existing files # in all cases. #log_rotation_age = 1d # Automatic rotation of logfiles will # happen after that time. 0 disables. #log_rotation_size = 10MB # Automatic rotation of logfiles will # happen after that much log output. # 0 disables. # These are relevant when logging to syslog: #syslog_facility = 'LOCAL0' #syslog_ident = 'postgres' #syslog_sequence_numbers = on #syslog_split_messages = on # This is only relevant when logging to eventlog (win32): # (change requires restart) #event_source = 'PostgreSQL' # - When to Log - #log_min_messages = warning # values in order of decreasing detail: # debug5 # debug4 # debug3 # debug2 # debug1 # info # notice # warning # error # log # fatal # panic #log_min_error_statement = error # values in order of decreasing detail: # debug5 # debug4 # debug3 # debug2 # debug1 # info # notice # warning # error # log # fatal # panic (effectively off) #log_min_duration_statement = -1 # -1 is disabled, 0 logs all statements # and their durations, > 0 logs only # statements running at least this number # of milliseconds #log_transaction_sample_rate = 0.0 # Fraction of transactions whose statements # are logged regardless of their duration. 1.0 logs all # statements from all transactions, 0.0 never logs. # - What to Log - #debug_print_parse = off #debug_print_rewritten = off #debug_print_plan = off #debug_pretty_print = on #log_checkpoints = off #log_connections = off #log_disconnections = off #log_duration = off #log_error_verbosity = default # terse, default, or verbose messages #log_hostname = off log_line_prefix = '%m [%p] %q%u@%d ' # special values: # %a = application name # %u = username # %d = database name # %r = remote host and port # %h = remote host # %p = process ID # %t = timestamp without milliseconds # %m = timestamp with milliseconds # %n = timestamp with milliseconds (as a Unix epoch) # %i = command tag # %e = SQL state # %c = session ID # %l = session line number # %s = session start timestamp # %v = virtual transaction ID # %x = transaction ID (0 if none) # %q = stop here in non-session # processes # %% = '%' # e.g. '<%u%%%d> ' #log_lock_waits = off # log lock waits >= deadlock_timeout #log_statement = 'none' # none, ddl, mod, all #log_replication_commands = off #log_temp_files = -1 # log temporary files equal or larger # than the specified size in kilobytes; # -1 disables, 0 logs all temp files log_timezone = 'America/New_York' #------------------------------------------------------------------------------ # PROCESS TITLE #------------------------------------------------------------------------------ cluster_name = '12/main' # added to process titles if nonempty # (change requires restart) #update_process_title = on #------------------------------------------------------------------------------ # STATISTICS #------------------------------------------------------------------------------ # - Query and Index Statistics Collector - #track_activities = on #track_counts = on #track_io_timing = off #track_functions = none # none, pl, all #track_activity_query_size = 1024 # (change requires restart) stats_temp_directory = '/var/run/postgresql/12-main.pg_stat_tmp' # - Monitoring - #log_parser_stats = off #log_planner_stats = off #log_executor_stats = off #log_statement_stats = off #------------------------------------------------------------------------------ # AUTOVACUUM #------------------------------------------------------------------------------ #autovacuum = on # Enable autovacuum subprocess? 'on' # requires track_counts to also be on. #log_autovacuum_min_duration = -1 # -1 disables, 0 logs all actions and # their durations, > 0 logs only # actions running at least this number # of milliseconds. #autovacuum_max_workers = 3 # max number of autovacuum subprocesses # (change requires restart) #autovacuum_naptime = 1min # time between autovacuum runs #autovacuum_vacuum_threshold = 50 # min number of row updates before # vacuum #autovacuum_analyze_threshold = 50 # min number of row updates before # analyze #autovacuum_vacuum_scale_factor = 0.2 # fraction of table size before vacuum #autovacuum_analyze_scale_factor = 0.1 # fraction of table size before analyze #autovacuum_freeze_max_age = 200000000 # maximum XID age before forced vacuum # (change requires restart) #autovacuum_multixact_freeze_max_age = 400000000 # maximum multixact age # before forced vacuum # (change requires restart) #autovacuum_vacuum_cost_delay = 2ms # default vacuum cost delay for # autovacuum, in milliseconds; # -1 means use vacuum_cost_delay #autovacuum_vacuum_cost_limit = -1 # default vacuum cost limit for # autovacuum, -1 means use # vacuum_cost_limit #------------------------------------------------------------------------------ # CLIENT CONNECTION DEFAULTS #------------------------------------------------------------------------------ # - Statement Behavior - #client_min_messages = notice # values in order of decreasing detail: # debug5 # debug4 # debug3 # debug2 # debug1 # log # notice # warning # error #search_path = '"$user", public' # schema names #row_security = on #default_tablespace = '' # a tablespace name, '' uses the default #temp_tablespaces = '' # a list of tablespace names, '' uses # only default tablespace #default_table_access_method = 'heap' #check_function_bodies = on #default_transaction_isolation = 'read committed' #default_transaction_read_only = off #default_transaction_deferrable = off #session_replication_role = 'origin' #statement_timeout = 0 # in milliseconds, 0 is disabled #lock_timeout = 0 # in milliseconds, 0 is disabled #idle_in_transaction_session_timeout = 0 # in milliseconds, 0 is disabled #vacuum_freeze_min_age = 50000000 #vacuum_freeze_table_age = 150000000 #vacuum_multixact_freeze_min_age = 5000000 #vacuum_multixact_freeze_table_age = 150000000 #vacuum_cleanup_index_scale_factor = 0.1 # fraction of total number of tuples # before index cleanup, 0 always performs # index cleanup #bytea_output = 'hex' # hex, escape #xmlbinary = 'base64' #xmloption = 'content' #gin_fuzzy_search_limit = 0 #gin_pending_list_limit = 4MB # - Locale and Formatting - datestyle = 'iso, mdy' #intervalstyle = 'postgres' timezone = 'America/New_York' #timezone_abbreviations = 'Default' # Select the set of available time zone # abbreviations. Currently, there are # Default # Australia (historical usage) # India # You can create your own file in # share/timezonesets/. #extra_float_digits = 1 # min -15, max 3; any value >0 actually # selects precise output mode #client_encoding = sql_ascii # actually, defaults to database # encoding # These settings are initialized by initdb, but they can be changed. lc_messages = 'en_US.UTF-8' # locale for system error message # strings lc_monetary = 'en_US.UTF-8' # locale for monetary formatting lc_numeric = 'en_US.UTF-8' # locale for number formatting lc_time = 'en_US.UTF-8' # locale for time formatting # default configuration for text search default_text_search_config = 'pg_catalog.english' # - Shared Library Preloading - #shared_preload_libraries = '' # (change requires restart) #local_preload_libraries = '' #session_preload_libraries = '' #jit_provider = 'llvmjit' # JIT library to use # - Other Defaults - #dynamic_library_path = '$libdir' #------------------------------------------------------------------------------ # LOCK MANAGEMENT #------------------------------------------------------------------------------ #deadlock_timeout = 1s #max_locks_per_transaction = 64 # min 10 # (change requires restart) #max_pred_locks_per_transaction = 64 # min 10 # (change requires restart) #max_pred_locks_per_relation = -2 # negative values mean # (max_pred_locks_per_transaction # / -max_pred_locks_per_relation) - 1 #max_pred_locks_per_page = 2 # min 0 #------------------------------------------------------------------------------ # VERSION AND PLATFORM COMPATIBILITY #------------------------------------------------------------------------------ # - Previous PostgreSQL Versions - #array_nulls = on #backslash_quote = safe_encoding # on, off, or safe_encoding #escape_string_warning = on #lo_compat_privileges = off #operator_precedence_warning = off #quote_all_identifiers = off #standard_conforming_strings = on #synchronize_seqscans = on # - Other Platforms and Clients - #transform_null_equals = off #------------------------------------------------------------------------------ # ERROR HANDLING #------------------------------------------------------------------------------ #exit_on_error = off # terminate session on any error? #restart_after_crash = on # reinitialize after backend crash? #data_sync_retry = off # retry or panic on failure to fsync # data? # (change requires restart) #------------------------------------------------------------------------------ # CONFIG FILE INCLUDES #------------------------------------------------------------------------------ # These options allow settings to be loaded from files other than the # default postgresql.conf. Note that these are directives, not variable # assignments, so they can usefully be given more than once. include_dir = 'conf.d' # include files ending in '.conf' from # a directory, e.g., 'conf.d' #include_if_exists = '...' # include file only if it exists #include = '...' # include file #------------------------------------------------------------------------------ # CUSTOMIZED OPTIONS #------------------------------------------------------------------------------</pre> <p> All of my changes, highlighted in yellow, are at the bottom of <code>pg_hba.conf</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/postgresql/12/main/pg_hba.conf</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id81d8d6e5332e'><button class='copyBtn' data-clipboard-target='#id81d8d6e5332e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># PostgreSQL Client Authentication Configuration File # =================================================== # # Refer to the "Client Authentication" section in the PostgreSQL # documentation for a complete description of this file. A short # synopsis follows. # # This file controls: which hosts are allowed to connect, how clients # are authenticated, which PostgreSQL usernames they can use, which # databases they can access. Records take one of these forms: # # local DATABASE USER METHOD [OPTIONS] # host DATABASE USER ADDRESS METHOD [OPTIONS] # hostssl DATABASE USER ADDRESS METHOD [OPTIONS] # hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] # hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS] # hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS] # # (The uppercase items must be replaced by actual values.) # # The first field is the connection type: "local" is a Unix-domain # socket, "host" is either a plain or SSL-encrypted TCP/IP socket, # "hostssl" is an SSL-encrypted TCP/IP socket, and "hostnossl" is a # non-SSL TCP/IP socket. Similarly, "hostgssenc" uses a # GSSAPI-encrypted TCP/IP socket, while "hostnogssenc" uses a # non-GSSAPI socket. # # DATABASE can be "all", "sameuser", "samerole", "replication", a # database name, or a comma-separated list thereof. The "all" # keyword does not match "replication". Access to replication # must be enabled in a separate record (see example below). # # USER can be "all", a username, a group name prefixed with "+", or a # comma-separated list thereof. In both the DATABASE and USER fields # you can also write a file name prefixed with "@" to include names # from a separate file. # # ADDRESS specifies the set of hosts the record matches. It can be a # host name, or it is made up of an IP address and a CIDR mask that is # an integer (between 0 and 32 (IPv4) or 128 (IPv6) inclusive) that # specifies the number of significant bits in the mask. A host name # that starts with a dot (.) matches a suffix of the actual host name. # Alternatively, you can write an IP address and netmask in separate # columns to specify the set of hosts. Instead of a CIDR-address, you # can write "samehost" to match any of the server's own IP addresses, # or "samenet" to match any address in any subnet that the server is # directly connected to. # # METHOD can be "trust", "reject", "md5", "password", "scram-sha-256", # "gss", "sspi", "ident", "peer", "pam", "ldap", "radius" or "cert". # Note that "password" sends passwords in clear text; "md5" or # "scram-sha-256" are preferred since they send encrypted passwords. # # OPTIONS are a set of options for the authentication in the format # NAME=VALUE. The available options depend on the different # authentication methods -- refer to the "Client Authentication" # section in the documentation for a list of which options are # available for which authentication methods. # # Database and usernames containing spaces, commas, quotes and other # special characters must be quoted. Quoting one of the keywords # "all", "sameuser", "samerole" or "replication" makes the name lose # its special character, and just match a database or username with # that name. # # This file is read on server startup and when the server receives a # SIGHUP signal. If you edit the file on a running system, you have to # SIGHUP the server for the changes to take effect, run "pg_ctl reload", # or execute "SELECT pg_reload_conf()". # # Put your actual configuration here # ---------------------------------- # # If you want to allow non-local connections, you need to add more # "host" records. In that case you will also need to make PostgreSQL # listen on a non-local interface via the listen_addresses # configuration parameter, or via the -i or -h command line switches. # DO NOT DISABLE! # If you change this first entry you will need to make sure that the # database superuser can access the database using some other method. # Noninteractive access to all databases is required during automatic # maintenance (custom daily cronjobs, replication, and similar tasks). # # Database administrative login by Unix domain socket <span style="background-color: yellow">local all postgres md5</span> # TYPE DATABASE USER ADDRESS METHOD # "local" is for Unix domain socket connections only <span style="background-color: yellow">local all all peer</span> # IPv4 local connections: <span style="background-color: yellow">host all all 0.0.0.0/0 md5</span> <span style="background-color: yellow">host all all 0.0.0.0/32 md5</span> # IPv6 local connections: host all all ::1/128 md5 # Allow replication connections from localhost, by a user with the # replication privilege. local replication all peer host replication all 127.0.0.1/32 md5 host replication all ::1/128 md5</pre> <p> Now that PostgreSQL was configured, I restarted it: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id775de5fd41a7'><button class='copyBtn' data-clipboard-target='#id775de5fd41a7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl restart postgresql</pre> <h2 id="together">Connecting PostgreSQL to PHP</h2> <p> I created a new database called <code>opencart</code> with this command: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida0bfb683f4ad'><button class='copyBtn' data-clipboard-target='#ida0bfb683f4ad' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>psql -U postgres -c "create database opencart;" <span class='unselectable'>CREATE DATABASE Time: 1057.887 ms (00:01.058)</span></pre> <p> I entered PHP interactive mode to verify that PHP could connect properly to the new PostgreSQL database. This command sequence just creates a simple table and deletes it. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6637b94d4b83'><button class='copyBtn' data-clipboard-target='#id6637b94d4b83' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>php -a <span class='unselectable'>Interactive mode enabled php > </span>pg_connect("host=localhost dbname=opencart user=postgres password=hithere"); <span class='unselectable'>php > </span>pg_query("create table test(id integer)"); <span class='unselectable'>php > </span>pg_query("drop table test"); <span class='unselectable'>php > </span>exit</pre> <h2 id="opencart_configuration1">Configuring OpenCart</h2> <p> The two configuration files that OpenCart provides are empty. They need to be renamed before OpenCart can be installed. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0dce9ca08cbc'><button class='copyBtn' data-clipboard-target='#id0dce9ca08cbc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo mv /work/ecommerce/opencart/upload/config{-dist,}.php <span class='unselectable'>$ </span>sudo mv /work/ecommerce/opencart/upload/admin/config{-dist,}.php</pre> <p> These files will contain configuration information after the OpenCart <code>admin</code> user configures the system. The files also need to have their owner or group set to the same user that the web server runs as. For <code>nginx</code>, this username and group are both called <code>www-data</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0e4620426226'><button class='copyBtn' data-clipboard-target='#id0e4620426226' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>find . -name config.php -exec sudo chown www-data:dev {} \;</pre> <p> I dislike the idea of having a web application modify its configuration data while running. This is inherently insecure. However, many PHP programs from the era that OpenCart was originally written operated that way. I have always been acutely uncomfortable with this practice. </p> <p> Equally distasteful to me was the hack that PHP programmers often do in order to support multi-tenant web applications where users who self-administer their sites have limited storage options (20 years ago this was an issue, the rest of the world has moved on): storing logs within the program file structure. This is insecure. OpenCart logs belong in <code>/var/log/opencart</code>. I did not modify the code, instead I rolled my eyes and made the log files in <code>opencart/upload/system/storage/logs/</code> group writable. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idea65ac62376b'><button class='copyBtn' data-clipboard-target='#idea65ac62376b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo chmod g+w opencart/upload/system/storage/logs/*.log</pre> <h2 id="install">Installing OpenCart</h2> <h3 id="web-install">Web-Based Installer</h3> <p> The web-based OpenCart installer is fragile and <a href='https://github.com/opencart/opencart/commits/master/upload/install/index.php' target='_blank' rel='nofollow'>not well maintained</a>. It dies near the end of its work when attempting to install using a Postgres database. </p> <p> Clicking on <code><a href='http://localhost/upload/install/' target='_blank' rel='nofollow'>http://localhost/upload/install/</a></code> starts the web-based OpenCart installation process by displaying the GNU license agreement from 2007. The installation fails on page 3. This problem was first reported on <a href='https://github.com/opencart/opencart/issues/7521' target='_blank' rel='nofollow'>July 15, 2019</a> but it was not addressed. </p> <p> Completing the installation only requires that the database be set up. <code>system/helper/db_schema.php</code> contains PHP code for defining the database schema using MySQL, and the SQL to populate the database is found in <code>upload/install/opencart.sql</code>. </p> <p> At this point I gave up and tried the command-line installer. </p> <h3 id="cli-install">Command-Line Installer</h3> <p> OpenCart has a command-line installer which is not mentioned in the online installation documentation. I always prefer to use a command line installer, if possible because any problems encountered are easier to diagnose and fix than with web-based installers. </p> <p> In contrast to the publicly promoted web-based installer, the command-line installer appears to be <a href='https://github.com/opencart/opencart/commits/master/upload/install/cli_install.php' target='_blank' rel='nofollow'>well maintained</a> for and by the current authors, who obviously also operate OpenCart Cloud (more on that in a minute). Once again, we see the inherent conflict of interest in traditional open-source software. </p> <p> Here is a sample command line for installing OpenCart. The script should be run from the <code>opencart/upload/install</code> directory. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idda861d1ee332'><button class='copyBtn' data-clipboard-target='#idda861d1ee332' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>php cli_install.php install \ --db_database opencart \ --db_driver postgre \ --db_hostname localhost \ --db_password postgres_password \ --db_port 5432 \ --db_username postgres \ --email email@example.com \ --http_server http://localhost/opencart/ \ --password admin_password \ --username admin \ | lynx -stdin</pre> <p> I needed to provide proper values for the following options: </p> <dl> <dt><code>--db_database</code></dt> <dd> There is no default value for this option. It makes sense to name the database <code>opencart</code>, but one might have reasons to give it another name. </dd> <dt><code>--db_hostname</code></dt> <dd>The database might not run on the same network node as OpenCart's web server.</dd> <dt><code>--db_password</code></dt> <dd>Friends do not let friends use empty passwords, even on personal machines.</dd> <dt><code>--db_port</code></dt> <dd>I usually use the default PostgreSQL port, 5432.</dd> <dt><code>--db_username</code></dt> <dd>It is more secure to not use the <code>postgres</code> default username.</dd> <dt><code>--email</code></dt> <dd>Email address of the OpenCart administrator.</dd> <dt><code>--http_server</code></dt> <dd>More than just the domain name, this option also specifies the protocol, HTTP port and the path to the OpenCart directory on the web server.</dd> <dt><code>--password</code></dt> <dd>OpenCart admin user password.</dd> </dl> <h4 id="db_prefix">The <span class="code">--db-prefix</span> Option</h4> <p> The above omits the <code>--db_prefix</code> option, whose default value is <code>oc_</code>. This is because the installer uses <code>upload/install/opencart.sql</code>, which is hard-coded to use the default value. </p> <h4 id="cloud">The <span class="code">--cloud</span> Option</h4> <p> The above also omits the <code>--cloud</code> option. This option has no documentation. After looking at the source code, I think this parameter is exclusively for <a href='https://www.opencart.com/index.php?route=cloud/landing' target='_blank' rel='nofollow'>OpenCart Cloud</a> installations. This means that most people could omit the option because its value defaults to 0, which means the installation is not intended for Open Cloud. </p> <p> Why do I think that? <a href='https://github.com/opencart/opencart/blob/master/upload/install/cli_install.php' target='_blank' rel='nofollow'>Looking at the code</a> I see this parameter suppresses database configuration and saving of configuration information. Also, cloud installations require that the admin user password be pre-hashed, which suggests to me that this script can be initiated from another installation script used by Open Cloud. </p> <h3 id="cloud">Using Default Values</h3> <p> If you are installing on a development machine, it is likely to run both the Postgres database and the PHP website, and software is likely to be set up using default values. Assuming that the PostgreSQL username is the default, <code>postgres</code>, and the database is called <code>opencart</code>, you just need to specify the following parameters: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5ed7e3cebe45'><button class='copyBtn' data-clipboard-target='#id5ed7e3cebe45' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>php cli_install.php install \ --db_driver postgre \ --db_database opencart \ --db_password postgres_password \ --db_port 5432 \ --db_username postgres \ --email email@example.com \ --http_server http://localhost/opencart/ \ --password admin_password \ | lynx -stdin</pre> <h3 id="patch">Patching Source Code</h3> <p> I ran <code>cli_install.php</code>, found problems, fixed them, reran <code>cli_install.php</code>, found more problems, fixed them, etc. etc. </p> <p> Clearly no-one has ever run <code>cli_install.php</code> to completion when <code>--cloud</code> option was set to 0. <p> <h4 id="constants">Defining Constants</h4> <p> The people who use <code>cli_install.php</code> provide values for two constants before the program runs. I found some code in <code>upload/index.php</code> that defined them. Using those statements as a guide, I added some lines after line 57 of <code>upload/system/startup.php</code> so these constants were defined: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide157e7e9105e'><button class='copyBtn' data-clipboard-target='#ide157e7e9105e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>// mslinn added: define('DIR_EXTENSION', DIR_OPENCART . 'extension/'); define('HTTP_SERVER', 'file:' . $_SERVER['HTTP_HOST'] . rtrim(dirname($_SERVER['SCRIPT_NAME']), '/.\\') . '/'); // end mslinn</pre> <h4 id="db_driver">Checking Postgres Driver</h4> <p> Clearly no-one has ever tried to install using the PostgreSQL driver before. I had to modify line 198 of <code>upload/install/cli_install.php</code> to add a check, highlighted in yellow, for the PostgreSQL driver extension: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1d36ae71998c'><button class='copyBtn' data-clipboard-target='#id1d36ae71998c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>if (!extension_loaded('mysqli') <span style="background-color: yellow">&& !extension_loaded('pgsql')</span>) { $error .= 'ERROR: MySQLi extension needs to be loaded for OpenCart to work!' . "\n"; }</pre> <h2 id="results">Running the Command-Line Installer</h2> <p> The command-line installer spewed out miles and miles of HTML (mostly the GNU license) and died with an error message. So, I fixed the problem and reran it. It would die somewhere else with a different error. So, I fixed that problem too and reran it again. It would die yet somewhere else with yet another error. </p> <p> The last problem I found before quitting was an error message resulting from the command-line installer attempting to rewrite an HTML header. Really! A command-line installer does not need to present HTML to the user. This command-line installer program is clearly a cheap hack. </p> <h2 id="stop">Evaluation Results</h2> <p> At this point I felt that I now had a good idea of the quality this open-source project: OpenCart is very poorly constructed. </p> <p> The business model for the company that stewards OpenCart is also clear: keep a few programmers of modest ability busy, and charge for their time by the hour. OpenCart is not something I would want to base an e-commerce business on. </p> <p> Since this is supposedly the best open-source shopping cart today, <a href='/django/index.html'>I will next look into</a> building something just for me that provides me with competitive advantage. </p> Jekyll Plugin Template Collection 2020-12-30T00:00:00-05:00 https://mslinn.github.io/blog/2020/12/30/jekyll-plugin-template-collection <p> Here are templates for you to start writing your next <a href='https://jekyllrb.com/docs/plugins/' target='_blank'>Jekyll plugin</a> in Ruby. Templates are provided for custom Jekyll filters, generators, tags and block tags. These templates all: </p> <p> <ul> <li> Set up their own custom logger as described in my <a href='/blog/2020/12/28/custom-logging-in-jekyll-plugins.html'>previous blog post</a>. </li> <li> Provide the <a href='https://jekyllrb.com/docs/variables/' target='_blank'>Jekyll <code>site</code>, <code>page</code> and <code>mode</code> variables</a> in all the places you need them but don't have them available in scope. Note that generators are only invoked once for the entire site, when all the pages have been scanned and the site structure is available for processing, but unlike the other templates generators do not have access to the <code>page</code> variable because they are invoked on a per-site basis, not a per-page basis. It is common for generators to include code that loops through various collections of pages. </li> <li> Come with documentation boilerplate for processing with <a href='https://yardoc.org/' target='_blank'>yard</a>. </li> </ul> <p> You can <a href='/jekyll/doc/top-level-namespace.html'>view the rendered documentation</a>. </p> <h2 id="download">Code Download Options</h2> <p> You have options for how you might download these F/OSS Jekyll plugin templates. Pick an option and save to your <code>_plugins/</code> directory of your Jekyll-powered site. Your download options are: </p> <ol> <li> <a href='/mslinn_jekyll_plugins.zip'>Download a zip file</a> containing <a href='/blog/2020/10/03/jekyll-plugins.html'>all the F/OSS Jekyll plugins I publish</a>. </li> <li> Copy the following code to your clipboard by clicking on the clipboard icon at the top right of the code container, then save the code to <code>_plugins/</code> directory of your Jekyll-powered site. </li> </ol> <h2 id="filter" class="code">jekyll_filter_template.rb</h2> <p> This Jekyll filter template converts strings passed to it to upper case. <a href='/jekyll/doc/JekyllFilterTemplate.html'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/jekyll_filter_template.rb" download="jekyll_filter_template.rb" title="Click on the file name to download the file">jekyll_filter_template.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idd4bcd2dad263"><button class='copyBtn' data-clipboard-target='#idd4bcd2dad263' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 # Template for Jekyll filters. module JekyllFilterTemplate require_relative 'logger_factory' # include the source of logger_factory.rb into this program @log = LoggerFactory.new.create_logger('my_filter_template', Jekyll.configuration(&#123;&#125;), :warn, $stderr) # Accessor allows classes in this module to use the logger def self.log @log end # Describe the filter here. # @param input_strings [Array&lt;String>] State what this parameter is for. # @return [String] # @example Describe an example of how to use it. # &#123;&#123; 1234 | my_filter_template &#125;&#125; def my_filter_template(input_strings) JekyllFilterTemplate.log.info "input_strings = #&#123;input_strings&#125;, upcased = #&#123;input_strings.upcase&#125;".cyan input_strings.upcase end end Liquid::Template.register_filter(JekyllFilterTemplate) </pre> <h3>Output</h3> <p> Given this markup in an HTML file: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida1b6d8657273'><button class='copyBtn' data-clipboard-target='#ida1b6d8657273' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ "Hello, world!" | my_filter_template }}</pre> <p> This is what is rendered to the web page after being passed through the filter: </p> HELLO, WORLD! <h2 id="generator" class="code">jekyll_generator_template.rb</h2> <p> <a href='/jekyll/doc/JekyllGeneratorTemplate.html'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/jekyll_generator_template.rb" download="jekyll_generator_template.rb" title="Click on the file name to download the file">jekyll_generator_template.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="ide92a092238f3"><button class='copyBtn' data-clipboard-target='#ide92a092238f3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 # Describe this Jekyll generator here. class JekyllGeneratorTemplate &lt; Jekyll::Generator require_relative 'logger_factory' def initialize(config) super(config) @log = LoggerFactory.new.create_logger('my_generator_template', config, :warn, $stderr) end # Method prescribed by the Jekyll plugin lifecycle. # @param site [Jekyll.Site] Automatically provided by Jekyll plugin mechanism # @return [void] def generate(site) @config = site.config @mode = @config['env']['JEKYLL_ENV'] @log.info "mode=#&#123;@mode&#125;".green end end </pre> <h3>Output</h3> <p> Generators do not display anything on the generated web site. They can create files containing pages in a directory. Generators usually log information to the console whenever a problem occurs, or progress needs to be shown. </p> <h2 id="tag" class="code">jekyll_tag_template.rb</h2> <p> <a href='/jekyll/doc/JekyllTagTemplate.html'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/jekyll_tag_template.rb" download="jekyll_tag_template.rb" title="Click on the file name to download the file">jekyll_tag_template.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id6ebe80ce3045"><button class='copyBtn' data-clipboard-target='#id6ebe80ce3045' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 # # Module-level description goes here. # # @example Heading for this example # Describe what this example does # &#123;% my_tag_template "parameter" %&#125; # # @example Heading for this example # Describe what this example does # &#123;% my_tag_template "parameter" %&#125; module MyTagTemplate # Start of custom logger definition require_relative 'logger_factory' # include the source of logger_factory.rb into this program @log = LoggerFactory.new.create_logger('my_tag_template', Jekyll.configuration(&#123;&#125;), :warn, $stderr) # This accessor allows classes in this module to use the logger. def self.log @log end # End of custom logger definition # This class implements the Jekyll tag functionality class MyTag &lt; Liquid::Tag # Constructor. # @param tag_name [String] is the name of the tag, which we already know. # @param arguments [Hash, String, Liquid::Tag::Parser] the arguments from the tag. # @param tokens [Liquid::ParseContext] tokenized command line # @return [void] def initialize(tag_name, arguments, tokens) super(tag_name, arguments, tokens) MyTagTemplate.log.info "tag_name [#&#123;tag_name.class&#125;] = '#&#123;tag_name&#125;' [#&#123;tag_name.class&#125;]".green MyTagTemplate.log.info "arguments [#&#123;arguments.class&#125;] = '#&#123;arguments&#125;'".green # @site = context.registers[:site] # This variable is handy but not required # @config = @site.config # This variable is handy but not required # @mode = @config['env']['JEKYLL_ENV'] # This variable is handy but not required # MyTagTemplate.log.info "mode=#&#123;@mode&#125;".green @arguments = arguments @arguments = '' if @arguments.nil? || @arguments.empty? end # Method prescribed by the Jekyll plugin lifecycle. # @return [String] def render(context) @site = context.registers[:site] @config = @site.config @mode = @config['env']['JEKYLL_ENV'] MyTagTemplate.log.info "mode='#&#123;@mode&#125;'".green @page = context.registers[:page] MyTagTemplate.log.info "page.path='#&#123;@page.path&#125;'".green MyTagTemplate.log.info "page.url='#&#123;@page.url&#125;'".green &lt;&lt;~HEREDOC &lt;p style="color: green; background-color: yellow; padding: 1em; border: solid thin grey;"> #&#123;@arguments&#125; &lt;/p> HEREDOC end end private # Describe the function's purpose # This is a link &#123;https://domain.com with some text&#125;. # @param parameter [String] Describe this parameter's purpose # @return [String, nil] Describe the return value def my_private_function(parameter) log.info "my_private_function.parameter=#&#123;parameter&#125;" end # parse, or return the args # @note you can pass in parsed args # @return [Liquid::Tag::Parser] def parse_args(args) return args if args.is_a?(Liquid::Tag::Parser) || args.is_a?(Hash) Liquid::Tag::Parser.new( @args ) end end Liquid::Template.register_tag(MyTagTemplate.log.progname, MyTagTemplate::MyTag) </pre> <h3>Output</h3> <p> Given this markup in an HTML file: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id760c01c5c4a1'><button class='copyBtn' data-clipboard-target='#id760c01c5c4a1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{% my_tag_template This is a little song I wrote, I hope you sing it note for note... %}</pre> <p> This is how it looks after the block tag is rendered to the page: </p> <p style="color: green; background-color: yellow; padding: 1em; border: solid thin grey;"> This is a little song I wrote, I hope you sing it note for note... </p> <h2 id="block_template" class="code">jekyll_block_template.rb</h2> <p> <a href='/jekyll/doc/JekyllBlockTemplate.html'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/jekyll_block_template.rb" download="jekyll_block_template.rb" title="Click on the file name to download the file">jekyll_block_template.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id9b679d47f47a"><button class='copyBtn' data-clipboard-target='#id9b679d47f47a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 # # Module-level description goes here. # # @example Heading for this example # Describe what this example does # &#123;% my_block_template "parameter" %&#125; # Hello, world! # &#123;% endmy_block_template %&#125; # # @example Heading for this example # Describe what this example does # &#123;% my_block_template "parameter" %&#125; # Hello, world! # &#123;% endmy_block_template %&#125; module MyBlockTagTemplate # Start of custom logger definition require_relative 'logger_factory' # include the source of logger_factory.rb into this program @log = LoggerFactory.new.create_logger('my_block_template', Jekyll.configuration(&#123;&#125;), :warn, $stderr) # This accessor allows classes in this module to use the logger. def self.log @log end # End of custom logger definition # This class implements the Jekyll tag functionality class MyBlock &lt; Liquid::Block # Constructor. # @param tag_name [String] the name of the tag, which we already know. # @param text [Hash, String, Liquid::Tag::Parser] the arguments from the tag. # @param tokens [Liquid::ParseContext] parsed and tokenized command line # @return [void] def initialize(tag_name, arguments, tokens) super(tag_name, arguments, tokens) MyTagTemplate.log.info "tag_name [#&#123;tag_name.class&#125;] = '#&#123;tag_name&#125;' [#&#123;tag_name.class&#125;]".green MyTagTemplate.log.info "arguments [#&#123;arguments.class&#125;] = '#&#123;arguments&#125;'".green @arguments = arguments @arguments = '' if @arguments.nil? || @arguments.empty? end # Method prescribed by the Jekyll plugin lifecycle. # @return [String] def render(context) content = super # This magically underdocumented assignment somehow returns the text within the block. @site = context.registers[:site] @config = @site.config @mode = @config['env']['JEKYLL_ENV'] MyBlockTagTemplate.log.info "mode='#&#123;@mode&#125;'".green @page = context.registers[:page] MyBlockTagTemplate.log.info "page.path='#&#123;@page.path&#125;'".green MyBlockTagTemplate.log.info "page.url='#&#123;@page.url&#125;'".green &lt;&lt;~HEREDOC &lt;p style="color: green; background-color: yellow; padding: 1em; border: solid thin grey;"> #&#123;content&#125; &lt;/p> HEREDOC end end end Liquid::Template.register_tag(MyBlockTagTemplate.log.progname, MyBlockTagTemplate::MyBlock) </pre> <h3>Output</h3> <p> Given this markup in an HTML file: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id738674f2f3ea'><button class='copyBtn' data-clipboard-target='#id738674f2f3ea' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{% my_block_template %}Hello, world!{% endmy_block_template %}</pre> <p> This is how it looks after the block tag is rendered to the page: </p> <p style="color: green; background-color: yellow; padding: 1em; border: solid thin grey;"> Hello, world! </p> Custom Logging in Jekyll Plugins 2020-12-28T00:00:00-05:00 https://mslinn.github.io/blog/2020/12/28/custom-logging-in-jekyll-plugins <p> Debugging Jekyll plugins can be a challenge, but a configurable logging facility can be a big help. Writing <code>puts</code> statements to generate output does work, but they are tedious to control, because you must comment and uncomment lines that generate output as you work through issues. It is better to set up another standard Ruby logger, perhaps outputting to <code>STDERR</code>, and control the output by setting the log level for the plugin&rsquo;s logger. </p> <p> Loggers are cheap &ndash; and easy to set up. Be sure to set the log level to <code>warn</code> or <code>error</code> when you are not debugging, so site generation goes full speed. </p> <p> Use a new logger for each plugin that you write. At the end of this article I have provided the source code for <code>logger_factory.rb</code>, a Ruby library routine that I wrote which allows you to quickly make custom loggers for your plugins. </p> <h2 id="color">Colored Log Output</h2> <p> It can be difficult to find what you are looking for as you watch miles and miles of log output spew onto your console, hour after hour, while you work on a problem. Colored output can be a big help. Jekyll is configured with a <a href='https://github.com/octopress/colorator' target='_blank' rel='nofollow'>colorizer</a>, so you can use colors on any terminal output written by <code>puts</code>, or sent to a log that writes to <code>STDERR</code> or <code>STDOUT</code>. It is easy to do this, just append a suffix to a string that indicates the color you want that string to be displayed with. For example: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9c9fcfca13b8'><button class='copyBtn' data-clipboard-target='#id9c9fcfca13b8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>puts "This green text is written to STDOUT".green log.warn "This cyan text is written to STDERR because that is how the logger is configured".cyan</pre> <p style="color: green;">This green text is written to STDOUT</p> <p style="color: cyan;">This cyan text is written to STDERR because that is how the logger is configured</p> <p> Supported colors are: </p> <ul> <li><code>red</code></li> <li><code>black</code></li> <li><code>green</code></li> <li><code>yellow</code></li> <li><code>magenta</code></li> <li><code>white</code></li> <li><code>blue</code></li> <li><code>cyan</code></li> <li><code>bold</code></li> </ul> <h2 id="wrong">First, the Obvious But Wrong Way</h2> <p> You can send a message to <code>Jekyll.logger</code> from a plugin like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0a57a09f560f'><button class='copyBtn' data-clipboard-target='#id0a57a09f560f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>Jekyll.logger.debug "Hello, world" Jekyll.logger.info "Hello, world" Jekyll.logger.warn "Hello, world" Jekyll.logger.error "Hello, world"</pre> <p> However, the default <code>Jekyll.logger.log_level</code> is <code>:info</code>. This is a global setting that affects all of Jekyll because only one <code>Logger</code> is used throughout Jekyll. It outputs to <code>STDERR</code>. </p> <p> If you want to control the output by adjusting log levels you will quickly realize that attempting to adjust the <code>log_level</code> for Jekyll&rsquo;s logger singleton is problematic. This is because when you need to see verbose output from your plugin, verbose output from the rest of Jekyll will spew all over your terminal. </p> <h3 id="bad">Adjusting <span class="code">Jekyll.logger.log_level</span></h3> <p> <code>Jekyll.logger</code> is under-documented. This section is just provided to round out your understanding of how logging works in Jekyll. Please do not attempt to use Jekyll&rsquo;s logger in your plugins &mdash; instead, use the <code>logger_factory.rb</code> I provide later in this article. </p> <p> To set <code>Jekyll.logger.log_level</code> (globally), specify the <code>--verbose</code> / <code>-V</code> or <code>--quiet</code> / <code>-q</code> options when starting Jekyll: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4b910b4a1b98'><button class='copyBtn' data-clipboard-target='#id4b910b4a1b98' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>bundle exec jekyll build --verbose # Sets log_level to :debug <span class='unselectable'>$ </span>bundle exec jekyll build --quiet # Sets log_level to :error <span class='unselectable'>$ </span>JEKYLL_ENV=development bundle exec jekyll serve --verbose <span class='unselectable'>$ </span>JEKYLL_ENV=development bundle exec jekyll serve --quiet</pre> <p> You can also reset the Jekyll log level from within your Jekyll plugin like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id93d9d625c1d9'><button class='copyBtn' data-clipboard-target='#id93d9d625c1d9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>Jekyll.logger.adjust_verbosity(:verbose => true)</pre> <p> I found that <code>_config.yml</code> has no effect on <code>Jekyll.logger.log_level</code>; the following entry does nothing: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id58e6f2ed5a63'><button class='copyBtn' data-clipboard-target='#id58e6f2ed5a63' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>quiet: true</pre> <h2 id="custom">Custom Ruby Logger To the Rescue!</h2> <p> Here is a working code example for a tag plugin that shows how to set up a custom logger in a plugin. While this code works, and is quite useful, I have packaged this code into a library routine that is more convenient to set up. You will read all about the library routine later in this article. But first, please read through this section so you understand how the library routine works. </p> <ul> <li> The logger, called <code>log</code>, is defined at the module scope. Output can be sent to <code>STDOUT</code> or <code>STDERR</code>. </li> <li> An accessor is defined called <code>self.log</code>, which is invoked as <code>MyPlugin.log</code> from within module classes. </li> <li>The logger output does not include a timestamp because that is generally not helpful.</li> <li> The program name that is passed to <code>@@log.progname</code> is also used to register the plugin with Jekyll. This ensures that the program name displayed in the log output corresponds to the registered name for the Jekyll plugin. </li> </ul> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idba0e384775c4'><button class='copyBtn' data-clipboard-target='#idba0e384775c4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>require 'logger' module MyPlugin # See https://ruby-doc.org/stdlib-2.7.1/libdoc/logger/rdoc/Logger.html @@log = Logger.new(STDERR) # Could send output to STDOUT @@log.level = Logger::WARN @@log.progname = 'my_tag_name' @@log.formatter = proc { |severity, datetime, progname, msg| "#{severity} #{progname}: #{msg}\n" } def self.log @@log end class MyClass < Liquid::Tag # Write output to the custom logger like this: MyPlugin.log.info "Hello, world" MyPlugin.log.warn "You can colorize the log output".yellow end end Liquid::Template.register_tag(MyPlugin.log.progname, MyPlugin::MyClass)</pre> <p> The above works fine, but it is tedious to apply to every plugin. We need a logger factory for creating loggers in a standard way. </p> <h2 id="factory" class="code">logger_factory.rb</h2> <p> You have options for how you might download this F/OSS library plugin. Pick an option and save to a file in your Jekyll-powered site called <code>_plugins/logger_factory.rb</code>. Placing the file within the <code>_plugins/</code> directory will make this library class available to all your Jekyll plugins by following the instructions I will provide in a moment. Your options are: </p> <ol> <li> <a href='/mslinn_jekyll_plugins.zip'>Download a zip file</a> containing <a href='/blog/2020/10/03/jekyll-plugins.html'>all the F/OSS Jekyll plugins I publish</a>. </li> <li> Copy the following code to your clipboard by clicking on the clipboard icon at the top right of the code container, then save the code to a file in your Jekyll-powered site called <code>_plugins/logger_factory.rb</code>. </li> </ol> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/logger_factory.rb" download="logger_factory.rb" title="Click on the file name to download the file">logger_factory.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idf70267425783"><button class='copyBtn' data-clipboard-target='#idf70267425783' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 Michael Slinn # @license SPDX-License-Identifier: Apache-2.0 # Looks within _config.yml for a key corresponding to the plugin progname. # For example, if the plugin's progname has value "abc" then an entry called logger_factory.abc # will be read from the config file, if present. # If the entry exists, its value overrides the value specified when create_logger() was called. # If no such entry is found then the log_level value specified when create_logger() was called is used. # # @example Create a new logger using this code like this: # LoggerFactory.new.create_logger('my_tag_name', site.config, Logger::WARN, $stderr) # # For more information about the logging feature in the Ruby standard library, # @see https://ruby-doc.org/stdlib-2.7.1/libdoc/logger/rdoc/Logger.html class LoggerFactory require 'logger' # @param log_level [String, Symbol, Integer] can be specified as $stderr or $stdout, # or an integer from 0..3 (inclusive), # or as a case-insensitive string # (`debug`, `info`, `warn`, `error`, or `DEBUG`, `INFO`, `WARN`, `ERROR`), # or as a symbol (`:debug`, `:info`, `:warn`, `:error` ). # # @param config [YAML] is normally created by reading a YAML file such as Jekyll's `_config.yml`. # When invoking from a Jekyll plugin, provide `site.config`, # which is available from all types of Jekyll plugins as `Jekyll.configuration(&#123;&#125;)`. # # @example If `progname` has value `abc`, then the YAML to override the programmatically set log_level to `debug` is: # logger_factory: # abc: debug def create_logger(progname, config, log_level, stream_name) config_log_level = check_config_log_level(config: config, progname: progname) logger = Logger.new(stream_name) logger.level = config_log_level || log_level logger.progname = progname logger.formatter = proc &#123; |severity, _, prog_name, msg| "#&#123;severity&#125; #&#123;prog_name&#125;: #&#123;msg&#125;\n" &#125; logger end private # @param config [YAML] Configuration data that might contain a entry for `logger_factory` # @param progname [String] The name of the `config` subentry to look for underneath the `logger_factory` entry # @return [String, FalseClass] def check_config_log_level(config:, progname:) log_config = config['logger_factory'] return false if log_config.nil? progname_log_level = log_config[progname] return false if progname_log_level.nil? progname_log_level end end </pre> <p> The contents of <code>logger_factory.rb</code> defines a class called <code>LoggerFactory</code>. This class could be used to easily create custom loggers in any Ruby program. Following is a complete example of how you could use <code>LoggerFactory</code> in your Jekyll tag plugin. </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/jekyll_tag_template.rb" download="jekyll_tag_template.rb" title="Click on the file name to download the file">jekyll_tag_template.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id65425fcf5dbb"><button class='copyBtn' data-clipboard-target='#id65425fcf5dbb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 # # Module-level description goes here. # # @example Heading for this example # Describe what this example does # &#123;% my_tag_template "parameter" %&#125; # # @example Heading for this example # Describe what this example does # &#123;% my_tag_template "parameter" %&#125; module MyTagTemplate # Start of custom logger definition require_relative 'logger_factory' # include the source of logger_factory.rb into this program @log = LoggerFactory.new.create_logger('my_tag_template', Jekyll.configuration(&#123;&#125;), :warn, $stderr) # This accessor allows classes in this module to use the logger. def self.log @log end # End of custom logger definition # This class implements the Jekyll tag functionality class MyTag &lt; Liquid::Tag # Constructor. # @param tag_name [String] is the name of the tag, which we already know. # @param arguments [Hash, String, Liquid::Tag::Parser] the arguments from the tag. # @param tokens [Liquid::ParseContext] tokenized command line # @return [void] def initialize(tag_name, arguments, tokens) super(tag_name, arguments, tokens) MyTagTemplate.log.info "tag_name [#&#123;tag_name.class&#125;] = '#&#123;tag_name&#125;' [#&#123;tag_name.class&#125;]".green MyTagTemplate.log.info "arguments [#&#123;arguments.class&#125;] = '#&#123;arguments&#125;'".green # @site = context.registers[:site] # This variable is handy but not required # @config = @site.config # This variable is handy but not required # @mode = @config['env']['JEKYLL_ENV'] # This variable is handy but not required # MyTagTemplate.log.info "mode=#&#123;@mode&#125;".green @arguments = arguments @arguments = '' if @arguments.nil? || @arguments.empty? end # Method prescribed by the Jekyll plugin lifecycle. # @return [String] def render(context) @site = context.registers[:site] @config = @site.config @mode = @config['env']['JEKYLL_ENV'] MyTagTemplate.log.info "mode='#&#123;@mode&#125;'".green @page = context.registers[:page] MyTagTemplate.log.info "page.path='#&#123;@page.path&#125;'".green MyTagTemplate.log.info "page.url='#&#123;@page.url&#125;'".green &lt;&lt;~HEREDOC &lt;p style="color: green; background-color: yellow; padding: 1em; border: solid thin grey;"> #&#123;@arguments&#125; &lt;/p> HEREDOC end end private # Describe the function's purpose # This is a link &#123;https://domain.com with some text&#125;. # @param parameter [String] Describe this parameter's purpose # @return [String, nil] Describe the return value def my_private_function(parameter) log.info "my_private_function.parameter=#&#123;parameter&#125;" end # parse, or return the args # @note you can pass in parsed args # @return [Liquid::Tag::Parser] def parse_args(args) return args if args.is_a?(Liquid::Tag::Parser) || args.is_a?(Hash) Liquid::Tag::Parser.new( @args ) end end Liquid::Template.register_tag(MyTagTemplate.log.progname, MyTagTemplate::MyTag) </pre> <h2 id="ack">Acknowledgements</h2> <p> Thanks to <a href='https://github.com/ashmaroli' target='_blank' rel='nofollow'>Ashwin Maroli (@ashmaroli)</a> for his kind assistance. </p> Propagating Git Template Changes Downstream 2020-11-30T00:00:00-05:00 https://mslinn.github.io/blog/2020/11/30/propagating-git-template-changes <p> I have developed a <a href='https://jekyllrb.com/' target='_blank' rel='nofollow'>Jekyll</a> template that I use as the starting point for most of my websites. Whenever I improve the template I can easily incorporate the changes into all the websites that are based on it. This article describes how I set that up, and the information applies to all templates in general &ndash; Jekyll is not required. Templates do not need to have any special characteristics, beyond being generally useful in some sense. </p> <p> <a href='https://docs.github.com/en/free-pro-team@latest/github/creating-cloning-and-archiving-repositories/creating-a-template-repository' target='_blank' rel='nofollow'><code>GitHub</code> has a template feature</a> but this article does not require <code>GitHub</code> and works with all <code>git</code> hosts. </p> <p> This diagram shows the local and hosted versions of a template repository and a project repository based on the template. </p> <div style=""> <picture> <source srcset="/blog/images/gitTemplates.webp" type="image/webp"> <source srcset="/blog/images/gitTemplates.png" type="image/png"> <img src="/blog/images/gitTemplates.png" title="Local and Hosted versions of a project and its template" class=" liImg2 rounded shadow" alt="Local and Hosted versions of a project and its template" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> Local and Hosted versions of a project and its template </figcaption> </figure> </div> <h2 id="template_clone">Copy from the Template Repository</h2> <p> To make a new local repository called <code>new_project</code> based on the repository called <code>template</code> from GitHub user with ID <code>mslinn</code>, type the following incantation. Please modify this command to suit your project, which might be hosted on AWS CodeCommit, Bitbucket, GitLab, etc. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0014763dde89'><button class='copyBtn' data-clipboard-target='#id0014763dde89' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git clone git@github.com:mslinn/template.git new_project</pre> <p><i>I have no <code>git</code> repository called <code>template</code>, so the above is just for explanation purposes.</i></p> <p> <code>git</code> automatically sets up a remote origin from the local repository pointing to <code>template</code> for <code>git fetch</code> and <code>git push</code> commands, as we can see from the following: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3ece5c668d51'><button class='copyBtn' data-clipboard-target='#id3ece5c668d51' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git remote -v <span class='unselectable'>origin git@github.com:mslinn/template.git (fetch) origin git@github.com:mslinn/template.git (push)</span></pre> <h2 id="upstream_downstream">Define Upstream and Downstream Repositories</h2> <p> To separately obtain updates from the <code>new_project</code> repository tracked at <code>origin</code> and updates from the <code>upstream</code> template, we need to define remote URLs for both the <code>origin</code> and <code>template</code> repositories. </p> <h3 id="upstream">Define Upstream Repository</h3> <p> The template will be an upstream remote repository. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfecfde9363a6'><button class='copyBtn' data-clipboard-target='#idfecfde9363a6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git remote rename origin upstream</pre> <p> To ensure read-only status we should disable pushing from <code>new_project</code> to the <code>upstream</code> repository, like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4ccd7af75fff'><button class='copyBtn' data-clipboard-target='#id4ccd7af75fff' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git remote set-url --push upstream no_push</pre> <h3 id="create_downstream">Create New Downstream Repository</h3> <p> We need to create a new hosted repository for <code>new_project</code>. All the git hosting sites provide a way to do this using a web browser. However, I much prefer to use command line interfaces (CLIs). </p> <h4 id="gitlab_create">AWS CodeCommit</h4> <p> <a href='https://docs.aws.amazon.com/codecommit/latest/userguide/how-to-create-repository.html#how-to-create-repository-cli' target='_blank' rel='nofollow'>The AWS CLI</a> incantation looks something like this: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2043b9ac8cca'><button class='copyBtn' data-clipboard-target='#id2043b9ac8cca' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws codecommit create-repository --repository-name new_project \ --repository-description "My downstream project"</pre> </p> <h4 id="bitbucket_create">Bitbucket</h4> <p> <a href='https://marketplace.atlassian.com/apps/1211193/bitbucket-command-line-interface-cli?hosting=server&tab=overview' target='_blank' rel='nofollow'>The Bitbucket CLI</a> incantation looks something like this: </p> <p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id31290133c868'><button class='copyBtn' data-clipboard-target='#id31290133c868' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>acli bobswift9 --action createRepository \ --project new_project --repository new_project --name new_project</pre> </p> <h4 id="github_create">GitHub</h4> <p> The shiny new official <a href='https://cli.github.com' target='_blank' rel='nofollow'>GitHub CLI</a> unfortunately cannot do something that the tried-and-true <a href='https://hub.github.com/' target='_blank' rel='nofollow'>Hub <code>gh</code></a> does: decouple the creation of a local git repo from the creation of the remote repo on GitHub. That means we must use Hub. like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id35ee4158b240'><button class='copyBtn' data-clipboard-target='#id35ee4158b240' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cd new_project <span class='unselectable'>$ </span>hub create new_project <span class='unselectable'>Existing repository detected Updating origin Warning: No xauth data; using fake authentication data for X11 forwarding. X11 forwarding request failed on channel 0 https://github.com/mslinn/new_project</span></pre> <h4 id="gitlab_create">GitLab</h4> <p> All the GitLab CLIs I found had been abandoned. </p> <h3 id="git_all">All Git Host Sites</h3> <p> We can verify that the remotes for the downstream git project on your computer are now set up appropriately. The output shown shows that I used the GitHub CLI: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id72f2e61594ed'><button class='copyBtn' data-clipboard-target='#id72f2e61594ed' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git remote -v <span class='unselectable'>origin git@github.com:mslinn/new_project.git (fetch) origin git@github.com:mslinn/new_project.git (push) upstream git@github.com:mslinn/template.git (fetch) upstream no_push (push)</span></pre> <h2 id="template_update">Updating From the Downstream Repository</h2> <p> We can pull changes from the downstream <code>new_project</code> <code>origin</code> repository into the local copy of the downstream project like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id230d0db92417'><button class='copyBtn' data-clipboard-target='#id230d0db92417' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git pull <span class='unselectable'>From github.com:mslinn/template * branch master -> FETCH_HEAD Already up-to-date.</span></pre> <p> We could have typed this more verbose version, which accomplishes the same thing: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2d65c7ba1ff2'><button class='copyBtn' data-clipboard-target='#id2d65c7ba1ff2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git pull origin <span class='unselectable'>Everything up-to-date </span></pre> <h2 id="template_update">Updating From the Upstream Template</h2> <p> We can pull changes from the upstream <code>template</code> repository into the local copy of the downstream <code>new_project</code> repository like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id17357b13459f'><button class='copyBtn' data-clipboard-target='#id17357b13459f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git pull upstream <span class='unselectable'>From github.com:mslinn/template * branch master -> FETCH_HEAD Already up-to-date.</span></pre> <h2 id="pushing">Pushing Changes</h2> <p> We can push changes from the local copy of the <code>new_project</code> repository to the hosted copy. From the top-level <code>new_project</code> directory, type: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb72938ae8b27'><button class='copyBtn' data-clipboard-target='#idb72938ae8b27' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git add -A <span class='unselectable'>$ </span>git commit -m "Commit message goes here" <span class='unselectable'>$ </span>git push origin <span class='unselectable'>Everything up-to-date </span></pre> <p> However, if we try to push changes from <code>new_project</code> to <code>upstream</code>, we get the following error message. This is good because it means we cannot accidentally modify the upstream template when working on a project derived from the template. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id45d57135e34d'><button class='copyBtn' data-clipboard-target='#id45d57135e34d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git push upstream <span class='unselectable'>fatal: 'no_push' does not appear to be a git repository fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.</span></pre> <h2 id="update_upstream">Updating the Upstream Template From a Downstream Repo</h2> <p> It is convenient to use two-way merge utilities to propagate selected changes in a downstream repository with the upstream repository. My favorite such utilities are: </p> <ul> <li><a href='https://meldmerge.org' target='_blank' rel='nofollow'>Meld</a> &ndash; Available for Linux and Windows. Also, available for Mac without support. </li> <li><a href='https://www.jetbrains.com/help/idea/command-line-merge-tool.html' target='_blank' rel='nofollow'>IntelliJ</a> &ndash; Available as a command-line utility for Windows, Mac, and Linux. <a href='https://www.jetbrains.com/help/idea/settings-tools-diff-and-merge.html' target='_blank' rel='nofollow'>Also integrated with the IDEA GUI.</a> </li> </ul> <p> Once you have propagated selected changes from the downstream project to the upstream template repo, commit the changes to the upstream repo. From the top-level template project directory, type: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2ad89b617299'><button class='copyBtn' data-clipboard-target='#id2ad89b617299' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git add -A <span class='unselectable'>$ </span>git commit -m "Commit message goes here" <span class='unselectable'>$ </span>git push origin <span class='unselectable'># the word 'origin' is optional here</span> <span class='unselectable'>Everything up-to-date </span></pre> <!-- <p> The following makes it possible to push changes to <code>new_project</code> without affecting <code>template</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id914b0e3b1c04'><button class='copyBtn' data-clipboard-target='#id914b0e3b1c04' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git remote add origin git@github.com:mslinn/new_project.git <span class='unselectable'>$ </span>git push -u origin master</pre> --> <h2 id="2nd">Setting Up Another Computer</h2> <p> You might need to work on your project on another computer, and update from <code>upstream</code> the same way you set up the first computer. The process to do this is much the same as what was just described, but with fewer steps. It looks something like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida7d075b1f74b'><button class='copyBtn' data-clipboard-target='#ida7d075b1f74b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git clone git@github.com:mslinn/new_project.git <span class='unselectable'>Cloning into 'new_project'... Warning: No xauth data; using fake authentication data for X11 forwarding. X11 forwarding request failed on channel 0 remote: Enumerating objects: xxxx, done. remote: Total xxxx (delta 0), reused 0 (delta 0), pack-reused 1139 Receiving objects: 100% (xxxx/xxxx), xx.xx MiB | x.x MiB/s, done. Resolving deltas: 100% (xxx/xxx), done.</span> <span class='unselectable'>$ </span>cd new_project <span class='unselectable'>$ </span>git remote add upstream https://github.com/mslinn/template <span class='unselectable'>$ </span>git remote set-url --push upstream no_push</pre> <h2 id="script">GitHub Script</h2> <p> The following bash script is an example of how to automate the process of creating a project based on a template, using GitHub as the repository service. </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/cloneTemplate" download="cloneTemplate" title="Click on the file name to download the file">cloneTemplate</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idef47bed03446"><button class='copyBtn' data-clipboard-target='#idef47bed03446' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash # See https://www.mslinn.com/blog/2020/11/30/propagating-git-template-changes.html function help &#123; if [ "$1" ]; then printf "\nError: $1\n\n"; fi echo "Usage: $0 templateUrl newProjectName" exit 1 &#125; if [ -z "$(which git)" ]; then echo "Please install git and rerun this script" exit 2 fi if [ -z "$(which hub)" ]; then echo "Please install hub and rerun this script" exit 3 fi if [ -z "$1" ]; then help "No git project was specified as a template."; fi if [ -z "$2" ]; then help "Please provide the name of the new project based on the template"; fi git clone "$1" "$2" cd "$2" git remote rename origin upstream git remote set-url --push upstream no_push # Add the -p option to create a private repository hub create "$2" git branch -M master git push -u origin master </pre> <h2 id="acks">Acknowledgements</h2> <p> This posting was inspired by <a href='https://medium.com/@smrgrace/having-a-git-repo-that-is-a-template-for-new-projects-148079b7f178' target='_blank' rel='nofollow'>this article</a>. </p> Installing a New SSH Key on AWS EC2 with User Data 2020-10-27T00:00:00-04:00 https://mslinn.github.io/blog/2020/10/27/installing-a-new-ssh-key-on-awc-ec2-with-user-data <editor-fold Intro> <p> For some reason the <code>ssh</code> certificates that AWS generated for me 3 years ago are no longer recognized by Ubuntu 20.10. This article shows how to create new certificates and push them to an AWS server that was just upgraded to Ubuntu 20.01, and now cannot be logged into. I decided to use OpenSSH to generate the new keypairs instead of AWS to generate the keypairs because the current problem stems from <a href='https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html' target='_blank' rel='nofollow'>AWS-generated keys</a> gradually becoming incompatible with OpenSSH servers. </p> <p> This article describes the following: </p> <ol> <li>Tracking down the problem</li> <li>Create a new <code>ssh</code> certificate keypair.</li> <li>Even though the system cannot accept logins, the new <code>ssh</code> public key must be copied to the <code>ubuntu</code> user&rsquo;s <code>~/.ssh</code> directory on the problem server. This is done by defining a user data script on the server instance prior to booting it.</li> <li>Log into the problem server using the new certificates.</li> <li>Complete the upgrade to XUbuntu 20.10.</li> <li>Remove the user data script from the problem server instance.</li> </ol> </editor-fold Intro> <editor-fold Discovery> <h2 class="numbered" id="discovery">Discovery</h2> <p> I was able to log into another of my machines (<code>gojira</code>), so first I wanted to know if the problem machines (<code>va</code> and <code>france</code>) had OpenSSH configured differently. I used the <a href='https://linux.die.net/man/1/comm' target='_blank' rel='nofollow'>comm</a> Linux utility to perform the comparison. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb771afd8c287'><button class='copyBtn' data-clipboard-target='#idb771afd8c287' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>for x in cipher mac key kex; do comm -3 <(ssh -Q $x france|sort) <(ssh -Q $x gojira|sort) done <span class='unselectable'>$ </span>for x in cipher mac key kex; do comm -3 <(ssh -Q $x france|sort) <(ssh -Q $x gojira|sort) done</pre> <p> So the problem was not OpenSSH configuration per se. Next I wondered if the ssh connections to the problem machines were different somehow from the ssh connection to the working machine. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbfd805bdcd54'><button class='copyBtn' data-clipboard-target='#idbfd805bdcd54' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>comm -3 <(ssh -Gva|sort) <(ssh -G gojira|sort) <span class='unselectable'>hostname gojira hostname va identityfile ~/.ssh/id_rsa identityfile ~/.ssh/sslTest user mslinn user ubuntu </span></pre> <p>So the only differences were the hostnames and the keys offered.</p> <p> One of the problem machines, <code>france</code>, resided on <a href='https://scaleway.com' target='_blank' rel='nofollow'><code>scaleway</code></a>. I used the most recently available <a href='https://medium.com/@moul/scaleway-bootscript-simple-kernel-management-for-your-c1-server-de0c301de721' target='_blank' rel='nofollow'>bootscript</a> to launch the server and examined <code>/var/log/auth.log</code>. I found this: <code>sshd[27025]: Unable to negotiate with 205.185.123.173 port 40555: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1 [preauth]</code> </p> <p> This error message is produced by OpenSSH 7.0+. <a href='https://www.openssh.com/txt/release-7.0' target='_blank' rel='nofollow'>The release notes say</a> &ldquo;Support for the 1024-bit diffie-hellman-group1-sha1 key exchange is disabled by default at run-time. It may be re-enabled using the instructions at <code><a href='http://www.openssh.com/legacy.html' target='_blank' rel='nofollow'>http://www.openssh.com/legacy.html</a></code> &rdquo; </p> <p> So it seems that the version of OpenSSH installed with Ubuntu 20.10 rejects my old keys. The release notes for the new version of OpenSSH also indicate that OpenSSH 7.1 will be even stricter: </p> <ul> <li> &ldquo;This focus of this release is primarily to deprecate weak, legacy and/or unsafe cryptography&rdquo; </li> <li> &ldquo;Refusing all RSA keys smaller than 1024 bits (the current minimum is 768 bits)&rdquo; </li> <li> &ldquo;Several ciphers will be disabled by default: <code>blowfish-cbc</code>, <code>cast128-cbc</code>, all <code>arcfour</code> variants and the <code>rijndael-cbc</code> aliases for AES.&rdquo; </li> <li> &ldquo;MD5-based <code>HMAC</code> algorithms will be disabled by default.&rdquo; </li> </ul> <p> Clearly I need to generate better <code>ssh</code> keys. </p> <p> The version of OpenSSH installed by Ubuntu 20.10 is 8.3: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4de3ad4b63cd'><button class='copyBtn' data-clipboard-target='#id4de3ad4b63cd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ sshd -V <span class='unselectable'>unknown option -- V OpenSSH_8.3p1 Ubuntu-1, OpenSSL 1.1.1f 31 Mar 2020 usage: sshd [-46DdeiqTt] [-C connection_spec] [-c host_cert_file] [-E log_file] [-f config_file] [-g login_grace_time] [-h host_key_file] [-o option] [-p port] [-u len]</span> $ ssh -V <span class='unselectable'>OpenSSH_8.3p1 Ubuntu-1, OpenSSL 1.1.1f 31 Mar 2020 </span></pre> </editor-fold Discovery> <editor-fold Setup> <h2 class="numbered" id="setup">Setup</h2> <h3 class="numbered" id="setupAwsCli">AWS CLI</h3> <p> I prefer to use the AWS CLI instead of the <a href='https://console.aws.amazon.com' target='_blank' rel='nofollow'>web console</a>. Installation instructions are <a href='https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html' target='_blank' rel='nofollow'>here</a>. This article uses the AWS CLI exclusively in favor of the AWS web console. </p> <h3 class="numbered" id="setupJq"><span class="code">jq</span></h3> <p> I also use <a href='https://stedolan.github.io/jq/' target='_blank' rel='nofollow'>jq</a> for parsing JSON in the bash shell. Install it on Debian-style Linux distros such as Ubuntu like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3fc02f98f883'><button class='copyBtn' data-clipboard-target='#id3fc02f98f883' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install jq</pre> </editor-fold Setup> <editor-fold nameKeys> <h2 class="numbered" id="nameKeys">Name the New Keypair</h2> <p> I wanted to make new <code>ecdsa</code> keys because this algorithm is the currently accepted best practice for commercial security concerns. <code>ecdsa</code> stands for <a href='https://en.bitcoin.it/wiki/Elliptic_Curve_Digital_Signature_Algorithm' target='_blank' rel='nofollow'>Elliptic Curve Digital Signature Algorithm</a>. </p> <p> Unfortunately, AWS EC2 only accepts <code>RSA</code> keys. The name of the new key pair will be of the form <code>~/.ssh/rsa-YYYY-MM-DD</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd7197900acf4'><button class='copyBtn' data-clipboard-target='#idd7197900acf4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_KEY_PAIR_FILE "$HOME/.ssh/rsa-$( date '+%Y-%m-%d' )" <span class='unselectable'>AWS_KEY_PAIR_FILE=/home/mslinn/.ssh/rsa-2020-11-03</span></pre> <p> The new public key will be called <code>~/.ssh/rsa-2020-11-03.pub</code> and the new private key will be called <code>~/.ssh/rsa-2020-11-03</code>. </p> </editor-fold nameKeys> <editor-fold createKeys> <h2 class="numbered" id="createKeys">Create a New Keypair</h2> <ol> <li type='a'>This is how I would have created a new <code>ECDSA</code> keypair. <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3ef600008544'><button class='copyBtn' data-clipboard-target='#id3ef600008544' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ssh-keygen -b 521 -C "mslinn@mslinn.com" -f "$AWS_KEY_PAIR_FILE" -P "" -t ecdsa <span class='unselectable'>Generating public/private ecdsa key pair. Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-03 Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-03.pub The key fingerprint is: SHA256:HEKjAA1GZxHbpwqjm85DXQpQEeIWrcjZ6fl84RHQaHE mslinn@mslinn.com The key's randomart image is: +---[RSA 2048]----+ |=O*Bo.*E | |+.=oo*.o | |+o+.+.o.. | |o= o .o+ . | | o+ +. S | |..o=. o | |o .o . o | |.+ o o | |+o. . | +----[SHA256]-----+ $ </span>chmod 400 $AWS_KEY_PAIR_FILE</pre> </li> <li type='a'> Instead, I created a new <code>RSA</code> keypair like this: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id278105f4ba4e'><button class='copyBtn' data-clipboard-target='#id278105f4ba4e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ssh-keygen -b 4096 -C "mslinn@mslinn.com" -f "$AWS_KEY_PAIR_FILE" -m PEM -P "" -t rsa <span class='unselectable'>Generating public/private rsa key pair. Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-03 Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-03.pub The key fingerprint is: SHA256:bQScX0UMn0xGDorxSvElMZzwMyyk7hgs2FNbshBNenA mslinn@mslinn.com The key's randomart image is: +---[RSA 4096]----+ | ooE .*++o+** | | =. ooXo=.B.. | | o + o +.X. = | | o = * . =.o | |. + = . S o | | o + . | | . . | | | | | +----[SHA256]-----+ $ </span>chmod 400 $AWS_KEY_PAIR_FILE</pre> </li> <li type='a'> I would have liked to copy the keypair to the problem system using <code>ssh-copy-id</code>, but that only works when login is possible. <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2f1cdb65bd5f'><button class='copyBtn' data-clipboard-target='#id2f1cdb65bd5f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ssh-copy-id -i $AWS_KEY_PAIR_FILE ubuntu@$AWS_PROBLEM_IP</pre> </li> <li type='a'> Instead, I decided to paste the public key into an AWS user data script and execute that script on the problem server the next time it booted. The purpose of the script is to copy the new public key that was just made to <code>~/.ssh/</code> on the problem server. This is the user data script I wrote to install the new public key, called <code>rescue_ubuntu2010.sh</code>: <br /><br /> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/factor12/rescue_ubuntu2010.sh" download="rescue_ubuntu2010.sh" title="Click on the file name to download the file">rescue_ubuntu2010.sh</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id331e25ec36e0"><button class='copyBtn' data-clipboard-target='#id331e25ec36e0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash KEY_FILE_NAME=/home/ubuntu/.ssh/rsa-2020-11-03.pub cat > "$KEY_FILE_NAME" &lt;&lt;EOF ssh-rsa ABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFA== mslinn@mslinn.com EOF chown ubuntu: /home/ubuntu/.ssh/* chmod 400 "$KEY_FILE_NAME" cat "$KEY_FILE_NAME" >> /home/ubuntu/.ssh/authorized_keys </pre> The script runs on the problem server as <code>root</code> next time the system boots, and it reboots the server on the last line. </li> <li type='a'> The script need to be converted into base 64, in a file called <code>rescue_ubuntu2010.b64</code>. <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida7fd3fc6830c'><button class='copyBtn' data-clipboard-target='#ida7fd3fc6830c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>base64 rescue_ubuntu2010.sh > rescue_ubuntu2010.b64</pre> </li> <li type='a'> The problem EC2 instance can be shut down like this: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1ef4644d3346'><button class='copyBtn' data-clipboard-target='#id1ef4644d3346' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 stop-instances --instance-id $AWS_PROBLEM_INSTANCE_ID <span class='unselectable'>$ </span>aws ec2 wait instance-stopped --instance-ids $AWS_PROBLEM_INSTANCE_ID</pre> </li> <li type='a'> With the problem EC2 instance stopped, its user data was set to the base64-encoded version of the rescue script. <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id17b245e9412f'><button class='copyBtn' data-clipboard-target='#id17b245e9412f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 modify-instance-attribute \ --instance-id $AWS_PROBLEM_INSTANCE_ID \ --attribute userData \ --value file://rescue_ubuntu2010.b64</pre> </li> <li type='a'> Now the problem EC2 instance can be restarted. The script will add the new key to <code>/home/ubuntu/.ssh/authorized_keys</code> and login should be possible. <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id48d5964a4782'><button class='copyBtn' data-clipboard-target='#id48d5964a4782' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 start-instances --instance-id $AWS_PROBLEM_INSTANCE_ID <span class='unselectable'>{ "StartingInstances": [ { "CurrentState": { "Code": 0, "Name": "pending" }, "InstanceId": "i-d3b03954", "PreviousState": { "Code": 80, "Name": "stopped" } } ] }</span> <span class='unselectable'>$ </span>aws ec2 wait instance-running --instance-ids $AWS_PROBLEM_INSTANCE_ID</pre> </li> </ol> </editor-fold createKeys> <editor-fold resetData> <h2 class="numbered" id="resetData">Reset User Data for Next Time</h2> <p> Next time the problem server is stopped, clear the user data so it is not provided the next time the server restarts. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9ea2fe76c3d9'><button class='copyBtn' data-clipboard-target='#id9ea2fe76c3d9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 modify-instance-attribute \ --instance-id $AWS_PROBLEM_INSTANCE_ID \ --user-data Value=</pre> </editor-fold resetData> Rescuing a Catastrophic Upgrade to Ubuntu 20.10 2020-10-25T00:00:00-04:00 https://mslinn.github.io/blog/2020/10/25/rescuing-catastrophic-upgrades-to-ubuntu-20_10 <editor-fold Intro> <p> The upgrade from Ubuntu 20.04 to 20.10 has been especially problematic for each of the half-dozen XUbuntu systems that I manage. One important server that I run on <a href='https://www.scaleway.com' target='_blank' rel='nofollow'>Scaleway</a> became unresponsive and would not boot shortly after starting the installation, and another important server on <a href='https://aws.amazon.com' target='_blank' rel='nofollow'>AWS</a> ran fine, but did not allow logins. </p> <p> This blog post details what I did to recover the AWS server using a standard <a href='https://en.wikipedia.org/wiki/Unix-like' target='_blank' rel='nofollow'>*nix</a> procedure that any competent system administrator would be comfortable with: <a href='https://en.wikipedia.org/wiki/Chroot' target='_blank' rel='nofollow'><code>chroot</code></a>. Because the <code>chroot</code> environment will be set up in a way that it shares the rescue system&rsquo;s <code>/var/run</code> directory, the rescue system should have all upgrades in place and should be rebooted if <code>/var/run/reboot-required</code> exists. </p> <p> AWS also provides a tool called <a href='https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-ec2rescue.html' target='_blank' rel='nofollow'><code>EC2Rescue</code></a>, which does a complicated series of actions to accomplish something similar. <a href='https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html' target='_blank' rel='nofollow'>Here is additional documentation.</a> Because I find the AWS documentation is frequently obtuse, and the approach taken by most AWS products and tools is extremely general, I often find myself wasting a lot of time trying to get things to work. I don&rsquo;t subscribe to AWS support; if I had subscribed to expensive enterprise-level support, complete with an AWS expert to hold my hand while I attempted to resurrect the server, I might have tried using <code>EC2Rescue</code>. On the other hand, when pressed with an emergency, I prefer to lean on tried-and-true methods like <code>chroot</code>. </p> </editor-fold Intro> <editor-fold Setup> <h2 class="numbered" id="setup">Setup</h2> <h3 class="numbered" id="setupAwsCli">AWS CLI</h3> <p> I prefer to use the AWS CLI instead of the <a href='https://console.aws.amazon.com' target='_blank' rel='nofollow'>web console</a>. Installation instructions are <a href='https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html' target='_blank' rel='nofollow'>here</a>. This article uses the AWS CLI exclusively in favor of the AWS web console. </p> <h3 class="numbered" id="setupJq"><span class="code">jq</span></h3> <p> I also use <a href='https://stedolan.github.io/jq/' target='_blank' rel='nofollow'>jq</a> for parsing JSON in the bash shell. Install it on Debian-style Linux distros such as Ubuntu like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb2291e1a32ff'><button class='copyBtn' data-clipboard-target='#idb2291e1a32ff' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install jq</pre> </editor-fold Setup> <editor-fold info> <h2 class="numbered" id="info">Discover information about the Problem EC2 instance</h2> <h3 class="numbered" id="volumeId">Getting the AWS EC2 Instance Information</h3> <p> Because my problem EC2 instance has a tag called <code>Name</code> with <code>Value</code> <code>production</code>, I was able to easily obtain a JSON representation of all the information about it. I stored the JSON in an environment variable called <code>AWS_EC2_PRODUCTION</code>. </p> <p> The results are shown in unselectable text. This is so you can easily use this sample code yourself. You can copy the code to run into your clipboard. Just click on the little copy icon at the top right hand corner of the scrolling code display area. Because the prompt and the results and are unselectable, your clipboard will only pick up the code you need to paste in order to run the code example yourself. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf8ab6c42c309'><button class='copyBtn' data-clipboard-target='#idf8ab6c42c309' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_EC2_PRODUCTION "$( aws ec2 describe-instances | \ jq '.Reservations[].Instances[] | select((.Tags[].Key=="Name") and (.Tags[].Value=="production"))' )" <span class='unselectable'>AWS_EC2_PRODUCTION= '{ "AmiLaunchIndex": 0, "ImageId": "ami-e29b9388", "InstanceId": "i-825eb905", "InstanceType": "t2.small", "KeyName": "sslTest", "LaunchTime": "2017-10-12T16:24:14.000Z", "Monitoring": { "State": "disabled" }, "Placement": { "AvailabilityZone": "us-east-1c", "GroupName": "", "Tenancy": "default" }, "PrivateDnsName": "ip-10-0-0-201.ec2.internal", "PrivateIpAddress": "10.0.0.201", "ProductCodes": [], "PublicDnsName": "", "PublicIpAddress": "52.207.225.143", "State": { "Code": 16, "Name": "running" }, "StateTransitionReason": "", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "Architecture": "x86_64", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "AttachTime": "2016-04-05T19:07:17.000Z", "DeleteOnTermination": true, "Status": "attached", "VolumeId": "vol-1c8903b4" } } ], "ClientToken": "GykZz1459883236367", "EbsOptimized": false, "Hypervisor": "xen", "NetworkInterfaces": [ { "Association": { "IpOwnerId": "amazon", "PublicDnsName": "", "PublicIp": "52.207.225.143" }, "Attachment": { "AttachTime": "2016-04-05T19:07:16.000Z", "AttachmentId": "eni-attach-a58bd15f", "DeleteOnTermination": true, "DeviceIndex": 0, "Status": "attached" }, "Description": "Primary network interface", "Groups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "Ipv6Addresses": [], "MacAddress": "0a:a4:be:1b:8e:eb", "NetworkInterfaceId": "eni-fa4f65bb", "OwnerId": "031372724784", "PrivateIpAddress": "10.0.0.201", "PrivateIpAddresses": [ { "Association": { "IpOwnerId": "amazon", "PublicDnsName": "", "PublicIp": "52.207.225.143" }, "Primary": true, "PrivateIpAddress": "10.0.0.201" } ], "SourceDestCheck": true, "Status": "in-use", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "InterfaceType": "interface" } ], "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SecurityGroups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "SourceDestCheck": true, "Tags": [ { "Key": "Name", "Value": "production" } ], "VirtualizationType": "hvm", "CpuOptions": { "CoreCount": 1, "ThreadsPerCore": 1 }, "CapacityReservationSpecification": { "CapacityReservationPreference": "open" }, "HibernationOptions": { "Configured": false }, "MetadataOptions": { "State": "applied", "HttpTokens": "optional", "HttpPutResponseHopLimit": 1, "HttpEndpoint": "enabled" } }'</span></pre> <h3 class="numbered" id="problemInstanceId">Getting the AWS EC2 Problem Instance Id</h3> <p> The instance ID for the problem EC2 instance can be extracted from the JSON returned by the preceding results easily: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id53a1780ce9d8'><button class='copyBtn' data-clipboard-target='#id53a1780ce9d8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_PROBLEM_INSTANCE_ID "$( jq -r .InstanceId <<< $AWS_EC2_PRODUCTION )" <span class='unselectable'>AWS_PROBLEM_INSTANCE_ID=i-825eb905</span></pre> <h3 class="numbered" id="problemInstanceId">Getting the AWS EC2 Problem Instance IP Address</h3> <p> The IP address for the problem EC2 instance can be extracted from the JSON returned by the preceding results easily: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide5159582876d'><button class='copyBtn' data-clipboard-target='#ide5159582876d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_PROBLEM_IP "$( jq -r .PublicIpAddress <<< $AWS_EC2_PRODUCTION )" <span class='unselectable'>AWS_PROBLEM_IP=52.207.225.143</span></pre> <h3 class="numbered" id="problemVolumeId">Getting the AWS EC2 Problem Availability Zone</h3> <p> The AWS availability zone for the problem EC2 instance can be extracted from the JSON returned by the preceding results easily: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2a6a0e8109e3'><button class='copyBtn' data-clipboard-target='#id2a6a0e8109e3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_AVAILABILITY_ZONE "$( jq -r .Placement.AvailabilityZone <<< $AWS_EC2_PRODUCTION )" <span class='unselectable'>AWS_AVAILABILITY_ZONE=us-east-1c</span></pre> <h3 class="numbered" id="volumeId">Getting the AWS EC2 Problem Volume ID</h3> <p> The following command line extracts the volume id of the problem server&rsquo;s system drive into an environment variable called <code>$AWS_PROBLEM_VOLUME_ID</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd330097444d4'><button class='copyBtn' data-clipboard-target='#idd330097444d4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_PROBLEM_VOLUME_ID "$( jq -r '.BlockDeviceMappings[].Ebs.VolumeId' <<< "$AWS_EC2_PRODUCTION" )" <span class='unselectable'>AWS_PROBLEM_VOLUME_ID=vol-1c8903b4</span></pre> </editor-fold info> <editor-fold snapshotProblem> <h2 class="numbered" id="snapshotProblem">Make a Snapshot of the Problem Server</h2> <p> One approach, which would be living dangerously, would be to mount the system volume of the problem server on another server, set up <code>chroot</code>, attempt to repair the drive image, remount the repaired drive on the problem server, and reboot the server. I am never that optimistic. Things invariably go wrong. Instead, we will take a snapshot of the problem drive, turn the snapshot into a volume, repair the volume, swap in the repaired volume on the problem system, and reboot that system. </p> <p> It is better to shut down the EC2 instance before making a snapshot, however a snapshot can be taken whenever the server is idling. We will need to shut down the server anyway, so that could be done now, or at the last minute. </p> <p> I made a snapshot with a tag called <code>Name</code> with the value like <code>production 2020-10-25</code> and saved the snapshot id in an environment variable called <code>AWS_PROBLEM_SNAPSHOT_ID</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2b541357a2c9'><button class='copyBtn' data-clipboard-target='#id2b541357a2c9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_PROBLEM_SNAPSHOT_ID "$( aws ec2 create-snapshot --volume-id $AWS_PROBLEM_VOLUME_ID \ --description "production `date '+%Y-%m-%d'`" \ --tag-specifications "ResourceType=snapshot,Tags=[{Key=Created, Value=`date '+%Y-%m-%d'`},{Key=Name, Value=\"Broken do-release-upgrade 20.{04,10\"}]" | \ jq -r .SnapshotId )" <span class='unselectable'>$ </span>aws ec2 wait volume-available --volume-id $AWS_RESCUE_VOLUME_ID <span class='unselectable'>AWS_PROBLEM_SNAPSHOT_ID=snap-0a856be1f58b8a856</span></pre> <p> Snapshots only take a few minutes to complete. The <code>aws ec2 wait</code> command blocks until the specified operation finishes. </p> </editor-fold snapshotProblem> <editor-fold rescueVolume> <h2 class="numbered" id="rescueVolue">Create Rescue Volume From Snapshot</h2> <p> Once the snapshot process has completed, create a new volume from the snapshot. The default volume type is <a href='https://aws.amazon.com/ebs/features/#Amazon_EBS_volume_types' target='_blank' rel='nofollow'><code>gp2</code></a>. We&rsquo;ll refer to this volume as <code>$AWS_RESCUE_VOLUME_ID</code>. It is important to create the volume in the same availability zone as the problem EC2 instance so that it can easily be attached. This command applies a tag called <code>Name</code>, with the value <code>rescue</code>, for easy identification. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0529396538fa'><button class='copyBtn' data-clipboard-target='#id0529396538fa' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_RESCUE_VOLUME_ID "$( aws ec2 create-volume \ --availability-zone $AWS_AVAILABILITY_ZONE \ --snapshot-id $AWS_PROBLEM_SNAPSHOT_ID \ --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=rescue}]' | \ jq -r .VolumeId )" <span class='unselectable'>AWS_RESCUE_VOLUME_ID=vol-0e20fd22d2dc5a933</span> <span class='unselectable'>$ </span>aws ec2 wait volume-available --volume-id $AWS_RESCUE_VOLUME_ID</pre> </editor-fold rescueVolume> <editor-fold snapshotRescue> <h2 class="numbered" id="spot">Use an EC2 Spot Instance For the Rescue Server</h2> <p> Now that the rescue volume is <code>available</code>, we need to mount it on a server, which I&rsquo;ll call the rescue server. We&rsquo;ll refer to the server where the rescue volume is prepared via its instance id, saved as <code>AWS_EC2_RESCUE_ID</code>. You can either create a new EC2 instance for this purpose, or use an existing EC2 instance. </p> <p> The rescue server does not need to be anything special; a tiny virtual machine of any description will do fine. However, some rescue operations will be much easier if the type of operating system is the same as that on the problem drive. Volumes can be attached to running and stopped server instances. The load on the rescue server will likely be light and short-lived. An EC2 spot instance is ideal, and only costs two cents per hour! The spot instance will likely only be needed for 15 minutes. I specified my VPC id as <code>SubnetId</code>, the security group <code>sg-4cbc6f35</code> and the <code>AvailabilityZone</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8b517ed5234c'><button class='copyBtn' data-clipboard-target='#id8b517ed5234c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_EC2_RESCUE aws ec2 run-instances \ --image-id ami-0dba2cb6798deb6d8 \ --instance-market-options '{ "MarketType": "spot" }' \ --instance-type t2.small \ --key-name rsa-2020-11-03.pub \ --network-interfaces '[ { "DeviceIndex": 0, "Groups": ["sg-4cbc6f35"], "SubnetId": "subnet-49de033f", "DeleteOnTermination": true, "AssociatePublicIpAddress": true } ]' \ --placement '{ "AvailabilityZone": "us-east-1c" }' <span class='unselectable'>{ "Groups": [], "Instances": [ { "AmiLaunchIndex": 0, "ImageId": "ami-0dba2cb6798deb6d8", "InstanceId": "i-012a54aefcd333de9", "InstanceType": "t2.small", "KeyName": "rsa-2020-11-03.pub", "LaunchTime": "2020-11-03T23:19:50.000Z", "Monitoring": { "State": "disabled" }, "Placement": { "AvailabilityZone": "us-east-1c", "GroupName": "", "Tenancy": "default" }, "PrivateDnsName": "ip-10-0-0-210.ec2.internal", "PrivateIpAddress": "10.0.0.210", "ProductCodes": [], "PublicDnsName": "", "State": { "Code": 0, "Name": "pending" }, "StateTransitionReason": "", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "Architecture": "x86_64", "BlockDeviceMappings": [], "ClientToken": "026583fb-c94e-4bca-bdd2-8dcdcaa3aae9", "EbsOptimized": false, "EnaSupport": true, "Hypervisor": "xen", "InstanceLifecycle": "spot", "NetworkInterfaces": [ { "Attachment": { "AttachTime": "2020-11-03T23:19:50.000Z", "AttachmentId": "eni-attach-04feb4d36cf5c6792", "DeleteOnTermination": true, "DeviceIndex": 0, "Status": "attaching" }, "Description": "", "Groups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "Ipv6Addresses": [], "MacAddress": "0a:6d:ba:c5:65:4b", "NetworkInterfaceId": "eni-09ef90920cfb29dd9", "OwnerId": "031372724784", "PrivateIpAddress": "10.0.0.210", "PrivateIpAddresses": [ { "Primary": true, "PrivateIpAddress": "10.0.0.210" } ], "SourceDestCheck": true, "Status": "in-use", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "InterfaceType": "interface" } ], "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SecurityGroups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "SourceDestCheck": true, "SpotInstanceRequestId": "sir-rrs9gm3j", "StateReason": { "Code": "pending", "Message": "pending" }, "VirtualizationType": "hvm", "CpuOptions": { "CoreCount": 1, "ThreadsPerCore": 1 }, "CapacityReservationSpecification": { "CapacityReservationPreference": "open" }, "MetadataOptions": { "State": "pending", "HttpTokens": "optional", "HttpPutResponseHopLimit": 1, "HttpEndpoint": "enabled" } } ], "OwnerId": "031372724784", "ReservationId": "r-0d45e1919e7bad5c9" }</span></pre> <p> We need to retrieve the IP address of the newly created EC2 spot instance. This instance will disappear (terminate) once it shuts down, so do not reboot it. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8e4235a37a6c'><button class='copyBtn' data-clipboard-target='#id8e4235a37a6c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 describe-instances --instance-ids i-012a54aefcd333de9 <span class='unselectable'>{ "Reservations": [ { "Groups": [], "Instances": [ { "AmiLaunchIndex": 0, "ImageId": "ami-0dba2cb6798deb6d8", "InstanceId": "i-012a54aefcd333de9", "InstanceType": "t2.small", "KeyName": "rsa-2020-11-03.pub", "LaunchTime": "2020-11-03T23:19:50.000Z", "Monitoring": { "State": "disabled" }, "Placement": { "AvailabilityZone": "us-east-1c", "GroupName": "", "Tenancy": "default" }, "PrivateDnsName": "ip-10-0-0-210.ec2.internal", "PrivateIpAddress": "10.0.0.210", "ProductCodes": [], "PublicDnsName": "", "PublicIpAddress": "54.242.88.254", "State": { "Code": 16, "Name": "running" }, "StateTransitionReason": "", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "Architecture": "x86_64", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "AttachTime": "2020-11-03T23:19:51.000Z", "DeleteOnTermination": true, "Status": "attached", "VolumeId": "vol-0c44c8c009d1fafda" } } ], "ClientToken": "026583fb-c94e-4bca-bdd2-8dcdcaa3aae9", "EbsOptimized": false, "EnaSupport": true, "Hypervisor": "xen", "InstanceLifecycle": "spot", "NetworkInterfaces": [ { "Association": { "IpOwnerId": "amazon", "PublicDnsName": "", "PublicIp": "54.242.88.254" }, "Attachment": { "AttachTime": "2020-11-03T23:19:50.000Z", "AttachmentId": "eni-attach-04feb4d36cf5c6792", "DeleteOnTermination": true, "DeviceIndex": 0, "Status": "attached" }, "Description": "", "Groups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "Ipv6Addresses": [], "MacAddress": "0a:6d:ba:c5:65:4b", "NetworkInterfaceId": "eni-09ef90920cfb29dd9", "OwnerId": "031372724784", "PrivateIpAddress": "10.0.0.210", "PrivateIpAddresses": [ { "Association": { "IpOwnerId": "amazon", "PublicDnsName": "", "PublicIp": "54.242.88.254" }, "Primary": true, "PrivateIpAddress": "10.0.0.210" } ], "SourceDestCheck": true, "Status": "in-use", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "InterfaceType": "interface" } ], "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SecurityGroups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "SourceDestCheck": true, "SpotInstanceRequestId": "sir-rrs9gm3j", "VirtualizationType": "hvm", "CpuOptions": { "CoreCount": 1, "ThreadsPerCore": 1 }, "CapacityReservationSpecification": { "CapacityReservationPreference": "open" }, "HibernationOptions": { "Configured": false }, "MetadataOptions": { "State": "applied", "HttpTokens": "optional", "HttpPutResponseHopLimit": 1, "HttpEndpoint": "enabled" } } ], "OwnerId": "031372724784", "ReservationId": "r-0d45e1919e7bad5c9" } ] }</span></pre> <!-- <p> It is possible that the work necessary to rescue the problem disk image might make changes to the rescue system. The rescue system should therefore have a snapshot taken before going any further. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida8ee9f754962'><button class='copyBtn' data-clipboard-target='#ida8ee9f754962' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_RESCUE_SNAPSHOT_ID "$( aws ec2 create-snapshot --volume-id $AWS_RESCUE_VOLUME_ID %} \ --description "production `date '+%Y-%m-%d'`" \ --tag-specifications "ResourceType=snapshot,Tags=[{Key=Created, Value=`date '+%Y-%m-%d'`},{Key=Name, Value=\"Broken do-release-upgrade 20.{04,10\"}]" | \ jq -r .SnapshotId )" <span class='unselectable'>AWS_RESCUE_SNAPSHOT_ID=snap-0a856be1f58b8359a</span> <span class='unselectable'>$ </span>aws ec2 wait snapshot-completed --snapshot-ids $AWS_RESCUE_SNAPSHOT_ID</pre> --> </editor-fold snapshotRescue> <editor-fold mount> <h2 class="numbered" id="mount">Mount the Rescue Volume On the Rescue Server</h2> <!--<p> We need to select a device name for the rescue disk. The name depends on what names are already in use on the rescue server. After logging into the rescue server, I ran the <code>lsblk</code> Linux command to see the available disk devices and their mount points. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7d35b69545ce'><span class='unselectable'>$ </span>lsblk <span class="unselectable">NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop1 7:1 0 53.1M 1 loop /snap/lxd/10984 loop2 7:2 0 88.4M 1 loop /snap/core/7169 loop3 7:3 0 97.8M 1 loop /snap/core/10185 loop4 7:4 0 53.1M 1 loop /snap/lxd/11348 xvda 202:0 0 8G 0 disk └─xvda1 202:1 0 8G 0 part /</span></pre> <p> The lsblk output does not show full device paths, instead, the <code>/dev/</code> prefix is omitted. With that in mind we can see that the only <code>disk</code> device on the rescue server is <code>/dev/xvda</code>, and its only partition called <code>/dev/xvda1</code> is mounted on the root directory. Because Linux drives are normally named sequentially, we should name the rescue disk <code>/dev/xvdb</code>. Let&rsquo;s define an environment variable called <code>AWS_RESCUE_DRIVE</code> to memorialize that decision. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id910ead67481c'><span class='unselectable'>$ </span>AWS_RESCUE_DRIVE=/dev/xvdb</pre> --> <p> The <code>aws ec2 attach-volume</code> command will attach the rescue volume to the rescue server. It automatically selects an appropriate device name for the rescue volume, which in the following example is <code>/dev/xvdb</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id598af680f657'><button class='copyBtn' data-clipboard-target='#id598af680f657' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_ATTACH_VOLUME "$( aws ec2 attach-volume \ --device $AWS_RESCUE_DRIVE \ --instance-id $AWS_EC2_RESCUE_ID \ --volume-id $AWS_RESCUE_VOLUME_ID )" <span class='unselectable'>AWS_ATTACH_VOLUME={ "AttachTime": "2020-10-26T14:34:55.222Z", "InstanceId": "i-d3b03954", "VolumeId": "vol-0e20fd22d2dc5a933", "State": "attaching", "Device": "/dev/xvdb" }</span> <span class='unselectable'>$ </span>aws ec2 wait volume-in-use --volume-id $AWS_RESCUE_VOLUME_ID</pre> <p> We need to use the rescue volume&rsquo;s device name later, so we&rsquo;ll save it in <code>AWS_RESCUE_DRIVE</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide197a86635cf'><button class='copyBtn' data-clipboard-target='#ide197a86635cf' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_RESCUE_DRIVE "$( jq -r .Device <<< $AWS_ATTACH_VOLUME )" <span class='unselectable'>AWS_RESCUE_DRIVE=/dev/xvdb</span></pre> <p> The details of the mounted rescue drive are provided by <code>fdisk -l</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id02ae49f65ecb'><button class='copyBtn' data-clipboard-target='#id02ae49f65ecb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo fdisk -l | sed -n -e '/xvdb/,$p' <span class='unselectable'>Disk /dev/xvdb: 12 GiB, 12884901888 bytes, 25165824 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x00000000 Device Boot Start End Sectors Size Id Type /dev/xvdb1 * 16065 25165790 25149726 12G 83 Linux</span></pre> <p> Now it is time to mount the rescue drive on the rescue server. Ubuntu has a directory called <code>/mnt</code> whose purpose is to act as a mount point: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4723ca84d8b7'><button class='copyBtn' data-clipboard-target='#id4723ca84d8b7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo mount /dev/xvdb1 /mnt</pre> <p> Let&rsquo;s confirm that the drive is mounted: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id05e27523319f'><button class='copyBtn' data-clipboard-target='#id05e27523319f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>df -h | grep '^/dev/' | grep -v '^/dev/loop' <span class='unselectable'>/dev/xvda1 7.8G 6.3G 1.1G 86% / /dev/xvdb1 12G 9.0G 2.2G 82% /mnt</span></pre> <p> The last line shows that this drive is mounted on <code>/mnt</code> and it is 82% full. </p> </editor-fold mount> <editor-fold chroot> <h2 class="numbered" id="chroot">Set Up a <span class='code'>chroot</span> to Establish an Environment for Making Repairs</h2> <p> We need to mount some more file systems before we perform the <code>chroot</code>. The following mounts the rescue server&rsquo;s <code>/dev</code>, <code>/dev/shm</code>, <code>/sys</code>, and <code>/run</code> to the same paths within the rescue volume. Because programs like <code>do-release-upgrade</code> need a <code>tty</code>, I also mount <code>devtps</code> and <code>proc</code>. These mounts only last until the next server reboot. After all the mounts the <code>chroot</code> is issued. </p> <p class="warning"> <b>Warning</b> - mounting <code>/run</code> and then updating the system on the rescue disk from within a chroot may change the host system&rsquo;s <code>/run</code> contents; if the package managers (<code>apt</code> and <code>dpkg</code>) get out of sync with the actual state on the host system you won&rsquo;t be able to update the host system until you restore the host system&rsquo;s image from the snapshot that we made <a href='#snapshotRescue'>earlier</a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb1a9f1e6df07'><button class='copyBtn' data-clipboard-target='#idb1a9f1e6df07' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo mount -o bind /dev /mnt/dev <span class='unselectable'>$ </span>sudo mount -o bind /dev/shm /mnt/dev/shm <span class='unselectable'>$ </span>sudo mount -o bind /sys /mnt/sys <span class='unselectable'>$ </span>sudo mount -o bind /run /mnt/run <span class='unselectable'>$ </span>sudo mount -t proc proc /mnt/proc <span class='unselectable'>$ </span>sudo mount -t devpts devpts /mnt/dev/pts <span class='unselectable'>$ </span>sudo chroot /mnt <span class='unselectable'>root@ip-10-0-0-189:/#</span></pre> <p> Notice how the prompt changed after the <code>chroot</code>. That is your clue that it is active. </p> <!-- <p> I edited <code>/etc/hosts</code> in the <code>chroot</code> to add the name of the system to the entry for <code>localhost</code> (<code>127.0.0.1</code>): </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id343e962abbda'><button class='copyBtn' data-clipboard-target='#id343e962abbda' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>127.0.0.1 localhost ip-10-0-0-189</pre> --> </editor-fold chroot> <editor-fold fix> <h2 class="numbered" id="fix">Correct the Problem</h2> <p> This step depends on whatever is wrong. I won&rsquo;t bore you with the problem I had. </p> </editor-fold fix> <editor-fold unmountRescue> <h2 class="numbered" id="unmount">Unmount the New Volume</h2> <p> Exit the <code>chroot</code> and unmount the rescue volume from the rescue server. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id321f908517b8'><button class='copyBtn' data-clipboard-target='#id321f908517b8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'># </span>exit <span class='unselectable'>$ </span>sudo umount /mnt/dev <span class='unselectable'>$ </span>sudo umount /mnt/dev/shm <span class='unselectable'>$ </span>sudo umount /mnt/sys <span class='unselectable'>$ </span>sudo umount /mnt/run <span class='unselectable'>$ </span>sudo umount /mnt/proc <span class='unselectable'>$ </span>sudo umount /mnt/dev/pts <span class='unselectable'>$ </span>sudo umount /mnt</pre> <p> Detach the rescue volume from the rescue server. This can be done from any machine that is configured with <code>aws cli</code> for use with your account credentials. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id46b142e3cfe6'><button class='copyBtn' data-clipboard-target='#id46b142e3cfe6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 detach-volume --volume-id $AWS_RESCUE_VOLUME_ID <span class='unselectable'>$ </span>aws ec2 wait volume-available --volume-id $AWS_RESCUE_VOLUME_ID</pre> </editor-fold unmountRescue> <editor-fold unmountProblem> <h2 class="numbered" id="unmount">Unmount the Problem Volume</h2> <p> The problem server must be shut down for this to work. Detach the problem volume from the problem server. This can be done from any machine that is configured with <code>aws cli</code> for use with your account credentials. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idaf4ecf5bee6b'><button class='copyBtn' data-clipboard-target='#idaf4ecf5bee6b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 stop-instances --instance-id $AWS_PROBLEM_INSTANCE_ID <span class='unselectable'>$ </span>aws ec2 wait instance-stopped --instance-ids $AWS_PROBLEM_INSTANCE_ID <span class='unselectable'>$ </span>aws ec2 detach-volume --volume-id $AWS_PROBLEM_VOLUME_ID <span class='unselectable'>$ </span>aws ec2 wait volume-available --volume-id $AWS_PROBLEM_VOLUME_ID</pre> </editor-fold unmountProblem> <editor-fold replace> <h2 class="numbered" id="replace">Replace the Problem Volume</h2> <p> Now it is time to replace the problem volume containing the problem boot drive on the problem system with the newly created volume. AWS EC2 always refers to boot drives as <code>/dev/sda1</code>, even when the device has a different name, such as <code>/dev/xvdb1</code>. </p> <h3 id="replaceSystemVolume"><span>replaceSystemVolume</span> Bash function</h3> <p> This Bash function detaches the volume containing the current boot drive of an EC2 instance and replaces it with another volume. If the EC2 instance is running then it is first stopped. </p> <p>Paste the above into</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2756396e1534'><button class='copyBtn' data-clipboard-target='#id2756396e1534' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>replaceSystemVolume "$AWS_PROBLEM_INSTANCE_ID" "$AWS_RESCUE_VOLUME_ID"</pre> Preview 2 instance id is <code>AWS_EC2_RESCUE_ID</code>. Replace rescue volume on preview with preview's original volume: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id18ff0151d9b0'><button class='copyBtn' data-clipboard-target='#id18ff0151d9b0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>replaceSystemVolume "$AWS_EC2_RESCUE_ID" "$AWS_PREVIEW_VOLUME_ID"</pre> </editor-fold replace> <editor-fold boot> <h2 class="numbered" id="boot">Boot the problem system</h2> <p> Boot the problem system and verify the problem is solved. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd97732ba3021'><button class='copyBtn' data-clipboard-target='#idd97732ba3021' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 start-instances --instance-ids $AWS_PROBLEM_INSTANCE_ID <span class='unselectable'>$ </span>aws ec2 wait instance-started --instance-ids $AWS_PROBLEM_INSTANCE_ID</pre> </editor-fold boot> <editor-fold acknowledgements> <h2 id="acknowledgements">Acknowledgements</h2> <p> This article was inspired by <a href='https://www.rootusers.com/how-to-repair-an-aws-ec2-instance-without-console' target='_blank'>this excellent article</a>, which uses the AWS web console to achieve similar results. </p> </editor-fold acknowledgements> Working With EC2 Spot Instances From AWS CLI 2020-10-24T00:00:00-04:00 https://mslinn.github.io/blog/2020/10/24/ec2-spot-instances-cli <editor-fold Intro> <p> AWS EC2 <code>T2.medium</code> spot instances <a href='https://aws.amazon.com/ec2/spot/pricing/' target='_blank' rel='nofollow'>cost less than 2 cents per hour</a> for Linux and can be created very easily from the command line. They self-destruct once shut down. These powerful virtual machines can do an incredible amount of work in an hour for less than 2 cents! </p> <p> Permanent storage can either be on S3 or an EBS volume, which can easily be mounted. This article shows how all this can be done via the command line. I also provide an <a href='#createEc2Spot'>interactive bash script</a> for automating this process. </p> </editor-fold Intro> <editor-fold nameKeys> <h2 class="numbered" id="nameKeys">Create and Import a New Keypair</h2> <p> I want to create a new temporary ssh keypair that will just be used for this spot instance. The name of the new key pair will be of the form <code>~/.ssh/rsa-YYYY-MM-DD-mm-ss</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2e50841a43cc'><button class='copyBtn' data-clipboard-target='#id2e50841a43cc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_KEY_PAIR_NAME rsa-$( date '+%Y-%m-%d-%M-%S' ) <span class='unselectable'>AWS_KEY_PAIR_NAME=rsa-2020-11-04-43-54</span> <span class='unselectable'>$ </span>mem AWS_KEY_PAIR_FILE ~/.ssh/$AWS_KEY_PAIR_NAME <span class='unselectable'>AWS_KEY_PAIR_FILE=~/.ssh/rsa-2020-11-04-43-54</span></pre> <p> Now we can make the keypair. AWS EC2 does not accept keys longer than 2048 bits. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id518d449970f8'><button class='copyBtn' data-clipboard-target='#id518d449970f8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ssh-keygen -b 2048 -f "$AWS_KEY_PAIR_FILE" -P "" -N "" -t rsa <span class='unselectable'>Generating public/private rsa key pair. Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-43-54 Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-43-54.pub The key fingerprint is: SHA256:bQScX0UMn0xGDorxSvElMZzwMyyk7hgs2FNbshBNenA mslinn@mslinn.com The key's randomart image is: +---[RSA 4096]----+ | ooE .*++o+** | | =. ooXo=.B.. | | o + o +.X. = | | o = * . =.o | |. + = . S o | | o + . | | . . | | | | | +----[SHA256]-----+</span></pre> <p> The new public key will be stored in <code>~/.ssh/2020-11-04-43-54.pub</code> and the new private key will be stored in <code>~/.ssh/2020-11-04-43-54</code>. </p> <p> Now set the permissions for the key. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8690ff0f732c'><button class='copyBtn' data-clipboard-target='#id8690ff0f732c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>chmod 400 $AWS_KEY_PAIR_FILE</pre> <p> Now we can import the key pair into AWS: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id91f3fbf72cb6'><button class='copyBtn' data-clipboard-target='#id91f3fbf72cb6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 import-key-pair \ --key-name $AWS_KEY_PAIR_NAME \ --public-key-material fileb://${AWS_KEY_PAIR_FILE}.pub <span class='unselectable'>{ "KeyFingerprint": "c7:76:90:53:17:d0:fc:ba:45:dd:93:d2:93:03:c2:19", "KeyName": "2020-11-04-43-54", "KeyPairId": "key-092a2306ec3f4aff6" }</span></pre> </editor-fold nameKeys> <editor-fold AMI> <h2 class="numbered" id="ami">Select an AMI</h2> <p> New <a href='https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html' target='_blank' rel='nofollow'>AMIs</a> become available every day. You probably want your EC2 spot instance to be created from the most recent AMI that matches your needs. For most of my work I want an Ubuntu 64-bit Intel/AMD server distribution. <a href='https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/finding-an-ami.html' target='_blank' rel='nofollow'>AWS documentation</a> is helpful and gives us a head start in automating the AMI selection. </p> <p> The following incantation sets an environment variable called <code>AWS_AMI</code> to the details in JSON syntax of the AMI for the most recent 64-bit Ubuntu server release for Intel/AMD architecture. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9888ae442f1f'><button class='copyBtn' data-clipboard-target='#id9888ae442f1f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_AMI "$( aws ec2 describe-images \ --owners 099720109477 \ --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-????????-????????-amd64-server-????????" \ "Name=state,Values=available" \ --query "reverse(sort_by(Images, &CreationDate))[:1]" | \ jq -r '.[0]' )" <span class='unselectable'>AWS_AMI={ "Architecture": "x86_64", "CreationDate": "2020-10-30T14:07:42.000Z", "ImageId": "ami-0c71ec98278087e60", "ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030", "ImageType": "machine", "Public": true, "OwnerId": "099720109477", "PlatformDetails": "Linux/UNIX", "UsageOperation": "RunInstances", "State": "available", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "DeleteOnTermination": true, "SnapshotId": "snap-00bf581086dd686e5", "VolumeSize": 8, "VolumeType": "gp2", "Encrypted": false } }, { "DeviceName": "/dev/sdb", "VirtualName": "ephemeral0" }, { "DeviceName": "/dev/sdc", "VirtualName": "ephemeral1" } ], "Description": "Canonical, Ubuntu, 20.10, amd64 groovy image build on 2020-10-30", "EnaSupport": true, "Hypervisor": "xen", "Name": "ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030", "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SriovNetSupport": "simple", "VirtualizationType": "hvm" }</span></pre> <p> Now let's extract the ID of the AMI image and save it as <code>AWS_AMI_ID</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb51f7332670b'><button class='copyBtn' data-clipboard-target='#idb51f7332670b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_AMI_ID "$( jq -r '.[0].ImageId' <<< "$AWS_AMI" )" <span class='unselectable'>AWS_AMI_ID=ami-0c71ec98278087e60</span></pre> </editor-fold AMI> <editor-fold Spot> <h2 class="numbered" id="create">Create an EC2 Spot Instance</h2> <p> For my work I often want my spot instance to be created in the same VPC subnet as my other resources, with the same security group. That is why the following environment variables are defined for the <code>Groups</code> and <code>SubnetId</code> values within the <code>network-interfaces</code> option, as well as the AWS region. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9ac1df02e320'><button class='copyBtn' data-clipboard-target='#id9ac1df02e320' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_GROUP sg-4cbc6f35 <span class='unselectable'>AWS_GROUP=sg-4cbc6f35</span> <span class='unselectable'>$ </span>mem AWS_SUBNET subnet-49de033f <span class='unselectable'>AWS_SUBNET=subnet-49de033f</span> <span class='unselectable'>$ </span>mem AWS_ZONE us-east-1c <span class='unselectable'>AWS_ZONE=us-east-1c</span> <span class='unselectable'>$ </span>mem AWS_EC2_TYPE t2.medium <span class='unselectable'>AWS_EC2_TYPE t2.medium</span></pre> <p> The following creates an AWS EC2 spot instance with a public IP address and runs it. Details about the newly created spot instance are stored as JSON in <code>AWS_SPOT_INSTANCE</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfb0262fb6319'><button class='copyBtn' data-clipboard-target='#idfb0262fb6319' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_SPOT_INSTANCE "$( aws ec2 run-instances \ --image-id $AWS_AMI_ID \ --instance-market-options '{ "MarketType": "spot" }' \ --instance-type $AWS_EC2_TYPE \ --key-name $AWS_KEY_PAIR_NAME \ --network-interfaces "[ { \"DeviceIndex\": 0, \"Groups\": [\"$AWS_GROUP\"], \"SubnetId\": \"$AWS_SUBNET\", \"DeleteOnTermination\": true, \"AssociatePublicIpAddress\": true } ]" \ --placement "{ \"AvailabilityZone\": \"$AWS_ZONE\" }" | \ jq -r .Instances[0] )" <span class='unselectable'>AWS_SPOT_INSTANCE={ "AmiLaunchIndex": 0, "ImageId": "ami-0dba2cb6798deb6d8", "InstanceId": "i-012a54aefcd333de9", "InstanceType": "t2.small", "KeyName": "rsa-2020-11-03.pub", "LaunchTime": "2020-11-03T23:19:50.000Z", "Monitoring": { "State": "disabled" }, "Placement": { "AvailabilityZone": "us-east-1c", "GroupName": "", "Tenancy": "default" }, "PrivateDnsName": "ip-10-0-0-210.ec2.internal", "PrivateIpAddress": "10.0.0.210", "ProductCodes": [], "PublicDnsName": "", "State": { "Code": 0, "Name": "pending" }, "StateTransitionReason": "", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "Architecture": "x86_64", "BlockDeviceMappings": [], "ClientToken": "026583fb-c94e-4bca-bdd2-8dcdcaa3aae9", "EbsOptimized": false, "EnaSupport": true, "Hypervisor": "xen", "InstanceLifecycle": "spot", "NetworkInterfaces": [ { "Attachment": { "AttachTime": "2020-11-03T23:19:50.000Z", "AttachmentId": "eni-attach-04feb4d36cf5c6792", "DeleteOnTermination": true, "DeviceIndex": 0, "Status": "attaching" }, "Description": "", "Groups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "Ipv6Addresses": [], "MacAddress": "0a:6d:ba:c5:65:4b", "NetworkInterfaceId": "eni-09ef90920cfb29dd9", "OwnerId": "031372724784", "PrivateIpAddress": "10.0.0.210", "PrivateIpAddresses": [ { "Primary": true, "PrivateIpAddress": "10.0.0.210" } ], "SourceDestCheck": true, "Status": "in-use", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "InterfaceType": "interface" } ], "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SecurityGroups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "SourceDestCheck": true, "SpotInstanceRequestId": "sir-rrs9gm3j", "StateReason": { "Code": "pending", "Message": "pending" }, "VirtualizationType": "hvm", "CpuOptions": { "CoreCount": 1, "ThreadsPerCore": 1 }, "CapacityReservationSpecification": { "CapacityReservationPreference": "open" }, "MetadataOptions": { "State": "pending", "HttpTokens": "optional", "HttpPutResponseHopLimit": 1, "HttpEndpoint": "enabled" } }</span></pre> <p> Now extract the EC2 spot instance id and save it in <code>AWS_SPOT_ID</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8c44793fc20c'><button class='copyBtn' data-clipboard-target='#id8c44793fc20c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_SPOT_ID "$( jq -r .InstanceId <<< "$AWS_SPOT_INSTANCE" )" <span class='unselectable'>i-012a54aefcd333de9</span></pre> <p> Wait for the instance to start. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id55ec8eead200'><button class='copyBtn' data-clipboard-target='#id55ec8eead200' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 wait instance-running --instance-ids $AWS_SPOT_ID</pre> </editor-fold Spot> <editor-fold Connect> <h2 class="numbered" id="connect">Connect to the Spot Instance</h2> <p> In order to <code>ssh</code> into the spot instance we first need to discover its IP address, which is saved in <code>AWS_SPOT_IP</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddddf26bebea1'><button class='copyBtn' data-clipboard-target='#iddddf26bebea1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_SPOT_IP "$( aws ec2 describe-instances \ --instance-ids $AWS_SPOT_ID | \ jq -r .Reservations[0].Instances[0].PublicIpAddress )" <span class='unselectable'>54.242.88.254</span></pre> <p> Now we can connect to the spot instance via <code>ssh</code>. The default userid for Ubuntu is <code>ubuntu</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4fbb54074f6c'><button class='copyBtn' data-clipboard-target='#id4fbb54074f6c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ssh ubuntu@$AWS_SPOT_IP</pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1e45c053c4b2'><button class='copyBtn' data-clipboard-target='#id1e45c053c4b2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span># Do your work on the spot instance now</pre> <p>We'll disconnect and clean up next.</p> </editor-fold Connect> <editor-fold Disconnect> <h2 class="numbered" id="connect">Disconnect from the Spot Instance and Clean Up</h2> <p> Once the spot instance stops it is automatically terminated. The instance will survive a <code>reboot</code>, but not a <code>halt</code>. From a prompt on the spot instance, type: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id449b825e7be1'><button class='copyBtn' data-clipboard-target='#id449b825e7be1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo halt</pre> <p> Back in the shell that launched the spot instance, wait for the spot instance to stop before cleaning up. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id052668cfda62'><button class='copyBtn' data-clipboard-target='#id052668cfda62' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 wait instance-stopped --instance-ids $AWS_SPOT_ID</pre> <p> Delete the temporary <code>ssh</code> keypair we created. Copies exist on AWS and the local machine; we need to remove all of them, like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id19e4c5718a3c'><button class='copyBtn' data-clipboard-target='#id19e4c5718a3c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 delete-key-pair --key-name $AWS_KEY_PAIR_NAME <span class='unselectable'>$ </span>rm $AWS_KEY_PAIR_FILE $AWS_KEY_PAIR_FILE.pub</pre> </editor-fold Disconnect> <editor-fold Script> <h2 class="numbered" id="createEc2Spot">Bash Script <span class="code">createEc2Spot</span></h2> <h3 class="numbered" id="createEc2Spot_code">Source Code</h3> <p> This script does everything discussed above, plus it prompts the user with default values for parameters unique to each invocation. Click on the name of the script and save it this script to a directory on your <code>PATH</code>. </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/factor12/createEc2Spot" download="createEc2Spot" title="Click on the file name to download the file">createEc2Spot</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id2be883747776"><button class='copyBtn' data-clipboard-target='#id2be883747776' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash set -e function readWithDefault &#123; >&2 printf "\n$1: " read -e -i "$2" VALUE echo "$VALUE" &#125; echo "Please answer a few questions so the AWS EC2 spot instance can be created." mem AWS_GROUP "$( readWithDefault "AWS security group" sg-4cbc6f35 )" mem AWS_SUBNET "$( readWithDefault "EC2 subnet" subnet-49de033f )" mem AWS_ZONE "$( readWithDefault "AWS availability zone" us-east-1c )" mem AWS_EC2_TYPE "$( readWithDefault "EC2 machine type" t2.medium )" mem AWS_KEY_PAIR_NAME rsa-$( date '+%Y-%m-%d-%M-%S' ) mem AWS_KEY_PAIR_FILE ~/.ssh/$AWS_KEY_PAIR_NAME ssh-keygen -b 4096 -f "$AWS_KEY_PAIR_FILE" -P "" -N "" -t rsa chmod 400 $AWS_KEY_PAIR_FILE aws ec2 import-key-pair \ --key-name $AWS_KEY_PAIR_NAME \ --public-key-material fileb://$AWS_KEY_PAIR_FILE echo "Searching for the latest 64-bit Intel/AMD Ubuntu AMI." mem AWS_AMI "$( aws ec2 describe-images \ --owners 099720109477 \ --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-????????-????????-amd64-server-????????" \ "Name=state,Values=available" \ --query "reverse(sort_by(Images, &CreationDate))[:1]" | \ jq -r '.[0]' )" source .env echo "Obtaining the AMI image ID." mem -d AWS_AMI_ID "$( jq -r '.ImageId' &lt;&lt;&lt; "$AWS_AMI" )" source .env echo "Creating the EC2 spot instance." mem AWS_SPOT_INSTANCE "$( aws ec2 run-instances \ --image-id $AWS_AMI_ID \ --instance-market-options '&#123; "MarketType": "spot" &#125;' \ --instance-type $AWS_EC2_TYPE \ --key-name $AWS_KEY_PAIR_NAME \ --network-interfaces "[ &#123; \"DeviceIndex\": 0, \"Groups\": [\"$AWS_GROUP\"], \"SubnetId\": \"$AWS_SUBNET\", \"DeleteOnTermination\": true, \"AssociatePublicIpAddress\": true &#125; ]" \ --placement "&#123; \"AvailabilityZone\": \"$AWS_ZONE\" &#125;" | \ jq -r .Instances[0] )" source .env echo "Obtaining the EC2 spot instance ID." mem AWS_SPOT_ID "$( jq -r .InstanceId &lt;&lt;&lt; "$AWS_SPOT_INSTANCE" )" source .env echo "Awaiting for the EC2 spot instance $AWS_SPOT_ID to enter the running state." aws ec2 wait instance-running --instance-ids $AWS_SPOT_ID echo "Obtaining the IP address of the new EC2 spot instance $AWS_SPOT_ID." mem AWS_SPOT_IP "$( aws ec2 describe-instances \ --instance-ids $AWS_SPOT_ID | \ jq -r .Reservations[0].Instances[0].PublicIpAddress )" source .env echo "About to ssh to the EC2 spot instance as ubuntu@$AWS_SPOT_IP. echo "When you are done, type: sudo halt." echo "The spot instance will then terminate and be gone forever." echo "Any predefined resources, such as volumes that you attach will be freed." ssh ubuntu@AWS_SPOT_IP echo "Waiting for the EC2 spot instance $AWS_SPOT_ID to enter the stopped state." aws ec2 wait instance-stopped --instance-ids $INSTANCE_ID echo "The spot instance is no longer available. Deleting its keypair." aws ec2 delete-key-pair --key-name AWS_KEY_PAIR_NAME rm $AWS_KEY_PAIR_FILE $AWS_KEY_PAIR_FILE.pub </pre> <p>Make the script executable.</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id572f4a61994a'><button class='copyBtn' data-clipboard-target='#id572f4a61994a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>chmod a+x createEc2Spot</pre> <h3 class="numbered" id="createEc2Spot_usage">Sample Usage</h3> <p> The script is easy to use: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd0486232b179'><button class='copyBtn' data-clipboard-target='#idd0486232b179' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>createEc2Spot <span class='unselectable'>Just a few questions before the AWS EC2 spot instance can be created. AWS security group: </span>sg-4cbc6f35 <span class='unselectable'>AWS_GROUP='sg-4cbc6f35' EC2 subnet: </span>subnet-49de033f <span class='unselectable'>AWS_SUBNET='subnet-49de033f' AWS availability zone: </span>us-east-1c <span class='unselectable'>AWS_ZONE='us-east-1c' EC2 machine type: </span>t2.medium <span class='unselectable'>AWS_EC2_TYPE='t2.medium' AWS_KEY_PAIR_NAME='rsa-2020-11-04-46-00' AWS_KEY_PAIR_FILE='/home/mslinn/.ssh/rsa-2020-11-04-46-00' Generating public/private rsa key pair. Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-43-54 Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-43-54.pub The key fingerprint is: SHA256:bQScX0UMn0xGDorxSvElMZzwMyyk7hgs2FNbshBNenA mslinn@mslinn.com The key's randomart image is: +---[RSA 4096]----+ | ooE .*++o+** | | =. ooXo=.B.. | | o + o +.X. = | | o = * . =.o | |. + = . S o | | o + . | | . . | | | | | +----[SHA256]-----+ { "KeyName": "2020-11-04-43-54", "KeyFingerprint": "1f:51:ae:28:bf:89:e9:d8:1f:25:5d:37:2d:7d:b8:ca" } Searching for the latest 64-bit Intel/AMD Ubuntu AMI. AWS_AMI='{ "Architecture": "x86_64", "CreationDate": "2020-10-30T14:07:42.000Z", "ImageId": "ami-0c71ec98278087e60", "ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030", "ImageType": "machine", "Public": true, "OwnerId": "099720109477", "PlatformDetails": "Linux/UNIX", "UsageOperation": "RunInstances", "State": "available", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "DeleteOnTermination": true, "SnapshotId": "snap-00bf581086dd686e5", "VolumeSize": 8, "VolumeType": "gp2", "Encrypted": false } }, { "DeviceName": "/dev/sdb", "VirtualName": "ephemeral0" }, { "DeviceName": "/dev/sdc", "VirtualName": "ephemeral1" } ], "Description": "Canonical, Ubuntu, 20.10, amd64 groovy image build on 2020-10-30", "EnaSupport": true, "Hypervisor": "xen", "Name": "ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030", "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SriovNetSupport": "simple", "VirtualizationType": "hvm" }' Obtaining the AMI image ID. AWS_AMI_ID='ami-0c71ec98278087e60' Creating the EC2 spot instance. AWS_SPOT_INSTANCE={ "AmiLaunchIndex": 0, "ImageId": "ami-0dba2cb6798deb6d8", "InstanceId": "i-012a54aefcd333de9", "InstanceType": "t2.small", "KeyName": "rsa-2020-11-03.pub", "LaunchTime": "2020-11-03T23:19:50.000Z", "Monitoring": { "State": "disabled" }, "Placement": { "AvailabilityZone": "us-east-1c", "GroupName": "", "Tenancy": "default" }, "PrivateDnsName": "ip-10-0-0-210.ec2.internal", "PrivateIpAddress": "10.0.0.210", "ProductCodes": [], "PublicDnsName": "", "State": { "Code": 0, "Name": "pending" }, "StateTransitionReason": "", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "Architecture": "x86_64", "BlockDeviceMappings": [], "ClientToken": "026583fb-c94e-4bca-bdd2-8dcdcaa3aae9", "EbsOptimized": false, "EnaSupport": true, "Hypervisor": "xen", "InstanceLifecycle": "spot", "NetworkInterfaces": [ { "Attachment": { "AttachTime": "2020-11-03T23:19:50.000Z", "AttachmentId": "eni-attach-04feb4d36cf5c6792", "DeleteOnTermination": true, "DeviceIndex": 0, "Status": "attaching" }, "Description": "", "Groups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "Ipv6Addresses": [], "MacAddress": "0a:6d:ba:c5:65:4b", "NetworkInterfaceId": "eni-09ef90920cfb29dd9", "OwnerId": "031372724784", "PrivateIpAddress": "10.0.0.210", "PrivateIpAddresses": [ { "Primary": true, "PrivateIpAddress": "10.0.0.210" } ], "SourceDestCheck": true, "Status": "in-use", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "InterfaceType": "interface" } ], "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SecurityGroups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "SourceDestCheck": true, "SpotInstanceRequestId": "sir-rrs9gm3j", "StateReason": { "Code": "pending", "Message": "pending" }, "VirtualizationType": "hvm", "CpuOptions": { "CoreCount": 1, "ThreadsPerCore": 1 }, "CapacityReservationSpecification": { "CapacityReservationPreference": "open" }, "MetadataOptions": { "State": "pending", "HttpTokens": "optional", "HttpPutResponseHopLimit": 1, "HttpEndpoint": "enabled" } } Obtaining the EC2 spot instance ID. i-012a54aefcd333de9 Awaiting for the EC2 spot instance i-012a54aefcd333de9 to enter the running state. Obtaining the IP address of the new EC2 spot instance i-012a54aefcd333de9D. 54.242.88.254 When you are done, type: sudo halt. The spot instance will then terminate and be gone forever. Any predefined resources, such as volumes that you attach will be freed. Waiting for the EC2 spot instance $AWS_SPOT_ID to enter the stopped state. The spot instance is no longer available. Deleting its keypair.</span></pre> </editor-fold Script> Scala-Style Lambda Function Placeholder Syntax in Python 3 2020-10-22T00:00:00-04:00 https://mslinn.github.io/blog/2020/10/22/scala-style-functional-programming-in-python-3 <p> Pipes are the ultimate in functional programming. It is clear when reading code that uses pipes that data is not mutated. Piping data into and out of lambda functions (and regular functions) is a succinct way of elegantly expressing a computation. This article discusses how to do this using similar syntax in Python 3, Scala 2 and Scala 3. </p> <p> After programming in Scala for more than ten years I have grown to appreciate Scala's implementation of lambda functions, including the ability to use the underscore character as a placeholder for variables. Scala 2.13 introduced pipelining between functions, which is rather like <a href='https://en.wikipedia.org/wiki/Unix-like' target='_blank' rel='nofollow'>*nix</a> pipes between processes. </p> <p> Python 3 can also do something similar. This article demonstrates how to use <a href='https://github.com/sspipe/sspipe' target='_blank' rel='nofollow'><code>sspipe</code></a> and <a href='https://github.com/JulienPalard/Pipe' target='_blank' rel='nofollow'>JulienPalard&rsquo;s <code>pipe</code></a> with Scala's underscore placeholder for <a href='https://stackoverflow.com/questions/29767310/pythons-lambda-with-underscore-for-an-argument' target='_blank' rel='nofollow'>Python 3 lambda functions</a>. </p> <h2 id="pythonSetup">Python 3 Setup</h2> The key concept is to use this specific Python import: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida9d9158d65b7'><button class='copyBtn' data-clipboard-target='#ida9d9158d65b7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>from sspipe import p, px, px as _</pre> <p> All the Python code examples that follow require this import. The Python code examples are modified versions of the <a href='https://github.com/sspipe/sspipe#examples' target='_blank' rel='nofollow'><code>sspipe</code> examples</a> to illustrate how to use underscores as placeholders. </p> <p> This import is unusual because it imports <code>px</code> twice: once as a normal import, and once aliased to <code>_</code>. I use the <code>_</code> alias to support Scala-like syntax, and I use <code>px</code> when I need to reference a parameter twice. Python is unlike Scala in that the Python compiler does not treat variables called <code>_</code> specially; those variables are merely called <code>_</code>. I could use <code>_</code> in Python code many times to refer to the same value, but a Scala programmer reading that code would expect that each reference to <code>_</code> would be another input parameter, not a regular variable reference. The examples that follow should make this clear. </p> <h3 id="installation">Python 3 Installation</h3> Install <code>sspipe</code> using <code>pip</code>: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf5ace1380f28'><button class='copyBtn' data-clipboard-target='#idf5ace1380f28' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pip install --upgrade sspipe</pre> <h2 id="scalaSetup">Scala 2.13+ Setup</h2> <p> Scala 2.13 introduced <a href='https://www.scala-lang.org/api/current/scala/util/ChainingOps.html' target='_blank' rel='nofollow'><code>ChainingOps</code></a>, which adds chaining methods <code>tap</code> and <code>pipe</code> to every type. The key concept is to import the following prior to attempting the code examples below: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida9de43ceda88'><button class='copyBtn' data-clipboard-target='#ida9de43ceda88' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>implicit class Smokin[A](val a: A) { import scala.util.chaining._ import scala.language.implicitConversions implicit def |>[B](f: (A) => B): B = a.pipe(f) }</pre> <h2 id="examples">Python and Scala Usage Examples</h2> <h3 id="2-3">One Lambda Function and 1 Pipe</h3> <p> This Python example employs one lambda function and 1 pipe to add 2 to the number 5: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id18fc39c99012'><button class='copyBtn' data-clipboard-target='#id18fc39c99012' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>5 | _ + 2 <span class='unselectable'>7 </span></pre> <p> The Scala equivalent of the above is: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id98200dfb51da'><button class='copyBtn' data-clipboard-target='#id98200dfb51da' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>scala> </span>5 |> ((_: Int) + 2) <span class='unselectable'>val res0: Int = 7 </span></pre> <h3 id="2-3">Two Lambda Functions and 2 Pipes</h3> <p> This Python example employs two lambda functions and 2 pipes to multiply the previous result by 5 and then add the previous result. Recall that I said that in Python, an underscore when used this way is the name of a normal variable and that the compiler does not treat underscores as placeholders for lambda parameters. A Scala programmer would complain about the following code, because they would expect that the second lambda function would require 2 inputs: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id73d55568356e'><button class='copyBtn' data-clipboard-target='#id73d55568356e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>5 | _ + 2 | _ * 5 + _ <span class='unselectable'>42 </span></pre> <p> A better way to write the above would be to use the special variable <code>px</code>, which was imported above. Now everyone either knows that <code>px</code> holds the piped value, or they complain about <code>px</code> being a magic variable. A possible solution to this complaint would be to alias <code>px</code> to a more descriptive name, such as <code>pipedValue</code> &mldr; which is still magical, but at least it is more descriptive. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0c3e59b44a9b'><button class='copyBtn' data-clipboard-target='#id0c3e59b44a9b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>5 | _ + 2 | px * 5 + px <span class='unselectable'>42 </span></pre> The Scala equivalent of the above is: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0dd002b27299'><button class='copyBtn' data-clipboard-target='#id0dd002b27299' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>scala> </span>5 |> ((_: Int) + 2) |> ((x: Int) => x * 5 + x) <span class='unselectable'>val res1: Int = 42 </span></pre> <h3 id="2-3">Two Lambda Functions and 3 Pipes</h3> <p> This Python example employs 2 lambda functions and 3 pipes to add 10 to the even numbers from 0 to 5, exclusive. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3e8f6c78c5a5'><button class='copyBtn' data-clipboard-target='#id3e8f6c78c5a5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>( range(5) | p(filter, _ % 2 == 0) | p(map, _ + 10) | p(list) ) <span class='unselectable'>[10, 12, 14] </span></pre> <p> Scala has a better way of performing this type of computation that does not require pipes or computation. It is better because it is simpler to understand. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd2cc34a3ac89'><button class='copyBtn' data-clipboard-target='#idd2cc34a3ac89' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>scala> </span>for { | x <- (0 until 5).toList if x % 2 == 0 | y = x + 10 | } yield y <span class='unselectable'>val res12: List[Int] = List(10, 12, 14) </span></pre> <h2 id="other">Other examples of placeholder syntax</h2> <p> <code>NumPy</code> expressions (NumPy is Python-specific): </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6e858453c836'><button class='copyBtn' data-clipboard-target='#id6e858453c836' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>range(10) | np.sin(_)+1 | p(plt.plot)</pre> <p> Pandas expressions (Pandas is Python-specific): </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idce1a305f21c9'><button class='copyBtn' data-clipboard-target='#idce1a305f21c9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>people_df | _.loc[_.age > 10, 'name']</pre> <p> Solution for the <a href='https://projecteuler.net/problem=2' target='_blank' rel='nofollow'>2nd Project Euler exercise</a>: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id120b22cf1a9f'><button class='copyBtn' data-clipboard-target='#id120b22cf1a9f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>def fib(): a, b = 0, 1 while True: yield a a, b = b, a + b <span class='unselectable'>>>> </span>euler2 = ( fib() | p.where(_ % 2 == 0) | p.take_while(_ < 4000000) | p.add() ) <span class='unselectable'>>>> </span>euler2 4613732</pre> <h2 id="dotty">Looking Ahead to Scala 3 (Dotty)</h2> <p> The next major version of Scala, due out in a few months, will probably allow a Scala 3 extension method to define the vertical bar as a method for more readabile code: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id426a1d554943'><button class='copyBtn' data-clipboard-target='#id426a1d554943' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>def [A,B](a: A) |(f: (A) => B): B = a.pipe(f) # Sample usage: val x = 5 | doSomething | doSomethingElse | doSomethingMore</pre> <h2 id="scalacourses">To Learn More</h2> <p> My <a href='https://www.scalacourses.com/showCourse/40' target='_blank'>Introduction to Scala</a> course on ScalaCourses.com teaches Scala lambda functions. </p> I've Been Writing Jekyll Plugins 2020-10-03T00:00:00-04:00 https://mslinn.github.io/blog/2020/10/03/jekyll-plugins <editor-fold Intro> <p> This is a Jekyll-powered web site. <a href='https://jekyllrb.com/' target='_blank' rel='nofollow'>Jekyll</a> is a free open-source preprocessor that generates static web sites. You can extend Jekyll by using the <a href='https://jekyllrb.com/docs/liquid/' target='_blank' rel='nofollow'>Liquid</a> language to write includes. Includes are just macros for Jekyll. </p> <p> I prefer to write and use Jekyll plugins instead of Jekyll includes. Non-trivial Jekyll includes require logic to be expressed in the Liquid language. Liquid is an interesting language, but it is quite verbose, syntax can be awkward, some expressions are impossible to formulate, and there are no debugging tools. </p> <p> In contrast, plugins are written in <a href='https://www.ruby-lang.org' target='_blank' rel='nofollow'>Ruby</a>. Plugin usage syntax is more flexible and require less typing for users. </p> <p> The argument against writing plugins is that the Ruby language is subtle and powerful, and could be overwhelming for novice programmers. However, just as the <a href='https://spark.apache.org/' target='_blank' rel='nofollow'>Apache Spark</a> framework allows novice Scala programmers to write in <a href='https://databricks.com/session/just-enough-scala-for-spark' target='_blank' rel='nofollow'>Just Enough Scala for Spark</a>, and the <a href='https://rubyonrails.org' target='_blank' rel='nofollow'>Ruby on Rails</a> framework allows novice Ruby programmers to write <a href='https://ivanoats.github.io/just_enough_ruby' target='_blank' rel='nofollow'>Just Enough Ruby for Rails</a>, writing plugins for the Jekyll framework generally does not require total mastery of Ruby. </p> <p> Here are some of my plugins. The source code for a plugin can be copied to the clipboard whenever you click on this icon at the top right corner of the code: <img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'>. </p> <p> <a href='../../../../mslinn_jekyll_plugins.zip'>A zip file containing all the plugins</a> is available. </p> <h2 id="update">Update 2020-12-28</h2> <p> ... but wait, there is more! I wrote a <code>LogFactory</code> Ruby library class after publishing this article. Most of these plugins have been retrofitted with <code>LogFactory</code> &ndash; calls to <code>LoggerFactory.new.create_logger</code> create a custom logger. </p> <p> You can use <code>LogFactory</code> in your Jekyll plugins for debugging. <a href='/blog/2020/12/28/custom-logging-in-jekyll-plugins.html'>I wrote it up separately</a> but <code>log_factory.rb</code> is included in the above zip file. </p> </editor-fold Intro> <editor-fold archive_display> <h2 id="archiveDisplay" class="spaceAbove"><span class="code">archive_display</span></h2> <p> Lists the names and contents of each file in a <code>tar</code> file. For each text file, the following HTML is emitted: </p> <pre data-lt-active="false" class="snippet" id="foo" style="position: relative">&lt;div class='codeLabel'>{tar_entry.full_name}&lt;/div> &lt;pre data-lt-active='false'>&lt;code>{tar_entry.file_contents}&lt;/pre> </pre> <p> Binary files are displayed like this: </p> <div class="codeLabel">usr/bin/ruby2.7 <span style="font-size: smaller">(application/x-sharedlib; charset=binary)</span></div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id29f181029026'><i>Binary file</i></pre> <h3 id="archiveDisplaySyntax">Syntax</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idad168cb5dfb0'><button class='copyBtn' data-clipboard-target='#idad168cb5dfb0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{% archive_display filename.tar %}</pre> <p> Sample output is: </p> <h3 id="archiveDisplaySource">Source Code</h3> <p> <a href='/jekyll/doc/ArchiveDisplayTag.html'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/archive_display.rb" download="archive_display.rb" title="Click on the file name to download the file">archive_display.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id5951a535d57c"><button class='copyBtn' data-clipboard-target='#id5951a535d57c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 # # Displays information about the contents of tar files # # Install dependencies: # - Ubuntu: `sudo apt install libmagic-dev` # - Mac: `brew install libmagic` module ArchiveDisplayTag require_relative './logger_factory.rb' @log = LoggerFactory.new.create_logger('my_tag', Jekyll.configuration(&#123;&#125;), :warn, $stderr) # accessor allows classes in this module to use the logger def self.log @log end class ArchiveDisplay &lt; Liquid::Tag # Constructor. # @param tag_name [String] is the name of the tag, which we already know. # @param archive_name [Hash, String, Liquid::Tag::Parser] the arguments from the web page. # @param tokens [Liquid::ParseContext] tokenized command line # @return [void] def initialize(tag_name, archive_name, tokens) super archive_name.strip! @archive_name = archive_name end # Method prescribed by the Jekyll plugin lifecycle. # @return [String] def render(context) source = context.registers[:site].config['source'] tar_name = "#&#123;source&#125;/#&#123;@archive_name&#125;" ArchiveDisplayTag.log.info "archive_display: tar_name=#&#123;tar_name&#125;" traverse_tar(tar_name) end private # Walks through a `tar` file. # # Modified from this &#123;https://gist.github.com/sinisterchipmunk/1335041/5be4e6039d899c9b8cca41869dc6861c8eb71f13 gist by sinisterchipmunk &#125;. # # @param tar_name [String] Name of tar file to examine. # @return [String] containing HTML describing the contents of the `tar`. def traverse_tar(tar_name) require 'rubygems/package' require 'ruby-filemagic' # sudo apt install libmagic-dev # brew install libmagic file_magic = FileMagic.new(FileMagic::MAGIC_MIME) File.open(tar_name, "rb") do |file| Gem::Package::TarReader.new(file) do |tar| return tar.each.map &#123; |entry| next if entry.file? content = entry.read fm_type = file_magic.buffer(content) &#123; name: entry.full_name, content: content.strip, is_text: (fm_type.start_with? "text"), fm_type: fm_type &#125; &#125;.compact.sort_by &#123; |entry| entry[:name] &#125;.map &#123; |entry| heading = "&lt;div class='codeLabel'>#&#123;entry[:name]&#125; &lt;span style='font-size: smaller'>(#&#123;entry[:fm_type]&#125;)&lt;/span>&lt;/div>" if entry[:is_text] "#&#123;heading&#125;\n&lt;pre data-lt-active='false'>#&#123;entry[:content]&#125;&lt;/pre>" else "#&#123;heading&#125;\n&lt;p>&lt;i>Binary file&lt;/i>&lt;/pre>" end &#125; end end end end end Liquid::Template.register_tag('archive_display', ArchiveDisplayTag::ArchiveDisplay) </pre> <h3 id="archiveDisplayInstall">Installation</h3> <ol> <li> Install <code>libmagic</code>. <br /> <div class='codeLabel' style="margin-top: 1.25em;">Ubuntu & WSL</div> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4add6556e7ed'><button class='copyBtn' data-clipboard-target='#id4add6556e7ed' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install libmagic-dev</pre> <div class='codeLabel'>Mac</div> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4ff0b0b13dc3'><button class='copyBtn' data-clipboard-target='#id4ff0b0b13dc3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>brew install libmagic</pre> </li> <li> Add this line to <code>Gemfile</code> in your Jekyll site's top-level directory: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7a07d0235d97'><button class='copyBtn' data-clipboard-target='#id7a07d0235d97' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>gem 'ruby-filemagic'</pre> </li> <li> Install the <code>ruby-filemagic</code> gem. From your Jekyll site's top-level directory, type: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc0d5cd7a2d26'><button class='copyBtn' data-clipboard-target='#idc0d5cd7a2d26' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>bundle install</pre> </li> <li> Copy <code>archive_display.rb</code> and <code>logger_factory.rb</code> into the <code>_plugins/</code> directory of your Jekyll site. </li> <li> Restart Jekyll. </li> </ol> </editor-fold archive_display> <editor-fold basename> <h2 id="basename" class="spaceAbove"><span class="code">basename</span>, <span class="code">dirname</span> and <span class="code">basename_without_extension</span></h2> <p> These filters all return portions of a string. They are all defined in the same plugin. </p> <ul> <li><code>basename</code> &mdash; Filters a string containing a path, returning the filename and extension.</li> <li><code>dirname</code> &mdash; Filters a string containing a path, returning the portion before the filename and extension.</li> <li><code>basename_without_extension</code> &mdash; Filters a string containing a path, returning the filename without the extension.</li> </ul> <h3 id="basename_syntax">Syntax</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3f469c09e382'>{{ "blah/blah/filename.ext" | basename }} {{ "blah/blah/filename.ext" | dirname }} {{ "blah/blah/filename.ext" | basename_without_extension }}</pre> <h3 id="archiveDisplaySource">Source Code</h3> <p> <a href='/jekyll/doc/Basename.html'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/basename.rb" download="basename.rb" title="Click on the file name to download the file">basename.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id44de425aeb30"><button class='copyBtn' data-clipboard-target='#id44de425aeb30' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 Michael Slinn # @license SPDX-License-Identifier: Apache-2.0 # # Jekyll filters for working with paths. module Basename # Filters a string containing a path. # @return [String] the filename extracted from the path, including the filetype. # @example Extracts "filename.ext" from the path # &#123;&#123; "blah/blah/filename.ext" | basename &#125;&#125; def basename(filepath) File.basename(filepath) end # Filters a string containing a path. # @return [String] the portion of th path before the filename and extension. # @example Extracts "blah/blah" from the path. # &#123;&#123; "blah/blah/filename.ext" | dirname &#125;&#125; def dirname(filepath) File.dirname(filepath) end # Filters a string containing a path. # @return the filename without the extension. # @example Extracts "filename" from the path. # &#123;&#123; "blah/blah/filename.ext" | basename_without_extension &#125;&#125; def basename_without_extension(filepath) File.basename(filepath).split('.')[0...-1].join('.') end end Liquid::Template.register_filter(Basename) </pre> </editor-fold basename> <editor-fold flexible_include> <h2 id="flexibleInclude" class="spaceAbove"><span class="code">flexible_include</span></h2> <p> Jekyll&#39;s built-in <code>include</code> tag does not support including files outside of the <code>_includes</code> folder. Originally called <code>include_absolute</code>, this plugin name is now called <code>flexible_include</code> because it no longer just includes absolute file names. This plugin now supports 4 types of includes: </p> <ol> <li>Absolute filenames (first character is <code>/</code>).</li> <li>Filenames relative to the top-level directory of the Jekyll web site (unnecessary to preface with <code>./</code>).</li> <li>Filenames relative to the user home directory (first character is <code>~</code>).</li> <li>Executable filenames on the <code>PATH</code> (first character is <code>!</code>).</li> </ol> <p> In addition, filenames that require environment expansion because they contain a <code>$</code> character are expanded according to the environment variables defined when <code>jekyll build</code> executes. </p> <h3 id="flexibleIncludeSyntax">Syntax</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idef24eb256849'><button class='copyBtn' data-clipboard-target='#idef24eb256849' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{% flexible_include 'path' %}</pre> <p> The optional parameters can have any name. The included file will have parameters substituted. </p> <h3 id="flexibleIncludeUsage">Usage Examples</h3> <ol> <li> Include files without parameters; all four types of includes are shown. <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1d6b6696adb9'>{% flexible_include '../../folder/outside/jekyll/site/foo.html' %} {% flexible_include 'folder/within/jekyll/site/bar.js' %} {% flexible_include '/etc/passwd' %} {% flexible_include '~/.ssh/config' %}</pre> Here is another <code>flexible_include</code> invocation using environment variables: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1e34d5f603ce'><button class='copyBtn' data-clipboard-target='#id1e34d5f603ce' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{% flexible_include '$HOME/.bash_aliases' %}</pre> </li> <li> Include a file and pass parameters to it. <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id83480d7a7bf0'>{% flexible_include '~/folder/under/home/directory/foo.html' param1='yes' param2='green' %}</pre> </li> </ol> <h3 id="flexibleIncludeSource">Source Code</h3> <p> This code lives in a <a href='https://github.com/mslinn/jekyll-flexible-include-plugin' target='_blank' rel='nofollow'>GitHub repository</a>. <a href='/jekyll/doc/FlexibleIncludeTag.html'>Yard docs are here.</a> </p> </editor-fold flexible_include> <editor-fold from_to_until> <h2 id="from_to_until" class="spaceAbove"><span class="code">from</span>, <span class="code">to</span> and <span class="code">until</span></h2> <p> These filters all return portions of a multiline string. They are all defined in the same plugin. A <a href='https://ruby-doc.org/core-2.5.7/Regexp.html' target='_blank' rel='nofollow'>regular expression</a> is used to specify the match; the simplest regular expression is a string. </p> <ul> <li><code>from</code> &mdash; returns the portion beginning with the line that satisfies a regular expression to the end of the multiline string.</li> <li><code>to</code> &mdash; returns the portion from the first line to the line that satisfies a regular expression, including the matched line.</li> <li><code>until</code> &mdash; returns the portion from the first line to the line that satisfies a regular expression, excluding the matched line.</li> </ul> <p> <a href='https://rubular.com/' target='_blank' rel='nofollow'>Rubular</a> is a handy online tool to try out regular expressions. </p> <h3 id="from_to_until_syntax">Syntax</h3> <p> The regular expression may be enclosed in single quotes, double quotes, or nothing. </p> <h4 id="from_syntax" class="code">from</h4> All of these examples perform identically. <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbfe6d7859cb5'><button class='copyBtn' data-clipboard-target='#idbfe6d7859cb5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ sourceOfLines | from: 'regex' }} {{ sourceOfLines | from: "regex" }} {{ sourceOfLines | from: regex }}</pre> <h4 id="to_syntax" class="code">to</h4> All of these examples perform identically. <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1b04fdbf4688'><button class='copyBtn' data-clipboard-target='#id1b04fdbf4688' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ sourceOfLines | to: 'regex' }} {{ sourceOfLines | to: "regex" }} {{ sourceOfLines | to: regex }}</pre> <h4 id="until_syntax" class="code">until</h4> All of these examples perform identically. <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf85944f4168c'><button class='copyBtn' data-clipboard-target='#idf85944f4168c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ sourceOfLines | until: 'regex' }} {{ sourceOfLines | until: "regex" }} {{ sourceOfLines | until: regex }}</pre> <p> <b>Important:</b> the name of the filter must be followed by a colon (:). If you fail to do that an error will be generated and the Jekyll site building process will halt. The error message looks something like this: <code>Liquid Warning: Liquid syntax error (line 285): Expected end_of_string but found string in "{{ lines | from '2' | until: '4' | xml_escape }}" in /some_directory/some_files.html Liquid Exception: Liquid error (line 285): wrong number of arguments (given 1, expected 2) in /some_directory/some_file.html Error: Liquid error (line 285): wrong number of arguments (given 1, expected 2)</code> </p> <h3 id="from_to_until_examples">Usage Examples</h3> <p> Some of the following examples use a multiline string containing 5 lines, called <code>lines</code>, which was created this way: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb9a519b4364b'><button class='copyBtn' data-clipboard-target='#idb9a519b4364b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{% capture lines %}line 1 line 2 line 3 line 4 line 5 {% endcapture %}</pre> <p> Other examples use a multiline string containing the contents of <code>.gitignore</code>, which looks like this: </p> <div class="codeLabel">.gitignore</div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id64446a526b23">*.code-workspace .bsp/ project/ target/ *.gz *.sublime* *.swp *.out *.Identifier *.log .idea* *.iml *.tmp *~ ~* .DS_Store .idea .jekyll-cache/ .jekyll-metadata .makeAwsBucketAndDistribution.log .sass-cache/ .yardoc/ __pycache__/ __MACOSX _build/ _package/ _site/ bin/*.class doc/ jekyll/doc/ node_modules/ Notepad++/ out/ package/ instances.json rescue_ubuntu2010 rescue_ubuntu2010.b64 landingPageShortName.md test.html RUNNING_PID mslinn_jekyll_plugins.zip cloud9.tar cloud9.zip mslinn_aws.tar </pre> <h4 id="until_example_line3">From the third line of string</h4> <p> These examples return the lines of the file from the beginning of the until a line with the string <code>"3"</code> is found, including the matched line. The only difference between the examples is the delimiter around the regular expression. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id19e94976603a'><button class='copyBtn' data-clipboard-target='#id19e94976603a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ lines | from: '3' }}</pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id553b675ee993'><button class='copyBtn' data-clipboard-target='#id553b675ee993' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ lines | from: "3" }}</pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ideef9020e2b23'><button class='copyBtn' data-clipboard-target='#ideef9020e2b23' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ lines | from: 3 }}</pre> <p> These all generate: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd21f0b079bf0'><button class='copyBtn' data-clipboard-target='#idd21f0b079bf0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>line 3 line 4 line 5</pre> <h4 id="until_example_filec">From Line In a File Containing 'PID'</h4> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc70023762ad7'><button class='copyBtn' data-clipboard-target='#idc70023762ad7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{% capture gitignore %}{% flexible_include '.gitignore' %}{% endcapture %} {{ gitignore | from: 'PID' | xml_escape }}</pre> <p> This generates: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id228fe320e3cf'><button class='copyBtn' data-clipboard-target='#id228fe320e3cf' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>RUNNING_PID mslinn_jekyll_plugins.zip cloud9.tar cloud9.zip mslinn_aws.tar</pre> <h4 id="to_example_line3">To the third line of string</h4> <p> These examples return the lines of the file from the first line until a line with the string <code>"3"</code> is found, including the matched line. The only difference between the examples is the delimiter around the regular expression. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id81aab1dd9f25'><button class='copyBtn' data-clipboard-target='#id81aab1dd9f25' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ lines | to: '3' }}</pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idae8ce451e937'><button class='copyBtn' data-clipboard-target='#idae8ce451e937' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ lines | to: "3" }}</pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfab9f6e7a641'><button class='copyBtn' data-clipboard-target='#idfab9f6e7a641' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ lines | to: 3 }}</pre> <p> These all generate: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id10428aa2af10'><button class='copyBtn' data-clipboard-target='#id10428aa2af10' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>line 1 line 2 line 3</pre> <h4 id="until_example_file">To Line In a File Containing 'idea'</h4> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id570ea61c3ad1'><button class='copyBtn' data-clipboard-target='#id570ea61c3ad1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ gitignore | to: 'idea' }}</pre> <p> This generates: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idee94dbcbc02b'><button class='copyBtn' data-clipboard-target='#idee94dbcbc02b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>*.code-workspace .bsp/ project/ target/ *.gz *.sublime* *.swp *.out *.Identifier *.log .idea*</pre> <h4 id="until_example_line3b">Until the third line of string</h4> <p> These examples return the lines of the file until a line with the string <code>"3"</code> is found, excluding the matched line. The only difference between the examples is the delimiter around the regular expression. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf6429942120d'><button class='copyBtn' data-clipboard-target='#idf6429942120d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ lines | until: '3' }}</pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb02775cd04c3'><button class='copyBtn' data-clipboard-target='#idb02775cd04c3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ lines | until: "3" }}</pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd46b85a233ba'><button class='copyBtn' data-clipboard-target='#idd46b85a233ba' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ lines | until: 3 }}</pre> <p> These all generate: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb0c8bec6641a'><button class='copyBtn' data-clipboard-target='#idb0c8bec6641a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>line 1 line 2</pre> <h4 id="until_example_fileb">Until Line In a File Containing 'idea'</h4> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id320f8d812a26'><button class='copyBtn' data-clipboard-target='#id320f8d812a26' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ gitignore | until: 'idea' }}</pre> <p> This generates: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7c443e3b1835'><button class='copyBtn' data-clipboard-target='#id7c443e3b1835' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>*.code-workspace .bsp/ project/ target/ *.gz *.sublime* *.swp *.out *.Identifier *.log</pre> <h4 id="until_example_lines_2_4">From the string "2" until the string "4"</h4> <p> These examples return the lines of the file until a line with the string <code>"3"</code> is found, excluding the matched line. The only difference between the examples is the delimiter around the regular expression. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc9f0054936c2'><button class='copyBtn' data-clipboard-target='#idc9f0054936c2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ lines | from: '2' | until: '4' }}</pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9e00c54d4f7a'><button class='copyBtn' data-clipboard-target='#id9e00c54d4f7a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ lines | from: "2" | until: "4" }}</pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id187ac5e4101d'><button class='copyBtn' data-clipboard-target='#id187ac5e4101d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ lines | from: 2 | until: 4 }}</pre> <p> These all generate: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3652f15018e7'><button class='copyBtn' data-clipboard-target='#id3652f15018e7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>line 2 line 3</pre> <h4 id="until_example_file_no_match">From Line In a File Containing 'idea' Until no match</h4> <p> The <code>.gitignore</code> file does not contain the string <code>xx</code>. If we attempt to match against that string the remainder of the file is returned for the <code>to</code> and <code>until</code> filter, and the empty string is returned for the <code>from</code> filter. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida357a7b161b7'><button class='copyBtn' data-clipboard-target='#ida357a7b161b7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ gitignore | from: 'PID' | until: 'xx' }}</pre> <p> This generates: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id59278e681330'><button class='copyBtn' data-clipboard-target='#id59278e681330' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>RUNNING_PID mslinn_jekyll_plugins.zip cloud9.tar cloud9.zip mslinn_aws.tar</pre> <h4 id="until_example_complex">More Complex Regular Expressions</h4> <p> The <code>from</code>, <code>to</code> and <code>until</code> filters can all accept more complex regular expressions. This regular expression matches lines that have either the string <code>sun</code> or <code>cloud</code> at the beginning of the line. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc7cce98fbeac'><button class='copyBtn' data-clipboard-target='#idc7cce98fbeac' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{{ gitignore | from: '^(cloud|sun)' }}</pre> <p> This generates: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id43d51fbc2228'><button class='copyBtn' data-clipboard-target='#id43d51fbc2228' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>cloud9.tar cloud9.zip mslinn_aws.tar</pre> <h3 id="includeSource">Source Code</h3> <p> <a href='/jekyll/doc/FromToUntil.html'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/from_to_until.rb" download="from_to_until.rb" title="Click on the file name to download the file">from_to_until.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id3461e1ad0d45"><button class='copyBtn' data-clipboard-target='#id3461e1ad0d45' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 Michael Slinn # @license SPDX-License-Identifier: Apache-2.0 # Jekyll filters for working with multiline strings. module FromToUntil # Filters a multiline string, returning the portion beginning with the line that satisfies a regex. # The regex could be enclosed in single quotes, double quotes, or nothing. # @param input_strings [String] The multi-line string to scan # @param regex [String] The regular expression to match against each line of `input_strings` until found # @return [String] The remaining multi-line string # @example Returns remaining lines starting with the line containing the word `module`. # &#123;&#123; flexible_include '/blog/2020/10/03/jekyll-plugins.html' | from 'module' &#125;&#125; def from(input_strings, regex) return '' unless check_parameters(input_strings, regex) regex = remove_quotations(regex.to_s.strip) matched = false result = '' input_strings.each_line do |line| matched = true if !matched && line =~ /#&#123;regex&#125;/ result += line if matched end result end # Filters a multiline string, returning the portion from the beginning until and including the line that satisfies a regex. # The regex could be enclosed in single quotes, double quotes, or nothing. # @example Returns lines up to and including the line containing the word `module`. # &#123;&#123; flexible_include '/blog/2020/10/03/jekyll-plugins.html' | to 'module' &#125;&#125; def to(input_strings, regex) return '' unless check_parameters(input_strings, regex) regex = remove_quotations(regex.to_s.strip) result = '' input_strings.each_line do |line| result += line return result if line =~ /#&#123;regex&#125;/ end result end # Filters a multiline string, returning the portion from the beginning until but not including the line that satisfies a regex. # The regex could be enclosed in single quotes, double quotes, or nothing. # @example Returns lines up to but not including the line containing the word `module`. # &#123;&#123; flexible_include '/blog/2020/10/03/jekyll-plugins.html' | until 'module' &#125;&#125; def until(input_strings, regex) return '' unless check_parameters(input_strings, regex) regex = remove_quotations(regex.to_s.strip) result = '' input_strings.each_line do |line| return result if line =~ /#&#123;regex&#125;/ result += line end result end private def check_parameters(input_strings, regex) if input_strings.nil? || input_strings.empty? then puts "Warning: Plugin 'from' received no input." return false end regex = regex.to_s if regex.nil? || regex.empty? then puts "Warning: Plugin 'from' received no regex." return false end true end def remove_quotations(str) str = str.slice(1..-2) if (str.start_with?('"') && str.end_with?('"')) || (str.start_with?("'") && str.end_with?("'")) str end end Liquid::Template.register_filter(FromToUntil) </pre> <h3 id="includeInstall">Installation</h3> <ol> <li> Copy <code>from_to_until.rb</code> and <code>logger_factory.rb</code> into the <code>_plugins/</code> directory of your Jekyll site. </li> <li> Restart Jekyll. </li> </ol> </editor-fold from_to_until> <editor-fold href> <h2 id="href" class="spaceAbove"><span class="code">href</span></h2> <p>Generates an <code>a href</code> tag with <code>target=&quot;_blank&quot;</code> and <code>rel=nofollow</code>.</p> <h3 id="hrefSyntax">Syntax</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idaa64109fff2f'><button class='copyBtn' data-clipboard-target='#idaa64109fff2f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{% href url text to display %}</pre> <p>The url should not be enclosed in quotes.</p> <h3 id="hrefExamples">Usage Examples</h3> <h4 id="hrefExample1">Default</h4> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6c62138df3e3'><button class='copyBtn' data-clipboard-target='#id6c62138df3e3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{% href https://www.mslinn.com The Awesome %}</pre> <p> This generates: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id53ef05851ec4'><button class='copyBtn' data-clipboard-target='#id53ef05851ec4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>&lt;a href='https://www.mslinn.com' target='_blank' rel='nofollow'&gt;The Awesome&lt;/a&gt;</pre> <p> Which renders as: <a href='https://www.mslinn.com'>The Awesome</a> </p> <h4 id="hrefExample2"><span class='code'>follow</span></h4> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3265d5312b9a'><button class='copyBtn' data-clipboard-target='#id3265d5312b9a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{% href follow https://www.mslinn.com The Awesome %}</pre> <p> This generates: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida0e90f1c5b25'><button class='copyBtn' data-clipboard-target='#ida0e90f1c5b25' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>&lt;a href='https://www.mslinn.com' target='_blank'&gt;The Awesome&lt;/a&gt;</pre> <h4 id="hrefExample3"><span class="code">notarget</span></h4> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1592dc54316e'><button class='copyBtn' data-clipboard-target='#id1592dc54316e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{% href notarget https://www.mslinn.com The Awesome %}</pre> <p> This generates: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2cbdfe2ed79a'><button class='copyBtn' data-clipboard-target='#id2cbdfe2ed79a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>&lt;a href='https://www.mslinn.com' rel='nofollow'&gt;The Awesome&lt;/a&gt;</pre> <h4 id="hrefExample4"><span class="code">follow notarget</span></h4> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id961032b4df7e'><button class='copyBtn' data-clipboard-target='#id961032b4df7e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{% href follow notarget https://www.mslinn.com The Awesome %}</pre> <p> This generates: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id760be64e4395'><button class='copyBtn' data-clipboard-target='#id760be64e4395' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>&lt;a href='https://www.mslinn.com'&gt;The Awesome&lt;/a&gt;</pre> <h3 id="hrefSource">Source Code</h3> <p> <a href='/jekyll/doc/HrefTag.html'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/href.rb" download="href.rb" title="Click on the file name to download the file">href.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="ide7e62c2fc3ef"><button class='copyBtn' data-clipboard-target='#ide7e62c2fc3ef' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># @author Copyright 2020 Michael Slinn # @license SPDX-License-Identifier: Apache-2.0 # # Generates an href. # Note that the url should not be enclosed in quotes. # # If the link starts with 'http' or `match` is specified: # The link will open in a new tab or window # The link will include `rel=nofollow` for SEO purposes. # # To suppress the `nofollow` attribute, preface the link with the word `follow`. # To suppress the `target` attribute, preface the link with the word `notarget`. # # The `match` option looks through the pages collection for a URL with containing the provided substring. # Match implies follow and notarget. # # If a section called plugin-vars exists then its name/value pairs are available for substitution. # plugin-vars: # django-github: 'https://github.com/django/django/blob/3.1.7' # django-oscar-github: 'https://github.com/django-oscar/django-oscar/blob/3.0.2' # # # @example General form # &#123;% href [follow] [notarget] [match] url text to display %&#125; # # @example Generates `nofollow` and `target` attributes. # &#123;% href https://mslinn.com The Awesome %&#125; # # @example Does not generate `nofollow` or `target` attributes. # &#123;% href follow notarget https://mslinn.com The Awesome %&#125; # # @example Does not generate `nofollow` attribute. # &#123;% href follow https://mslinn.com The Awesome %&#125; # # @example Does not generate `target` attribute. # &#123;% href notarget https://mslinn.com The Awesome %&#125; # # @example Matches page with URL containing abc. # &#123;% href match abc The Awesome %&#125; # @example Matches page with URL containing abc. # &#123;% href match abc.html#tag The Awesome %&#125; # # @example Substitute name/value pair for the django-github variable: # &#123;% href &#123;&#123;django-github&#125;&#125;/django/core/management/__init__.py#L398-L401 # &lt;code>django.core.management.execute_from_command_line&lt;/code> %&#125; module HrefTag class ExternalHref &lt; Liquid::Tag # Constructor. # @param tag_name [String] is the name of the tag, which we already know. # @param command_line [Hash, String, Liquid::Tag::Parser] the arguments from the web page. # @param tokens [Liquid::ParseContext] tokenized command line # @return [void] def initialize(tag_name, command_line, tokens) super @follow = " rel='nofollow'" @match = false @target = " target='_blank'" tokens = command_line.strip.split(" ") followIndex = tokens.index("follow") if followIndex then tokens.delete_at(followIndex) @follow = "" end targetIndex = tokens.index("notarget") if targetIndex then tokens.delete_at(targetIndex) @target = "" end matchIndex = tokens.index("match") if matchIndex then tokens.delete_at(matchIndex) @follow = "" @match = true @target = "" end @link = tokens.shift unless @link.start_with? "http" @follow = "" @target = "" end @text = tokens.join(" ").strip @text = if @text.empty? then @link else @text end end def match(context) site = context.registers[:site] config = site.config['href'] die_if_nomatch = !config.nil? && config['nomatch'] && config['nomatch']=='fatal' path, fragment = @link.split('#') # puts "@link=#&#123;@link&#125;" # puts "site.posts[0].url = #&#123;site.posts.docs[0].url&#125;" # puts "site.posts[0].path = #&#123;site.posts.docs[0].path&#125;" posts = site.posts.docs.select &#123; |x| x.url.include?(path) &#125; case posts.length when 0 if die_if_nomatch then abort "href error: No url matches '#&#123;@link&#125;'" else @link = "#" @text = "&lt;i>#&#123;@link&#125; is not available&lt;/i>" end when 1 @link = "#&#123;@link&#125;\##&#123;fragment&#125;" if fragment else abort "Error: More than one url matched: #&#123; matches.join(", ")&#125;" end end def replaceVars(context, link) variables = context.registers[:site].config['plugin-vars'] variables.each do |name, value| #puts "#&#123;name&#125;=#&#123;value&#125;" link = link.gsub("&#123;&#123;#&#123;name&#125;&#125;&#125;", value) end link end # Method prescribed by the Jekyll plugin lifecycle. # @return [String] def render(context) if (@match) then match(context) end link = replaceVars(context, @link) # puts "@link=#&#123;@link&#125;; link=#&#123;link&#125;" "&lt;a href='#&#123;link&#125;'#&#123;@target&#125;#&#123;@follow&#125;>#&#123;@text&#125;&lt;/a>" end end end Liquid::Template.register_tag('href', HrefTag::ExternalHref) </pre> <h3 id="hrefInstall">Installation</h3> <ol> <li> Copy <code>href.rb</code> and <code>logger_factory.rb</code> into the <code>_plugins/</code> directory of your Jekyll site. </li> <li> Restart Jekyll. </li> </ol> </editor-fold href> <editor-fold link> <h2 id="link" class="spaceAbove"><span class="code">link</span></h2> <p> This plugin generates a link to the given URI, which must be a file on the server. The file name can be absolute or relative to the top-level directory of the web site. </p> <h3 id="linkSyntax">Syntax</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida24dd5a493bd'><button class='copyBtn' data-clipboard-target='#ida24dd5a493bd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{% link uri %}</pre> <h3 id="linkExample">Usage Example</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc5b880d01f85'><button class='copyBtn' data-clipboard-target='#idc5b880d01f85' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{% link cloud9.tar %}</pre> <p> Generates: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0e8e067ce955'><button class='copyBtn' data-clipboard-target='#id0e8e067ce955' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>&lt;a href="/cloud9.tar">&lt;code>cloud9.tar&lt;/code>&lt;/a> (4.5 KB)</pre> <p> Which renders as: <a href='/cloud9.tar'><code>cloud9.tar</code></a> (4.5 KB) </p> <h3 id="linkSource">Source Code</h3> <p> <a href='/jekyll/doc/LinkTag.html'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/link.rb" download="link.rb" title="Click on the file name to download the file">link.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id8b708d174ee0"><button class='copyBtn' data-clipboard-target='#id8b708d174ee0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># @author Copyright 2020 Michael Slinn # @license SPDX-License-Identifier: Apache-2.0 module LinkTag # Generates an href to a file for the user to download from the site. # Also shows the file size in a human-readable format. class Linker &lt; Liquid::Tag # Constructor. # @param tag_name [String] is the name of the tag, which we already know. # Contains the name of the file, relative to the website top level directory # @param text [Hash, String, Liquid::Tag::Parser] the arguments from the web page. # @param tokens [Liquid::ParseContext] tokenized command line # @return [void] def initialize(tag_name, text, tokens) super(tag_name, text, tokens) @filename = text.delete('"').delete("'").strip end # Method prescribed by the Jekyll plugin lifecycle. # @return [String] def render(context) source = context.registers[:site].config['source'] file_fq = File.join(source, @filename) abort("Error: '#&#123;file_fq&#125;' not found. See the link tag in") unless File.exist?(file_fq) "&lt;a href='/#&#123;@filename&#125;'>&lt;code>#&#123;@filename&#125;&lt;/code>&lt;/a> (#&#123;as_size(File.size(file_fq))&#125;)" end def as_size(s) units = %w[B KB MB GB TB] size, unit = units.reduce(s.to_f) do |(fsize, _), utype| fsize > 512 ? [fsize / 1024, utype] : (break [fsize, utype]) end "#&#123;size > 9 || size.modulo(1) &lt; 0.1 ? '%d' : '%.1f'&#125; %s" % [size, unit] end end end Liquid::Template.register_tag('link', LinkTag::Linker) </pre> <h3 id="linkInstall">Installation</h3> <ol> <li> Copy <code>link.rb</code> and <code>logger_factory.rb</code> into the <code>_plugins/</code> directory of your Jekyll site. </li> <li> Restart Jekyll. </li> </ol> </editor-fold link> <editor-fold make_archive> <h2 id="make_archive" class="spaceAbove"><span class="code">make_archive</span></h2> <p> Creates <code>tar</code> and <code>zip</code> archives according to the <code>make_archive</code> entry in <code>_config.yml</code>. In <code>production</code> mode, the archives are built each time Jekyll generates the web site. In <code>development</code> mode, the archives are only built if they do not already exist, or if <code>delete: true</code> is set for that archive in <code>_config.yml</code>. Archives are placed in the top-level of the Jekyll project, and are copied to <code>_site</code> by Jekyll's normal build process. Entries are created in <code>.gitignore</code> for each of the generated archives. </p> <h3 id="make_archive_files">File Specifications</h3> <p>This plugin supports 4 types of file specifications:</p> <ol> <li>Absolute filenames (start with <code>/</code>).</li> <li>Filenames relative to the top-level directory of the Jekyll web site (Do not preface with <code>.</code> or <code>/</code>).</li> <li>Filenames relative to the user home directory (preface with <code>~</code>).</li> <li>Executable filenames on the <code>PATH</code> (preface with <code>!</code>).</li> </ol> <h3 id="make_archive_code"><span class="code">_config.yml</span> Syntax</h3> <p> Any number of archives can be specified. Each archive has 3 properties: <code>archive_name</code>, <code>delete</code> (defaults to <code>true</code>) and <code>files</code>. Take care that the dashes have exactly 2 spaces before them, and that the 2 lines following each dash have exactly 4 spaces in front. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2320cb2efa84'><button class='copyBtn' data-clipboard-target='#id2320cb2efa84' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>make_archive: - archive_name: cloud9.zip delete: true # This is the default, and need not be specified. files: [ index.html, error.html, ~/.ssh/config, /etc/passwd, '!update' ] - archive_name: cloud9.tar delete: false # Do not overwrite the archive if it already exists files: [ index.html, error.html, ~/.ssh/config, /etc/passwd, '!update' ]</pre> <h3 id="make_archive_source">Source Code</h3> <p> <a href='/jekyll/doc/MakeArchive.html'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/make_archive.rb" download="make_archive.rb" title="Click on the file name to download the file">make_archive.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id9db400200812"><button class='copyBtn' data-clipboard-target='#id9db400200812' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 Michael Slinn # @license SPDX-License-Identifier: Apache-2.0 require 'fileutils' require 'ptools' require 'rubygems' require 'rubygems/package' require 'tmpdir' require 'zlib' # Makes tar or zip file based on _config.yml entry class MakeArchive &lt; Jekyll::Generator require_relative 'logger_factory' priority :high def initialize(config) super(config) @log = LoggerFactory.new.create_logger('make_archive', config, :warn, $stderr) end # Method prescribed by the Jekyll plugin lifecycle. # @param site [Jekyll.Site] Automatically provided by Jekyll plugin mechanism # @return [void] def generate(site) @live_reload = site.config['livereload'] archive_config = site.config['make_archive'] return if archive_config.nil? archive_config.each do |config| @archive_name = config['archive_name'] # Relative to _site abort 'Error: archive_name was not specified in _config.yml.' if @archive_name.nil? if @archive_name.end_with? '.zip' @archive_type = :zip elsif @archive_name.end_with? '.tar' @archive_type = :tar else abort "Error: archive must be zip or tar; #&#123;@archive_name&#125; is of an unknown archive type." end @archive_files = config['files'].compact abort 'Error: archive files were not specified in _config.yml.' if @archive_files.nil? delete_archive = config['delete'] @force_delete = delete_archive.nil? ? !@live_reload : delete_archive @log.info "@archive_name=#&#123;@archive_name&#125;; @live_reload=#&#123;@live_reload&#125;; @force_delete=#&#123;@force_delete&#125;; @archive_files=#&#123;@archive_files&#125;" doit site.source site.keep_files &lt;&lt; @archive_name end end private def doit(source) archive_name_full = "#&#123;source&#125;/#&#123;@archive_name&#125;" archive_exists = File.exist?(archive_name_full) return if archive_exists && @live_reload @log.info "#&#123;archive_name_full&#125; exists? #&#123;archive_exists&#125;" if archive_exists && @force_delete @log.info "Deleting old #&#123;archive_name_full&#125;" File.delete(archive_name_full) end if !archive_exists || @force_delete @log.info "Making #&#123;archive_name_full&#125;" case @archive_type when :tar make_tar(archive_name_full, source) when :zip make_zip(archive_name_full, source) end end @log.info "Looking for #&#123;@archive_name&#125; in .gitignore..." return if File.foreach('.gitignore').grep(/^#&#123;@archive_name&#125;\n?/).any? @log.info "#&#123;@archive_name&#125; not found in .gitignore, adding entry." File.open('.gitignore', 'a') do |f| f.puts File.basename(@archive_name) end end def make_tar(tar_name, source) Dir.mktmpdir do |dirname| @archive_files.each do |filename| fn, filename_full = qualify_file_name(filename, source) @log.info "Copying #&#123;filename_full&#125; to temporary directory #&#123;dirname&#125;; filename=#&#123;filename&#125;; fn=#&#123;fn&#125;" FileUtils.copy(filename_full, dirname) end write_tar(tar_name, dirname) end end def write_tar(tar_name, dirname) # Modified from https://gist.github.com/sinisterchipmunk/1335041/5be4e6039d899c9b8cca41869dc6861c8eb71f13 File.open(tar_name, 'wb') do |tarfile| Gem::Package::TarWriter.new(tarfile) do |tar| Dir[File.join(dirname, '**/*')].each do |filename| write_tar_entry(tar, dirname, filename) end end end end def write_tar_entry(tar, dirname, filename) mode = File.stat(filename).mode relative_file = filename.sub(%r&#123;^#&#123;Regexp.escape dirname&#125;/?&#125;, '') if File.directory?(filename) tar.mkdir relative_file, mode else tar.add_file relative_file, mode do |tf| File.open(filename, 'rb') &#123; |f| tf.write f.read &#125; end end end def make_zip(zip_name, source) require 'zip' Zip.default_compression = Zlib::DEFAULT_COMPRESSION Zip::File.open(zip_name, Zip::File::CREATE) do |zipfile| @archive_files.each do |filename| filename_in_archive, filename_original = qualify_file_name(filename, source) @log.info "make_zip: adding #&#123;filename_original&#125; to #&#123;zip_name&#125; as #&#123;filename_in_archive&#125;" zipfile.add(filename_in_archive, filename_original) end end end # @return tuple of filename (without path) and fully qualified filename def qualify_file_name(path, source) case path[0] when '/' # Is the file absolute? @log.info "Absolute filename: #&#123;path&#125;" [File.basename(path), path] when '!' # Should the file be found on the PATH? clean_path = path[1..-1] filename_full = File.which(clean_path) abort "Error: #&#123;clean_path&#125; is not on the PATH." if filename_full.nil? @log.info "File on PATH: #&#123;clean_path&#125; -> #&#123;filename_full&#125;" [File.basename(clean_path), filename_full] when '~' # Is the file relative to user's home directory? clean_path = path[2..-1] filename_full = File.join(ENV['HOME'], clean_path) @log.info "File in home directory: #&#123;clean_path&#125; -> #&#123;filename_full&#125;" [File.basename(clean_path), filename_full] else # The file is relative to the Jekyll website top-level directory @log.info "Relative filename: #&#123;path&#125;" [File.basename(path), File.join(source, path)] # join yields the fully qualified path end end end </pre> <h3 id="make_archive_install">Installation</h3> <ol> <li> Copy <code>make_archive.rb</code> and <code>logger_factory.rb</code> into the <code>_plugins/</code> directory of your Jekyll site. </li> <li> Restart Jekyll. </li> </ol> </editor-fold make_archive> <editor-fold pre> <h2 id="pre_noselect" class="spaceAbove"><span class="code">pre</span> and <span class="code">noselect</span></h2> <p> This plugin provides 2 tags that frequently work together: </p> <ol> <li>A <code>pre</code> block tag that can optionally display a copy button.</li> <li>A <code>noselect</code> tag that can renders HTML content passed to it unselectable.</li> </ol> <h3 id="preSyntax">Syntax</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id70c7501f6b24'><button class='copyBtn' data-clipboard-target='#id70c7501f6b24' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{% pre [copyButton] [shell] [headline words] %} Contents of pre tag {% endpre %}</pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8b1f10f00fa8'>{% pre [copyButton] %} {% noselect [text string]%}Contents of pre tag {% endpre %}</pre> <h3 id="preUse1">Usage Example 1</h3> <p> This example does not generate a copy button and does not demonstrate <code>noselect</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id85122cfcc1bf'>{% pre %} Contents of pre tag {% endpre %}</pre> <p> Generates: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6d5974dd29dc'>&lt;pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id586ffaf9225b'&gt;Contents of pre tag&lt;/pre&gt;</pre> <p> Which renders as: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id586ffaf9225b'>Contents of pre tag</pre> <h3 id="preUse2">Usage Example 2</h3> <p> This example generates a copy button and does not demonstrate <code>noselect</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id63bc6be635a5'>{% pre copyButton %}Contents of pre tag {% endpre %}</pre> <p> Generates: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5ba02df14b6a'>&lt;pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8015f05b154a'&gt;&lt;button class='copyBtn' data-clipboard-target='#id8015f05b154a' title='Copy to clipboard'&gt;&lt;img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'&gt;&lt;/button&gt;Contents of pre tag&lt;/pre&gt;</pre> <p> Which renders as: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8015f05b154a'><button class='copyBtn' data-clipboard-target='#id8015f05b154a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>Contents of pre tag</pre> <h3 id="preUse3">Usage Example 3</h3> <p> This example generates a copy button and does demonstrates the default usage of <code>noselect</code>, which renders an unselectable dollar sign followed by a space. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id385820745dda'>{% pre copyButton %} {% noselect %}Contents of pre tag {% endpre %}</pre> <p> Generates: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id634d36906aca'>&lt;pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfe7a85b0f047'&gt;&lt;button class='copyBtn' data-clipboard-target='#idfe7a85b0f047' title='Copy to clipboard'&gt;&lt;img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'&gt;&lt;/button&gt;&lt;span class='unselectable'&gt;$ &lt;/span&gt;Contents of pre tag&lt;/pre&gt;</pre> <p> Which renders as: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfe7a85b0f047'><button class='copyBtn' data-clipboard-target='#idfe7a85b0f047' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>Contents of pre tag</pre> <h3 id="preUse4">Usage Example 4</h3> <p> This example generates a copy button and does demonstrates the <code>noselect</code> being used twice: the first time to render an unselectable custom prompt, and the second time to render unselectable output. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id705a838ed132'>{% pre copyButton %}{% noselect >>> %}Contents of pre tag {% noselect How now brown cow%} {% endpre %}</pre> <p> Generates: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7b8f97a556cc'>&lt;pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idafd42ad914fd'&gt;&lt;button class='copyBtn' data-clipboard-target='#idafd42ad914fd' title='Copy to clipboard'&gt;&lt;img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'&gt;&lt;/button&gt;&lt;span class='unselectable'&gt;&gt;&gt;&gt; &lt;/span&gt;contents of pre tag &lt;span class='unselectable'&gt;How now brown cow&lt;/span&gt;&lt;/pre&gt;</pre> <p> Which renders as: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idafd42ad914fd'><button class='copyBtn' data-clipboard-target='#idafd42ad914fd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>contents of pre tag <span class='unselectable'>How now brown cow</span></pre> <h3 id="preCss">CSS</h3> <p> Here are the CSS declarations that I defined pertaining to the <code>pre</code> and <code>noselect</code> tags: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6b6acf99a36d'><button class='copyBtn' data-clipboard-target='#id6b6acf99a36d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>.maxOneScreenHigh { max-height: 500px; } .unselectable { color: #7922f9; -moz-user-select: none; -khtml-user-select: none; user-select: none; }</pre> <h3 id="fyi">Comprehensive Example</h3> <p> The code I wrote to generate the above CSS was a good example of how the plugins work together: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id881c09bdb454'>{% capture css %}{% flexible_include '_sass/mystyle.scss' %}{% endcapture %} {% pre copyButton %}{{ css | from: '.copyBtn' | to: '^$' | strip }} {{ css | from: '.copyContainer' | to: '^$' | strip }} {{ css | from: '.maxOneScreenHigh' | to: '^$' | strip }} {{ css | from: '.unselectable' | to: '^$' | strip }} {% endpre %}</pre> <h3 id="preSource">Source Code</h3> <p> <a href='/jekyll/doc/PreTag.html'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/pre.rb" download="pre.rb" title="Click on the file name to download the file">pre.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id1cfc737ff48f"><button class='copyBtn' data-clipboard-target='#id1cfc737ff48f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 require 'securerandom' module PreTag # """ # \\&#123;% pre %&#125; # Content here # \\&#123;% endpre %&#125; # # \\&#123;% pre copyButton %&#125; # Content here # \\&#123;% endpre %&#125;""" # # \\&#123;% pre shell %&#125; # Content here # \\&#123;% endpre %&#125; # # \\&#123;% pre copyButton shell %&#125; # Content here # \\&#123;% endpre %&#125; # # \\&#123;% pre copyButton label %&#125; # Content here # \\&#123;% endpre %&#125;""" class PreTagBlock &lt; Liquid::Block @@prefix = "&lt;button class='copyBtn' data-clipboard-target=" @@suffix = " title='Copy to clipboard'>&lt;img src='/assets/images/clippy.svg' " \ "alt='Copy to clipboard' style='width: 13px'>&lt;/button>" def self.make_copy_button(pre_id) "#&#123;@@prefix&#125;'##&#123;pre_id&#125;'#&#123;@@suffix&#125;" end def self.make_pre(make_copy_button, label, content) label = if label.to_s.empty? then '' elsif label.to_s.downcase.strip == 'shell' "&lt;div class='codeLabel unselectable' data-lt-active='false'>Shell&lt;/div>" else "&lt;div class='codeLabel unselectable' data-lt-active='false'>#&#123;label&#125;&lt;/div>" end pre_id = "id#&#123;SecureRandom.hex(6)&#125;" copy_button = make_copy_button ? PreTagBlock.make_copy_button(pre_id) : '' "#&#123;label&#125;&lt;pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='#&#123;pre_id&#125;'>#&#123;copy_button&#125;#&#123;content.strip&#125;&lt;/pre>" end # Constructor. # @param tag_name [String] is the name of the tag, which we already know. # @param text [Hash, String, Liquid::Tag::Parser] the arguments from the web page. # @param tokens [Liquid::ParseContext] tokenized command line # @return [void] def initialize(tag_name, text, tokens) super(tag_name, text, tokens) text = '' if text.nil? text.strip! @make_copy_button = text.include? 'copyButton' remaining_text = text.sub('copyButton', '').strip #puts "@make_copy_button = '#&#123;@make_copy_button&#125;'; text = '#&#123;text&#125;'; remaining_text = '#&#123;remaining_text&#125;'" @label = remaining_text end # Method prescribed by the Jekyll plugin lifecycle. # @return [String] def render(context) content = super #puts "@make_copy_button = '#&#123;@make_copy_button&#125;'; @label = '#&#123;@label&#125;'" PreTagBlock.make_pre(@make_copy_button, @label, content) end end # """\\&#123;% noselect %&#125; or \\&#123;% noselect this all gets copied. # Also, space before the closing percent is signficant %&#125;""" class UnselectableTag &lt; Liquid::Tag def initialize(tag_name, text, tokens) super(tag_name, text, tokens) @content = text # puts "UnselectableTag: content1= '#&#123;@content&#125;'" @content = '$ ' if @content.nil? || @content.empty? # puts "UnselectableTag: content2= '#&#123;@content&#125;'" end def render(_) "&lt;span class='unselectable'>#&#123;@content&#125;&lt;/span>" end end end Liquid::Template.register_tag('pre', PreTag::PreTagBlock) Liquid::Template.register_tag('noselect', PreTag::UnselectableTag) </pre> <h3 id="preInstall">Installation</h3> <ol> <li> Copy <code>pre.rb</code> and <code>logger_factory.rb</code> into the <code>_plugins/</code> directory of your Jekyll site. </li> <li> Restart Jekyll. </li> </ol> </editor-fold pre> <editor-fold random_hex_string> <h2 id="random_hex_string" class="spaceAbove"><span class="code">random_hex_string</span></h2> <p> This Liquid filter generates a random hexadecimal string of any length. Each byte displays as two characters. You can specify the number of bytes in the hex string; if you do not, 6 random bytes (12 characters) will be generated. </p> <h3 id="random_hex_string_usage">Usage Example</h3> <p> This example generates a random hex string 6 bytes long and stores the result in a Liquid variable called <code>id</code>. Both of the following do the same thing: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id309e2cf0ecae'>{% assign id = random_hex_string %} {% assign id = random_hex_string 6 %}</pre> <p> The generated 6 bytes (12 characters) might be: <code>fd78ebdc1aa5</code>. </p> <h3 id="random_hex_string_source">Source Code</h3> <p> <a href='/jekyll/doc/RandomHex.html'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/random_hex.rb" download="random_hex.rb" title="Click on the file name to download the file">random_hex.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="ida4fd26b42edb"><button class='copyBtn' data-clipboard-target='#ida4fd26b42edb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 module RandomHex # Outputs a string of random hexadecimal characters of any length. # Defaults to a six-character string. # @example Generate 6 random characters. # &#123;&#123; random_hex_string &#125;&#125; # @example Generate 20 random characters. # &#123;&#123; random_hex_string 10 &#125;&#125; class RandomNumberTag &lt; Liquid::Tag # Called by Jekyll only once to register the module. # @param tag_name [String] Describe this parameter's purpose # @param text [String] Describe this parameter's purpose # @param context [String] Describe this parameter's purpose # @return [String, nil] Describe the return value def initialize(tag_name, text, context) super#(tag_name, text, context) text.to_s.strip! if text.empty? @n = 6 else tokens = text.split(' ') abort "random_hex_string error - more than one token was provided: '#&#123;text&#125;'" if tokens.length > 1 not_integer = !Integer(text, exception: false) abort "random_hex_string error: '#&#123;text&#125;' is not a valid integer" if not_integer @n = text.to_i end end def render(_) require 'securerandom' SecureRandom.hex(@n) end end end Liquid::Template.register_tag('random_hex_string', RandomHex::RandomNumberTag) </pre> <h3 id="random_hex_string_install">Installation</h3> <ol> <li> Copy <code>random_hex.rb</code> and <code>logger_factory.rb</code> into the <code>_plugins/</code> directory of your Jekyll site. </li> <li> Restart Jekyll. </li> </ol> </editor-fold random_hex_string> <editor-fold site_inspector> <h2 id="site_inspector" class="spaceAbove"><span class="code">site_inspector</span></h2> <p> Dumps lots of information from <code>site</code> when enabled by the <code>site_inspector</code> setting in <code>_config.yml</code>. </p> <h3 id="site_inspector_syntax"><span class="code">_config.yml</span> Syntax</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5692c0be15b4'>site_inspector: true # Run in development mode</pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8503ebfd4aa2'>site_inspector: force # Run in development and production modes</pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6cc8d11d4301'>site_inspector: false # The default is to not run</pre> <h3 id="site_inspector_output">Sample Output</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7d69e97f8ab6'>site is of type Jekyll::Site site.time = 2020-10-05 05:18:27 -0400 site.config['env']['JEKYLL_ENV'] = development site.collections.posts site.collections.expertArticles site.config.source = '/mnt/_/www/www.mslinn.com' site.config.destination = '/mnt/_/www/www.mslinn.com/_site' site.config.collections_dir = '' site.config.plugins_dir = '_plugins' site.config.layouts_dir = '_layouts' site.config.data_dir = '_data' site.config.includes_dir = '_includes' site.config.collections = '{"posts"=>{"output"=>true, "permalink"=>"/blog/:year/:month/:day/:title:output_ext"}, "expertArticles"=>{"output"=>true, "relative_directory"=>"_expertArticles", "sort_by"=>"order"}}' site.config.safe = 'false' site.config.include = '[".htaccess"]' site.config.exclude = '["_bin", ".ai", ".git", ".github", ".gitignore", "Gemfile", "Gemfile.lock", "script", ".jekyll-cache/assets"]' site.config.keep_files = '[".git", ".svn", "cloud9.tar"]' site.config.encoding = 'utf-8' site.config.markdown_ext = 'markdown,mkdown,mkdn,mkd,md' site.config.strict_front_matter = 'false' site.config.show_drafts = 'true' site.config.limit_posts = '0' site.config.future = 'true' site.config.unpublished = 'false' site.config.whitelist = '[]' site.config.plugins = '["classifier-reborn", "html-proofer", "jekyll", "jekyll-admin", "jekyll-assets", "jekyll-docs", "jekyll-environment-variables", "jekyll-feed", "jekyll-gist", "jekyll-sitemap", "kramdown"]' site.config.markdown = 'kramdown' site.config.lsi = 'false' site.config.excerpt_separator = ' ' site.config.incremental = 'true' site.config.detach = 'false' site.config.port = '4000' site.config.host = '127.0.0.1' site.config.baseurl = '' site.config.show_dir_listing = 'false' site.config.permalink = '/blog/:year/:month/:day/:title:output_ext' site.config.paginate_path = '/page:num' site.config.timezone = '' site.config.quiet = 'false' site.config.verbose = 'false' site.config.defaults = '[]' site.config.liquid = '{"error_mode"=>"warn", "strict_filters"=>false, "strict_variables"=>false}' site.config.rdiscount = '{"extensions"=>[]}' site.config.redcarpet = '{"extensions"=>[]}' site.config.kramdown = '{"auto_ids"=>true, "toc_levels"=>"1..6", "entity_output"=>"as_char", "smart_quotes"=>"lsquo,rsquo,ldquo,rdquo", "input"=>"GFM", "hard_wrap"=>false, "footnote_nr"=>1, "show_warnings"=>false}' site.config.author = 'Mike Slinn' site.config.compress_html = '{"blanklines"=>false, "clippings"=>"all", "comments"=>["<!-- ", " -->"], "endings"=>"all", "ignore"=>{"envs"=>["development"]}, "profile"=>false, "startings"=>["html", "head", "body"]}' site.config.email = 'mslinn@mslinn.com' site.config.feed = '{"categories"=>["AI", "Blockchain", "Scala", "Software-Expert"]}' site.config.ignore_theme_config = 'true' site.config.site_inspector = 'false' site.config.make_archive = '[{"archive_name"=>"cloud9.tar", "delete"=>true, "files"=>["!killPortFwdLocal", "!killPortFwdOnJumper", "!tunnelToJumper"]}]' site.config.sass = '{"style"=>"compressed"}' site.config.title = 'Mike Slinn' site.config.twitter = '{"username"=>"mslinn", "card"=>"summary"}' site.config.url = 'http://localhost:4000' site.config.livereload = 'true' site.config.livereload_port = '35729' site.config.serving = 'true' site.config.watch = 'true' site.config.assets = '{}' site.config.tag_data = '[]' site.keep_files: [".git", ".svn", "cloud9.tar"]</pre> <h3 id="site_inspector_source">Source Code</h3> <p> <a href='/jekyll/doc/SiteInspector.html'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/site_inspector.rb" download="site_inspector.rb" title="Click on the file name to download the file">site_inspector.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id34a0678fed79"><button class='copyBtn' data-clipboard-target='#id34a0678fed79' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 # # Dumps lots of information from `site` if in `development` mode and `site_inspector: true` in `_config.yml`. class SiteInspector &lt; Jekyll::Generator require_relative 'logger_factory' def initialize(config) super(config) @log = LoggerFactory.new.create_logger('site_inspector', config, :warn, $stderr) end # Displays information about the Jekyll site # # @param site [Jekyll.Site] Automatically provided by Jekyll plugin mechanism # @return [void] def generate(site) mode = site.config['env']['JEKYLL_ENV'] config = site.config['site_inspector'] return if config.nil? inspector_enabled = config != false return unless inspector_enabled force = config == 'force' return unless force || mode == 'development' @log.info "site is of type #&#123;site.class&#125;" @log.info "site.time = #&#123;site.time&#125;" @log.info "site.config['env']['JEKYLL_ENV'] = #&#123;mode&#125;" site.collections.each do |key, _| puts "site.collections.#&#123;key&#125;" end # key env contains all environment variables, quite verbose so output is suppressed site.config.sort.each &#123; |key, value| @log.info "site.config.#&#123;key&#125; = '#&#123;value&#125;'" unless key == 'env' &#125; site.data.sort.each &#123; |key, value| @log.info "site.data.#&#123;key&#125; = '#&#123;value&#125;'" &#125; # site.documents.each &#123;|key, value| @log.info "site.documents.#&#123;key&#125;" &#125; # Generates too much output! @log.info "site.keep_files: #&#123;site.keep_files.sort&#125;" # site.pages.each &#123;|key, value| @log.info "site.pages.#&#123;key&#125;'" &#125; # Generates too much output! # site.posts.each &#123;|key, value| @log.info "site.posts.#&#123;key&#125;" &#125; # Generates too much output! site.tags.sort.each &#123; |key, value| @log.info "site.tags.#&#123;key&#125; = '#&#123;value&#125;'" &#125; end end </pre> <h3 id="site_inspector_install">Installation</h3> <ol> <li> Copy <code>site_inspector.rb</code> and <code>logger_factory.rb</code> into the <code>_plugins/</code> directory of your Jekyll site. </li> <li> Restart Jekyll. </li> </ol> </editor-fold site_inspector> Bash Script to Create a New Jekyll Post 2020-08-16T00:00:00-04:00 https://mslinn.github.io/blog/2020/08/16/new-jekyll-post <p> I use <a href='https://jekyllrb.com/' target='_blank' rel='nofollow'>Jekyll</a> to build this website. Some material is published as articles, some as blog posts. I wrote a script called <code>newPost</code> that creates a new draft blog post with SEO considerations. SEO rankings are improved when the description and title tag are neither too long nor too short. </p> <h2 id="usage">Sample Usage</h2> <p> Here is an example of how I used it on 2020-08-16: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id423b721e5d8e'><button class='copyBtn' data-clipboard-target='#id423b721e5d8e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>_bin/newPost Post Title (30-60 characters): ______________________________123456789012345678901234567890 This is a test of newPost for the greater good 46 characters, excellent! Publication date: 2020-08-16 Post Description (30-60 characters): ____________________________________________________________123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890 I wish newPost used some sort of web service to generate an SEO-optimized description 85 characters, excellent! Post Categories (comma delimited): Post Tags (comma delimited): Post Keywords (comma delimited): Goofiness, Silliness Banner image (bg_ .jpg):</pre> <p> For the above example the generated file is called <code> _drafts/2020-08-16-this-is-a-test-of-newpost-for-the-greater-good.html</code>. When you are happy with the new posting, move it from <code>_drafts</code> to <code>_posts</code> like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id84910ea02eec'><button class='copyBtn' data-clipboard-target='#id84910ea02eec' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mv _drafts/2020-08-16-new-jekyll-post.html _posts/</pre> <h3 id="generated">Generated Posting</h3> <p> Here is the generated file: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2e435caab2b8'><button class='copyBtn' data-clipboard-target='#id2e435caab2b8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>--- categories: [] description: I wish newPost used some sort of web service to generate an SEO-optimized description image: keywords: [Goofiness, Silliness] last_modified_at: 2020-08-16 layout: blog title: This is a test of newPost for the greater good tags: [] ---</pre> <h2 id="error">Error Handling</h2> <p> The script checks the length of the title and the posting description for SEO purposes. If either of these are too long or too short, the script allows the user to edit their input over and over until they get it right. For example, here you can see that at first the user just types in <code>xx</code> for the title, then they provide a string that is too long, then they edit it until it has an acceptable length: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id050a9b58fbd7'><button class='copyBtn' data-clipboard-target='#id050a9b58fbd7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>_bin/newPost Post Title (30-60 characters): ______________________________123456789012345678901234567890 xx 28 characters too short, please edit Post Title (30-60 characters): ______________________________123456789012345678901234567890 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 14 characters too long, please edit Post Title (30-60 characters): ______________________________:123456789012345678901234567890 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 59 characters, excellent!</pre> <h2 id="code">Source Code</h2> <p>This is the source code for the <code>newPost</code> bash script.</p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/bin/newPost" download="newPost" title="Click on the file name to download the file">newPost</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idfb07bb66674f"><button class='copyBtn' data-clipboard-target='#idfb07bb66674f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash # This bash script will setup a new HTML Jekyll draft blog post and open it for editing in Notepad++ # # See https://www.mslinn.com/blog/2020/08/16/new-jekyll-post.html # # Copyright Original by Katie Harron - @pibby from https://gist.github.com/pibby/6911493 # Copyright 2020 Modified by Mike Slinn - mslinn@mslinn.com # - added length checks for title and description # - added reprompt for when length check fails # - launch notepad instead of Mac editor # - modified front matter # - Added optional date as command line parameter # - Added date modification dialog # # SPDX-License-Identifier: Apache-2.0 DIR="$( cd "$( dirname "$&#123;BASH_SOURCE[0]&#125;" )" >/dev/null 2>&1 && pwd )/.." cd "$DIR" function checkLength &#123; # $1 - minimum length # $2 - maximum length # $3 - string to test length=$&#123;#3&#125; if (( length &lt; $1 )); then >&2 echo "$(($1 - $length)) characters too short, please edit" return 1 elif (( length > $2 )); then >&2 echo "$(( $length - $2 )) characters too long, please edit" return 2 else >&2 echo "$length characters, excellent!" return 0 fi &#125; function edit &#123; if [ `which notepad` ]; then notepad "$filename" # Invokes mslinn's notepad++ script elif [ `which gedit` ]; then `which gedit` "$filename" else echo "No editor defined, please edit '$filename' somehow" exit 1 fi &#125; function emit_array &#123; if [ "$2" ]; then echo "$1: [$2]\n" else echo -n "" fi &#125; function emit_scalar &#123; if [ "$2" ]; then echo "$1: $2\n" else echo -n "" fi &#125; function reprompt &#123; # $1 - name of front matter variable # $2 - minimum length of user-provided value # $3 - maximum length of user-provided value if [ -z "$1" ]; then 2> echo "Error: no front matter variable provided"; exit 1; fi if [ -z "$2" ]; then 2> echo "Error: no minimum length provided"; exit 1; fi if [ -z "$3" ]; then 2> echo "Error: no maximum length provided"; exit 1; fi NAME="$1" MIN="$2" MAX="$3" LEADIN="'_%0.s'" SPACES="$( eval $(echo printf "$LEADIN" &#123;1..$MIN&#125;) )" (( COUNT= (($MAX * 10) - ($MIN * 10) + 5) / 10 )) NUMBERS="$( eval $(echo printf '"0123456789%0.s"' &#123;1..$COUNT&#125;) )" (( CHARS= $MAX - $MIN )) VALUE="" #>&2 printf "SPACES='$SPACES'\n" #>&2 printf "NUMBERS='$NUMBERS'\n" while : do >&2 printf "Post $1 (30-60 characters):\n$SPACES$&#123;NUMBERS:1:$CHARS&#125;\n" read -e -i "$VALUE" VALUE if $(checkLength "$MIN" "$MAX" "$VALUE"); then break; fi done >&2 echo echo "$VALUE" &#125; # Set cwd to project root GIT_ROOT="$( git rev-parse --show-toplevel )" cd "$&#123;GIT_ROOT&#125;" if [ ! -f _bin/loadConfigEnvVars ]; then echo -e "Error: _bin/loadConfigEnvVars was not found.\ncd \$msp" exit 1 fi source _bin/loadConfigEnvVars title="$( reprompt Title 30 60 )" ptitle=$&#123;title// /-&#125; # convert spaces in title to hyphens # Convert title to lowercase and remove slashes plc="$( echo "$ptitle" | tr '[:upper:]' '[:lower:]' | tr -d '[/,]' )" >&2 printf "Filename slug (without date or filetype): " read -e -i "$plc" plc if [ "$1" ]; then pdate="$1" # TODO verify $1 is YYYY-mm-dd else pdate="$( date +%Y-%m-%d )" # create date as YYYY-mm-dd fi read -p 'Publication date: ' -e -i "$pdate" pdate filename=_drafts/$pdate-$plc.html # location to create the new file as year-month-day-title.md mkdir -p _drafts touch "$filename" # create the new blank post desc="$( reprompt Description 60 150 )" printf "Post CSS (comma delimited): "; read css printf "Post Categories (comma delimited): "; read categories printf "Post Tags (comma delimited): "; read tags #printf "Post Keywords (comma delimited): "; read keyw printf "Banner image (.png & .webp): "; read img printf "Enable code example clipboard icon (Y/n): "; read clipboard if [[ ! "$clipboard" =~ ^(n|N).* ]]; then javascriptEnd="/assets/js/clipboard.min.js" javascriptInline="new ClipboardJS('.copyBtn');" fi contents="---\n" contents="$contents$( emit_array css "$css" )" contents="$contents$( emit_array categories "$categories" )" contents="$contents$( emit_scalar description "$desc" )" contents="$contents$( emit_scalar image "$img" )" contents="$contents$( emit_scalar javascript "$javascript" )" contents="$contents$( emit_scalar javascriptEnd "$javascriptEnd" )" contents="$contents$( emit_scalar javascriptInline "$javascriptInline" )" #contents="$contents$( emit_array keywords "$keyw" )" contents="$contents$( emit_scalar last_modified_at "$pdate" )" contents="$contents$( emit_scalar layout blog )" contents="$contents$( emit_array tags "$tags" )" contents="$contents$( emit_scalar title "$title" )" contents="$contents---\n" echo -e "$contents" | sed '/^$/d' > "$filename" # fill out YAML Front Matter and insert into the new file echo "Created '$filename'" #edit printf "Use mem to append code examples to this post (y/N): "; read clipboard if [[ "$clipboard" =~ ^(y|Y).* ]]; then ps ax | grep '[j]ekyll' | awk -F ' ' '&#123;print $1&#125;' | xargs sudo kill -15 _bin/mem "$filename" & _bin/serve -c fi </pre> Converting All Images in a Website to webp Format 2020-08-15T00:00:00-04:00 https://mslinn.github.io/blog/2020/08/15/converting-all-images-to-webp-format <p> I first launched this website in 1996. Since then, it has been re-incarnated using many different technologies. Presently I use <a href='https://jekyllrb.com/' target='_blank' rel='nofollow'>Jekyll</a> to assemble the site, then push the image to a web-enabled AWS S3 bucket that is edge-cached by an AWS CloudFront distribution. </p> <p> Until yesterday, the site contained images with a mixture of image formats. I decided to convert them all to the new <a href='https://developers.google.com/speed/webp' target='_blank' rel='nofollow'><code>webp</code></a> format. Because there are hundreds of images in over 120 web pages, I wrote a bash script called <code>toWebP</code> to do the work. This posting provides the <code>toWebP</code> script plus instructions on how you could use it for your website. </p> <p> The script converts image types <code>gif</code>, <code>jpg</code>, <code>jpeg</code>, <code>png</code>, <code>tif</code>, and <code>tiff</code>. It also modifies the HTML pages, CSS and SCSS that reference those images. </p> <p> The conversions are set for maximum fidelity (lossless where possible), and maximum compression. This means the images look great and load quickly. </p> <h3 id='caveat'>Caveat</h3> <p> The script assumes that all images are local to your website, which makes sense because the converted images need to be stored, and local storage is the only sensible option. It renames all references to images in HTML, CSS and SCSS files to <code>webp</code> format. If the images are remote (for example, on a CDN), they are not converted, but the image file types in the HTML, CSS and SCSS are adjusted anyway. I suppose I could fix the script, but I don't need to do that for myself. If someone needs that feature, go ahead and enhance the script... and please provide me the enhanced script, so I can update this blog posting. </p> <h2 id="prerequisites">Prerequisites</h2> <p> You need to install the WebP package.<br> </p> <h3 id="mac">Mac</h3> <p> Use <a href='https://formulae.brew.sh/formula/webp' target='_blank' rel='nofollow'>Homebrew</a> or <a href='https://ports.macports.org/?search=webp&search_by=name' target='_blank' rel='nofollow'>Macports</a>. </p> <h3 id="ubuntu">Ubuntu (this is the default Linux distribution for Windows Subsystem for Linux)</h3> <p>At a shell prompt type:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id29c187c4bc0a'><button class='copyBtn' data-clipboard-target='#id29c187c4bc0a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install webp</pre> <h2 id="running">Running <span class="code">toWebp</span></h2> <p> The program may emit warnings when it runs. Those warnings can be safely ignored. </p> <p> Hopefully, your website is managed by git. I suggest that you commit your work before running the script. That way if something goes wrong you just have to type <code>git stash</code> to return your website to its previous state. </p> <h3 id="usage">Usage</h3> <p>The general form of the command to convert all images and modify the HTML pages that they are referenced from is:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id482c30f8d767'><button class='copyBtn' data-clipboard-target='#id482c30f8d767' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>toWebp &lt;directoryName></pre> <h3 id="examples">Examples</h3> <p>To convert the website (images, html, scss & css) rooted at the current directory, type:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfd5c48a24cfb'><button class='copyBtn' data-clipboard-target='#idfd5c48a24cfb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>toWebp .</pre> <p>To convert the website called <code>mySite</code> rooted under your home directory, type:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1ee3ea9772f2'><button class='copyBtn' data-clipboard-target='#id1ee3ea9772f2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>toWebp ~/mySite</pre> <p>To just convert 1 specific image to <code>webp</code>, type:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8351ed8a2536'><button class='copyBtn' data-clipboard-target='#id8351ed8a2536' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>toWebp images/blah.jpg</pre> <h2 id="gist">Gist Containing the <code>toWebP</code> Bash Script.</h2> <p>Put this file in one of the directories on your <code>PATH</code>, for example <code>/usr/local/bin</code>: <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/d87f13c921456a21070c4d96366c6778.js"> </script> <h3 id="chmod">Make it Executable</h3> <p> Remember to make the <code>toWebp</code> script executable before trying to use it: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id75224a6a6b0b'><button class='copyBtn' data-clipboard-target='#id75224a6a6b0b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>chmod a+x /usr/local/bin/toWebp</pre> Dotty (Scala 3 Preview) Presentation at Hopper, Montreal 2019-11-28T00:00:00-05:00 https://mslinn.github.io/blog/2019/11/28/dotty-scala-3-preview <div style=""> <picture> <source srcset="/blog/images/dottyLambda_690x388.webp" type="image/webp"> <source srcset="/blog/images/dottyLambda_690x388.png" type="image/png"> <img src="/blog/images/dottyLambda_690x388.png" title="Mike Slinn presents" class=" liImg " alt="Mike Slinn presents" /> </picture> </div> <div style="text-align: center"> <p> Yesterday I presented <a href='https://www.meetup.com/lambda-montreal/events/266306046/' target='_blank' rel='nofollow'>Dotty (Scala 3 Preview)</a> to Lambda Montreal. </p> <p> The slides are <a href='https://www.slideshare.net/mslinn/dotty-scala-3-preview' target='_blank' rel='nofollow'>here</a>. </p> <p> The code is <a href='https://github.com/mslinn/dotty-example-project/' target='_blank' rel='nofollow'>here</a>. </p> <p> The video recording is <a href='https://www.youtube.com/watch?v=7S68TY0S2e0' target='_blank' rel='nofollow'>here</a>. </p> </div> <div style="text-align: center;"> <picture> <source srcset="/blog/images/lambdaMontreal.webp" type="image/webp"> <source srcset="/blog/images/lambdaMontreal.png" type="image/png"> <img src="/blog/images/lambdaMontreal.png" title="Mike Slinn presents" class="center quartersize liImg2 rounded shadow" alt="Mike Slinn presents" /> </picture> </div> A Hybrid Machine Learning / Personality Simulation Platform 2019-10-24T00:00:00-04:00 https://mslinn.github.io/blog/2019/10/24/hybrid-ml-simulation <div style="text-align: center;"> <a href="https://www.meetup.com/MTL-Machine-Learning/events/265039754/" target="_blank" rel="nofollow"><picture> <source srcset="/blog/images/mtlMLconference.webp" type="image/webp"> <source srcset="/blog/images/mtlMLconference.png" type="image/png"> <img src="/blog/images/mtlMLconference.png" class="center liImg2 rounded shadow" /> </picture></a> </div> <p> Yesterday I presented <a href='https://www.meetup.com/MTL-Machine-Learning/events/265039754/' target='_blank' rel='nofollow'>&ldquo;EmpathyWorks: A Hybrid Machine Learning / Personality Simulation Platform&rdquo;</a> to the <a href='https://www.meetup.com/MTL-Machine-Learning/events/265039754/' target='_blank' rel='nofollow'>Fall 2019 Montreal Machine Learning Mini-Conference</a>. </p> <p> The slides are <a href='https://www.slideshare.net/mslinn/empathyworks-towards-an-eventbased-simulationml-hybrid-platform' target='_blank' rel='nofollow'>here</a>. The video recording is <a href='https://youtu.be/PiDsiyJIMmo' target='_blank' rel='nofollow'>here</a>. </p> <div style="text-align: center"> <div style="display: inline-block; margin: 0.5em; vertical-align: top;"> <picture> <source srcset="/assets/images/robotCircle207x207.webp" type="image/webp"> <source srcset="/assets/images/robotCircle207x207.png" type="image/png"> <img src="/assets/images/robotCircle207x207.png" title="Mike Slinn presents" class=" liImg " alt="Mike Slinn presents" /> </picture> </div> <div style="display: inline-block; margin: 0.5em; vertical-align: top;"> <picture> <source srcset="/blog/images/montrealMachineLearningMeetup.webp" type="image/webp"> <source srcset="/blog/images/montrealMachineLearningMeetup.png" type="image/png"> <img src="/blog/images/montrealMachineLearningMeetup.png" title="Montreal Machine Learning Meetup" class=" quartersize liImg2 rounded shadow" style="margin-left: 3em; margin-top: 3em;" alt="Montreal Machine Learning Meetup" /> </picture> </div> </div> Decentralized Ponytails 2018-09-13T00:00:00-04:00 https://mslinn.github.io/blog/2018/09/13/decentralized-ponytails <p> I&rsquo;d like to point out the similarity of the early days of the open-source movement with today&rsquo;s decentralized blockchain movement. </p> <p> Open-source software was brought to mainstream attention during the last technology bubble at the end of the last millennium. The open-source software movement had a loyal cadre of zealots who believed that their cause would overcome any need for a business case. Sun Microsystems was the hardware company whose servers powered the Internet, and their software included the Java programming language, plus many other important networking-related products. Sun's slogan was &ldquo;The network is the computer&rdquo;. </p> <div style="text-align: center;"> <picture> <source srcset="/assets/images/Sun-Logo_225x99.webp" type="image/webp"> <source srcset="/assets/images/Sun-Logo_225x99.png" type="image/png"> <img src="/assets/images/Sun-Logo_225x99.png" title="Sun Microsystems logo" class="center quartersize liImg2 rounded shadow" style="padding: 1em" alt="Sun Microsystems logo" /> </picture> </div> <p> Jonathan Schwartz, the CEO of Sun Microsystems was one of the open-source zealots. He was famous for his ponytail. Unfortunately, zealotry and dogma is bad for business, and as a result Sun Microsystems is no longer with us. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/jonathanSchwartz.webp" type="image/webp"> <source srcset="/blog/images/jonathanSchwartz.png" type="image/png"> <img src="/blog/images/jonathanSchwartz.png" title="Jonathan Schwartz and his ponytail" class="center liImg rounded shadow" alt="Jonathan Schwartz and his ponytail" /> </picture> </div> <p> Eventually companies like <a href='https://redhat.com' target='_blank' rel='nofollow'>Red Hat</a> developed solid business models for open-source software, but that took years to develop. Today we see many companies attempting using decentralized blockchain technology to create cryptocurrencies, other token-based economies, and evangelizing decentralized dogma without a solid business case. Most of these ventures will die a horrible death, and the investors will get nothing. It will take years for solid business models based on decentralization to be proven. </p> <p> Mr. Schwartz's ponytail was the fashion statement that fueled the YouTube parody below. For background, <a href='https://en.wikipedia.org/wiki/Scott_McNealy' target='_blank' rel='nofollow'>Scott McNealy</a> was the previous CEO at Sun Microsystems. </p> <iframe width="690" height="388" src="https://www.youtube.com/embed/5r3JSciJf5M" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen class="rounded shadow liImg"></iframe> <p> Full disclosure: I also had a ponytail in 2008, and for a few years I had my Sun Spark 2 workstation at home. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/mikeclose3.webp" type="image/webp"> <source srcset="/blog/images/mikeclose3.png" type="image/png"> <img src="/blog/images/mikeclose3.png" title="Mike Slinn and his ponytail back in 2008" class="center quartersize liImg2 rounded shadow" alt="Mike Slinn and his ponytail back in 2008" /> </picture> </div> <p> Here is a transcription of the video, which I paraphrased for clarity:</p> </p> <div class="quote"> <p><b>Steve Gilmore:</b> Hi, this is Steve Gilmore and this is a video special edition of the Gilmore gang. I'm here with Jonathan Schwartz. It's a great pleasure &ndash; it's been a long, long time coming &ndash; I haven't seen Jonathan for quite a while. Jonathan Schwartz, who is the president and CEO of Sun Microsystems, agreed to sit down for the first time in three or four years and talk about what's going on with Sun. I wanna start, Jonathan, by thanking you for joining us. </p> <p><b>Jonathan Schwartz:</b> Thank you for having me, Steve. It has been a long time, nice to see you. </p> <p><b>Steve Gilmore:</b> So, you know there's been a lot of turmoil on Wall Street as I know you know. </p> <p><b>Jonathan Schwartz:</b> Yes. </p> <p><b>Steve Gilmore:</b> What's your take on that? </p> <p><b>Jonathan Schwartz:</b> Well, I think that it's a cyclical thing as you know Sun was been very prepared for this because we took <a href='https://www.thestreet.com/story/10334514/1/pipe-deal-shines-up-sun.html' target='_blank' rel='nofollow'>three quarters of a billion dollars off of KKR</a> a few years ago so that's in our war chest and I think that we're in a very good position moving forward, Steve. </p> <p><b>Steve Gilmore:</b> Ahh, and specifically what are you doing? </p> <p><b>Jonathan Schwartz:</b> Well, what we're doing is as you know Sun has always been very proactive in the open-source movement. I know you're very familiar with that. What we want to do to help our customers during this very difficult time is keep up with that trend of open-source. So I'm actually very pleased to announce, Steve, that Sun is starting a new open-source initiative. We're going to be releasing the source code to my ponytail as open-source, Steve. </p> <p><b>Steve Gilmore:</b> How's that going to help the situation? </p> <p><b>Jonathan Schwartz:</b> It's open-source, Steve. </p> <p><b>Steve Gilmore:</b> Yeah. </p> <p><b>Jonathan Schwartz:</b> It's my ponytail, Steve. </p> <p><b>Steve Gilmore:</b> You know, we've had this conversation in the past. </p> <p><b>Jonathan Schwartz:</b> Yes. </p> <p><b>Steve Gilmore:</b> Something is open-source, fine, and you get a lot of adoption, you get a lot of exposure in the in the marketing arena... </p> <p><b>Jonathan Schwartz:</b> Yes. </p> <p><b>Steve Gilmore:</b> ... but how do you make money on your ponytail? </p> <p><b>Jonathan Schwartz:</b> Well, what we're going to be doing is releasing my ponytail as open-source. So what we're hoping is that our developers take my ponytail and develop some kind of revenue stream with my ponytail. Did I mention this is open-source, Steve? </p> <p><b>Steve Gilmore:</b> Yeah, so how do you open-source a ponytail? What does that mean? </p> <p><b>Jonathan Schwartz:</b> Well, basically what we do is, we will have some of our best and brightest engineers here at Sun go through my ponytail and find out the unique attributes about what makes my ponytail so successful in the valley. And what we've done Steve, we've open-sourced my ponytail, Steve. </p> <p><b>Steve Gilmore:</b> Jonathan, you're just repeating this over and over again; it doesn't necessarily arrive at a business model. </p> <p><b>Jonathan Schwartz:</b> Umm, Steve? </p> <p><b>Steve Gilmore:</b> Yeah. </p> <p><b>Jonathan Schwartz:</b> We're gonna take my ponytail right and make it open-source. Now I know that this is a big concept for you but I really think it's a game changer, Steve. </p> <p><b>Steve Gilmore:</b> So, who do you see as your competition in the open-source ponytail arena? </p> <p><b>Jonathan Schwartz:</b> I think that we pretty much have it locked up. I don't see anybody who can compete with Sun Microsystems, when it comes to the open-source ponytail market. I think that we're in very good shape, Steve. How much do you miss Scott McNealy? </p> <p><b>Steve Gilmore:</b> Right now, a lot. </p> <p><b>Jonathan Schwartz:</b> Not nearly as much as I do, Steve. </p> <p><b>Steve Gilmore:</b> Okay, so what are you gonna do about, uh, you've laid off a lot of people in the last few months. </p> <p><b>Jonathan Schwartz:</b> Yes, it's going well in fact we have another round of layoffs coming. Once the ponytail is released into the wild, we'll be releasing the team that open-sourced my ponytail, Steve. </p> <p><b>Steve Gilmore:</b> Well you know I have to say that I was hoping for something a little bit more visionary from you Jonathan. </p> <p><b>Jonathan Schwartz:</b> Well I think that this is quite visionary. I don't think that IBM will be releasing a ponytail, and if they did it certainly wouldn't be open-source, Steve. We are very, very excited about our open-source ponytail program if you want you can go to <code>sunmicrosystems.com/ponytail</code>. Any other questions, Steve? </p> <p><b>Steve Gilmore:</b> Yeah I got one that I hope will be a stumper for you, which is a do you see the relationship between your open-source ponytail strategy and micro-messaging as popularized by Twitter? </p> <p><b>Jonathan Schwartz:</b> I'd like to open the pipe to the ponytail. Except as you know, Twitter is not exactly handling XMPP correctly at this time. I don't see the correlation between an open-source ponytail, and a closed-off micro-blogging system. Steve, I think that the ponytail is much bigger than Twitter. </p> <p><b>Steve Gilmore:</b> And the business model again? </p> <p><b>Jonathan Schwartz:</b> Let me see this real slowly and clearly: OPEN. SOURCE. PONYTAIL. </p> <p><b>Steve Gilmore:</b> This has been Jonathan Schwartz along with me, Steve Gilmore. Good luck, Jon. </p> <p><b>Jonathan Schwartz:</b> Thank you Steve. God, I miss McNealy. Think it's going to work? </p> <p><b>Steve Gilmore:</b> No. </p> </div> IBM Personality Insights 2018-08-29T00:00:00-04:00 https://mslinn.github.io/blog/2018/08/29/personality-assessment <style> .comment { color: magenta; font-style: italic; } .diff { color: green; } </style> <p> EmpathyWorks&trade; is my name for the original research I've done on modeling personality and behavior of individuals and groups since 2007. For me, the work has been both therapeutic and insightful, and I now have a better basis for understanding myself and others as a result. </p> <p> Recently, an IBM employee pointed me to <a href='https://console.bluemix.net/docs/services/personality-insights' target='_blank' rel='nofollow'>IBM Personality Insights</a>, and I eagerly visited the site. The <a href='https://console.bluemix.net/docs/services/personality-insights/science.html#science' target='_blank' rel='nofollow'>scientific basis</a> for the results is interesting. I am greatly interested in the analysis that IBM Personality Insights provided of the blog posting I wrote on my birthday last year. I also submitted a <a href='/blog/2008/04/28/cult-of-software-god.html'>short humorous posting</a> I wrote ten years ago for analysis. </p> <h2 id="salt">A Modern Horoscope?</h2> <div style=""> <picture> <source srcset="/blog/images/zodiac_690x690.webp" type="image/webp"> <source srcset="/blog/images/zodiac_690x690.png" type="image/png"> <img src="/blog/images/zodiac_690x690.png" title="Horoscope" class=" liImg2 rounded shadow" alt="Horoscope" /> </picture> </div> <p> I think the results from IBM Personality Insights are about as accurate as a horoscope. <a href='https://www.quora.com/How-accurate-is-IBMs-Watson-Personality-Insights-application/answer/Abhishek-Srivastava-198' target='_blank' rel='nofollow'>Abhishek Srivastava&rsquo;s Quora posting</a> of November 18, 2016, expresses this well. </p> <div class="quote"> I think most answers here have missed the point of IBM Watson’s personality insight service. It is a NLP based approach to find scores of individuals on some well-established scales in psychology like Big 5, Basic Humans Values and Needs. So, when those numbers are arrived using those standard questionnaires or through text analysis done by Watson, they are pretty close statistically. In that respect Watson is definitely very accurate. As far as interpretation of those results are concerned, that’s beyond the purview of current scope of Watson and rather is a question for psychologist. If I was a team member at IBM Watson, I would have actually not given the interpretation and would rather just give the scores and let people interpret them. </div> <p> Furthermore, until a peer review of IBM Personality Insights concludes that the results are well-founded, I would not be comfortable using the technology for decision-making. </p> <p> To be fair, <a href='https://www.news.vcu.edu/article/An_untested_foundation_A_VCU_study_finds_that_many_published' target='_blank' rel='nofollow'>a recent examination</a> of nearly 350 published psychological experiments found that 42% failed to show that they were based on a valid foundation of empirical evidence, suggesting that a wide swath of psychological science is based on an untested foundation. With that in mind, let's see what this modern horoscope serves up! </p> <h2 id="highLevel">High Level Results</h2> <p> Results were fairly consistent between the two blog postings. When I concatenated the two posts, the longer post dominated. <span class="diff">Differences between the two personality assessments are shown this way.</span> <span class="comment">My comments are shown this way.</span> </p> <table class="table table_striped table_cell_top table_cell_vspace table_cell_justify" width="100%"> <tr> <th width="50%">Birthday Posting Summary</th> <th width="50%"><a href='/blog/2008/04/28/cult-of-software-god.html'>Humorous Posting</a> Summary</th> </tr> <tr> <td> The results in JSON format are <a href='/blog/ibmPersonality/ibmPersonality.json'>here</a>. </td> <td> The results in JSON format are <a href='/blog/ibmPersonality/ibmPersonality2.json'>here</a>. </td> </tr> <tr> <td> You are shrewd and <span class="diff">skeptical</span>. </td> <td> You are shrewd<span class="diff">, inner-directed and guarded</span>. </td> </tr> <tr> <td> You are philosophical: you are open to and intrigued by new ideas and love to explore them. You are independent: you have a strong desire to have time to yourself. <span class="diff">And you are authority-challenging: you prefer to challenge authority and traditional values to help bring about positive changes.</span> <td> You are philosophical: you are open to and intrigued by new ideas and love to explore them. You are independent: you have a strong desire to have time to yourself. <span class="diff">And you are solemn: you are generally serious and do not joke much.</span> <span class="comment">I guess IBM did not like my humor!</span> </td> </tr> <tr> <td> Your choices are driven by a desire for <span class="diff">discovery</span> <span class='comment'>often true</span>. </td> <td> Your choices are driven by a desire for <span class="diff">organization</span> <span class='comment'>often true</span>. </td> </tr> <tr> <td> You are relatively unconcerned with both <span class="diff">tradition</span> and taking pleasure in life. <span class="diff">You care more about making your own path than following what others have done. And you prefer activities with a purpose greater than just personal enjoyment.</span> </td> <td> You are relatively unconcerned with both <span class="diff">achieving success</span> and taking pleasure in life. <span class="diff">You prefer activities with a purpose greater than just personal enjoyment. And you make decisions with little regard for how they show off your talents.</span> </td> </tr> </table> <p> An old friend, who I've known since university, is a clinical psychologist with a PhD and works as a doctor in a mental hospital. His comments on the results were:</p> <div class="quote"> <p> I think that some aspects of the profile are pretty accurate; however, other aspects such as calling you shrewd and skeptical are a bit evaluate: I would instead say intelligent and not naïve. The openness results would seem pretty accurate. </p> <p> I think you're probably a bit more extraverted than this analysis suggests. </p> <p> The ‘emotional range’ factor of the Big 5 is typically labeled Neuroticism. I kind of think of Agreeableness as a compliance/ conformity dimension, and I if I am not mistaken, it is not unusual for (male) entrepreneurs to score low on that dimension. </p> </div> <h2 id="more">But Wait, There's More!</h2> <p> IBM's Personality Insights provides additional information that explains the above in more detail: </p> <table class="table table_striped table_cell_top table_cell_vspace table_cell_justify"> <tr> <th width="50%">Birthday Posting Summary</th> <th width="50%"><a href='/blog/2008/04/28/cult-of-software-god.html'>Humorous Posting</a> Summary</th> </tr> <tr> <td> You are <b>likely</b> to: <ul> <li> <span class="diff">like musical movies</span><br /> <span class="comment">Oops, that was way off!</span> </li> <li> be sensitive to ownership cost when buying automobiles </li> <li> have experience playing music <br /> <span class="comment">True: I am a multi-instrumentalist</span> </li> </ul> </td> <td> You are <b>likely</b> to: <ul> <li> <span class="diff">like historical movies</span> <br /> <span class="comment">True!</span></li> <li>be sensitive to ownership cost when buying automobiles </li> <li> have experience playing music</li> </ul> </td> </tr> <tr> <td> You are <b>unlikely</b> to: <ul> <li> be influenced by social media during product purchases <span class="comment">The truth is more complex</span> </li> <li> prefer style when buying clothes <span class="comment">True, and to compensate I try to shop for clothes with carefully selected friends</span> </li> <li><span class="diff">like country music</span> <span class="comment">Spot on!</span></li> </ul> </td> <td> <p style="bold"> You are <b>unlikely</b> to: </p> <ul> <li> be influenced by social media during product purchases <br> <br> </li> <li> prefer style when buying clothes <br> <br> <br> </li> <li> <span class="diff">be influenced by brand name when making product purchases</span> </li> </ul> </td> </tr> </table> <p> I think that the above was a pretty good assessment of me, and the detailed breakdowns which follow are interesting. Before you look at that, however, you should know that the <a href='https://www.verywellmind.com/the-big-five-personality-dimensions-2795422' target='_blank' rel='nofollow'>Big 5 Personality Model</a> defines the 5 traits using words with meanings that might seem different from the meanings you might expect. The 5 traits are: <a href='https://console.bluemix.net/docs/services/personality-insights/openness.html' target='_blank' rel='nofollow'><i>openness</i></a>, <a href='https://console.bluemix.net/docs/services/personality-insights/conscientiousness.html' target='_blank' rel='nofollow'><i>conscientiousness</i></a>, <a href='https://console.bluemix.net/docs/services/personality-insights/extroversion.html' target='_blank' rel='nofollow'><i>extroversion</i></a>, <a href='https://console.bluemix.net/docs/services/personality-insights/agreeableness.html' target='_blank' rel='nofollow'><i>agreeableness</i></a>, and <a href='https://console.bluemix.net/docs/services/personality-insights/emotional-range.html' target='_blank' rel='nofollow'><i>emotional range</i></a>. </p> <h2 id="pnv">Personality, Needs and Values</h2> <p> Each of the 5 traits are broken down into various aspects. For example, openness consists of <i>adventurousness</i>, <i>artistic interests</i>, <i>emotionality</i>, <i>imagination</i>, <i>intellect</i>, and <i>authority-challenging</i>. Again, these terms are <a href='https://console.bluemix.net/docs/services/personality-insights/openness.html#dimensions' target='_blank' rel='nofollow'>defined in specific ways</a> that might be different from the definitions that you might expect. </p> <table class="table table_striped table_cell_top table_cell_vspace table_cell_justify"> <tr> <th width="50%">Birthday Posting Summary</th> <th width="50%"><a href='/blog/2008/04/28/cult-of-software-god.html'>Humorous Posting</a> Summary</th> </tr> <tr> <td> <p><br /><br /> <i>1724 words; decent analysis.</i> </p> <div style=""> <picture> <source srcset="/blog/images/ibmPersonalityInsightMslinnSliders.webp" type="image/webp"> <source srcset="/blog/images/ibmPersonalityInsightMslinnSliders.png" type="image/png"> <img src="/blog/images/ibmPersonalityInsightMslinnSliders.png" title="Birthday posting summary by IBM Personality Insights" class=" rounded shadow zoom" alt="Birthday posting summary by IBM Personality Insights" /> </picture> </div> </td> <td> <p> <i>516 words; we need a minimum of 600, preferably 1,200 or more, to compute statistically significant estimates.</i> </p> <div style=""> <picture> <source srcset="/blog/images/ibmPersonalityInsightMslinnSliders2.webp" type="image/webp"> <source srcset="/blog/images/ibmPersonalityInsightMslinnSliders2.png" type="image/png"> <img src="/blog/images/ibmPersonalityInsightMslinnSliders2.png" title="Humorous Posting summary by IBM Personality Insights" class=" rounded shadow zoom" alt="Humorous Posting summary by IBM Personality Insights" /> </picture> </div> </td> </tr> </table> <h2 id="detail">Detailed Breakdown</h2> <p> Here is my detailed breakdown, shown as sunburst charts: </p> <table class="table table_striped table_cell_top table_cell_vspace table_cell_justify" style="width: 100%;"> <tr> <th width="50%"> Birthday Posting Summary </th> <th width="50%"> <a href='/blog/2008/04/28/cult-of-software-god.html'>Humorous Posting</a> Summary </th> </tr> <tr> <td> <div style=""> <picture> <source srcset="/blog/images/ibmPersonalityInsightMslinnSunburst.webp" type="image/webp"> <source srcset="/blog/images/ibmPersonalityInsightMslinnSunburst.png" type="image/png"> <img src="/blog/images/ibmPersonalityInsightMslinnSunburst.png" title="Birthday posting summary by IBM Personality Insights" class=" rounded shadow zoom" alt="Birthday posting summary by IBM Personality Insights" /> </picture> </div> </td> <td> <div style=""> <picture> <source srcset="/blog/images/ibmPersonalityInsightMslinnSunburst2.webp" type="image/webp"> <source srcset="/blog/images/ibmPersonalityInsightMslinnSunburst2.png" type="image/png"> <img src="/blog/images/ibmPersonalityInsightMslinnSunburst2.png" title="Humorous posting summary by IBM Personality Insights" class=" rounded shadow zoom" alt="Humorous posting summary by IBM Personality Insights" /> </picture> </div> </td> </tr> </table> <p> The sunburst charts paint me as an unusual person. If this personality assessment is accurate, I am rather complex. However, further reading suggests that people with very high openness scores defy most structured evaluations, and I score in the 99<sup>th</sup> percentile for openness. </p> <table class="table table_striped table_cell_top table_cell_vspace table_cell_justify" style="width: 100%"> <tr> <th width="50%"> Birthday Posting Summary </th> <th width="50%"> <a href='/blog/2008/04/28/cult-of-software-god.html'>Humorous Posting</a> Summary </th> </tr> <tr> <td> <b>Openness to experience</b>: high (99<sup>th</sup> percentile). This means I have an unusually high fluid intelligence (I am able to learn complex concepts and tasks very quickly), and I am likely to be eccentric. <br /> <span class="comment">I believe this to be true</span>. </td> <td> <b>Openness to experience</b>: high (99<sup>th</sup> percentile) &ndash; <span class="comment">Same</span> </td> </tr> <tr> <td> <b>Extraversion</b>: Low (15th percentile) &ndash; Cold, withdrawn, unfriendly. <span class="comment">That feels harsh.</span> </td> <td> <b>Extraversion</b>: Even lower (6<sup>th</sup> percentile). <span class="comment">Yikes!</span> </td> </tr> <tr> <td> <b>Agreeableness</b>: Low (5<sup>th</sup> percentile) &ndash; Independent, tough, dominant, possibly manipulative. <span class="comment">Yes, I make up my mind and I follow what I believe to be the appropriate course, regardless of what others might say or do.</span> <br /> Additionally, I have very strong sympathy (97<sup>th</sup> percentile) and I am rather uncompromising (82nd percentile), strongly cooperative (82<sup>nd</sup> percentile) with strong altruism (87<sup>th</sup> percentile). <br /> <span class="comment">Perhaps that means that I am a crusty individual with a warm heart.</span> <br /> According to the breakdown, I am quite trusting (82<sup>nd</sup> percentile), yet I am also very cautious (90<sup>th</sup> percentile), so for me I go with &ldquo;trust but verify&rdquo;. </td> <td> <b><a href='https://cloud.ibm.com/docs/services/personality-insights?topic=personality-insights-agreeableness' target='_blank' rel='nofollow'>Agreeableness</a></b>: Even lower (0<sup>th</sup> percentile). <span class="comment">Yikes!</span> </td> </tr> <tr> <td> <b>Emotional range</b>: average (> 59<sup>th</sup> percentile). <span class="comment"><a href='https://cloud.ibm.com/docs/services/personality-insights?topic=personality-insights-emotionalRange' target='_blank' rel='nofollow'>IBM defines this</a> as &ldquo;the extent to which a person's emotions are sensitive to the individual's environment&rdquo;</span> </td> <td> <b>Emotional range</b>: very high (> 91<sup>st</sup> percentile). When coupled with low agreeableness, IBM predicts: temperamental, irritable, quarrelsome, impatient, grumpy. When coupled with low conscientiousness, IBM predicts: compulsive, nosy, self-indulgent, forgetful, impulsive. When coupled with low extroversion, IBM predicts: guarded, fretful, insecure, pessimistic, secretive. When coupled with high openness, IBM predicts: excitable, passionate, sensual. <span class="comment">Hmm, quarrelsome, impatient, compulsive, self-indulgent, forgetful, insecure, pessimistic and yet passionate and sensual. What a combination!</span> </td> </tr> <tr> <td> <b>Low Conservation (1<sup>st</sup> percentile)</b> &ndash; <span class="comment">This is called Hedonism in the other graph.</span> </td> <td> <b>Hedonism</b>: very low (1<sup>st</sup> percentile) &ndash; <span class="comment">This is called Conservation in the other graph.</span> Within this category I scored low self-enhancement, low Hedonism, low Openness to change and low conservation. </td> </tr> <tr> <td> <b>Low Harmony (5<sup>th</sup> percentile)</b> &ndash; <span class="comment">This is called Closeness in the other graph.</span> </td> <td> <b>Closeness:</b> low (1<sup>th</sup> percentile) &ndash; <span class="comment">This is called Harmony in the other graph.</span> Within this category I scored low Excitement, Harmony, Ideal, Liberty, Love, Self-expression and Stability. I also scored high Curiosity (78<sub>th</sub> percentile) and high Structure (88<sup>th</sup> percentile). <span class="comment">Sounds like I'm pretty much a curious robot.</span> </td> </tr> </table> <p> These traits, if true, would make me a good expert witness, and a good evaluator for technical due diligence. This might also explain my propensity for constantly inventing new things. </p> <p> According to these results, I also registered some extreme <a href='https://console.bluemix.net/docs/services/personality-insights/needs.html#needs' target='_blank' rel='nofollow'>needs</a> and <a href='https://console.bluemix.net/docs/services/personality-insights/values.html#values' target='_blank' rel='nofollow'>values</a>. Because the results of analyzing both documents were almost the same, I show them together. </p> <p> First, let's look at the results that I identify with: </p> <table class="table table_striped table_cell_top table_cell_vspace table_cell_justify" style="width: 100%"> <tr> <th> Both articles </th> </tr> <tr> <td> An extremely liberal mindset (<b>conservation</b>: 1st percentile) </td> </tr> <tr> <td> Virtually no interest in comfort (<b>hedonism</b>: 2nd percentile) </td> </tr> <tr> <td> Almost no interest in social power, authority, wealth, success, capability, etc. (<a href='https://www.researchgate.net/publication/322627891_The_five_pillars_of_self-enhancement_and_self-protection' target='_blank' rel='nofollow'><b>self-enhancement</b></a>: 4th percentile) </td> </tr> <tr> <td> Almost no interest in <b>harmony</b> (5th percentile) </td> </tr> <tr> <td> Strong <b>curiosity</b> (87th percentile) </td> </tr> <tr> <td> Low connection to family and setting up a home (<b>closeness</b>: 9%) </td> </tr> <tr> <td> Very little respect, commitment, and acceptance of the customs and ideas that culture and/or religion provides. </td> </tr> </table> <p> I think the following needs assessments are way off the mark. Perhaps I protest too much? </p> <table class="table table_striped table_cell_top table_cell_vspace table_cell_justify" style="width: 100%"> <tr> <td> Very little interest in just having fun for its own sake (<b>excitement</b>: 7th percentile). </td> </tr> <tr> <td> Low desire for perfection or a sense of community <span class="comment">The truth is complicated: I have spent decades building professional and musical communities, yet I am isolated in many ways</span> </td> </tr> <tr> <td> Low interest in discovering and asserting their identities (<b>self-expression</b>: 9%) <span class="comment">This seems way off the mark, for example consider the motivation for my spending thousands of hours on EmpathyWorks.</span> </td> </tr> </table> <h2 id="another">Another Assessment</h2> <p> The more clocks you have, the less certain you are of the correct time. Some of the following is true, but my personality expresses itself differently according to circumstance, so no one single description could be realistic. </p> <p> <a href='https://We3app.com' target='_blank' rel='nofollow'>We3app.com</a> rated my personality type as: </p> <p> <b>The Good-Timer</b><br> Calm, messy, sociable, curious, egoistic<br> 3.3% of women | 5.3% of men<br><br> You’re independent, imaginative, and intelligent, and you want to make fun where you can. Life is great, and you’ve got a good idea of how to live it—it’d take a lot to make you change your mind.<br><br> Your positive vibe and strong opinions mean that people tend to give you what you need, and are happy to follow your lead. You’ll listen to what they have to say, but you usually end up being right anyway. Besides, you’re the only one who knows how to get away with cutting corners: nobody can beat the system like you.<br><br> You’re never sure exactly how much is in your bank account, you regularly have to improvise when you run out of underwear, and you are disgusted at the concept of a cleaning schedule. You might agree to washing the car, but only because it will inevitably turn into a water fight.<br><br> You’re not one to say no to temptation, and there are certain habits you know it would be better to kick, but self-discipline is for monks and bodybuilders. </p> Evaluating Blockchain Companies 2018-08-29T00:00:00-04:00 https://mslinn.github.io/blog/2018/08/29/evaluating-blockchain-companies <div style=""> <picture> <source srcset="/blog/images/2018-10-19_10-51-13_690x511.webp" type="image/webp"> <source srcset="/blog/images/2018-10-19_10-51-13_690x511.png" type="image/png"> <img src="/blog/images/2018-10-19_10-51-13_690x511.png" title="Mike Slinn presents" class=" liImg " alt="Mike Slinn presents" /> </picture> </div> <div> <p> I have performed <a href='/evaluation/index.html'>technical due diligence</a> for investors since the mid-1980s. In this 40-minute presentation I explain how I evaluate blockchain-related technology companies. </p> <p> This talk was presented at the 6th Annual Global Big Data Conference in Santa Clara, California on August 29, 2018. The promotional video recording is <a href='https://www.youtube.com/watch?v=o_9q2USRzzI' target='_blank' rel='nofollow'>here</a>. </p> <p> <a href='https://www.infoq.com/presentations/evaluate-blockchain-companies/' target='_blank' rel='nofollow'>InfoQ produced the presentation</a> in their unique format. </p> <p> The slides are <a href='https://www.slideshare.net/mslinn/evaluating-blockchain-companies/mslinn/evaluating-blockchain-companies' target='_blank' rel='nofollow'>here</a>. </p> <p> The video recording is <a href='https://big.mslinn.com/video/18-aug-evaluatingbccompanies.mp4' target='_blank' rel='nofollow'>here</a>. </p> </div> <div style="text-align: center;"> <picture> <source srcset="/blog/images/6thGlobalBlockchain.webp" type="image/webp"> <source srcset="/blog/images/6thGlobalBlockchain.png" type="image/png"> <img src="/blog/images/6thGlobalBlockchain.png" title="Mike Slinn presents" class="center liImg2 rounded shadow" alt="Mike Slinn presents" /> </picture> </div> Bob Summerwill 2018-08-28T00:00:00-04:00 https://mslinn.github.io/blog/2018/08/28/summerwill <p> This is a great photo of Bob Summerwill and me in San Francisco August 2018! </p> <div style=""> <picture> <source srcset="/assets/images/ethereum/meBobSummerwill_690x518.webp" type="image/webp"> <source srcset="/assets/images/ethereum/meBobSummerwill_690x518.png" type="image/png"> <img src="/assets/images/ethereum/meBobSummerwill_690x518.png" title="Mike Slinn and Bob Summerwill in San Francisco August 2018" class=" liImg2 rounded shadow" alt="Mike Slinn and Bob Summerwill in San Francisco August 2018" /> </picture> </div> Keynote Panel Discussion - The Future of Blockchain 2018-08-23T00:00:00-04:00 https://mslinn.github.io/blog/2018/08/23/blockchain-conference-2018-08-28 <div style="text-align: right;"> <picture> <source srcset="/blog/images/mikeBigData_500x431.webp" type="image/webp"> <source srcset="/blog/images/mikeBigData_500x431.png" type="image/png"> <img src="/blog/images/mikeBigData_500x431.png" title="Global Big Data Conference featuring Mike Slinn as a speaker" class="right liImg2 rounded shadow" style="width: 100%; height: auto" alt="Global Big Data Conference featuring Mike Slinn as a speaker" /> </picture> </div> <p> I will participate in a keynote panel discussion on the Future of Blockchain on August 23 from 4:50 PM to 6:00 PM at the <a href='https://sanjose.eventful.com/events/blockchain-new-infrastructure-ai-/E0-001-112130527-9' target='_blank' rel='nofollow'>6th Annual Global Big Data Conference</a> in Santa Clara, California. </p> <p> Following are some ideas I hope to discuss with the other members of the panel. First, I would like to remind the reader that blockchain data is passive; without a program to access it blockchain data is inert. Now I'd like to give the definition that I will use for this discussion. </p> <p> The word <i>blockchain</i> is defined by <a href='https://en.wikipedia.org/wiki/Blockchain' target='_blank' rel='nofollow'>Wikipedia</a> as: &ldquo;a growing list of records, called blocks, which are linked using cryptography.&rdquo; The main theme I&rsquo;d like to personally bring forward during this discussion is that Blockchain does not imply or require decentralization, or even distributed systems. Instead, blockchain is a useful technology in its own right, even if it is not distributed / decentralized. I have nothing against decentralization, when used appropriately; the point I want to make is that decentralization is not necessary for blockchain to provide value. </p> <p> Here are the use cases I want to talk about: </p> <h2 id="dbs">#1: Secure Mobile and Embedded Databases</h2> <p> For me, <i>blockchain</i> is a file format that yields immutable data, highly resistant to modification. It is not a database, instead, it is a generic type of storage technology. Yes, an API could be provided that provides a structured interface to blockchain data, but there is no universally accepted standard in use for this in any blockchain implementation that I am aware of. Such a standard should exist, and I would be interested in learning about any standards work in this area. </p> <p> A database built using blockchain would not support the full <a href='https://en.wikipedia.org/wiki/Create,_read,_update_and_delete' target='_blank' rel='nofollow'>CRUD</a> API (create, read, update, and delete) functionality traditionally supported by SQL. Instead, only create and read functionality would be supported. </p> <p> This seems especially useful for mobile and embedded data recorders, for example flight recorders used in airplanes. I have not yet attempted to compute how feasible it might be to use blockchain to store video or audio streams from devices such as <a href='https://www.amazon.com/slp/police-body-camera/4r72vj5jucggbyv' target='_blank' rel='nofollow'>police body cameras</a>. </p> <h2 id="data">#2: Secure Data Distribution</h2> <p> Blockchain is a data storage technology that is known to be highly resistant to corruption or modification. Because data can only be created or read, but not modified or deleted, if you have a file of blockchain data that passes inspection you can be confident that you have all the data that was available at the time the file was last updated. </p> <p> Blockchain data could be stored in RFID tags. This is possible because RFID tags exist which can <a href='https://www.rfidjournal.com/faq/show?66' target='_blank' rel='nofollow'>store up to 66 KB of data</a>, and read/write RFID tags are available. Read-write RFID tags usually have a serial number that can&rsquo;t be written over. Additional blocks of data can be used to store additional information about the items the tag is attached to (these can usually be locked to prevent overwriting of data). </p> <p> This would allow blockchain to be used for <a href='https://www.theglobalist.com/smart-passports-making-travel-safer/' target='_blank' rel='nofollow'>smart passports</a> and smart ID cards. Secure, smart passports would provide extra security for passport/ID cardholders and would make counterfeiting extremely difficult, and would provide a digital record of countries visited, instead of relying on passport stamps, which are easy to counterfeit. </p> <p> Similarly, smart ID cards could hold mutable yet secure data. </p> <h2 id="builders">#3: Secure Software Build Systems</h2> <p> There have been many instances of corporate spies infiltrating software companies or their repositories. The perpetrators altered the source code, accompanying data or modified other build dependencies. When the next version of the software is published, all the customers who install updates will potentially be compromised. </p> <p> <a href='https://en.wikipedia.org/wiki/Git' target='_blank' rel='nofollow'><code>git</code></a> uses similar cryptography techniques as does blockchain. A blockchain-based build system would use a git tag to kick off build, and would need to also store and compare hashes for all dependencies to ensure that they had not changed. Because the software build toolchain is also a dependency, hashes for all the software used in the toolchain would need to be validated at build time. </p> <h2 id="dist">#4: Secure Software Distribution</h2> <p> Computer viruses have been a problem for decades. There have been many instances where programs have viruses embedded after they are published. This is generally true for any software downloaded from a torrent site. If software was packaged using blockchain technology, it could be retrieved for installation with confidence. Software installers that used blockchain in this way could be trusted. </p> <hr /> <h2 id="panel">The Panelists</h2> <p> After the panel, we panelists were photographed together. From left to right: Hayden Kirkpatrick, VP of Strategy at Esurance (moderator); Karen Hsu, CRO at BlockXypher; Steve Beauregard, CRO at Bloq; Mike Slinn, CTO at Micronautics Research (holding the microphone); Mark Javier, Account Executive at Ambisafe; and Camille Sanandaji, CEO at Foodstems. </p> <div style=""> <picture> <source srcset="/assets/images/ethereum/futureOfBlockchainPanelCrop2_690x192.webp" type="image/webp"> <source srcset="/assets/images/ethereum/futureOfBlockchainPanelCrop2_690x192.png" type="image/png"> <img src="/assets/images/ethereum/futureOfBlockchainPanelCrop2_690x192.png" title="Hayden Kirkpatrick (moderator), Karen Hsu, Steve Beauregard, Mike Slinn, and Camille Sanandaji" class=" liImg2 rounded shadow" alt="Hayden Kirkpatrick (moderator), Karen Hsu, Steve Beauregard, Mike Slinn, and Camille Sanandaji" /> </picture> </div> <div style=""> <picture> <source srcset="/assets/images/ethereum/blockchainFuturePanel_690x690.webp" type="image/webp"> <source srcset="/assets/images/ethereum/blockchainFuturePanel_690x690.png" type="image/png"> <img src="/assets/images/ethereum/blockchainFuturePanel_690x690.png" title="Mark Javier, Mike Slinn, Hayden Kirkpatrick (moderator), Steve Beauregard, Karen Hsu, and Camille Sanandaji" class=" liImg2 rounded shadow" alt="Mark Javier, Mike Slinn, Hayden Kirkpatrick (moderator), Steve Beauregard, Karen Hsu, and Camille Sanandaji" /> </picture> </div> Windows Subsystem for Linux Revisited 2018-08-20T00:00:00-04:00 https://mslinn.github.io/blog/2018/08/20/wsl-revisited <div style=""> <picture> <source srcset="/blog/images/wsl1_690x431.webp" type="image/webp"> <source srcset="/blog/images/wsl1_690x431.png" type="image/png"> <img src="/blog/images/wsl1_690x431.png" title="Windows Subsystem for Linux (WSL)" class=" liImg2 rounded shadow" alt="Windows Subsystem for Linux (WSL)" /> </picture> </div> <p> I&rsquo;ve been using <a href='https://docs.microsoft.com/en-us/windows/wsl/install-win10' target='_blank' rel='nofollow'>Windows Subsystem for Linux</a> (WSL) headless since it was first released with Windows 10 version 1607 in August 2016. The <a href='https://blogs.msdn.microsoft.com/commandline/2018/03/07/windows10v1803/' target='_blank' rel='nofollow'>April 2018 release</a> of Windows 10 (version 1803) significantly improved WSL. </p> <p> It is possible to work with Ubuntu graphically on a vanilla Windows machine. No special drivers are required. No special Linux or Ubuntu support is required from the computer vendor. </p> <p> This is my setup for running an X client like <a href='https://www.x.org/releases/X11R7.5/doc/man/man1/xeyes.1.html' target='_blank' rel='nofollow'><code>xeyes</code></a> or <a href='https://www.jetbrains.com/idea/' target='_blank' rel='nofollow'>IntelliJ IDEA</a> from WSL or WSL2, accessed via a Windows X server like <a href='https://sourceforge.net/projects/vcxsrv' target='_blank' rel='nofollow'>VcXsrv</a>. These scripts assume <a href='https://www.microsoft.com/en-us/p/ubuntu-1804/9n9tngvndl3q' target='_blank' rel='nofollow'>Ubuntu 18.04 was installed</a> under WSL. </p> <div style=""> <picture> <source srcset="/blog/images/wsl.webp" type="image/webp"> <source srcset="/blog/images/wsl.png" type="image/png"> <img src="/blog/images/wsl.png" title="Desktop showing Windows Subsystem for Linux (WSL)" class=" liImg2 rounded shadow" alt="Desktop showing Windows Subsystem for Linux (WSL)" /> </picture> </div> <h2 id="essential">Essential Scripts</h2> <p>Here are bash scripts to install everything:</p> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_init"> </script> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_setup"> </script> <h2 id="optional">Optional Scripts</h2> <p>The remaining scripts are all optional.</p> <h3 id="vnc">VNC</h3> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_vnc"> </script> <p>These bash scripts allow VNC to connect to a remote machine or to WSL on the local machine.</p> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=vncRun"> </script> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl"> </script> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=vncSample"> </script> <h2 id="packages">Packages and Programming Environments</h2> <p>Installation of various packages and programming environments follow.</p> <h3 id="git">Git</h3> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_git"> </script> <h3 id="python">Python</h3> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_python"> </script> <h3 id="nodejs">NodeJS</h3> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_node"> </script> <h3 id="jvm">Java Virtual Machine</h3> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_jvm"> </script> <h3 id="misc">Miscellaneous</h3> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_misc"> </script> <h3 id="atom">Atom Editor</h3> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_atom"> </script> <h3 id="wine">Wine</h3> <p> The advent of WSL means that Wine is no longer required! </p> <h2 id="update">Update Aug 17, 2020</h2> <p> Here are my notes on upgrading WSL and WSL2 from Ubuntu 19.11 to Ubuntu 20.04. </p> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/50a3a8d26cbab51c634ed8e2edf129ae.js"> </script> Ethereum Source Code Walkthrough 2018-06-13T00:00:00-04:00 https://mslinn.github.io/blog/2018/06/13/evm-source-walkthrough <div class="formalNotice rounded shadow" id="about"> <h2 id='about'>April 27, 2020</h2> <p> <div style=""> <picture> <source srcset="/assets/images/ethereum/Ethereum_logo_2014.svg" type="image/webp"> <source srcset="/assets/images/ethereum/Ethereum_logo_2014.svg" type="image/png"> <img src="/assets/images/ethereum/Ethereum_logo_2014.svg" title="Ethereum logo" class=" liImg right" style="height: 100px;" alt="Ethereum logo" /> </picture> </div> This was a sample work in progress as part of a proposal to the Ethereum Foundation Grants committee. They declined to fund this activity, but gave no reason and no feedback as to what might be acceptable. The preparation of my proposal took weeks, and the preliminary feedback that I had received from many knowledgeable people was that it had great value. </p> <p> I felt that the evaluation process was broken, in fact the entire organization was broken, and there was little hope that Ethereum would ever become a professional organization. 2 years later, I believe history has proved me right. This was the last blockchain-related initiative I participated in. </p> <p> I no longer maintain <a href='/blog/2017/11/29/web3j-scala.html'><code>web3j-scala</code></a>, an Ethereum-related project for Scala programmers. I created that open source project and worked on it for free for 3 years. It has been forked and I&rsquo;ve been told it is or was used in production. However, I see no reason to continue working for free on it. Others can carry the project forward if they want. </p> </div> <p> This is a brief walkthrough of some of the core source files for smart contracts in the official <a href='https://golang.org/' target='_blank' rel='nofollow'>Go language</a> Ethereum implementation, which includes the <a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/core/vm' target='_blank' rel='nofollow'><code>geth</code></a> command-line Ethereum client program, along with many other programs. Ethereum clients include an implementation of the Ethereum Virtual Machine (EVM), which are able to parse and verify the Ethereum blockchain, including smart contracts, and provides interfaces to create transactions and mine blocks. </p> <p> I&rsquo;ve added some suggestions for how the source code might be improved. If there is general agreement that these suggestions make sense (tell me in the comments!) then I&rsquo;ll create a pull request. </p> <h2 id='license'>License</h2> <p> <div style=""> <picture> <source srcset="/assets/images/lgplv3-147x51.webp" type="image/webp"> <source srcset="/assets/images/lgplv3-147x51.png" type="image/png"> <img src="/assets/images/lgplv3-147x51.png" title="LGPL logo" class=" right" alt="LGPL logo" /> </picture> </div> This Ethereum client project was released under the <a href='https://www.gnu.org/licenses/lgpl-3.0.en.html' target='_blank' rel='nofollow'>GNU Lesser General Public License, version 3</a> or later, which permits use of the code as a library in proprietary programs. </p> <h2 id='src'>Source Files</h2> <p> The <a href='https://github.com/hhatto/gocloc' target='_blank' rel='nofollow'><code>gocloc</code></a> program counted the following source files and lines: </p> <table class="table table_striped table_right"> <tr> <th>Language</th> <th>Files</th> <th>Blank Lines</th> <th>Comment Lines</th> <th>Code Lines</th> </tr> <tr> <th>Go</th> <td>1824</td> <td>58,134</td> <td>81,861</td> <td>639,435</td> </tr> <tr> <th>C</th> <td>55</td> <td>17,257</td> <td>30,909</td> <td>84,719</td> </tr> <tr> <th>C Header</th> <td>97</td> <td>2,559</td> <td>6,318</td> <td>15,083</td> </tr> <tr> <th>Markdown</th> <td>88</td> <td>3,152</td> <td>0</td> <td>9,175</td> </tr> <tr> <th>JavaScript</th> <td>13</td> <td>1,845</td> <td>4,495</td> <td>7,986</td> </tr> <tr> <th>Assembly</th> <td>39</td> <td>557</td> <td>957</td> <td>3,783</td> </tr> <tr> <th>JSON</th> <td>17</td> <td>0</td> <td>0</td> <td>2,065</td> </tr> <tr> <th>Protocol Buffers</th> <td>2</td> <td>113</td> <td>40</td> <td>1,030</td> </tr> <tr> <th>Plain Text</th> <td>11</td> <td>217</td> <td>0</td> <td>954</td> </tr> <tr> <th>C++</th> <td>4</td> <td>132</td> <td>102</td> <td>937</td> </tr> <tr> <th>BASH</th> <td>10</td> <td>178</td> <td>315</td> <td>931</td> </tr> <tr> <th>Perl</th> <td>10</td> <td>268</td> <td>1,289</td> <td>879</td> </tr> <tr> <th>JSX</th> <td>11</td> <td>119</td> <td>245</td> <td>722</td> </tr> <tr> <th>XML</th> <td>9</td> <td>0</td> <td>0</td> <td>651</td> </tr> <tr> <th>M4</th> <td>4</td> <td>79</td> <td>99</td> <td>649</td> </tr> <tr> <th>YAML</th> <td>20</td> <td>77</td> <td>42</td> <td>581</td> </tr> <tr> <th>NSIS</th> <td>5</td> <td>86</td> <td>154</td> <td>446</td> </tr> <tr> <th>Java</th> <td>4</td> <td>143</td> <td>187</td> <td>438</td> </tr> <tr> <th>Makefile</th> <td>11</td> <td>101</td> <td>84</td> <td>381</td> </tr> <tr> <th>Python</th> <td>6</td> <td>154</td> <td>250</td> <td>339</td> </tr> <tr> <th>HTML</th> <td>3</td> <td>15</td> <td>9</td> <td>245</td> </tr> <tr> <th>Solidity</th> <td>7</td> <td>56</td> <td>171</td> <td>213</td> </tr> <tr> <th>Bourne Shell</th> <td>6</td> <td>23</td> <td>25</td> <td>119</td> </tr> <tr> <th>CMake</th> <td>1</td> <td>9</td> <td>0</td> <td>35</td> </tr> <tr> <th>Awk</th> <td>1</td> <td>4</td> <td>4</td> <td>17</td> </tr> <tr> <th>TOML</th> <td>1</td> <td>0</td> <td>0</td> <td>3</td> </tr> <tr> <th>Total</th> <th class="table_right">2,260</th> <th class="table_right">85,278</th> <th class="table_right">127,556</th> <th class="table_right">771,825</th> </tr> </table> <h2 id='packages'>Packages</h2> <p> I used the following incantation to discover that <code>geth</code> defines 244 packages: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf0b055e71ea9'><button class='copyBtn' data-clipboard-target='#idf0b055e71ea9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>grep -rh "^package" | grep -v "not installed" | \ tr -d ';' | sed 's^//.*^^' | awk '{$1=$1};1' | \ sort | uniq | wc -l</pre> <p> I won't list them all. The <a href='https://godoc.org/github.com/ethereum/go-ethereum#pkg-subdirectories' target='_blank' rel='nofollow'><code>godoc</code></a> for the project contains much of the <a href='https://godoc.org/github.com/ethereum/go-ethereum#pkg-subdirectories' target='_blank' rel='nofollow'>following documentation</a> for the top-level packages. I provided the rest of the information from disparate sources, including reading the source code: </p> <table class="table table_striped"> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/accounts' target='_blank' rel='nofollow'>accounts</a></th> <td>implements high-level Ethereum account management.</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/blob/master/trie/trie.go' target='_blank' rel='nofollow'><code>trie</code></a></th> <td>provides a binary Merkle tree implementation.</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/cmd' target='_blank' rel='nofollow'>cmd</a></th> <td>Contains the following command-line tools. Most tools support the <code>--help</code> option. <table class="table table_striped" style="margin-top: 0.5em"> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/cmd/abigen' target='_blank' rel='nofollow'>abigen</a></th> <td>source code generator to convert Ethereum contract definitions into easy to use, compile-time type-safe Go packages. It operates on plain <a href='https://github.com/ethereum/wiki/wiki/Ethereum-Contract-ABI' target='_blank' rel='nofollow'>Ethereum contract ABIs</a> with expanded functionality if the contract bytecode is also available. However it also accepts Solidity source files, making development much more streamlined. Please see the <a href='https://github.com/ethereum/go-ethereum/wiki/Native-DApps:-Go-bindings-to-Ethereum-contracts' target='_blank' rel='nofollow'>Native DApps wiki page</a> for details. </td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8//cmd/bootnode' target='_blank' rel='nofollow'>bootnode</a></th> <td>runs a bootstrap node for the Ethereum Discovery Protocol. This is a stripped-down version of <code>geth</code> that only takes part in the network node discovery protocol, and does not run any of the higher level application protocols. It can be used as a lightweight bootstrap node to aid in finding peers in private networks.</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/cmd/clef' target='_blank' rel='nofollow'>clef</a></th> <td>a standalone signer that manages keys across multiple Ethereum-aware apps such as Geth, Metamask, and cpp-ethereum. <i>Alpha quality, not released yet.</i></td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/cmd/ethkey' target='_blank' rel='nofollow'>ethkey</a></th> <td>a key/wallet management tool for Ethereum keys. Allows user to add, remove and change their keys, and supports cold wallet device-friendly transaction inspection and signing. <a href='https://github.com/ethereum/guide/blob/master/ethkey.md' target='_blank' rel='nofollow'>This documentation</a> was written for the C++ Ethereum client implementation, but it is probably suitable for the Go implementation as well. </td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8//cmd/evm' target='_blank' rel='nofollow'>evm</a></th> <td>a version of the EVM (Ethereum Virtual Machine) for running bytecode snippets within a configurable environment and execution mode. Allows isolated, fine-grained debugging of EVM opcodes. Example usage: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide6a478f360ca'><button class='copyBtn' data-clipboard-target='#ide6a478f360ca' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>evm --code 60ff60ff --debug</pre></td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/cmd/faucet' target='_blank' rel='nofollow'>faucet</a></th> <td>an Ether faucet backed by a light client.</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/cmd/geth' target='_blank' rel='nofollow'>geth</a></th> <td>official command-line client for Ethereum. It provides the entry point into the Ethereum network (main-, test- or private net), capable of running as a full node (default) archive node (retaining all historical state) or a light node (retrieving data live). It can be used by other processes as a gateway into the Ethereum network via JSON RPC endpoints exposed on top of HTTP, WebSocket and/or IPC transports. For more information see the <a href='https://github.com/ethereum/go-ethereum/wiki/Command-Line-Options' target='_blank' rel='nofollow'>CLI Wiki page</a>.</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8//cmd/p2psim' target='_blank' rel='nofollow'>p2psim</a></th> <td>a simulation HTTP API. <a href='https://godoc.org/github.com/ethereum/go-ethereum/cmd/p2psim' target='_blank' rel='nofollow'>Docs are here</a>.</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/cmd/puppeth' target='_blank' rel='nofollow'>puppeth</a></th> <td>assembles and maintains private networks.</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/cmd/rlpdump' target='_blank' rel='nofollow'>rlpdump</a></th> <td>a pretty-printer for RLP data. RLP (Recursive Length Prefix) is the data encoding used by the Ethereum protocol. Sample usage: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc0c349fd4180'><button class='copyBtn' data-clipboard-target='#idc0c349fd4180' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>rlpdump --hex CE0183FFFFFFC4C304050583616263</pre></td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/blob/master/swarm/README.md' target='_blank' rel='nofollow'>swarm</a></th> <td>provides the <code>bzzhash</code> command, which computes a swarm tree hash, and implements the swarm daemon and tools. See <a href='https://swarm-guide.readthedocs.io' target='_blank' rel='nofollow'>the swarm documentation</a> for more information.</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/cmd/wnode' target='_blank' rel='nofollow'>wnode</a></th> <td>simple Whisper node. It could be used as a stand-alone bootstrap node. Also could be used for different test and diagnostics purposes.</td> </tr> </table> </td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/common' target='_blank' rel='nofollow'>common</a></th> <td>contains various helper functions worth checking out</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/consensus' target='_blank' rel='nofollow'>consensus</a></th> <td>implements different Ethereum consensus engines (which must conform to the <a href='https://godoc.org/github.com/ethereum/go-ethereum/consensus#Engine' target='_blank' rel='nofollow'><code>Engine</code> interface</a>): <a href='https://godoc.org/github.com/ethereum/go-ethereum/consensus/clique' target='_blank' rel='nofollow'><code>clique</code></a> implements proof-of-authority consensus, and <a href='https://godoc.org/github.com/ethereum/go-ethereum/consensus/ethash' target='_blank' rel='nofollow'>ethash</a> implements proof-of-work consensus.</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/console' target='_blank' rel='nofollow'>console</a></th> <td> Ethereum implements a JavaScript runtime environment (JSRE) that can be used in either interactive (console) or non-interactive (script) mode. Ethereum's JavaScript console exposes the full web3 JavaScript Dapp API and the admin API. <a href='https://github.com/ethereum/go-ethereum/wiki/JavaScript-Console' target='_blank' rel='nofollow'>More documentation is here.</a> This package implements JSRE for the <code>geth console</code> and <code>geth console</code> subcommands. </td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/containers' target='_blank' rel='nofollow'>containers</a></th> <td></td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/contracts' target='_blank' rel='nofollow'>contracts</a></th> <td></td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/core' target='_blank' rel='nofollow'>core</a></th> <td>implements the Ethereum consensus protocol, implements the Ethereum Virtual Machine, and other miscellaneous important bits</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/crypto' target='_blank' rel='nofollow'>crypto</a></th> <td>cryptographic implementations</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/dashboard' target='_blank' rel='nofollow'>dashboard</a></th> <td></td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/eth' target='_blank' rel='nofollow'>eth</a></th> <td>implements the Ethereum protocol</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/ethclient' target='_blank' rel='nofollow'>ethclient</a></th> <td>provides a client for the Ethereum RPC API</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/ethdb' target='_blank' rel='nofollow'>ethdb</a></th> <td></td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8//ethstats' target='_blank' rel='nofollow'>ethstats</a></th> <td>implements the network stats reporting service</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8//event' target='_blank' rel='nofollow'>event</a></th> <td>deals with subscriptions to real-time events</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/internal' target='_blank' rel='nofollow'>internal</a></th> <td>Debugging support, JavaScript dependencies, testing support</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/les' target='_blank' rel='nofollow'>les</a></th> <td>implements the Light Ethereum Subprotocol</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/light' target='_blank' rel='nofollow'>light</a></th> <td>implements on-demand retrieval capable state and chain objects for the Ethereum Light Client</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/log' target='_blank' rel='nofollow'>log</a></th> <td>provides an opinionated, simple toolkit for best-practice logging that is both human and machine readable</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/metrics' target='_blank' rel='nofollow'>metrics</a></th> <td>port of Coda Hale's Metrics library. Unclear why this was not implemented as a separate library, like <a href='https://github.com/rcrowley/go-metrics' target='_blank' rel='nofollow'>this one</a>.</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/miner' target='_blank' rel='nofollow'>miner</a></th> <td>implements Ethereum block creation and mining</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/mobile' target='_blank' rel='nofollow'>mobile</a></th> <td>contains the simplified mobile APIs to go-ethereum</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/node' target='_blank' rel='nofollow'>node</a></th> <td>sets up multi-protocol Ethereum nodes</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/p2p' target='_blank' rel='nofollow'>p2p</a></th> <td>implements the Ethereum p2p network protocols: Node Discovery Protocol, RLPx v5 Topic Discovery Protocol, Ethereum Node Records as defined in EIP-778, common network port mapping protocols, and p2p network simulation.</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/params' target='_blank' rel='nofollow'>params</a></th> <td></td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/rlp' target='_blank' rel='nofollow'>rlp</a></th> <td>implements the RLP serialization format</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/rpc' target='_blank' rel='nofollow'>rpc</a></th> <td>provides access to the exported methods of an object across a network or other I/O connection</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/signer' target='_blank' rel='nofollow'>signer</a></th> <td></td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/swarm' target='_blank' rel='nofollow'>swarm</a></th> <td></td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/tests' target='_blank' rel='nofollow'>tests</a></th> <td>implements execution of Ethereum JSON tests</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/trie' target='_blank' rel='nofollow'>trie</a></th> <td>implements Merkle Patricia Tries</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/vendor' target='_blank' rel='nofollow'>vendor</a></th> <td>contains a minimal framework for creating and organizing command line Go applications, and a rich testing extension for Go's testing package</td> </tr> <tr> <th class="code table_left"><a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/whisper' target='_blank' rel='nofollow'>whisper</a></th> <td>implements the Whisper protocol</td> </tr> </table> <p> <i>I used the following incantation to list the package names:</i> </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5d4f4ccbfe8d'><button class='copyBtn' data-clipboard-target='#id5d4f4ccbfe8d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>find . -maxdepth 1 -type d | sed 's^\./^^' | sed '/\..*/d'</pre> <p> The <a href='https://github.com/ethereum/go-ethereum/tree/release/1.8/build' target='_blank' rel='nofollow'><code>build/</code></a> directory does not contain a Go source package; instead, it contains scripts and configurations for building the package in various environments. </p> <h2 id='smartSrc'>Smart Contract Source Code</h2> <p> The <code>core/vm</code> directory contains the files that implement the EVM. These files are part of the <code>vm</code> package. Let's look at two of them: </p> <ul> <li> <a href='https://github.com/ethereum/go-ethereum/blob/master/core/vm/contract.go' target='_blank' rel='nofollow'><code>contract.go</code></a>, which defines smart contract behavior. </li> <li> <a href='https://github.com/ethereum/go-ethereum/blob/master/core/vm/contracts.go' target='_blank' rel='nofollow'><code>contracts.go</code></a>, responsible for executing smart contracts on the EVM. </li> </ul> <h3 id='refs'>Referenced Types</h3> <p> Two of the types used in the source files that we would like to understand are defined in <a href='https://github.com/ethereum/go-ethereum/blob/master/common/types.go' target='_blank' rel='nofollow'><code>common/types.go</code></a>. Let's look at them first. </p> <p> <a href='https://github.com/ethereum/go-ethereum/blob/master/common/types.go#L137-L138"' target='_blank' rel='nofollow'><code>Address</code></a> is defined as an array of 20 <code>byte</code>s: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id36f16a3c3884'><button class='copyBtn' data-clipboard-target='#id36f16a3c3884' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>const ( HashLength = 32 AddressLength = 20 ) // Address represents the 20 byte address of an Ethereum account. type Address [AddressLength]byte</pre> <p> <a href='https://github.com/ethereum/go-ethereum/blob/master/common/types.go#L43' target='_blank' rel='nofollow'><code>Hash</code></a> is defined as an array of 32 <code>byte</code>s: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4b7dc615f5cd'><button class='copyBtn' data-clipboard-target='#id4b7dc615f5cd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>// Hash represents the 32 byte Keccak256 hash of arbitrary data. type Hash [HashLength]byte</pre> <p> The opcodes for each version of the EVM are defined in <a href='https://github.com/ethereum/go-ethereum/blob/master/core/vm/jump_table.go' target='_blank' rel='nofollow'><code>jump_table.go</code></a>. The <a href='https://github.com/ethereum/go-ethereum/blob/master/core/vm/jump_table.go#L35-L51' target='_blank' rel='nofollow'><code>operation</code></a> <code>struct</code> defines the properties: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7ee0197ac460'><button class='copyBtn' data-clipboard-target='#id7ee0197ac460' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>type operation struct { // execute is the operation function execute executionFunc // gasCost is the gas function and returns the gas required for execution gasCost gasFunc // validateStack validates the stack (size) for the operation validateStack stackValidationFunc // memorySize returns the memory size required for the operation memorySize memorySizeFunc halts bool // indicates whether the operation should halt further execution jumps bool // indicates whether the program counter should not increment writes bool // determines whether this a state modifying operation valid bool // indication whether the retrieved operation is valid and known reverts bool // determines whether the operation reverts state (implicitly halts) returns bool // determines whether the operations sets the return data content }</pre> <p> Notice the <code>jumps</code> property, a Boolean, which if set indicates that the program counter should not increment after executing any form of jump opcode. </p> <p> The <code>destinations</code> type maps the hash of a smart contract to a bit vector for each the smart contract's entry points. If a bit is set, that indicates the EMV's program counter should increment after executing the entry point. <a href='https://github.com/ethereum/go-ethereum/blob/master/core/vm/analysis.go#L25-L28' target='_blank' rel='nofollow'><code>analysis.go</code></a> defines the <code>destinations</code> type like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbff08f22eb47'><button class='copyBtn' data-clipboard-target='#idbff08f22eb47' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>// destinations stores one map per contract (keyed by hash of code). // The maps contain an entry for each location of a JUMPDEST instruction. type destinations map[common.Hash]bitvec</pre> <h3 class="code">contract.go</h3> <p> This file defines smart contract behavior. </p> <h4 id='imports'>Imports</h4> <p> This comment applies to all of the Go source files in the entire project. I think the following absolute import would have been better specified as a relative import: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id02cda5350539'><button class='copyBtn' data-clipboard-target='#id02cda5350539' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>"github.com/ethereum/go-ethereum/common"</pre> <p> The relative import would look like this instead: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id06fc1e88cbfd'><button class='copyBtn' data-clipboard-target='#id06fc1e88cbfd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>"../../common"</pre> <p> If relative imports were used instead of absolute imports that point to the github repo, local changes to the project made by a developer would automatically be picked up. As currently written, absolute imports cause local changes to be ignored, in favor of the version on github. It might take a software developer a while to realize that the reason why their changes are ignored by most of the code base is because absoluate imports were used. It would then be painful to for the developer to modify the affected source files throughout the project such that they used relative imports. </p> <h4 id='types'>Types</h4> <p> The publicly visible <a href='https://github.com/ethereum/go-ethereum/blob/master/core/vm/contract.go#L30-L40' target='_blank' rel='nofollow'><code>AccountRef</code></a> type is defined as: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0605370cd245'><button class='copyBtn' data-clipboard-target='#id0605370cd245' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>// Account references are used during EVM initialisation and // it's primary use is to fetch addresses. Removing this object // proves difficult because of the cached jump destinations which // are fetched from the parent contract (i.e. the caller), which // is a ContractRef. type AccountRef common.Address</pre> <p> The same file defines a type cast from <code>AccountRef</code> to <code>Address</code>: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id01d096255569'><button class='copyBtn' data-clipboard-target='#id01d096255569' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>// Address casts AccountRef to a Address func (ar AccountRef) Address() common.Address { return (common.Address)(ar) }</pre> <p> The <a href='https://github.com/ethereum/go-ethereum/blob/master/core/vm/contract.go#L25-L28' target='_blank' rel='nofollow'><code>ContractRef</code></a> interface is used by the <code>Contract</code> <code>struct</code>, which we'll see in a moment. This <code>ContractRef</code> interface just consists of an <code>Address</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3d7f1831c438'><button class='copyBtn' data-clipboard-target='#id3d7f1831c438' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>// ContractRef is a reference to the contract's backing object type ContractRef interface { Address() common.Address }</pre> <p> The <a href='https://github.com/ethereum/go-ethereum/blob/master/core/vm/contract.go#L42-L65' target='_blank' rel='nofollow'><code>Contract</code></a> struct defines the behavior of Ethereum smart contracts, and is central to the topic, so here it is in all its glory: </p><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id632b4b86e782'><button class='copyBtn' data-clipboard-target='#id632b4b86e782' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>type Contract struct { CallerAddress common.Address caller ContractRef self ContractRef jumpdests destinations // result of JUMPDEST analysis. Code []byte CodeHash common.Hash CodeAddr *common.Address Input []byte Gas uint64 value *big.Int Args []byte DelegateCall bool }</pre> <p> <code>CallerAddress</code> is a publicly visible <code>Address</code> of the caller. <code>caller</code> and <code>self</code> are private <code>ContractRef</code>s, which as we know are really just <code>Address</code>es. </p> <p> <code>jumpdests</code>, a private field, has type <code>destinations</code>, which as we've already discussed defines if the entry point in the smart contract that need the program counter to be incremented after executing. </p> <p> <code>Code</code> is a a publicly visible <code>byte</code> slice. We don't yet know if this is the smart contract source code, compiled code, or something else. </p> <p> <code>CodeHash</code> is the publicly visible hash of the <code>Code</code>, while <code>CodeAddr</code> is a publicly visible pointer to the <code>Address</code> (of the code, presumably). </p> <p> <code>Gas</code> is the publicly visible amount of Ethereum gas allocated by the user for executing this smart contract, stored as an unsigned 64-bit integer. </p> <p> <code>Value</code> is a private pointer to a big integer. Possibly this might be the result of executing the contract? </p> <p> <code>Args</code> is a publicly visible <code>byte</code> slice, not sure what it is for. </p> <p> <code>DelegateCall</code> is a publicly visible Boolean value, unclear if this means the smart contract was invoked using <a href='https://solidity.readthedocs.io/en/v0.4.24/introduction-to-smart-contracts.html#delegatecall-callcode-and-libraries' target='_blank' rel='nofollow'><code>delegatecall</code></a>. From the documentation: "This means that a contract can dynamically load code from a different address at runtime. Storage, current address and balance still refer to the calling contract, only the code is taken from the called address. This makes it possible to implement the “library” feature in Solidity: Reusable library code that can be applied to a contract’s storage, e.g. in order to implement a complex data structure." </p> <h3 class="code">contracts.go</h3> <p> This file is responsible for executing smart contracts on the EVM. </p> <h4 id='imps'>Imports</h4> <p> The following imports are used: </p> <ul> <li> <a href='https://golang.org/pkg/crypto/sha256/' target='_blank' rel='nofollow'>Package <code>sha256</code></a> from the <code>crypto</code> project implements the SHA224 and SHA256 hash algorithms as defined in FIPS 180-4. </li> <li> <a href='https://godoc.org/github.com/pkg/errors' target='_blank' rel='nofollow'><code>errors</code></a>, the Go language simple error handling primitives, such as <code>error</code>. </li> <li> <a href='https://golang.org/pkg/math/big/' target='_blank' rel='nofollow'><code>math/big</code></a> implements arbitrary-precision arithmetic (big numbers). </li> <li> Other packages in this project (<code>go-ethereum</code>): <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1af3b2479eed'>"github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/common/math" "github.com/ethereum/go-ethereum/crypto" "github.com/ethereum/go-ethereum/crypto/bn256" "github.com/ethereum/go-ethereum/params"</pre> <p> Again, I think the above imports would have been better specified as relative imports: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7c3ec36f375d'>"../../common" "../../common/math" "../../crypto" "../../crypto/bn256" "../../params"</pre> </li> <li> <a href='https://godoc.org/golang.org/x/crypto/ripemd160' target='_blank' rel='nofollow'><code>ripemd160</code></a> implements the <a href='https://homes.esat.kuleuven.be/~bosselae/ripemd160.html' target='_blank' rel='nofollow'>RIPEMD-160 hash algorithm</a>, a secure replacement for the MD4 and MD5 hash functions. These hashes are also termed RIPE message digests. </li> </ul> <h4 id='pres'>Type <span class="code">PrecompiledContract</span></h4> <p><a href='https://github.com/ethereum/go-ethereum/blob/master/core/vm/contracts.go#L32-L38' target='_blank' rel='nofollow'><code>PrecompiledContract</code></a> is the interface for native Go smart contracts. This interface is used by precompiled contracts, as we will see next. <code>Contract</code> is a <code>struct</code> defined in <a href='https://github.com/ethereum/go-ethereum/blob/master/core/vm/contract.go' target='_blank' rel='nofollow'><code>contract.go</code></a>. </p> <h4 id='maps'>Pre-Compiled Contract Maps</h4> <p> These maps specify various types of cryptographic hashes and utility functions, accessed via their address. </p> <p> <a href='https://github.com/ethereum/go-ethereum/blob/master/core/vm/contracts.go#L40-L47' target='_blank' rel='nofollow'><code>PrecompiledContractsHomestead</code></a> contains the default set of pre-compiled contract addresses used in the Frontier and Homestead releases of Ethereum: <code>ecrecover</code>, <code>sha256hash</code>, <code>ripemd160hash</code> and <code>dataCopy</code>.</p> <p><a href='https://github.com/ethereum/go-ethereum/blob/master/core/vm/contracts.go#L49-L60' target='_blank' rel='nofollow'><code>PrecompiledContractsByzantium</code></a> contains the default set of pre-compiled contract addresses used in the Byzantium Ethereum release. All of the previously defined pre-compiled contract addresses are provided in Byzantium, plus: <code>bigModExp</code>, <code>bn256Add</code>, <code>bn256ScalarMul</code> and <code>bn256Pairing</code>.</p> <p> I'm not happy about the code duplication, whereby the contents of <code>PrecompiledContractsHomestead</code> are incorporated into <code>PrecompiledContractsByzantium</code> by listing the values again; this would be better expressed by referencing the values of <code>PrecompiledContractsHomestead</code> instead of duplicating them. </p> <h4 id='eval'>Contract Evaluator Function</h4> <p> The <code>RunPrecompiledContract</code> function runs and evaluates the output of a precompiled contract. It accepts three parameters: </p> <ul> <li> A <code>PrecompiledContract</code> instance. </li> <li> A byte array of input data. </li> <li> A reference to a <code>Contract</code>, defined in <a href='https://github.com/ethereum/go-ethereum/blob/master/core/vm/contract.go#L44-L65' target='_blank' rel='nofollow'><code>contract.go</code></a>, discussed above. </li> </ul> <p> The function returns: </p> <ul> <li> A byte array containing the output of the contract. </li> <li> An <code>error</code> value, which could be <code>nil</code>. </li> </ul> <h4 id='other'>Other Functions</h4> <ul> <li> <code>RunPrecompiledContract</code> &ndash; runs and evaluates the output of a precompiled contract; returns the output as a byte array and an <code>error</code>. </li> <li> <code>RequiredGas</code> (overloaded) &ndash; Computes the gas required for input data, specified as a byte array and returns a <code>uint64</code>. </li> <li> <code>Run</code> (overloaded) &ndash; Computes the smart contract for input data, specified as a byte array and returns the result as a left-padded byte array and an <code>error</code>. </li> <li> <code>newCurvePoint</code> &ndash; Unmarshals a binary blob into a bn256 elliptic curve point. BN-curves are an elliptic curves suitable for cryptographic pairings that provide high security and efficiency cryptographic schemes. See the IETF paper on <a href='https://tools.ietf.org/id/draft-kasamatsu-bncurves-01.html' target='_blank' rel='nofollow'>Barreto-Naehrig Curves</a> for more information. </li> </ul> Smart Contracts That Learn 2018-04-03T00:00:00-04:00 https://mslinn.github.io/blog/2018/04/03/smart-contracts-that-learn <p> It seems natural that machine learning and smart contracts will commonly be integrated over the next few years. </p> <iframe width="853" height="480" src="https://www.youtube.com/embed/a9PjjkMXsi0?rel=0" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen class="rounded shadow liImg" style="width: 100%"></iframe> <p> I presented <b>Smart Contracts That Learn</b> at the Global Blockchain Conference on April 3, 2018, in Santa Clara. This was a 40-minute technical presentation. Slides are <a href='https://www.slideshare.net/mslinn/smart-contracts-that-learn' target='_blank' rel='nofollow'>here</a>. </p> <p> <a href='https://www.infoq.com/presentations/smart-contracts-learn' target='_blank' rel='nofollow'>InfoQ.com</a> published the presentation in their unique format, featuring synchronized video and transcripts. </p> <div style=""> <picture> <source srcset="/assets/images/ethereum/gbc2018MikeCrop.webp" type="image/webp"> <source srcset="/assets/images/ethereum/gbc2018MikeCrop.png" type="image/png"> <img src="/assets/images/ethereum/gbc2018MikeCrop.png" title="Mike Slinn presenting Smart Contracts That Learn" class=" liImg2 rounded shadow" alt="Mike Slinn presenting Smart Contracts That Learn" /> </picture> </div> <p> A full-length (70 minute) version of this presentation was presented at IBM Ottawa on May 28, 2018. <a href='https://www.slideshare.net/mslinn/fullsize-smart-contracts-that-learn' target='_blank' rel='nofollow'>Here are the slides for the full-length version</a>. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/ibmOttawa.webp" type="image/webp"> <source srcset="/blog/images/ibmOttawa.png" type="image/png"> <img src="/blog/images/ibmOttawa.png" class="center liImg2 rounded shadow" /> </picture> </div> Svief 2018-03-02T00:00:00-05:00 https://mslinn.github.io/blog/2018/03/02/svief <p> Images from the <a href='https://twitter.com/SVIEF1/status/978771845796134912' target='_blank' rel='nofollow'>SVIEF Blockchain Conference</a> at Stanford University March 24, 2018. My presentation was entitled &ldquo;Smart Contracts that Learn&rdquo;. </p> <div style=""> <a href="https://us5.campaign-archive.com/?e=&id=29967a597f&u=21313a52472a2961f2cb58a34" target="_blank" ><picture> <source srcset="/assets/images/ethereum/bottos_svief_690x388.webp" type="image/webp"> <source srcset="/assets/images/ethereum/bottos_svief_690x388.png" type="image/png"> <img src="/assets/images/ethereum/bottos_svief_690x388.png" title="Chainovation with Michael Slinn and Sima Yazdani" class=" liImg2 rounded shadow" alt="Chainovation with Michael Slinn and Sima Yazdani" /> </picture></a> </div> <div style=""> <picture> <source srcset="/assets/images/ethereum/mike_690x460.webp" type="image/webp"> <source srcset="/assets/images/ethereum/mike_690x460.png" type="image/png"> <img src="/assets/images/ethereum/mike_690x460.png" title="Mike Slinn presenting at SVIEF Blockchain Conference" class=" liImg2 rounded shadow" alt="Mike Slinn presenting at SVIEF Blockchain Conference" /> </picture> </div> <div style=""> <picture> <source srcset="/assets/images/ethereum/mike2_690x460.webp" type="image/webp"> <source srcset="/assets/images/ethereum/mike2_690x460.png" type="image/png"> <img src="/assets/images/ethereum/mike2_690x460.png" title="Mike Slinn talking about fradulent events" class=" liImg2 rounded shadow" alt="Mike Slinn talking about fradulent events" /> </picture> </div> <div style=""> <picture> <source srcset="/assets/images/ethereum/mike5_690x460.webp" type="image/webp"> <source srcset="/assets/images/ethereum/mike5_690x460.png" type="image/png"> <img src="/assets/images/ethereum/mike5_690x460.png" title="Mike Slinn talking about self-optimizing contracts" class=" liImg2 rounded shadow" alt="Mike Slinn talking about self-optimizing contracts" /> </picture> </div> <div style=""> <picture> <source srcset="/assets/images/ethereum/mike6_690x460.webp" type="image/webp"> <source srcset="/assets/images/ethereum/mike6_690x460.png" type="image/png"> <img src="/assets/images/ethereum/mike6_690x460.png" title="Mike Slinn talking about the security / complexity tradeoff" class=" liImg2 rounded shadow" alt="Mike Slinn talking about the security / complexity tradeoff" /> </picture> </div> Smart Contracts For Enterprises 2018-01-18T00:00:00-05:00 https://mslinn.github.io/blog/2018/01/18/smart-contracts-for-enterprises <p> &lsquo;Polyglot&rsquo; means &ldquo;of many languages&rdquo;, and I believe that enterprises that integrate smart contracts into their infrastructure will require solutions that incorporate many simultaneous software languages, libraries and runtime environments, all working as one in a distributed environment. </p> <p> I anticipate significant need for system integration. I am interested in distributed consensus / blockchain / cryptocurrency architecture that support enterprise needs by providing useful features that are easy to use effectively. </p> <p text-align="left"> I presented &ldquo;Polyglot Ethereum Smart Contracts for the Enterprise&rdquo; at the <a href='https://wcef.co/' target='_blank' rel='nofollow'>World Crypto Economic Forum</a> in San Francisco, January 16, 2018. <a href='https://www.slideshare.net/mslinn/polyglot-ethereum-smart-contracts-for-the-enterprise' target='_blank' rel='nofollow'>Here is my slide deck.</a> </p> <div style=""> <picture> <source srcset="/assets/images/ethereum/WorldCryptoEconForum_690x439.webp" type="image/webp"> <source srcset="/assets/images/ethereum/WorldCryptoEconForum_690x439.png" type="image/png"> <img src="/assets/images/ethereum/WorldCryptoEconForum_690x439.png" title="Mike Slinn presents at the World Crypto Economic Forum" class=" liImg2 rounded shadow" style="width:100%" alt="Mike Slinn presents at the World Crypto Economic Forum" /> </picture> </div> <div style=""> <picture> <source srcset="/assets/images/ethereum/WCEF2_690x414.webp" type="image/webp"> <source srcset="/assets/images/ethereum/WCEF2_690x414.png" type="image/png"> <img src="/assets/images/ethereum/WCEF2_690x414.png" title="Mike Slinn presents at the World Crypto Economic Forum" class=" liImg2 rounded shadow" style="width:100%" alt="Mike Slinn presents at the World Crypto Economic Forum" /> </picture> </div> Tweet Stream Manager 2018-01-03T00:00:00-05:00 https://mslinn.github.io/blog/2018/01/03/tweet-stream-manager <p> I publish multiple tweet streams, so I wrote a Node.js web application using Express and Pug to help me manage them. This is what the user interface looks like. This is a silent video. Hopefully, it makes sense to everyone. </p> <iframe width="690px" height="388px" src="https://www.youtube.com/embed/r-47Jaa9Zek?rel=0" frameborder="1" class="liImg shadow" gesture="media" allow="encrypted-media" allowfullscreen></iframe> <p> For example, <a href='https://www.ScalaCourses.com' target='_blank' rel='nofollow'>ScalaCourses.com</a> currently has two tweet streams, each with their schedule (times shown are <code>HH:MM</code> for a 24-hour clock): </p> <ul> <li><b>ScalaCourses</b> &ndash; tweets are published at 6:45 every day</li> <li><b>SaleMore33</b> &ndash; This is a promotional campaign, starting on a certain day/time and ending on another day/time. This stream tweets at 3:24, 9:24, 13:24, 18:24, and 23:24.</li> </ul> <p><code>crontab</code>, running on a local XUbuntu machine, runs a Node.js command-line app that publishes the tweet streams:</p> <pre class="modest" style="font-size: small; padding: 0; margin-left: 0; margin-right:-20px"> NODE=/usr/bin/node TWEETER=/var/work/training/projects/tweeter 45 6 * * * $NODE $TWEETER/index.js $TWEETER/ScalaCourses 24 3,9,13,18,23 * * * $NODE $TWEETER/index.js $TWEETER/SaleMore33 2018-01-03 2018-01-15</pre> <p> I have been thinking about re-implementing this command-line program as an <a href='https://aws.amazon.com/lambda/' target='_blank' rel='nofollow'>AWS Lambda function</a> one day. Instead of using CSV files for persistence, I'll probably go with Dynamo. </p> The web3j-scala Ethereum Library 2017-11-29T00:00:00-05:00 https://mslinn.github.io/blog/2017/11/29/web3j-scala <div style="text-align: right;"> <picture> <source srcset="/assets/images/web3j.webp" type="image/webp"> <source srcset="/assets/images/web3j.png" type="image/png"> <img src="/assets/images/web3j.png" title="Web3j logo" class="right " style="margin-left: 1em; width: 15%" alt="Web3j logo" /> </picture> </div> <p> This video shows <a href='https://github.com/mslinn/web3j-scala' target='_blank' rel='nofollow'><code>web3j-scala</code></a>, an idiomatic Scala wrapper I wrote around <a href='https://web3j.io/' target='_blank' rel='nofollow'><code>web3j</code></a>, which is a Java 8 version of <a href='https://github.com/ethereum/web3.js' target='_blank' rel='nofollow'><code>web3.js</code></a>. These 3 libraries all leverage the <a href='https://github.com/ethereum/wiki/wiki/JSON-RPC' target='_blank' rel='nofollow'><code>json-rpc</code></a> protocol that all Ethereum clients support. <code>Web3j</code> is a lightweight, reactive, somewhat type-safe library for Java and Android that integrates with nodes on Ethereum blockchains. <code>Web3j-scala</code> provides type safety and enhanced scalability over its Java and JavaScript cousins, as well as the pleasure of writing solutions in Scala. </p> <iframe width="690" height="388" src="https://www.youtube.com/embed/8jZ7kICi4SE?rel=0" frameborder="0" allowfullscreen class="rounded shadow liImg"></iframe> <h2 id="transcript">Transcript</h2> <p> Much of the material for this presentation was taken from the <a href='https://github.com/mslinn/web3j-scala' target='_blank' rel='nofollow'><code>web3j-scala</code> GitHub page</a>. </p> How Much Do You Actually Program? 2017-08-07T00:00:00-04:00 https://mslinn.github.io/blog/2017/08/07/how-much-do-you-program <style> .modestPre { font-size: smaller; padding: 0; margin: 0; line-height: 130%; border-radius: 5px; } </style> <p> Last week someone asked me how much I programmed, or even if I actively programmed anymore. I recognized that the 20-something-year-old who asked me the question did not believe that people over 40 are capable of being skilled in relevant technology or productive programmers. Like all types of bigotry, ageism is ugliest when it is delivered with self-righteousness. </p> <h2 id="git-stats" style="font-family: Lucida Console, Courier, monospace">git-stats-scala</h2> <p> Following <a href='http://www.paulgraham.com/disagree.html' target='_blank' rel='nofollow'>Paul Graham&rsquo;s advice</a>, I decided to fight back with data. As the first step I wrote <a href='https://github.com/mslinn/git-stats-scala' target='_blank' rel='nofollow'><code>git-stats-scala</code></a>, a program that combs through an entire tree of git projects and summarizes the user's activity for a given time period. </p> <p>The resulting summary of commit statistics is intended to be placed near the top of one's resume.</p> <p> The output of this program merely answers the question: &ldquo;are you an active programmer?&rdquo; <code>Git-stats-scala</code> only reports textual additions and deletions, and includes the net change, which one hopes are indications of actual programming. Statistics are reported for each computer language found. The reader is free to impart any meaning they deem appropriate to this output. I make no claims regarding meaning. </p> <h2 id="myStats">My Statistics</h2> <p>Here is a summary of my git activity over the last 365 days:</p> <p style='font-weight: bold; margin-bottom: 0; padding-bottom: 0'>Subtotals By Language (lines changed across all projects)</p> <pre class="modest modestPre"> ┌───────────────────┬───────────────────┬───────────────────┬──────────────────┐ │Language │Lines added │Lines deleted │Net change │ ├───────────────────┼───────────────────┼───────────────────┼──────────────────┤ │Scala │+94,120 │-68,075 │+26,045 │ │SBT │+7,896 │-2,639 │+5,257 │ │Java │+23,955 │-20,597 │+3,358 │ │Markdown │+5,049 │-4,497 │+552 │ │Python │+371 │0 │+371 │ │HTML │+8,810 │-10,488 │-1,678 │ ├───────────────────┼───────────────────┼───────────────────┼──────────────────┤ │Total │+140,201 │-106,296 │+33,906 │ └───────────────────┴───────────────────┴───────────────────┴──────────────────┘ </pre> <p style="padding-top: 0"> Every project I was involved with had more features and better quality at the end of the time period compared to the state of the project at the beginning of the time period. You cannot value software by weighing it. </p> <div class="pullQuote"> You cannot value software by weighing it. </div> <p class="clear">I am curious to know how these numbers compare with those from other programmers.</p> <h2 id="cadenza" style="clear: both">The Cadenza Project</h2> <p> Here are stats over the same period for my contribution to the Cadenza core project. Cadenza is the web application that powers <a href='https://www.scalacourses.com' target='_blank' rel='nofollow'>ScalaCourses.com</a>. During this period I hived several F/OSS projects from the Cadenza code base, which is probably why it does not seem to be growing much even though I added many new features. </p> <pre class="modest modestPre"> ┌───────────────────┬───────────────────┬───────────────────┬──────────────────┐ │Language │Lines added │Lines deleted │Net change │ ├───────────────────┼───────────────────┼───────────────────┼──────────────────┤ │Scala │+42,687 │-38,237 │+4,450 │ │SBT │+233 │-120 │+113 │ │Java │+102 │-10 │+92 │ │Markdown │+519 │-2,617 │-2,098 │ │HTML │+1,134 │-5,345 │-4,211 │ ├───────────────────┼───────────────────┼───────────────────┼──────────────────┤ │Total │+44,675 │-46,329 │-1,654 │ └───────────────────┴───────────────────┴───────────────────┴──────────────────┘ </pre> Kafka Streams vs. Akka 2017-07-28T00:00:00-04:00 https://mslinn.github.io/blog/2017/07/28/kafka-vs-akka <div style="text-align: center; vertical-align: middle;"> <img src='/blog/images/kafka.webp' alt="Kafka logo" class="liImg" style="height: 150px" title="Kafka logo" /> <img src='/blog/images/akka_full_color.svg' alt="Akka logo" class="liImg" style="margin-left: 2em; height: 120px;" title="Akka logo" /> </div> <p> Context switches are expensive for CPUs to perform, and each incoming message in an <a href='https://akka.io/' target='_blank' rel='nofollow'>Akka</a> system requires a context switch when fairness (minimal latency deviation) is important <sup style='font-size: smaller'><a href='#1'>[1]</a></sup>. In contrast, <a href='https://kafka.apache.org/documentation/streams/' target='_blank' rel='nofollow'>Apache Kafka Streams</a> offers consistent processing times without requiring context switches, so Kafka Streams produces significantly higher throughput than Akka can when contending with a high volume of small computations that must be applied fairly. </p> <h2 id="fairness">Fairness</h2> <p> &ldquo;Fairness&rdquo; in a streaming system is a measure of how evenly computing resources are applied to each incoming message. A fair system is characterized by consistent and predictable latency for the processing of each message. An emergent effect of fair systems is that messages are journaled, processed, and transformed for downstream computation in approximately the order received. The output of a fair system roughly matches input in real time; a perfectly fair system would provide a perfect correspondence between input messages and output messages. </p> <p> &ldquo;Fair enough&rdquo; systems have a some acceptable amount of jitter in the ordering of the input vs. output messages. </p> <p> You might expect that streaming systems are generally fair, but systems based on Akka rarely are because of the implementation details of Akka's multithreaded architecture. Instead, Akka-based systems enqueue incoming messages for each actor, and the Akka scheduler periodically initiates the processing of each actor's input message queue. This is actually a type of batch processing. The reason Akka does this is so the high cost of CPU context switches can be amortized over several messages, to keep system throughput at an acceptable rate. </p> <h2 id="contextSwitching">Context Switching</h2> <p> It is desirable for the primary type of context switching in Akka systems to be between tasks on a thread because other types of context switching are more expensive. However, <a href='https://stackoverflow.com/questions/20288379/analysis-performance-of-forkjoinpool' target='_blank' rel='nofollow'>switching tasks on a thread</a> costs a surprisingly large amount of CPU cycles. To put this into context, the accompanying table shows various latencies compared to the amount of time necessary for a context switch on an <a href='https://ark.intel.com/products/27218/Intel-Xeon-Processor-5150-4M-Cache-2_66-GHz-1333-MHz-FSB' target='_blank' rel='nofollow'>Intel Xeon 5150</a> CPU <a href='#2'><sup style='font-size:smaller'>[2]</sup></a>. </p> <div style=""> <picture> <source srcset="/blog/images/latency.webp" type="image/webp"> <source srcset="/blog/images/latency.png" type="image/png"> <img src="/blog/images/latency.png" title="Latency numbers every programmer should know" class=" liImg2 rounded shadow" alt="Latency numbers every programmer should know" /> </picture> </div> <p> As you can see, a lot of computing can be done during the time that a CPU performs a context switch. My <a href='https://scalacourses.com/showCourse/45' target='_blank' rel='nofollow'>Intermediate Scala</a> course has several lectures that explore multithreading in great detail, with lots of code examples. </p> <h2 id="distributed">Building Distributed OSes is Expensive and Hard</h2> <p> Akka is rather low-level, and the actor model it is built around was originally conceived 45 years ago. Applications built using Akka require a lot of custom plumbing. Akka applications are rather like custom-built distributed operating systems spanning multiple <a href='https://en.wikipedia.org/wiki/OSI_model' target='_blank' rel='nofollow'>layers of the OSI model</a>, where application-layer logic is intertwined with transport-layer, session-layer and presentation-layer logic. </p> <p> Debugging and tuning a distributed system is inherently difficult. Building an operating system that spans multiple network nodes and continues to operate properly in the face of network partitions is even harder. Because each Akka application is unique, customers find distributed systems built with Akka to be expensive to maintain. </p> <h2 id="summary">Kafka vs. Akka</h2> <p> For most software engineering projects, it is better to use vendor-supplied or open-source libraries for building distributed systems because the library maintainers can amortize the maintenance cost over many systems. This allows users to focus on their problem, instead of having to develop and maintain the complex and difficult-to-understand plumbing for their distributed application. </p> <p> Akka is a poor choice for apps that need to do small amounts of computation on high volumes of messages where latency must be minimal and roughly equal for all messages. <a href='https://docs.confluent.io/current/streams/index.html' target='_blank' rel='nofollow'>Kafka Streams</a> is a better choice for those types of applications because programmers can focus on the problem at hand, without having to build and maintain a custom distributed operating system. <a href='https://docs.confluent.io/current/streams/concepts.html#ktable' target='_blank' rel='nofollow'>KTable</a> is also a nice abstraction to work with. </p> <!--p> Kafka also provides a higher-level mechanism for partitioning work among many servers than <a href='https://doc.akka.io/docs/akka/current/scala/common/cluster.html' target='_blank' rel='nofollow'>Akka clusters</a>, and it is no less efficient or fair. </p--> <div style=""> <picture> <source srcset="/assets/images/robotCircle400.webp" type="image/webp"> <source srcset="/assets/images/robotCircle400.png" type="image/png"> <img src="/assets/images/robotCircle400.png" title="EmpathyWorks™ – Artificial Personality For AI Systems" class=" right liImg" style="width: 150px" alt="EmpathyWorks™ – Artificial Personality For AI Systems" /> </picture> </div> <h2 id="motivation">Motivation</h2> <p> EmpathyWorks&trade; is a platform for modeling human emotions and network effects amongst groups of people as events impinge upon them. Each incoming event has the potential to cause a cascade of second-order events, which are spread throughout the network via people's relationships with one another. The actual processing required for each event is rather small, but given that the goal is to model everyone on earth (all 7.5 billion people), this is a huge computation to continuously maintain. For EmpathyWorks to function properly, incoming messages must be processed somewhat fairly. </p> <h2 id="ktables">KTables and Compacted Topics</h2> <p> Kafka Streams makes it easy to transform one or more immutable streams into another immutable stream. A KTable is an abstraction of a changelog stream, where each record represents an update. Instead of a using actors to track the state of multiple entities, a KTable provides an analogy to database tables, where the rows in a table contains the current state for each of the entities. Kafka is responsible for dealing with network partitions and data collisions, so the application layer does not become polluted with lower-layer concerns. </p> <p> Kafka&rsquo;s <a href='https://stackoverflow.com/questions/53390170/how-to-create-kafka-compacted-topic' target='_blank' rel='nofollow'>compacted topics</a> are a unique type of stream that only maintains the most recent message for each key. This produces something like a <i>materialized</i> or table-like view of a stream, with up-to-date values (subject to eventual consistency) for all key/value pairs in the log. </p> <hr class="separator"> <p id="1"> [1] From the <a href='https://letitcrash.com/post/40755146949/tuning-dispatchers-in-akka-applications' target='_blank' rel='nofollow'>Akka blog</a>: &ldquo;... pay close attention to your &lsquo;throughput&rsquo; setting on your dispatcher. This defines thread distribution &lsquo;fairness&rsquo; in your dispatcher &mdash; telling the actors how many messages to handle in their mailboxes before relinquishing the thread so that other actors do not starve. However, a context switch in CPU caches is likely each time actors are assigned threads, and warmed caches are one of your biggest friends for high performance. It may behoove you to be less fair so that you can handle quite a few messages consecutively before releasing it.&rdquo; </p> <p id="2">[2] Taken from <a href='https://gist.github.com/nelsnelson/3955759' target='_blank' rel='nofollow'>Latency numbers every programmer should know</a>. The Intel Xeon 5150 was released in 2006; for a more up-to-date information about CPU context switch time, please see <a href='https://www.quora.com/Linux-Kernel-How-much-processor-time-does-a-process-switching-cost-to-the-process-scheduler#pphKmT' target='_blank' rel='nofollow'>this Quora answer</a>. Although CPUs have gotten slightly faster since 2006, networks and SSDs have gotten much faster, so the relative latency penalty has actually increased over time. </p> Resume-Driven Development 2017-05-30T00:00:00-04:00 https://mslinn.github.io/blog/2017/05/30/resume-driven-development <div style="text-align: right;"> <a href="https://spark.apache.org/" target="_blank" rel="nofollow"><picture> <source srcset="/assets/images/spark.webp" type="image/webp"> <source srcset="/assets/images/spark.png" type="image/png"> <img src="/assets/images/spark.png" title="Apache Spark logo" class="right quartersize liImg2 rounded shadow" style="padding: 1em;" alt="Apache Spark logo" /> </picture></a> </div> <p>I had occasion to speak with a technical manager at a Very Large Corporation at length recently. He wanted advice on where to find Spark programmers. After my usual 20 questions I realized that his project did not have any requirement for the usual Spark capabilities:</p> <ul> <li>No need of machine learning or pattern recognition.</li> <li>He emphatically did not want to process streaming data.</li> <li>He had a rather small volume of data.</li> <li>He had no need of interactive analysis.</li> <li>There was no need for ETL.</li> <li>&ldquo;Data enrichment&rdquo; in his case was no more difficult than joining two SQL tables.</li> <li>There was no requirement for trigger event detection.</li> <li>... and there was no requirement for session analysis.</li> </ul> <p>This man just wanted to put &ldquo;I managed the development of an Apache Spark application&rdquo; on his resume.</p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/cart_before_the_horse1_450x313.webp" type="image/webp"> <source srcset="/blog/images/cart_before_the_horse1_450x313.png" type="image/png"> <img src="/blog/images/cart_before_the_horse1_450x313.png" title="The cart is before the horse" class="center quartersize liImg2 rounded shadow" alt="The cart is before the horse" /> </picture> </div> <h2 id="turd">Shine Up That Turd</h2> <p> Buffing up a resume with a nonsensical misapplication of the latest software fashion is nothing new. Few people involved with such a project have a positive experience, however. When complex technologies are misapplied they often fail to deliver value, or the project just fails, which contributes to the <a href='https://www.gartner.com/technology/research/methodologies/hype-cycle.jsp' target='_blank' rel='nofollow'>trough of disillusionment</a> in the Gartner Hype Cycle. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/WCFields_450x576.webp" type="image/webp"> <source srcset="/blog/images/WCFields_450x576.png" type="image/png"> <img src="/blog/images/WCFields_450x576.png" title="W. C. Fields" class="center quartersize liImg2 rounded shadow" alt="W. C. Fields" /> </picture> </div> <p> Consider the cynical and selfish world view that such a person must have to engage in this sort of activity. Software professionals are rewarded for being fashionable, and part of the mystique is to say and do things that others don&rsquo;t fully comprehend. If one keeps moving fast enough, and learns some key phrases to use as put downs that sound impressive but have no meaning when taken out of context (&ldquo;your approach would likely cause a data race condition&rdquo;), others are kept off balance long enough for the perpetrator to achieve advantage. </p> <p> As <a href='https://en.wikipedia.org/wiki/W._C._Fields' target='_blank' rel='nofollow'>W.C. Fields</a> said 100 years ago: &ldquo;If you can&rsquo;t dazzle them with brilliance, baffle them with bullshit.&rdquo; He also made a movie entitled <a href='https://www.youtube.com/watch?v=-AdXkJbW1Tg' target='_blank' rel='nofollow'>Never Give a Sucker an Even Break</a>... but I digress. </p> <h2 id="fashion">Fashion is the Enemy of Delivering Valuable Results on Time</h2> <div style="text-align: center;"> <picture> <source srcset="/assets/images/fashion_450x675.webp" type="image/webp"> <source srcset="/assets/images/fashion_450x675.png" type="image/png"> <img src="/assets/images/fashion_450x675.png" title="Fashion model" class="center quartersize liImg2 rounded shadow" alt="Fashion model" /> </picture> </div> <p> I often remark that the software business is just like the fashion business. The cycle goes like this: </p> <ol> <li>The latest thing is preannounced. No documentation is available. Big claims are made.</li> <li>Early code is provided, but it is slow, very buggy, there is no documentation, and the wild claims continue.</li> <li>Seminars pump up the hype. Arm-waving abounds. &lsquo;Experts&rsquo; associated with vendors are interviewed.</li> <li>A whole new vocabulary is invented to describe the features. However, since those terms are never defined, few people realize that the message is actually &ldquo;same old stuff with a new coat of paint.&rdquo; The cool kids learn the words and can parrot them back they way they heard them, but they don't actually know what they are saying. Even if you knew what the phrase &ldquo;a monad is a monoid in the category of endofunctors&rdquo; means, it would not help you add value to a business process... and for most programmers who work in the business world, that is what actually pays their bills.</li> <li>Gartner, ThoughtWorks and O&rsquo;Reilly Radar say nice things about the new tech.</li> <li>Marketing and sales activity shifts into high gear. Everyone&rsquo;s laptop has stickers on it that show the owner is hip.</li> <li>Some incomplete and mostly useless docs trickle out. Release candidates are eventually made available but it is not obvious how they provide value without significant magic being applied.</li> <li>Version 1 finally arrives, completely useless to everyone except those few customers who paid huge dollars to the vendor, so they could get the bragging rights for what is essentially a custom build.</li> <li>No useful benchmarks are available. Bugs crawl out from every crevasse. Documentation reads like it was badly translated from Chinese into Bantu, then into Navajo and then English.</li> <li>Version 1.1 comes out, to address stop-ship problems that somehow did not prevent version 1 from shipping.</li> <li>Version 2 is announced, completely incompatible with Version 1.x, and it is touted as revolutionary.</li> <li>The digital lemmings decide that this is the cliff they hold dearest in their hearts, and proceed to jump from it <i>en masse</i>.</li> <li>Gartner, ThoughtWorks and O&rsquo;Reilly Radar downgrade the new tech to &ldquo;meh&rdquo;</li> <li>... and the churn continues.</li> </ol> <p> Documentation is not written to be understood, they are written to encourage payments to the digital elite &ndash; who have no interest in providing value; they just want to make money from foolish lemmings. This encourages others to aspire to be elitists. The hype cycle only works because our society accepts bullshit, and software has become a cultural phenomenon. We have the software quality and utility that we deserve as a society. If we want better software, we as a society will have to modify our purchasing behavior accordingly. </p> <p> The software business has often demonstrated an insatiable need to be cool, at the expense of providing little or no value because evidencing value is either too hard for the many posers in the software business, or too boring for those who are capable but jaded. </p> Better Syntactic Sugar for Scala Futures 2017-05-25T00:00:00-04:00 https://mslinn.github.io/blog/2017/05/25/better-syntactic-sugar-for-scala-futures <div style="text-align: right;"> <picture> <source srcset="/blog/images/sugarCubes_225x169.webp" type="image/webp"> <source srcset="/blog/images/sugarCubes_225x169.png" type="image/png"> <img src="/blog/images/sugarCubes_225x169.png" title="Sugar cubes" class="right quartersize liImg2 rounded shadow" alt="Sugar cubes" /> </picture> </div> <p> Ever since Scala&rsquo;s <code>Future</code>s were initially provided as part of Akka 2.0, programmers have been confused by the non-intuitive syntax. That is why <a href='https://www.ScalaCourses.com' target='_blank' rel='nofollow'>ScalaCourses.com</a> dedicates an entire lecture to <a href='https://www.scalacourses.com/student/showLecture/172' target='_blank' rel='nofollow'>For-comprehensions With Futures</a>, as part of an 8 lecture series on Scala <code>Future</code>s within the <a href='https://www.scalacourses.com/showCourse/45' target='_blank' rel='nofollow'>Intermediate Scala</a> course. </p> <p> As Viktor Klang <a href='https://viktorklang.com/blog/Futures-in-Scala-protips-2.html' target='_blank' rel='nofollow'>points out</a> in his blog, the following code does not run 3 <code>Future</code>s in parallel: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id57989e28df33'><button class='copyBtn' data-clipboard-target='#id57989e28df33' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>def doSomething(someParameter: SomeType) (implicit ec: ExecutionContext): Future[Something] = for { v1 &lt;- Future(someCalculation()) v2 &lt;- Future(someOtherCalculation()) v3 &lt;- Future(someDifferentCalculation()) } yield doSomethingWith(v1, v2, v3)</pre> <p> The compiler has no way of ascertaining the programmer&rsquo;s intent &ndash; perhaps it is desirable for some reason to run the 3 <code>Future</code>s one after the other. </p> <p>Viktor suggests this syntax to make futures run in parallel:</p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9dafd1f66dff'><button class='copyBtn' data-clipboard-target='#id9dafd1f66dff' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>def doSomething(someParameter: SomeType) (implicit ec: ExecutionContext): Future[Something] = for { f1 = Future(someCalculation()) f2 = Future(someOtherCalculation()) f3 = Future(someDifferentCalculation()) v1 &lt;- f1 v2 &lt;- f2 v3 &lt;- f3 } yield doSomethingWith(v1, v2, v3)</pre> <p> While Viktor&rsquo;s solution works, it is verbose. Worse, it silently fails to run the futures in parallel if the programmer accidentally writes even one of the expressions out of order: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4081c424bd15'><button class='copyBtn' data-clipboard-target='#id4081c424bd15' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>def doSomething(someParameter: SomeType) (implicit ec: ExecutionContext): Future[Something] = for { f1 = Future(someCalculation()) v1 &lt;- f1 f2 = Future(someOtherCalculation()) v2 &lt;- f2 f3 = Future(someDifferentCalculation()) v3 &lt;- f3 } yield doSomethingWith(v1, v2, v3)</pre> <p> Again, the compiler has no way of ascertaining the programmer's intent, so it should not generate an error or warning message. </p> <h2 id='macro'>We Need A Macro</h2> <p> A new right-associative operator, implemented as a Scala macro, would make the programmer's intent clear. Let&rsquo;s call this operator <code>&lt;=:</code> (parallel generator). The above code could be rewritten using the parallel generation operator like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf5bc9afcc7f5'><button class='copyBtn' data-clipboard-target='#idf5bc9afcc7f5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>def doSomething(someParameter: SomeType) (implicit ec: ExecutionContext): Future[Something] = for { v1 &lt;=: Future(someCalculation()) v2 &lt;=: Future(someOtherCalculation()) v3 &lt;=: Future(someDifferentCalculation()) } yield doSomethingWith(v1, v2, v3)</pre> <p>There is no longer any doubt that the programmer intended for all 3 futures to run in parallel.</p> <p> The macro would examine all the for-expression&rsquo;s generators and expand consecutive expressions that use the <code>&lt;=:</code> operator to a series of variable declarations using the <code>=</code> operator followed by a series of assignments using the <code>&lt;-</code> operator, exactly as we saw earlier: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc12400f87fbc'><button class='copyBtn' data-clipboard-target='#idc12400f87fbc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>def doSomething(someParameter: SomeType) (implicit ec: ExecutionContext): Future[Something] = for { f1 = Future(someCalculation()) f2 = Future(someOtherCalculation()) f3 = Future(someDifferentCalculation()) v1 &lt;- f1 v2 &lt;- f2 v3 &lt;- f3 } yield doSomethingWith(v1, v2, v3)</pre> <p> Because the compiler &lsquo;knows&rsquo; that the programmer&rsquo;s intention was to run the <code>Future</code>s in parallel, this sort of error could cause an error or warning message to be generated: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5d734a6be1d5'><button class='copyBtn' data-clipboard-target='#id5d734a6be1d5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>def doSomething(someParameter: SomeType) (implicit ec: ExecutionContext): Future[Something] = for { v1 &lt;=: Future(someCalculation()) x <- List(1, 2, 3) v2 &lt;=: Future(someOtherCalculation()) y <- List("a", "b") v3 &lt;=: Future(someDifferentCalculation()) } yield doSomethingWith(v1, v2, v3)</pre> <p>... or the macro might reorder the generators and issue a warning that it did so.</p> <h2 id="macro">Include the Macro in Scala 2.12.x</h2> <p> This macro should become part of the Scala language so long as <code>Future</code>s are part of the standard runtime. If and when <code>Future</code>s are hived out of the standard runtime, the macro should be packaged with <code>Future</code>s. </p> <hr /> <p> PS: <a href='https://twitter.com/flaviusbraz' target='_blank' rel='nofollow'>@flaviusbraz</a> tweeted on May 25, 2017: <a href='https://twitter.com/flaviusbraz/status/867868408267685888' target='_blank' rel='nofollow'>&ldquo;Easily doable with a macro transformation. In fact, I&rsquo;ve implemented this transformation at Twitter, but it&rsquo;s not open source.&rdquo;</a> </p> Exploratory Conversation With AIs 2017-04-29T00:00:00-04:00 https://mslinn.github.io/blog/2017/04/29/exploratory-conversations-with-ais <p> This is a high-level proposal, not documentation for any specific system that exists today. I would be happy to discuss possible implementation details with interested parties, including the creation of a prototype. </p> <h2 id="overview">Intents and Subgoals</h2> <p> Some of today&rsquo;s interactive voice response systems, such as AWS&rsquo;s <a href='https://aws.amazon.com/lex/' target='_blank' rel='nofollow'>Lex</a>, Amazon&rsquo;s <a href='https://aws.amazon.com/alexa/' target='_blank' rel='nofollow'>Alexa</a>, and Google&rsquo;s <a href='https://cloud.google.com/dialogflow/docs/' target='_blank' rel='nofollow'>Dialogflow</a>, use <i>intent</i>s as a means of gathering the required parameters to fulfill a voice command. The next big step in voice interfaces to computation would be the ability to have an exploratory conversation, where one or more subgoals may or may not emerge. </p> <p> Subgoals might be fulfilled by intents. Continuous tracking of viable subgoals could be accomplished by a <a href='https://en.wikipedia.org/wiki/Constraint_programming' target='_blank' rel='nofollow'>constraint-based solver</a> that uses rules to participate in an exploratory conversation by recognizing and prioritizing potential subgoals, thereby activating and deactivating intents during the conversation. Once a user confirms that a potential subgoal is desirable, the system might consider that goal to be the only goal worth pursuing, or it might continuously elicit more information from the user regarding other potential subgoals. User-supplied information might be applied to a model associated with a subgoal/intent, and/or it might be retained as a user preference. </p> <div style="text-align: center;"> <picture> <source srcset="/assets/images/anthropomorph.webp" type="image/webp"> <source srcset="/assets/images/anthropomorph.png" type="image/png"> <img src="/assets/images/anthropomorph.png" title="Do not anthropomorphize computers. They really hate that!" class="center halfsize liImg2 rounded shadow" alt="Do not anthropomorphize computers. They really hate that!" /> </picture> </div> <h2 id="anthropomorphism">Anthropomorphism</h2> <p>If an AI system is a good conversationalist, even if it has no physical presence, people will <a href='https://en.wikipedia.org/wiki/Anthropomorphism#In_computing' target='_blank' rel='nofollow'>anthropomorphically</a> attribute the system with human traits, emotions, and intentions. This should not be discouraged. Rather, AI systems should aspirationally model our higher selves. The movie <a href='https://en.wikipedia.org/wiki/Her_(film)' target='_blank' rel='nofollow'>Her</a> explored this in a charming way. </p> <h2 id="breakdown">Breaking It Down</h2> <p>An interactive voice response system recognizes an <i>intent</i> by a key word or phrase uttered by the user. For example, if the user says: &ldquo;Computer, order dinner&rdquo;, and the system was previously &lsquo;taught&rsquo; to understand the word &ldquo;order&rdquo; as a key word for launching the <code>OrderDinner</code> intent, the system would then elicit the kind of food the user wanted, how many people would be eating, etc, and then order the desired dinner.</p> <p style="font-style: italic">Anthropomorphically: if intents are the only programming technique employed, they present interactive voice systems as slaves.</p> <p>Intents are useful for recognizing commands and gathering enough associated data to carry out the command. Intents are not useful for open-ended conversations where there is no explicit goal, or potential subgoals emerge as the result of a dialog. Once a subgoal is identified and confirmed, however, processing an intent is an efficient mechanism for fulfilling the emergent subgoal. </p> <p>Designers of chatbots and video games are familiar with goal-seeking behavior. <i>Exploratory conversation</i> is how people interact with each other IRL (in real life). Each individual uses a variety of strategies for initiating exploratory conversation, and participating in it. For example, small talk at a gathering is a useful skill for getting to know other people, and a socially adept practitioner of small talk can innocuously gather a lot of information in this way. Another strategy is to make a controversial statement, and use the resulting banter to learn about the conversationalists and the possible subgoals that they might request. A wide variety of such strategies exist, and could be utilized by conversational systems. </p> <p style="font-style: italic">Anthropomorphically: with exploratory conversation capability, interactive voice systems can be presented socially as co-operative entities.</p> <h2 id="agenda">Agenda and Strategy</h2> <p>Unlike people, computers can only do a finite number of tasks, and voice recognition systems are programmed with a finite number of intents. I define the potential <i>agenda</i> for a chatbot or video game to be the entire scope of its pre-programmed intents. Agendas may be fully disclosed, or they might be obvious, or they might be unveiled over time. Ethical considerations might apply to the design and implementation of conversational AI systems. </p> <p>Users should be apprised of the agenda of every autonomous computer entity they encounter. To mitigate potential problems, an industrial code of conduct should be established. The Europe Union will likely be the first government to require published standards and possibly a certification process. </p> <p>I define <i>strategy</i> to mean the autonomous modification to goal-seeking behavior when a criterion is met. For example, an insurance chatbot might begin to solicit sensitive information only after it has reason to believe that a modicum of trust has been established with the prospect. How a strategy is executed is quite significant; by this I mean the degree of <i>diplomacy</i>. Some circumstances might sacrifice diplomacy to save lives, while under normal circumstances the AI entity should treat everyone with respect and kindness. Again, the Europe Union is likely to be the first government to require published standards and possibly a certification process. </p> <h2 id="2017-05-05">Update May 5, 2017</h2> <p>Today, a week after I first published this blog posting, I received the following email from <a href='https://cloud.google.com/dialogflow/docs/' target='_blank' rel='nofollow'><code>api.ai</code></a>. Two points worth mentioning:</p> <ol> <li>I have no apps running on api.ai that use their Small Talk feature.</li> <li>The Small Talk feature, even after the changes mentioned below, does not approach the capability I proposed above.</li> </ol> <p>Nonetheless, it was great to get the information.</p> <div class="quote"> <p>Hi Michael,</p> <p> We noticed you are using the <a href='https://dialogflow.cloud.google.com/#/agent/smalltalk' target='_blank' rel='nofollow'>Small Talk customization feature</a> in your agent, and that's why we are getting in touch. </p> <p>The Small Talk customization has been updated.</p> <p> The new version will be even faster and more accurate. It also includes some additional intents that you can customize. The update also includes changes in actions and parameters returned with the responses. </p> <p> If you use them in your business logic, please review the changes <a href='https://cloud.google.com/dialogflow/docs/' target='_blank' rel='nofollow'>here</a>. If you’re ready, you can apply change right now here. Otherwise, the changes will be applied automatically on May 29, 2017. </p> <p>If you’d like a more flexible way to implement small talk, then get your <a href='https://console.api.ai/api-client/#/agent/prebuiltAgents/Small-Talk' target='_blank' rel='nofollow'>source agent</a> and implement the use cases you care about in your timeline. </p> <p>Regards,</p> <p><a href='https://cloud.google.com/dialogflow' target='_blank' rel='nofollow'><code>API.AI</code></a> Team</p> </div> <h2 id='author'>About the Author</h2> <p> Mike Slinn has been pursuing EmpathyWorks&trade; original AI research (augmenting machine learning with simulations and event-driven architecture) since 2007. </p> UI Considerations for the Visually Impaired 2017-04-04T00:00:00-04:00 https://mslinn.github.io/blog/2017/04/04/ui-considerations-for-the-visually-impaired <p> If you look at a computer screen all day, and the screen has a lot of glare, the resulting eyestrain can temporarily degrade your vision to the same degree as a visually impaired person. </p> <h2 id="background">Background</h2> <p> According to the Farlex Free Medical Dictionary, <a href='https://medical-dictionary.thefreedictionary.com/Visual+Impairment' target='_blank' rel='nofollow'>visual impairment</a>, or low vision, is a severe reduction in vision that cannot be corrected with standard glasses or contact lenses and reduces a person's ability to function at certain or all tasks. The World Health Organization <a href='https://www.who.int/mediacentre/factsheets/fs282/en/' target='_blank' rel='nofollow'>estimates</a> that 285 million people are visually impaired worldwide, of which 246 million have low vision and 39 million are blind. People's vision generally deteriorates with age. </p> <p> In the US, cataracts are the most common form of visual impairment, according to the <a href='https://www.nei.nih.gov/learn-about-eye-health/resources-for-health-educators/eye-health-data-and-statistics' target='_blank' rel='nofollow'>National Eye Institute</a>. The NEI estimates that more than 24 million Americans over the age of 40 have cataracts. &ldquo;The risk of cataracts increases with each decade of life starting around age 40. By age 75, half of white Americans have cataracts. By age 80, 70 percent of whites have cataracts, compared with 53 percent of blacks and 61 percent of Hispanic Americans.&rdquo; Males are twice as likely as females to get cataracts. </p> <div style=""> <picture> <source srcset="/assets/images/1_2010_Prevalence_Rates_Age_Race_Cataract_v4.webp" type="image/webp"> <source srcset="/assets/images/1_2010_Prevalence_Rates_Age_Race_Cataract_v4.png" type="image/png"> <img src="/assets/images/1_2010_Prevalence_Rates_Age_Race_Cataract_v4.png" title="2010 US prevalence rates for cataracts by age and race" class=" liImg2 rounded shadow" style="width: 100%" alt="2010 US prevalence rates for cataracts by age and race" /> </picture> </div> <p> Cataracts start slowly. If you wear glasses, at first you think you can&lsquo;t quite clean them properly. Then you notice your sunglasses are also always dirty. Then you notice that white screens with black text are hard to read because of the glare from the white areas. </p> <p> The medical treatment is to replace the cloudy and .webp lens in the eye that has the cataract with a plastic lens. In <a href='https://ec.europa.eu/eurostat/statistics-explained/index.php/Surgical_operations_and_procedures_statistics' target='_blank' rel='nofollow'>many European countries</a>, cataract surgery is the most common operation performed in that country. Usually people delay the surgery until their quality of life and independence his diminished markedly. </p> <h2 id="coping">Coping With Visual Impairment</h2> <h3 id="saturation">Saturated Color and White Backgrounds</h3> <p> Dark screens with a minimum of saturated color helps. Increasing font size helps, and sanserif fonts, which have fewer spindly features, are easier to read. </p> <p> Some OSes offer a dark theme. You can enable the Windows 10 dark theme from <b>Settings</b> / <b>Personalization</b> / <b>Colors</b>. Scroll down and select &lsquo;Dark&rsquo; under &lsquo;Choose your app mode&rsquo;. The Universal Windows Platform applications will immediately darken. However, if a developer did not make the effort to support the dark theme, their program will continue with their original color theme. Unfortunately, Windows File Explorer is one of those applications. </p> <div style=""> <picture> <source srcset="/assets/images/img_57a8f5e080b37.webp" type="image/webp"> <source srcset="/assets/images/img_57a8f5e080b37.png" type="image/png"> <img src="/assets/images/img_57a8f5e080b37.png" title="Enabling the Windows 10 dark theme" class=" liImg2 rounded shadow" style="width: 100%" alt="Enabling the Windows 10 dark theme" /> </picture> </div> <p>Some software offers a nice dark theme that does not require any fussing, for example IntelliJ IDEA's Darkula, Adobe Premiere Pro and Adobe Audition.</p> <p> The Firefox web browser&rsquo;s <a href='https://addons.mozilla.org/en-US/firefox/addon/dark-background-light-text/' target='_blank' rel='nofollow'>Dark Background and Light Text</a> plugin is very helpful. By default, the plugin overrides the CSS styles for all web pages, so the background is black, and text is white. The plugin can be trained to leave certain pages untouched, for those pages that become unusable with the modified default stylesheet, for example, <a href='https://calendar.google.com' target='_blank' rel='nofollow'>Google Calendar</a>. Other sites already offer a nice dark theme, so they do not need the modified theme thrust upon them, for example <a href='https://tweetdeck.twitter.com/' target='_blank' rel='nofollow'><code>tweetdeck.twitter.com</code></a>. </p> <p> Some software offers a dark theme which requires further modification before a good result is a achieved. For example, the Thunderbird email client with the <a href='https://addons.thunderbird.net/en-US/thunderbird/addon/tt-deepdark/' target='_blank' rel='nofollow'>TT DeepDark</a> addon gets you partway there, then <b>Tools</b> / <b>Options</b> / <b>Display</b> / <b>Fonts & Colors</b> / <b>Colors</b> shows the following dialog, where you can set the colors as shown: </p> <div style="text-align: center;"> <picture> <source srcset="/assets/images/ttDeepDarkColors.webp" type="image/webp"> <source srcset="/assets/images/ttDeepDarkColors.png" type="image/png"> <img src="/assets/images/ttDeepDarkColors.png" title="Enabling the Windows 10 dark theme" class="center halfsize liImg2 rounded shadow" alt="Enabling the Windows 10 dark theme" /> </picture> </div> <p> As another example, Adobe Reader has a dark theme, which can be enabled via <b>Edit</b> / <b>Preferences</b> / <b>Accessibility</b>; enable <b>Replace Document Colors</b> and click on <b>Custom Color</b>; change the <b>Page Background</b> to black and <b>Document Text</b> to light gray. Disable <b>Only change the color of black text or line art</b>, and ensure that <b>Change the color of line art as well as text</b> is enabled. </p> <div style=""> <picture> <source srcset="/assets/images/adobeReaderColors_690x525.webp" type="image/webp"> <source srcset="/assets/images/adobeReaderColors_690x525.png" type="image/png"> <img src="/assets/images/adobeReaderColors_690x525.png" title="Enabling the Adobe Reader dark theme" class=" liImg2 rounded shadow" alt="Enabling the Adobe Reader dark theme" /> </picture> </div> <h3 id="hard">Small Text and Low Contrast Colors</h3> <p> Users who know CSS can enforce a custom stylesheet on a per-site or a per-page basis for hard-to-read web pages. The <a href='https://chrome.google.com/webstore/detail/my-style/ljdhjpmbnkbengahefamnhmegbdifhlb?hl=en' target='_blank' rel='nofollow'>My Style</a> plugin for the Chrome browser activates with Ctrl-M, and users can enter CSS to suit. As any web developer knows, <kbd>Ctrl</kbd>-<kbd>Shift</kbd>-<kbd>C</kbd> starts HTML inspection, so the HTML in question can be viewed, and the CSS experimented with before immortalizing the changes with My Style. </p> <h2 id="design">Design Considerations</h2> <p> If your web page has images in it, try to use transparency instead of a white background. It would help visually impaired readers to only inflict strong primary colors, or a white background, when necessary. For example, this entry in a stylesheet would only show a light gray background when the user mouses over the image, otherwise, the background would remain transparent: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6e3fe4a4b6ba'><button class='copyBtn' data-clipboard-target='#id6e3fe4a4b6ba' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>img:hover { background-color: #666; }</pre> Are My Hands-Free Devices Always Listening? 2017-01-15T00:00:00-05:00 https://mslinn.github.io/blog/2017/01/15/are-my-handsfree-devices-always-listening <p> The short answer: probably not. </p> <p> A longer answer: Depends on what you mean by &lsquo;listening&rsquo;. If you mean &ldquo;is Alexa storing or sending every sound or voice utterance to a server&rdquo;, the answer is again &ldquo;probably not&rdquo;. </p> <p> A more complete answer: unless all the source code for a device is made available for scrutiny, and the build process is similarly examined, the only way to be sure that device does not have operating modes that are contrary to expectations would be to connect the device to a network monitor that only allows communication with specific servers at designated times. Normal computer security concerns also pose risks; for example, viruses and other malware could be injected into a device could cause it to behave differently. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/echoDevices.webp" type="image/webp"> <source srcset="/blog/images/echoDevices.png" type="image/png"> <img src="/blog/images/echoDevices.png" title="Amazon Echo device locations in homes" class="center liImg2 rounded shadow" alt="Amazon Echo device locations in homes" /> </picture> </div> <h2 id="update2021-01-12">At Least Echo&rsquo;s Mute Button Works</h2> <p> <b>Update 2021-01-12</b> &ndash; Amazon Echo&rsquo;s mute button was <a href='https://electronupdate.blogspot.com/2021/01/amazon-echo-flex-microphone-mute-real.html' target='_blank' rel='nofollow'>reverse engineered</a>. </p> <p class='quote'> The mute button appears to be very real and functional. When the button glows red the power is removed from the microphones. </p> <h2 id="paranoia" style='clear: both'>Some Paranoia is Healthy</h2> <div style="text-align: right;"> <picture> <source srcset="/blog/images/aristotle_300x344.webp" type="image/webp"> <source srcset="/blog/images/aristotle_300x344.png" type="image/png"> <img src="/blog/images/aristotle_300x344.png" title="Mattel Aristotle" class="right liImg" style="width: 300px" alt="Mattel Aristotle" /> </picture> </div> <p> <a href='https://www.engadget.com/2017/01/03/mattel-aristotle-echo-speaker-kids/' target='_blank' rel='nofollow'>Mattel&rsquo;s Aristotle</a> is targeted at children. It uses Microsoft Bing for searching, Microsoft Cortana for voice processing and streams video to the cloud. It can read bedtime stories and play soothing sounds if your child wakes up at the night. It can recognize your children&rsquo;s imperfect speech, and apparently adapts as they get older and become curious about the world. Aristotle is an AI to help raise your child. Did they do a terrific job or a terrible job? It depends on the factors you take into consideration. </p> <p> Aristotle can respond to adults differently than to children. Its logging capability is profound. It tracks things like wet diapers and feedings, and can order supplies when asked by an adult. </p> <p> Aristotle can use object recognition to identify flashcards, or co-opt a toy without electronics and thereby enhance it with sound effects or even another personality. </p> <p> Mattel has about 500 partners, and they have been invited to build connected toys and apps. The hardware uses 256-bit encryption for all transmissions to Aristotle&rsquo;s servers, and data is handled internally in compliance with COPAA and HIPAA. </p> <p class='pullQuote'>It is not difficult to write code that detects when a child is alone</p> <p> How hard would it be to write code that detects when the child is alone? If a malicious entity wanted to, they could embed their program in the device, use IP geolocation and other characteristics to identify specific families, and cause the device to behave differently and/or tell the child things when no adults were paying attention. Who knows what malicious software might do? </p> <h2 id="open">Open Source To The Rescue</h2> <div style="text-align: right;"> <picture> <source srcset="/assets/images/louis.webp" type="image/webp"> <source srcset="/assets/images/louis.png" type="image/png"> <img src="/assets/images/louis.png" title="Justice Louis Brandeis" class="right liImg2 rounded shadow" alt="Justice Louis Brandeis" /> </picture> </div> <p> Closed source systems cannot be adequately vetted for public usage. It is conceivable that selected children might become secretly radicalized by their toy. In 1913 <a href='https://www.brandeis.edu/legacyfund/bio.html' target='_blank' rel='nofollow'>Justice Louis Brandeis</a> said &ldquo;Sunlight is the best disinfectant&rdquo;. Open-source applications, with updates that can be vetted by any interested party, are the only way to ensure these devices are truly safe. </p> Hands-Free Voice as a User Interface 2017-01-10T00:00:00-05:00 https://mslinn.github.io/blog/2017/01/10/voice-activated-apps <p> Amazon&rsquo;s Alexa is a runaway success. Already, 5% of US households have a hands-free voice operated device. 50% of Echo owners keep one in the kitchen. <a href='https://www.gartner.com/doc/3021226/market-trends-voice-ui-consumer' target='_blank' rel='nofollow'>Gartner predicts</a> that by 2018, 30% of our interactions with technology will be via voice conversations. The near future will have many hands-free, voice-driven, Internet-aware applications, and some will also provide a web interface in order to satisfy use cases that require a text or graphic display. </p> <h2 id="near">The Near Future</h2> <p> The near future will have many hands-free, voice-driven, Internet-aware applications, and some will also provide a web interface to satisfy use cases that require a text or graphic display. Potential usages include hands-free voice control of applications running on computers, tablets and phones, hands-free voice control of home devices and vehicles, dictaphones, and games that provide anthropomorphic characters with the ability to generate and understand speech. </p> <div> <div style=""> <picture> <source srcset="/blog/images/google-home_384x500.webp" type="image/webp"> <source srcset="/blog/images/google-home_384x500.png" type="image/png"> <img src="/blog/images/google-home_384x500.png" title="Google Home smart speaker" class=" liImg right rounded shadow" style="height: 500px" alt="Google Home smart speaker" /> </picture> </div> <div style=""> <picture> <source srcset="/blog/images/echo_205x500.webp" type="image/webp"> <source srcset="/blog/images/echo_205x500.png" type="image/png"> <img src="/blog/images/echo_205x500.png" title="Amazon Echo smart speaker" class=" liImg left" style="height: 500px" alt="Amazon Echo smart speaker" /> </picture> </div> </div> <p style="clear: both"> Services that developers and integrators can use for hands-free, voice-operated applications include <a href='https://www.amazon.com/gp/help/customer/display.html?nodeId=201602040' target='_blank' rel='nofollow'>Amazon's Alexa</a>, Google's Assist and Voice, Apple's Siri, and Microsoft's Cortana. </p> <h2 id="present">The Present</h2> <p> Only recently has it become feasible to use hands-free voice as a user interface. The best hands-free, voice applications are innately distributed &mdash; that is, they perform some computation on a local device, and they also require a connection to a server to do the heavy computation necessary for a high-quality experience. Most companies developing applications that use hands-free voice as a user interface require and will continue to require services provided by third parties. However, from the point of view of the application developer's company, the data shared with these third parties can represent a significant security risk. From the user's point of view, this data can represent a potentially significant privacy breach. I'm going to discuss why this is so in this article, and what can be done about it. </p> <p> Privacy and security concerns for synthesizing speech are minimal because quality speech can be synthesized without context specific to an individual. This means that user data need not be associated with the words or phrases being synthesized, so anonymous phrases can be and should be sent to the remote service that generates the audio files produced by the speech synthesis. </p> <p> The best value for high-quality voice generation is achieved with a distributed solution. In this scenario, voice generation is simple to initiate: a text string is sent to a remote service, and an audio clip containing synthesized speech is returned. Most voice generators support embedded markup to control inflection. For example, <a href='https://docs.aws.amazon.com/polly/latest/dg/ssml.html' target='_blank' rel='nofollow'>Amazon Polly</a> and <a href='https://api.ai' target='_blank' rel='nofollow'>Google's Assistant</a> (formerly <code>api.ai</code>) both use <a href='https://developers.google.com/actions/reference/ssml' target='_blank' rel='nofollow'>SSML</a>; Apple uses a <a href='https://eclecticlight.co/2015/12/09/opening-access-text-speech/' target='_blank' rel='nofollow'>variety of techniques</a> across its products, and Microsoft uses <a href='https://en.wikipedia.org/wiki/Microsoft_Speech_API' target='_blank' rel='nofollow'>SAPI</a>. </p> <p> In contrast, the heavy lifting necessary to recognize unconstrained vocabulary requires lots of compute resource, and raises privacy and ethical issues. The main issues are: </p> <ol> <li><a href='#trigger'>Recognizing</a> a trigger word or phrase</li> <li><a href='#training'>Training</a> a voice recognition engine</li> <li>Determining the appropriate <a href='#privacy'>privacy/effectiveness</a> trade-off for your application</li> <li><a href='#integrating'>Integrating</a> with third-party or proprietary services</li> </ol> <h2 id="components">What Are These Things Made From?</h2> <p> Hands-free voice operated devices have most of the same components as a tablet, but often do not have a screen. An ARM A8 to A11 processor is typically embedded in a chip die that also contains a powerful digital signal processor. In other words, they can do a lot of computing; they are more powerful than any mobile phone, and they are more powerful than most tablets. </p> <div class="centered"> <div style=""> <picture> <source srcset="/blog/images/echoParts.webp" type="image/webp"> <source srcset="/blog/images/echoParts.png" type="image/png"> <img src="/blog/images/echoParts.png" title="Amazon Echo parts" class=" liImg" style="width: 100%; max-height: 45%" alt="Amazon Echo parts" /> </picture> </div> <div style=""> <picture> <source srcset="/blog/images/echoBoard.webp" type="image/webp"> <source srcset="/blog/images/echoBoard.png" type="image/png"> <img src="/blog/images/echoBoard.png" title="Amazon Echo PC board" class=" liImg2 rounded shadow" style="width: 100%; max-height: 45%" alt="Amazon Echo PC board" /> </picture> </div> <div style=""> <picture> <source srcset="/blog/images/tiDM37x.webp" type="image/webp"> <source srcset="/blog/images/tiDM37x.png" type="image/png"> <img src="/blog/images/tiDM37x.png" title="TI DM37x processors" class=" liImg2 rounded shadow" style="width: 500px; max-height: 45%" alt="TI DM37x processors" /> </picture> </div> </div> <p style="clear: both"> The open source ecosystem that has grown up around these DSP/CPU chips includes a variety of operating systems, including real-time and various Linux distributions, and many applications. The OSes are: Android, DSP/BIOS, Neutrino, Integrity, Windows Embedded CE, Linux, and VxWorks. It is quick and easy to design a complete device similar to Alexa and bring it to market. </p> <h2 id="trigger" style="clear: both">Recognizing a trigger word or phrase</h2> <p> One of the first major challenges one encounters when designing a device that responds to voice, is how to ignore silence, or recognizing noise that should be ignored. This requires some level of signal processing. In contrast, requiring the user to push a button makes for a a much simpler user interface, but this requirement would greatly restrict the types of applications possible. </p> <div style=""> <picture> <source srcset="/blog/images/Icom-IC-706MKIIG_300x180.webp" type="image/webp"> <source srcset="/blog/images/Icom-IC-706MKIIG_300x180.png" type="image/png"> <img src="/blog/images/Icom-IC-706MKIIG_300x180.png" title="Icom 706 Mark II g" class=" right" style="width: 300px" alt="Icom 706 Mark II g" /> </picture> </div> <p> I am a ham radio operator, and I use the well-known protocol for addressing a specific individual when using a broadcast medium. If I want to talk to another ham operator, I address them by their call sign three times and await their acknowledgement before giving my message. &ldquo;KG6LBG, KG6LBG, KG6LBG, this is KG6LDE, do you read me?&rdquo; If that person hears their call sign, they reply &ldquo;KG6LDE, this is KG6LBG, go ahead.&rdquo; &ldquo;KG6LBG, I just called to say hello, over.&rdquo; &ldquo;KG6LDE, Hello yourself, over and out.&rdquo; </p> <p> Hands-free voice applications also need to recognize a trigger word or phrase (also known as a hotword) that prefaces an audio stream which will be processed for voice recognition. Without such a trigger, either the user would need to press a button to start recording their speech, or a continuous audio stream would have to be processed. I'm interested in hands-free voice recognition. Since the voice recognition processing must be done on a remote service, a lot of bandwidth and CPU power would be wasted processing silence or irrelevant sound. Because it is undesirable to a continuous audio stream to a server to accurately recognize trigger words, the methods used to recognize them are less accurate. </p> <h3 id="alternatives">Proprietary Alternatives</h3> <p> Both of these products can be configured to perform the trigger word recognition without requiring any bandwidth between the CPU running the recognition program and a server. Neither of them provide a JavaScript implementation, which means that unless the trigger word recognition program in installed on the local machine, it must be installed on a connected server and bandwidth will be used at all times. </p> <ul> <li> <div style="text-align: right;"> <picture> <source srcset="/blog/images/THF-VC-72dpi.webp" type="image/webp"> <source srcset="/blog/images/THF-VC-72dpi.png" type="image/png"> <img src="/blog/images/THF-VC-72dpi.png" title="Truly Handsfree voice control" class="right " style="width: 250px" alt="Truly Handsfree voice control" /> </picture> </div> Sensory, Inc's <a href='https://www.sensory.com/products/technologies/trulyhandsfree/' target='_blank' rel='nofollow'>TrulyHandsfree library</a> (<a href='https://github.com/Sensory/alexa-rpi/blob/master/LICENSE.txt' target='_blank' rel='nofollow'>license</a>), based in Santa Clara, CA. &ldquo;We do not have any low cost or free license or library. We are unable to support any student or individual.&rdquo; </li> <li style='clear: both'> <div style="text-align: right;"> <picture> <source srcset="/blog/images/snowboyLogo_250x56.webp" type="image/webp"> <source srcset="/blog/images/snowboyLogo_250x56.png" type="image/png"> <img src="/blog/images/snowboyLogo_250x56.png" title="Snowboy Logo" class="right " style="width: 250px; margin-top: 1em" alt="Snowboy Logo" /> </picture> </div> kitt.ai's <a href='https://github.com/Kitt-AI/snowboy' target='_blank' rel='nofollow'>Snowboy</a>, based in Seattle, WA and <a href='https://www.geekwire.com/2016/backed-amazon-paul-allen-kitt-ai-launches-first-hotword-detection-software-toolkit/' target='_blank' rel='nofollow'>partially funded by Amazon</a>. </li> </ul> <h3 id="sphinx" style='clear: both'>CMU Sphinx</h3> <div style="text-align: right;"> <picture> <source srcset="/blog/images/CMUSphinx_300x72.webp" type="image/webp"> <source srcset="/blog/images/CMUSphinx_300x72.png" type="image/png"> <img src="/blog/images/CMUSphinx_300x72.png" title="CMU Sphinx Logo" class="right " style="width: 300px" alt="CMU Sphinx Logo" /> </picture> </div> <p> Designed for low-resource platforms, implementations of <a href='http://cmusphinx.sourceforge.net/' target='_blank' rel='nofollow'>CMU&rsquo;s Sphinx</a> exist for C (which supports Python) and Java. <a href='http://cmusphinx.sourceforge.net/wiki/faq#qhow_to_implement_hot_word_listening' target='_blank' rel='nofollow'>Hotword</a> spotting is supported. <a href='https://en.wikipedia.org/wiki/CMU_Sphinx' target='_blank' rel='nofollow'>Several versions</a> of Sphinx exist, with varying free and commercial licenses. Reports suggest that Sphinx works reasonably well but I have not tested yet. Sphinx powers <a href='https://jasperproject.github.io/documentation/faq/' target='_blank' rel='nofollow'>Jasper</a>. </p> <h3 id="js">JavaScript alternatives</h3> <ul> <li> <a href='https://github.com/TalAter/annyang' target='_blank' rel='nofollow'>Annyang</a> works well, but requires Google's web browsers and servers, so it is probably not reasonable to use it just for hotword detection. </li> <li> <a href='https://github.com/jimmybyrum/voice-commands.js' target='_blank' rel='nofollow'><code>Voice-commands.js</code></a> also requires Google's web browsers and servers. </li> <li> <a href='https://github.com/evancohen/sonus' target='_blank' rel='nofollow'>Sonus</a> is a Node framework which will be able to be configured to use a variety of back ends one day. It does hotword detection by using Snowboy; the authors obviously ignored Snowboy's license terms. </li> <li> <a href='https://github.com/zzmp/juliusjs' target='_blank' rel='nofollow'>JuliusJS</a> is a JavaScript port of the &ldquo;Large Vocabulary Continuous Speech Recognition Engine Julius&rdquo;. It does not call any servers and runs in most browsers. Recognition is weak and it requires a lot of CPU. </li> <li <a href='https://justbuildsomething.com/cross-browser-voice-recognition-with-pocketsphinx-js/' target='_blank' rel='nofollow'>PocketSphinx.js</a> is a free browser-based alternative, unfortunately it is horrible. </li> </ul> <h2 id="training">Training a voice recognition engine</h2> <p>Voice recognition engines need to be trained on a large dataset for the desired languages. Recognition effectiveness is less than linearly proportional to the size of the dataset, and high-quality datasets are important. Truly large amounts of data are required. Amazon, Apple, Google and Microsoft have commercial products that were trained using enormous proprietary datasets. This is a substantial investment, so only well-capitalized organizations will be able to offer their own voice recognition engines for unconstrained vocabularies. </p> <h2 id="privacy">Determining Your Application's Privacy / Effectiveness Tradeoff</h2> <p> A voice recognition's effectiveness increases for specific users if their voice streams are recorded and stored, then used for further training. However, this means that <a href='https://www.reddit.com/r/technology/comments/2wzmmr/everything_youve_ever_said_to_siricortana_has/' target='_blank' rel='nofollow'>privacy</a> and security are traded off for effectiveness. This service provider's tradeoff might not be optimal for your use case. Apple's Siri only associates the stored voice recordings with you for <a href='https://www.wired.com/2013/04/siri-two-years/' target='_blank' rel='nofollow'>6 months</a>. Google's Assist and Voice, and <a href='https://www.amazon.com/gp/help/customer/display.html?nodeId=201602040' target='_blank' rel='nofollow'>Amazon's Alexa</a> store all your voice recordings forever, unless you explicitly delete them. I could not discover how long Microsoft's Cortana and Skype store voice recordings, or how to delete them. <p> <h2 id="integrating">Integrating With Third-party or Proprietary Services</h2> <p> Because voice recognition returns a structured document like JSON or XML, and voice generation is simple to initiate, integration is well understood and many options exist. </p> Setting Up Jekyll with Ubuntu or WSL 2017-01-08T00:00:00-05:00 https://mslinn.github.io/blog/2017/01/08/setting-up-github-pages <p> Here is a script I wrote in April 2020 to set up a local development environment for this website. The script uses <a href='https://jekyllrb.com/' target='_blank' rel='nofollow'>Jekyll</a> to assemble the website that you are currently reading. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida40d0a1c40be'><button class='copyBtn' data-clipboard-target='#ida40d0a1c40be' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash # Installs the right version of Jekyll and all dependencies function installGem { sudo -H gem install $1 -v $2 --source 'https://rubygems.org/' } yes | sudo apt install bundler make ruby ruby-dev software-properties-common zlib1g-dev sudo -H gem update --system 3.0.6 sudo -H gem uninstall i18n jekyll jekyll-docs jekyll-sass-converter public_suffix --all installGem jekyll-timeago 0.13.1 installGem backports 3.17.1 installGem jekyll 3.3.0 #installGem jekyll-docs 3.3.0 sudo -H gem install bundler classifier-reborn github-pages jekyll-admin jekyll-assets \ jekyll-docs jekyll-gist jekyll-tagging jekyll-theme-architect html-proofer libz-dev sprockets installGem jekyll-sass-converter 1.5.2 installGem i18n 0.9.5 installGem nokogiri 1.10.9 bundle install</pre> <p> This remainder of this article is obsolete. <a href='https://jekyllrb.com/docs/' target='_blank' rel='nofollow'>Follow these instructions instead.</a> </p> <hr /> <p> These are my notes for setting up GitHub pages using Ubuntu or Windows Subsystem for Linux. I updated these notes Jan 3, 2018 to include instructions on <code>jekyll-admin</code>. Since then, Jekyll has continued to evolve and these instructions should no longer be followed. I leave this page merely for posterity's sake. </p> <p> Read the docs on the <a href='https://jekyllrb.com/docs/github-pages/#use-the-github-pages-gem' target='_blank' rel='nofollow'><code>github-pages</code> gem</a>. </p> <p>Make a <code>Gemfile</code> with the following contents:</p> <div class='codeLabel unselectable' data-lt-active='false'>Gemfile</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb75652a22041'><button class='copyBtn' data-clipboard-target='#idb75652a22041' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>source "https://rubygems.org" gem "classifier-reborn" gem "github-pages", group: :jekyll_plugins gem "html-proofer" gem "jekyll" gem 'jekyll-admin', group: :jekyll_plugins gem "jekyll-assets" gem "jekyll-docs" gem "jekyll-gist" gem "jekyll-theme-architect" gem "sprockets"</pre> <p> Create <code>_config.yml</code> with the following contents: </p> <div class='codeLabel unselectable' data-lt-active='false'>_config.yml</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id574613d50988'><button class='copyBtn' data-clipboard-target='#id574613d50988' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>exclude: [vendor] jekyll_admin: hidden_links: # - posts # - pages # - staticfiles # - datafiles # - configuration markdown: kramdown name: Mike Slinn, Connoisseur of Technology permalink: /blog/:year/:month/:day/:title plugins: [classifier-reborn, html-proofer, jekyll, jekyll-admin, jekyll-assets, jekyll-docs, jekyll-gist, jekyll-theme-cayman] title: Mike Slinn's Blog</pre> <p> Ruby 2.3+ is required, but Ubuntu defaults to an older version. I set up Ruby 2.3, with the option of installing other versions and making them default. For more background, see <a href='https://www.brightbox.com/docs/ruby/ubuntu/#Addingtherepository' target='_blank' rel='nofollow'>Ruby packages for Ubuntu</a>. I also installed various gems necessary to provide the Jekyll functionality I desired. </p> <div class='codeLabel unselectable' data-lt-active='false'>setup_ruby</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id72d856a46d52'><button class='copyBtn' data-clipboard-target='#id72d856a46d52' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>sudo apt-get install make software-properties-common sudo apt-add-repository ppa:brightbox/ruby-ng sudo apt-get update sudo apt install ruby2.3 ruby2.3-dev ruby-switch zlib1g-dev ruby-bundler ruby-switch --list sudo ruby-switch --set ruby2.3 sudo gem update --system sudo gem install bundler classifier-reborn jekyll github-pages jekyll-assets jekyll-gist \ jekyll-docs jekyll-theme-cayman html-proofer classifier-reborn jekyll-admin sprockets sudo bundle clean --force bundle install</pre> <p> Create repo <code>userId.github.io</code> (where <code>userId</code> is your GitHub user id) and clone it. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide6a720e2c34f'><button class='copyBtn' data-clipboard-target='#ide6a720e2c34f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git clone git@github.com:userId/userId.github.io.git <span class='unselectable'>$ </span>cd userId.github.io/</pre> <h2 id='running'>Running Jekyll</h2> <p>Read the docs.</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id61c2ab5dc00d'><button class='copyBtn' data-clipboard-target='#id61c2ab5dc00d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>bundle exec jekyll docs</pre> <p> <a href='https://github.com/Microsoft/BashOnWindows/issues/216'' target='_blank' rel='nofollow'>Read</a> about how Bash on Windows does not yet support watched directories properly. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8b8ceb2a8ddf'><button class='copyBtn' data-clipboard-target='#id8b8ceb2a8ddf' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>bundle exec jekyll serve --force_polling</pre> <p>For other OSes:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1b7584da55e1'><button class='copyBtn' data-clipboard-target='#id1b7584da55e1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>bundle exec jekyll serve</pre> <p> Use the <code>--drafts</code> option to preview draft blog posts in the <code>_drafts</code> directory. For Bash on Windows: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idff74d8bd7a3f'><button class='copyBtn' data-clipboard-target='#idff74d8bd7a3f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>bundle exec jekyll serve --force_polling --drafts</pre> <p>For other OSes:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6c33b55cf850'><button class='copyBtn' data-clipboard-target='#id6c33b55cf850' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>bundle exec jekyll serve --drafts</pre> <h2 id='editor'>Visual Blog Editor</h2> <p><a href='https://github.com/planetjekyll/awesome-jekyll-editors' target='_blank' rel='nofollow'>Many awesome Jekyll editors</a> exist. The above instructions installed <a href='https://github.com/jekyll/jekyll-admin' target='_blank' rel='nofollow'><code>jekyll-admin</code></a>. Run Jekyll as described above and navigate to <a href='http://localhost:4000/admin' target='_blank' rel='nofollow'>http://localhost:4000/admin</a> to access the administrative interface. </p> <p> Unfortunately, <code>jekyll-admin</code> does not provide an WYSIWYG editor like that provided by CKEditor. I <a href='https://github.com/jekyll/jekyll-admin/issues/437#issuecomment-355209137' target='_blank' rel='nofollow'>suggested this new feature</a>. </p> I Updated the Apache Spark Reference Applications 2016-11-15T00:00:00-05:00 https://mslinn.github.io/blog/2016/11/15/lessons-from-updating-the-twitter-classifier-apache-spark-reference-application <h2 id="intro">Overview</h2> <p> The Apache Spark committers just accepted my pull request that updated the official <a href='https://github.com/databricks/reference-apps/tree/master/twitter_classifier' target='_blank' rel='nofollow'>Twitter Classifier Reference Application</a> from Spark 1.4 / Scala 2.10 to Spark 2 / Scala 2.11. This post discusses some things I did in the pull request from the point of view of a Scala programmer. A primary goal was to rewrite the reference application using idiomatic and functional-style Scala. This post briefly discusses two unique aspects that I addressed: command-line parsing and DRYing up the code by importing scopes. I did several other things to improve the reference application, such as modularizing the code and providing run scripts, but this post does not address them because those techniques are generally well understood. </p> <p> I did not upgrade the reference application to Scala 2.12, which was released a couple of weeks ago because Spark does not yet support Scala 2.12. Josh Rosen of Databricks wrote me and said: </p> <div class="quote"> &ldquo;Some of Spark’s dependencies and by Scala-version-specific code changes necessary to work around method overloads became ambiguous in 2.12. The umbrella ticket tracking 2.12 support can be found at <a href='https://issues.apache.org/jira/browse/SPARK-14220' target='_blank' rel='nofollow'><code>issues.apache.org/jira/browse/SPARK-14220</code></a>. One of the hardest pieces will be <a href='https://issues.apache.org/jira/browse/SPARK-14643' target='_blank' rel='nofollow'><code>issues.apache.org/jira/browse/SPARK-14643</code></a> (see the linked design document on that issue). Lack of 2.12 support for Breeze and its dependencies is likely to be another serious blocker, but that might be avoidable by only publishing a subset of the projects with 2.12 to begin with (e.g. only Spark core / SQL at first).&rdquo; </div> <h2 id="cli">Command Line Parsing</h2> <p>I modified the reference applications’ command line parsing to use a Scala library that supported idiomatic Scala (<a href='https://github.com/acrisci/commander-scala' target='_blank' rel='nofollow'>Commander Scala</a>), instead of <a href='https://commons.apache.org/proper/commons-cli/' target='_blank' rel='nofollow'>Apache Commons CLI</a>, which is the Java library that was previously used. The result was simple, clean and very terse code that is intuitive to understand and easy to maintain. Commander Scala automatically generates the help message. Take a look at the <a href='https://github.com/databricks/reference-apps/blob/1793e3dc2335696e98a335130673b58b35086c26/twitter_classifier/scala/src/main/scala/com/databricks/apps/twitterClassifier/CollectOptions.scala' target='_blank' rel='nofollow'><code>collect</code> command&rsquo;s parsing</a>. You’ll notice that it uses some common code for parsing Twitter authentication parameters. This code is much shorter than the previous code, easier to understand and modify, and is more flexible.</p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf99b3ca2e1b4'><button class='copyBtn' data-clipboard-target='#idf99b3ca2e1b4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>import com.github.acrisci.commander.Program import java.io.File abstract sealed case class CollectOptions( twitterOptions: TwitterOptions, overWrite: Boolean = false, tweetDirectory: File = new File(System.getProperty("user.home"), "/sparkTwitter/tweets/"), numTweetsToCollect: Int = 100, intervalInSecs: Int = 1, partitionsEachInterval: Int = 1 ) object CollectOptions extends TwitterOptionParser { override val _program = super._program .option(flags="-w, --overWrite", description="Overwrite all data files from a previous run") .usage("Collect [options] &lt;tweetDirectory> &lt;numTweetsToCollect> &lt;intervalInSeconds> &lt;partitionsEachInterval>") def parse(args: Array[String]): CollectOptions = { val program: Program = _program.parse(args) if (program.args.length!=program.usage.split(" ").length-2) program.help new CollectOptions( twitterOptions = super.apply(args), overWrite = program.overWrite, tweetDirectory = new File(program.args.head.replaceAll("^~", System.getProperty("user.home"))), numTweetsToCollect = program.args(1).toInt, intervalInSecs = program.args(2).toInt, partitionsEachInterval = program.args(3).toInt ){} } }</pre> <p> Here is how to sidestep the Spark help message and display the help message for the collect entry point: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3b8aea5c3c21'><button class='copyBtn' data-clipboard-target='#id3b8aea5c3c21' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>spark-shell \ -class com.databricks.apps.twitterClassifier.Collect \ -jars target/scala-2.11/spark-twitter-lang-classifier-assembly-2.0.0.jar \ -- -help <span class='unselectable'>Usage: Collect [options] &lt;tweetDirectory> &lt;numTweetsToCollect> &lt;intervalInSeconds> &lt;partitionsEachInterval> Options: -h, — help output usage information -V, — version output the version number -w, — overWrite Overwrite all data files from a previous run -v, — accessTokenSecret [type] Twitter OAuth Access Token Secret -t, — accessToken [type] Twitter OAuth Access Token -s, — consumerSecret [type] Twitter OAuth Consumer Secret -c, — consumerKey [type] Twitter OAuth Consumer Key </span></pre> <h2 id="import">Importing Inner Scope Into Another Object</h2> <p> Apache Spark is unusual in that you cannot encapsulate a Spark streaming context in a type instance. A memory overflow occurs when you try to instantiate a Scala trait or class that creates a Spark context. The solution is to use a unique Scala feature: the ability to import inner scope from an object into another scope. This meant that the code was made DRY (common code was not repeated), without using classes or traits. </p> <p> Here is how I took advantage of this little-known Scala technique: first I defined the <a href='https://github.com/databricks/reference-apps/blob/1793e3dc2335696e98a335130673b58b35086c26/twitter_classifier/scala/src/main/scala/com/databricks/apps/twitterClassifier/package.scala#L7-L19' target='_blank' rel='nofollow'><code>SparkObject</code> object</a> within a package object so it was easily found: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2c687b5d2016'><button class='copyBtn' data-clipboard-target='#id2c687b5d2016' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>object SparkSetup { val spark = SparkSession .builder .appName(getClass.getSimpleName.replace("$", "")) .getOrCreate() val sqlContext = spark.sqlContext val sc: SparkContext = spark.sparkContext sc.setLogLevel("ERROR") }</pre> <p> Next I imported all the variables defined in <code>SparkSetup</code> into the <code>Collect</code> object’s scope, including <code>sc</code>, which was used twice, <a href='https://github.com/databricks/reference-apps/blob/1793e3dc2335696e98a335130673b58b35086c26/twitter_classifier/scala/src/main/scala/com/databricks/apps/twitterClassifier/Collect.scala' target='_blank' rel='nofollow'>like this</a>: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc853c04d3e6c'><button class='copyBtn' data-clipboard-target='#idc853c04d3e6c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>object Collect extends App { val options = CollectOptions.parse(args) import SparkSetup._ val ssc = new StreamingContext(sc, Seconds(options.intervalInSecs)) Collector.doIt(options, sc, ssc) }</pre> <p> Want to learn more practical Scala techniques? Head over to <a href='https://www.ScalaCourses.com' target='_blank' rel='nofollow'><code>ScalaCourses.com</code></a> and enroll! The combination of the Introduction to Scala and Intermediate Scala courses will teach you everything you need to know to start your journey with Apache Spark. </p> <p> Mike Slinn is the lead Scala instructor at <a href='https://www.ScalaCourses.com"' target='_blank' rel='nofollow'>ScalaCourses.com</a>. </p> Publishing Maven Artifacts to AWS S3 2013-07-07T00:00:00-04:00 https://mslinn.github.io/blog/2013/07/07/publishing-maven-artifacts-to-aws-s3 <p> I wanted a simple, flexible and cheap way of publishing proprietary Maven artifacts created by <code>sbt</code> projects such that they could be securely retrieved by authorized individuals. I liked the idea of versioned artifacts, but did not want to use GitHub or BitBucket to host the artifacts because of the hassle of maintaining ever-larger git repos. Instead, I opted for S3's optional versioning mechanism. </p> <p> The technique described here relies on the fact that publishing to a local file (using <code>sbt publish</code>) generates all the necessary artifacts, which merely need to be copied to the right directory on the Artifactory server. The server need not be anything special: a normal web server works fine, as does <code>webdav</code>. A variety of other protocols are also supported by <code>sbt</code>. </p> <p> I created two test projects on GitHub: <a href='https://github.com/mslinn/testPublishLib' target='_blank' rel='nofollow'><code>testPublishLib</code></a> and <a href='https://github.com/mslinn/testPublishApp' target='_blank' rel='nofollow'><code>testPublishApp</code></a>. You could clone them if you would like to try this. Each of them has a <code>README</code> that explains what they do.</p> I used <code>s3cmd</code> to manage the AWS S3 buckets that hold the repository. You can <a href='https://s3tools.org/s3cmd' target='_blank' rel='nofollow'>obtain <code>s3cmd</code></a> for most OSes. I found I needed to install <code>python-magic</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id994f80a13f0e'><button class='copyBtn' data-clipboard-target='#id994f80a13f0e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo pip install python-magic</pre> <ol> <li>Let <code>s3cmd</code> know your s3 keys: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id577bc5aa41b0'><button class='copyBtn' data-clipboard-target='#id577bc5aa41b0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd --configure</pre> </li> <li>Create the S3 bucket, which must be unique. If you want to repository to be publicly visible, be sure that the bucket name starts with <code>www.</code> If you already have the S3 bucket then just omit this step. <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idaad8d20db394'><button class='copyBtn' data-clipboard-target='#idaad8d20db394' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd mb s3://www.mymavenrepo</pre> </li> <li>If you want to repository to be publicly visible, you need to enable the S3 bucket web site option: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1ea5c2eb485c'><button class='copyBtn' data-clipboard-target='#id1ea5c2eb485c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd ws-create s3://www.mymavenrepo</pre></li> <li>Publish your library to a local repository. <a href='https://github.com/mslinn/testPublishLib' target='_blank' rel='nofollow'><code>testPublishLib</code></a> is an example of how to set up a project properly. <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ideaa785f1cf35'><button class='copyBtn' data-clipboard-target='#ideaa785f1cf35' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sbt publish</pre></li> <li>Copy your locally published artifacts to S3. You can either type it out longhand: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbf20f199ea9d'><button class='copyBtn' data-clipboard-target='#idbf20f199ea9d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd -P sync ~/.ivy2/local/com/micronautics/test_publish_lib \ s3://www.mymavenrepo/snapshots/com/micronautics/test_publish_lib</pre> ... (note that the <code>-P</code> option makes the files publicly visible), or you can use <code>s3publish</code>: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id858709d197e0'><button class='copyBtn' data-clipboard-target='#id858709d197e0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3publish com/micronautics/test_publish_lib</pre> </li> <li>If your repository is not public, you will need to provide authentication in a file which I called <tt>~/.sbt/awsCreds.sbt</tt> for convenience: <div class='codeLabel unselectable' data-lt-active='false'>~/.sbt/awsCreds.sbt</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3ddf3396c230'><button class='copyBtn' data-clipboard-target='#id3ddf3396c230' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>credentials += Credentials("AWS Realm", "www.mavenrepo.s3.amazonaws.com", "myUserId", "myPassword")</pre> </li> <li>Use the published artifact from your sbt project by including a resolver of the form: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id913b377bfd8a'><button class='copyBtn' data-clipboard-target='#id913b377bfd8a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>"AWS Snapshots" at "https://www.mavenrepo.s3.amazonaws.com/snapshots"</pre> The <code>TestPublishApp</code> project is a working example of how to do that.</li> </ol> <p> Following is the script I wrote to upload to S3, which I called <code>s3publish</code>. It assumes you are always publishing a snapshot; and that the files should be public. I leave it to you to extend this script to handle releases and private content if you have the need. </p> <noscript><pre>#!/bin/bash if [ $# -eq 0 ]; then echo &quot;Usage: `basename $0` pubpath/name [repo]&quot; echo &quot; Where pubpath might be something like com/micronautics&quot; echo &quot; name is name of artifact to publish&quot; echo &quot; repo is the optional name of the bucket to publish to&quot; echo &quot; Example: `basename $0` com/micronautics/test_publish_lib&quot; exit 1 fi REPO=www.mymavenrepo OPTIONS=-P if [ $# -eq 2 ]; REPO=$2; fi s3cmd $OPTIONS sync ~/.ivy2/local/$1 s3://$REPO/snapshots/$1</pre></noscript><script src="https://gist.github.com/mslinn/5945115.js"> </script> Load Testing ScalaCourses.com 2013-06-01T00:00:00-04:00 https://mslinn.github.io/blog/2013/06/01/load-testing-scalacoursescom <div style="text-align: center;"> <picture> <source srcset="/assets/images/ScalaCoursesEclipse.webp" type="image/webp"> <source srcset="/assets/images/ScalaCoursesEclipse.png" type="image/png"> <img src="/assets/images/ScalaCoursesEclipse.png" title="ScalaCourses logo" class="center liImg2 rounded shadow" alt="ScalaCourses logo" /> </picture> </div> <p> <a href='https://scalacourses.com/' target='_blank'>ScalaCourses.com</a>, which will be announced next week, is built using the entire Typesafe stack: Scala 2.10, Play 2.1, Slick 1.0 and Akka 2.1. It runs on Heroku. </p> <p> I ran a load test on the app running on only one Heroku dyno. I configured JMeter to use 300 threads, hammering at full speed (no pauses between hits). Testing was done from my desktop. The test generated 16.5Mb/s inbound and 4.1Mb/s outbound according to <code>iptraf</code>. The test did not download page assets because they are served directly from AWS S3. </p> <p> 50% of all responses were received in under 110ms, and 95% were received in under 165ms. The distance from my workstation in Half Moon Bay, CA, USA to the Heroku app server, running from an AWS server in Ashburn, Virginia, USA is about 3000 miles or 4,828 km away. Considering that average ping time is ~100ms, that is amazingly good! Ping measures round-trim time for a test packet, and <code>mtr</code> showed the average ping time as ~95ms with a standard deviation of 18. </p> <div style=""> <picture> <source srcset="/blog/images/jmeter_690x448.webp" type="image/webp"> <source srcset="/blog/images/jmeter_690x448.png" type="image/png"> <img src="/blog/images/jmeter_690x448.png" title="50% of all responses were received in under 110ms, and 95% were received in under 165ms" class=" liImg2 rounded shadow" alt="50% of all responses were received in under 110ms, and 95% were received in under 165ms" /> </picture> </div> <p> Yes, I did put a lot of care into the design of the app so that it would scale well, but I had not expected such fantastic results. Kudos to each of the Typesafe product teams, and to Heroku! </p> Cleaning the Heroku Cache 2013-03-18T00:00:00-04:00 https://mslinn.github.io/blog/2013/03/18/cleaning-heroku-cache <p> In an <a href='/blog/2013-02-27-command-line-sbt-on-heroku-dyno'>earlier post</a>, I talked about using a bash console to experiment with a Heroku app. I mentioned that you should not run <code>sbt</code>. Of course, that is exactly what I did, and I discovered the hard way that the cache can't be cleared from the Heroku bash shell. The build cache is held outside the dyno and cannot be accessed from inside a running dyno. </p> <p> Cleaning the cache is accomplished with a special cache cleaner buildpack. Set the buildpack in your app like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3174a6450003'><button class='copyBtn' data-clipboard-target='#id3174a6450003' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>heroku config:add BUILDPACK_URL=https://github.com/heroku/heroku-buildpack-scala.git#cleancache --app myapp</pre> <p>Push your code:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9f730f4b3a56'><button class='copyBtn' data-clipboard-target='#id9f730f4b3a56' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git push heroku master</pre> <p>Remove the cache cleaner buildpack:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbb3da1c16c36'><button class='copyBtn' data-clipboard-target='#idbb3da1c16c36' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>heroku config:remove BUILDPACK_URL --app myapp</pre> <p>Change a file, commit and push again:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id85554ace7780'><button class='copyBtn' data-clipboard-target='#id85554ace7780' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git push heroku master</pre> <p>All better!</p> Using Scala’s String Interpolation to Access a Map 2013-03-15T00:00:00-04:00 https://mslinn.github.io/blog/2013/03/15/using-scala-210s-string-interpolation <p> Given a map, would it not be nice to have a shorthand way of looking up a value from a key, and to provide a default value? I&rsquo;ve wanted to be able to do this for a long time. Scala 2.10 makes this really easy! </p> <p> The <code>MapLookup</code> class below contains a <code>Map[String, Int]</code> that can be looked up by using a dollar sign ($) from code that contains a reference to the implicit class. If the lookup key is not defined by the map, a zero is returned. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9f98ed9e4445'><button class='copyBtn' data-clipboard-target='#id9f98ed9e4445' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>implicit class MapLookup(val sc: StringContext) { val map = Map(("a", 1), ("b", 2), ("c", 3)).withDefaultValue(0)<br /> def $(args: Any*): Int = { val orig = sc.s (args : _*) map.get(orig) } }</pre> <p> Assuming that the above is stored in a file called <code>strInterp.scala</code>, here are some examples of usage: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc7d38ed27d49'><button class='copyBtn' data-clipboard-target='#idc7d38ed27d49' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>scala -i strInterp.scala <span class='unselectable'>Loading strInterp.scala... defined class MapLookup<br /> Welcome to Scala version 2.10.1 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_17). Type in expressions to have them evaluated. Type :help for more information.<br /> scala&gt; </span>$"a" <span class='unselectable'>res2: Int = 1<br /> scala&gt; </span>$"z" <span class='unselectable'>res3: Int = 0 </span></pre> <p>Short and sweet!</p> Listing of all AWS Elastic Transcoder Presets 2013-03-15T00:00:00-04:00 https://mslinn.github.io/blog/2013/03/15/listing-of-all-aws-elastic-transcoder <p> Here is a listing of all <a href='https://console.aws.amazon.com/elastictranscoder/home?region=us-east-1#presets:' target='_blank' rel='nofollow'>AWS Elastic Transcoder presets</a>, provided &lsquo;out of the box&rsquo; as system-wide presets. </p> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-000001</code><br> <b>Name:</b> <code>System preset: Generic 1080p</code><br> <b>Description:</b> <code>System preset generic 1080p</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=4},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 5400,FrameRate: 29.97,AspectRatio: 16:9,MaxWidth: 1920,MaxHeight: 1080,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-000010</code><br> <b>Name:</b> <code>System preset: Generic 720p</code><br> <b>Description:</b> <code>System preset generic 720p</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 2400,FrameRate: 29.97,MaxWidth: 1280,MaxHeight: 720,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-000020</code><br> <b>Name:</b> <code>System preset: Generic 480p 16:9</code><br> <b>Description:</b> <code>System preset generic 480p 16:9</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 1200,FrameRate: 29.97,MaxWidth: 854,MaxHeight: 480,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-000030</code><br> <b>Name:</b> <code>System preset: Generic 480p 4:3</code><br> <b>Description:</b> <code>System preset generic 480p 4:3</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 900,FrameRate: 29.97,MaxWidth: 640,MaxHeight: 480,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-000040</code><br> <b>Name:</b> <code>System preset: Generic 360p 16:9</code><br> <b>Description:</b> <code>System preset generic 360p 16:9</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 720,FrameRate: 29.97,MaxWidth: 640,MaxHeight: 360,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-000050</code><br> <b>Name:</b> <code>System preset: Generic 360p 4:3</code><br> <b>Description:</b> <code>System preset generic 360p 4:3</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 600,FrameRate: 29.97,MaxWidth: 480,MaxHeight: 360,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-000060</code><br> <b>Name:</b> <code>System preset: Generic 320x240</code><br> <b>Description:</b> <code>System preset generic 320x240</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 22050,BitRate: 64,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=1.3},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 300,FrameRate: 15,MaxWidth: 320,MaxHeight: 240,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-100010</code><br> <b>Name:</b> <code>System preset: iPhone4</code><br> <b>Description:</b> <code>System preset: iPod touch 5G, 4G, iPad 1G, 2G</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=3.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 2200,FrameRate: 30,MaxWidth: 1280,MaxHeight: 720,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-100020</code><br> <b>Name:</b> <code>System preset: iPhone4S</code><br> <b>Description:</b> <code>System preset: iPhone 5, iPad 3G, 4G, iPad mini, Samsung Galaxy S2/S3/Tab 2</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=high, Level=4.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 5000,FrameRate: 30,MaxWidth: 1920,MaxHeight: 1080,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-100030</code><br> <b>Name:</b> <code>System preset: iPhone3GS</code><br> <b>Description:</b> <code>System preset: iPhone 3GS</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 600,FrameRate: 30,MaxWidth: 640,MaxHeight: 480,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-100040</code><br> <b>Name:</b> <code>System preset: iPod Touch</code><br> <b>Description:</b> <code>System preset: iPhone 1, 3, iPod classic</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 1500,FrameRate: 30,MaxWidth: 640,MaxHeight: 480,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-100050</code><br> <b>Name:</b> <code>System preset: Apple TV 2G</code><br> <b>Description:</b> <code>System preset: Apple TV 2G</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=3.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 5000,FrameRate: 30,MaxWidth: 1280,MaxHeight: 720,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-100060</code><br> <b>Name:</b> <code>System preset: Apple TV 3G</code><br> <b>Description:</b> <code>System preset: Apple TV 3G, Roku HD/2 XD</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=high, Level=4},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 5000,FrameRate: 30,MaxWidth: 1920,MaxHeight: 1080,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div