Mike Slinn's Blog 2022-05-28T12:17:36-04:00 https://mslinn.github.io/blog Mike Slinn mslinn@gmail.com Montréal International vs. Bill 96 2022-05-27T00:00:00-04:00 https://mslinn.github.io/blog/2022/05/27/mi-bill96 <p> Last night I attended the 25th Annual General Meeting of <a href='https://www.montrealinternational.com/en/' target='_blank' rel='nofollow'>Montréal International</a>, an energetic organization dedicated to attracting foreign business to establish a presence in the greater Montréal region. My key takeaways were: </p> <ol> <li> I spoke with about 20 Montréal International employees; all of them were bilingual or trilingual, intelligent, motivated and well-informed. Their talent and commitment is palpable. Clearly, the hiring process is successful. </li> <li> The event was marred by long-winded, self-congratulatory speeches by senior management, and former senior managers who were invited to a panel discussion. This lack of discipline caused the event to significantly run overtime. </li> <li> The proceedings were conducted entirely in French. For an organization that calls itself &ldquo;Montréal International&rdquo;, this is unforgivable. </li> <li> I asked every employee I spoke with if <a href='https://qcgn.ca/bill-96/' target='_blank' rel='nofollow'>Quebec&rsquo;s new Bill 96</a> impacted their goals. All of them expressed grave concern, and said that their mandate to attract foreign business had become extremely difficult as a result. However, not one mention was made by anyone on stage about Bill 96. </li> </ol> <p> Montréal International&rsquo;s mandate is severely impacted by Quebec's Bill 96. If any organization has an intrinsic mandate to demonstrate public leadership towards repealing or modifying Bill 96 so it respects basic human rights, it would be Montréal International. Instead, Montréal International&rsquo;s leadership, which consists entirely of old white French-speaking alpha males, is complicit in their silence, and by their actions. </p> Protect Yourself From AWS Account Hijacking 2022-05-26T00:00:00-04:00 https://mslinn.github.io/blog/2022/05/26/aws-hijacking <p> <i>This blog post is a work in progress. I made it public so I could discuss it with others. </i> </p> <p style="display: none;"> </p> <p> In the time you take a quick shower, your AWS account can be highjacked and tens of thousands of dollars in fees can be incurred. EC2 is the service that provides the greatest financial risk. You can take steps to limit your liability, and this blog post shows you how. </p> <h2 id="nolimits">AWS IAM Users and Roles Have No Budget Limitations</h2> <p> I encountered two main issues when attempting to secure against AWS account hijacking: </p> <ol> <li> <b>AWS does not provide a mechanism to guard against launching expensive services.</b> For example, if an IAM user has a role that allows EC2 instances to be launched, then that user can launch an unlimited number of EC2 instances of any size. <br><br> The most expensive Amazon EC2 is currently the <a href='https://aws.amazon.com/ec2/instance-types/p4/' target='_blank' rel='nofollow'><code>p4de.24xlarge</code></a>, which costs $40.96 USD per hour on-demand in the US East (N. Virginia) region. The instance comes with 96 vCPUs, 1152 GiB of RAM and eight 1TB NVMe SSDs, and has eight NVIDIA A100 Tensor Core GPUs. <br><br> Using stolen credentials that allow EC2 instances to be launched, an attacker using scripts could spin up an armada of <code>p4de.24xlarge</code> instances and incur eye-popping costs in minutes. </li> <li> AWS Budgets provides a convenient way to shut down pre-designated services when a total budgetary amount is exceeded. <b>However, the AWS AMI security model is not integrated into real-time cost monitoring.</b> Thus there is no way to automatically shut down newly launched service instances that exceed a pre-authorized budgetary amount. </li> </ol> <h2 id="orElse">Shared Security Responsibility</h2> <p> The <a href='https://aws.amazon.com/agreement/' target='_blank' rel='nofollow'>AWS Customer Agreement</a> and <a href='https://aws.amazon.com/compliance/shared-responsibility-model/' target='_blank' rel='nofollow'>Shared Responsibility Model</a> describe how AWS shares responsibility for security with users. If you follow the AWS security recommendations, described next, you will not be held accountable for extra charges if your AWS account is hijacked. <i>Where is this publicly documented?</i> </p> <p> The AWS Security Blog published a nice overview, entitled <a href='https://aws.amazon.com/blogs/security/getting-started-follow-security-best-practices-as-you-configure-your-aws-resources/' target='_blank' rel='nofollow'>Getting Started: Follow Security Best Practices as You Configure Your AWS Resources</a>. Some of the same information is also provided in <a href='https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html' target='_blank' rel='nofollow'>Security best practices in IAM</a>. </p> <h2 id="recommendations">AWS Security Recommendations</h2> <p> Security needs to have depth. Any single measure can fail, and given enough scale, all measures will eventually fail. Layer your security measures so that one failure, no matter how grave, will not be fatal. </p> <p> AWS recommends the following. I have highlighted what AWS personnel have described to me as being the quickest and easiest path; this blog post walks through those steps to the extent that I have been able. </p> <ol> <li> Set up at least two of the following services to monitor cost and usage: <ol> <li> <a href='https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-managing-costs.html' target='_blank' rel='nofollow'><span class="bg_yellow">Managing your costs with AWS Budgets </span></a> </li> <li> <a href='https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/gs_monitor_estimated_charges_with_cloudwatch.html#gs_creating_billing_alarm' target='_blank' rel='nofollow'><span class="bg_yellow">Create a billing alarm Using CloudWatch</span></a>. </li> <li> <a href='https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html' target='_blank' rel='nofollow'>CloudTrail User Guide</a>. </li> <li> <a href='https://docs.aws.amazon.com/waf/latest/developerguide/what-is-aws-waf.html' target='_blank' rel='nofollow'>Web Application Firewall (WAF)</a>. </li> <li> <a href='https://docs.aws.amazon.com/awssupport/latest/user/get-started-with-aws-trusted-advisor.html' target='_blank' rel='nofollow'>Trusted Advisor</a>. <span class="bg_yellow">I found that Trusted Advisor was somewhat easier to use than most other options.</span> </li> </ol> For more information about managing your AWS cost and usage, see the <a href='https://docs.aws.amazon.com/cost-management/latest/userguide/what-is-costmanagement.html' target='_blank' rel='nofollow'>AWS Cost Management User Guide</a>. </li> <li>Set up at least one of the following security best practices: <ol> <li> <a href='https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa_enable.html' target='_blank' rel='nofollow'><span class="bg_yellow">Using multi-factor authentication (MFA) in AWS</span></a>. </li> <li> <a href='https://docs.aws.amazon.com/securityhub/index.html' target='_blank' rel='nofollow'>AWS Security Hub</a>. </li> <li> <a href='https://docs.aws.amazon.com/guardduty/index.html' target='_blank' rel='nofollow'>Amazon GuardDuty</a>. </li> </ol> </ol> <h2 id="easiestFirst">Do Easiest Things First</h2> <p> I usually prefer to do the easiest things first. <a href='https://www.lifehack.org/articles/productivity/easy-tasks-difficult-tasks-first-which-one-more-productive.html' target='_blank' rel='nofollow'>Not everyone agrees.</a> Doing even one highlighted item above provides a significant security improvement, so why wait? </p> <div class="quote"> “The best is the enemy of the good.”<br> &nbsp;&ndash; Voltaire.<br><br> “Better a diamond with a flaw than a pebble without.”<br> &nbsp;&ndash; Confucius.<br><br> “Striving to better, oft we mar what’s well.”<br> &nbsp;&ndash; Shakespeare. </div> <h2 id="root">Root Credentials</h2> <p> If a bad guy has your root credentials, they can change budget limits. For greatest security, <a href='https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/security_credentials' target='_blank' rel='nofollow'>delete your root credentials</a> by selecting <b>Access Keys (access key ID and secret access key)</b> as shown in the image below. However, if you do so, <a href='https://docs.aws.amazon.com/general/latest/gr/root-vs-iam.html#aws_tasks-that-require-root' target='_blank' rel='nofollow'>many things become impossible</a>. </p> <div style=""> <picture> <source srcset="/blog/images/aws/rootKeys.webp" type="image/webp"> <source srcset="/blog/images/aws/rootKeys.png" type="image/png"> <img src="/blog/images/aws/rootKeys.png" class=" liImg2 rounded shadow" /> </picture> </div> <h3 id="mfa">Enabling MFA</h3> <p> The easiest highlighted item above is to enable multi-factor authentication (MFA), which I <a href='https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa.html#enable-virt-mfa-for-root' target='_blank' rel='nofollow'>enabled for my root account</a> using Google Authenticator for <a href='https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2' target='_blank' rel='nofollow'>Android</a> and <a href='https://apps.apple.com/us/app/google-authenticator/id388497605' target='_blank' rel='nofollow'>iOS</a>. Google Authenticator features a <a href='https://tools.ietf.org/html/rfc6238' target='_blank' rel='nofollow'>RFC 6238</a> standards-based TOTP (time-based one-time password) algorithm. </p> <p class="pullQuote"> Only use root credentials when absolutely necessary. </p> <h2 id="trustedAdvisor">Trusted Advisor</h2> <p> After adding MFA to root credentials, I found that using <a href='https://docs.aws.amazon.com/awssupport/latest/user/get-started-with-aws-trusted-advisor.html' target='_blank' rel='nofollow'>AWS Trusted Advisor</a> was the next easiest thing to try in order to make my AWS account secure. </p> <p> Once you visit the <a href='https://console.aws.amazon.com/trustedadvisor/home' target='_blank' rel='nofollow'>Trusted Advisor Console</a>, and agree to enable Trusted Advisor, <a href='https://docs.aws.amazon.com/awssupport/latest/user/security-trusted-advisor.html' target='_blank' rel='nofollow'>IAM permissions are added to the AWS IAM user</a> that you are logged in as. </p> <p> Usage seems straightforward, however I found the information for securing S3 buckets puzzling at first: </p> <div style=""> <picture> <source srcset="/blog/images/aws/trustedAdvisor1.webp" type="image/webp"> <source srcset="/blog/images/aws/trustedAdvisor1.png" type="image/png"> <img src="/blog/images/aws/trustedAdvisor1.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <p> The column labeled <b>ACL Allows List</b> means that buckets flagged with <b>Yes</b> have an ACL that allows objects to be listed. AWS recommends that S3 buckets not allow objects to be listed by the public. To correct this: </p> <ol> <li>Click on the bucket name.</li> <li>Click on the <b>Permissions</b> tab in the page that opens next.</li> <li>Scroll down to <b>Access Control List</b>.</li> <li>Click on the <kbd>Edit</kbd> button.</li> <li>Disable the items marked in red, as shown below.</li> </ol> <div style=""> <picture> <source srcset="/blog/images/aws/trustedAdvisor2.webp" type="image/webp"> <source srcset="/blog/images/aws/trustedAdvisor2.png" type="image/png"> <img src="/blog/images/aws/trustedAdvisor2.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <p> Referring back to the previous screenshot, the <b>Policy Allows Access</b> column flags all S3 buckets used to serve webpages as insecure. In order to serve HTML pages and assets stored in an S3 bucket, the permissions must be set to allow objects to be read by everyone. Trusted Advisor flags these buckets as insecure, which does not make sense to me. </p> <p> Perhaps there is a better security policy for when webpages are served via CloudFront, but as is often the case with AWS documentation, it is difficult to piece together how to configure S3 permissions in conjunction with CloudFront options for optimally. In particular, CloudFront http/https promotion, how origins are specified, and S3 permissions, all interact in ways that are undocumented. This is likely a source of security problems and unwanted down time. </p> <p class="alert rounded shadow"> I have 15 buckets that are used to serve web pages. On my <a href='https://us-east-1.console.aws.amazon.com/support/home?region=us-east-1&skipRegion=true#/' target='_blank' rel='nofollow'>AWS Support Center console</a>, Trusted Advisor shows that &ldquo;You have a yellow check affecting 15 of 24 resources&rdquo;, however, the Trusted Advisor console displays those items as red triangles with exclamation marks. This is confusing. </p> <h2 id="budgets">Managing Costs With AWS Budgets</h2> <p> Short-lived hijacking is the most serious financial threat, and AWS Budgets was seemingly not designed to guard against that. Ticking off this item will enable AWS to absolve you of financial responsibility for being hacked, but you should not expect that it will do much to slow down an attacker. </p> <p> AWS Budgets are easy to set up in a passive manner, but they are clumsy to work with and not effective for preventing new services from being launched that would exceed the budget. The cost limiting functionality is not automatic, it must be set up manually. AWS documentation does not provide detailed how-to instructions for common scenarios. Tens of thousands of words of documentation must be digested before the full capabilities can be harnessed. Another significant issue is that only three types of actions are supported: </p> <ol> <li> <b>IAM Policy</b> &ndash; I am unclear on how to set this up so the bad guys cannot launch new services, and I am not certain this is possible. </li> <li> <a href='https://us-east-1.console.aws.amazon.com/organizations/v2/home/policies/service-control-policy' target='_blank' rel='nofollow'><b>Service Control Policy</b></a> &ndash; Hopefully this could prevent new services from being launched that would exceed the budget, however my head exploded when I researched this. <div style=""> <picture> <source srcset="/blog/images/aws/serviceControlPolicies1.webp" type="image/webp"> <source srcset="/blog/images/aws/serviceControlPolicies1.png" type="image/png"> <img src="/blog/images/aws/serviceControlPolicies1.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> Attempting to use this option to prevent new, expensive services from being launched would seem to introduce lots of complexity. Perhaps AWS did not consider this use case, or deemed it unimportant. </li> <li> <b>Automate Instances to Stop for EC2 or RDS</b> &ndash; If you run a <a href='https://aws.amazon.com/ec2/instance-types/' target='_blank' rel='nofollow'><code>t3.nano</code></a> instance, for example, bad guys can only incur a certain amount of financial damage. Instead of shutting down the EC2 or RDS instances that I want to run, my main concern is to prevent bad guys from launching expensive new services. This option is therefore unsuitable for my use case. </li> </ol> <p> If AWS Budgets provides an easy way of preventing new services to be launched that would exceed the budget, I could not find it after several hours of reading. </p> <h3 id="passive">Passively Monitoring Budget Spend</h3> <p> Simply follow <a href='https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-create.html' target='_blank' rel='nofollow'>the directions</a> and accept the defaults. Daily budgets do not support enabling forecasted alerts, or daily budget planning, so select <b>Monthly Budgets</b>. There is no need to set up SNS alerts or AWS Chatbot alerts, email notification works just as well for most users. </p> <h3 id="passive">Attaching Actions</h3> <p> Following are my notes from my attempt to enable cost limiting. I am currently stuck, and it is unclear if resolving the problem will in fact provide the desired benefit. </p> <ol> <li>When you get to the step labeled <b>Attach actions - Optional</b>, choose <b>Add Action</b>.</li> <li> You must choose between using an existing IAM role for AWS Budgets to use, or you can create a new IAM role expressly for this purpose. IAM roles are free. I like the idea of a dedicated IAM role, so I clicked on <a href='https://us-east-1.console.aws.amazon.com/iamv2/home?region=us-east-1#/roles' target='_blank' rel='nofollow'>manually create an IAM role</a>.<br> <div style=""> <picture> <source srcset="/blog/images/aws/budgetIamRole1.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole1.png" type="image/png"> <img src="/blog/images/aws/budgetIamRole1.png" class=" halfsize liImg2 rounded shadow" /> </picture> </div> </li> <li> A new browser tab opened, and I clicked on the blue <kbd>Create role</kbd> button. </li> <li> Now a confusing page appeared: <div style=""> <picture> <source srcset="/blog/images/aws/budgetIamRole2.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole2.png" type="image/png"> <img src="/blog/images/aws/budgetIamRole2.png" class=" liImg2 rounded shadow" /> </picture> </div> </li> <li> I selected the default <b>Trusted entity type</b>, AWS Service, then from the pulldown menu labeled <b>Use cases for other AWS services:</b> I selected <b>Budgets</b>. </li> <li> A new radio button appeared underneath the pull-down, also labeled <b>Budgets</b>, and I clicked on that. This does not win any usability awards. </li> <li> I clicked on the button labeled <kbd>Next</kbd>. </li> <li> <div class="quote" style="min-width: 35rem;"> Managed policy name: <code><a href='https://docs.aws.amazon.com/cost-management/latest/userguide/billing-permissions-ref.html#budget-managedIAM-SSM' target='_blank' rel='nofollow'><code>AWSBudgetsActionsRolePolicyForResourceAdministrationWithSSM</code></a></code><br><br> This managed policy is focused on specific actions that AWS Budgets takes on your behalf when completing a specific action. This policy gives AWS Budgets broad permission to control AWS resources. For example, starts and stops Amazon EC2 or Amazon RDS instances by running AWS Systems Manager (SSM) scripts.<br><br> &nbsp;&ndash; From <a href='https://docs.aws.amazon.com/cost-management/latest/userguide/billing-permissions-ref.html#budget-managedIAM-SSM' target='_blank' rel='nofollow'>Using identity-based policies (IAM policies) for AWS Cost Management</a> </div> On the page, I applied <code>AWSBudgetsActionsRolePolicyForResourceAdministrationWithSSM</code>, then I clicked on <kbd>Next</kbd>. <div style=""> <picture> <source srcset="/blog/images/aws/budgetIamRole3.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole3.png" type="image/png"> <img src="/blog/images/aws/budgetIamRole3.png" class=" liImg2 rounded shadow" /> </picture> </div> </li> <li> On the final page I named the IAM role <b>Budget</b> and clicked on <kbd>Create role</kbd>. <div style=""> <picture> <source srcset="/blog/images/aws/budgetIamRole4.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole4.png" type="image/png"> <img src="/blog/images/aws/budgetIamRole4.png" class=" liImg2 rounded shadow" /> </picture> </div> </li> <li> Back in the <b>Billing Management Console</b>, I refreshed the list of available IAM roles, then selected the new role called <b>Budget</b>. <div style=""> <picture> <source srcset="/blog/images/aws/budgetIamRole5.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole5.png" type="image/png"> <img src="/blog/images/aws/budgetIamRole5.png" class=" halfsize liImg2 rounded shadow" /> </picture> </div> </li> <li> I selected <b>Service Control Policy</b> from the pull-down that appeared next, labeled <b>Which action type should be applied when the budget threshold has been exceeded?</b>. I have no idea how to proceed, or even if going in this direction might provide the desired results. <div style=""> <picture> <source srcset="/blog/images/aws/serviceControlPolicies2.webp" type="image/webp"> <source srcset="/blog/images/aws/serviceControlPolicies2.png" type="image/png"> <img src="/blog/images/aws/serviceControlPolicies2.png" class=" liImg2 rounded shadow" /> </picture> </div> </li> </ol> Microsoft Clarity Lets Me Watch You Click and Scroll 2022-03-31T00:00:00-04:00 https://mslinn.github.io/blog/2022/03/31/clarity <p> I spent a fair bit of time designing the plugins for this Jekyll-powered website for <a href='https://moz.com/beginners-guide-to-seo' target='_blank' rel='nofollow'>SEO</a>. These plugins are published as open source; click on the word Jekyll in the sentence above this paragraph to learn more. You are welcome! </p> <p> I am currently getting strong month-over-month audience growth, and the rate of growth continues to accelerate. I do not use Google Analytics because it slows down page load time dramatically. I do use <a href='https://search.google.com' target='_blank' rel='nofollow'>Google Search Console</a>, however, and recently started trying out <a href='https://www.bing.com/webmasters/' target='_blank' rel='nofollow'>Microsoft Bing Webmaster Tools</a>. </p> <h2 id="clarity">Microsoft Clarity and Hotjar</h2> <p> Earlier today, I stumbled across <a href='https://clarity.microsoft.com/' target='_blank' rel='nofollow'>Microsoft Clarity</a>, a free product that I knew nothing about. I was curious to know what benefit it might provide. One of the things it can do is provide videos of actual user sessions as they interact with the website. </p> <p> Check out this video! </p> <style>.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }</style><div class='embed-container'> <iframe title="YouTube video player" width="640" height="390" src="//www.youtube.com/embed/N-Tip0Y4GMU" frameborder="0" allowfullscreen></iframe></div> <p style="margin-top: 1em"> Microsoft Clarity lets me watch movies of users clicking and scrolling through my website; spooky yet very informative. </p> <p> I have since learned that <a href='https://www.hotjar.com/' target='_blank' rel='nofollow'>Hotjar</a> is similar to Microsoft Clarity. </p> <h2 id="behavior">Online Behavior Matches Real-World Behavior</h2> <p> The user in the above video read a bit about my experience as a software expert witness, then straightaway downloaded my resume. They were on the website for just over one minute. Although they did call me, their online behavior showed a lack of urgency, and their &lsquo;real-world&rsquo; behavior mirrored what I saw in the movie. </p> <p> A week later another party visited my site, and spent 80 minutes carefully reading 3 pages, among others. Their &lsquo;real-world&rsquo; behavior also matched their online behavior, in that they exhibited a sense of urgency towards engaging a software expert. </p> <h2 id="downloads">Tracking Downloads And Other Behavior</h2> <p> Microsoft Clarity does not consider a download as a click. It does not even notice downloads. For me, downloads are what I care about most. If you never download my resume, you probably are not a candidate for hiring me as a <a href='/softwareexpert/index.html'>software expert</a>. I want to track downloads, not clicks. Clicks are nice, but there is a direct relationship between resume downloads and signing contracts with legal firms who represent new clients. </p> <p> AWS <a href='https://aws.amazon.com/premiumsupport/knowledge-center/view-iam-history/' target='_blank' rel='nofollow'>CloudTrail</a> and <a href='https://aws.amazon.com/aws-cost-management/aws-cost-optimization/monitor-track-and-analyze/' target='_blank' rel='nofollow'>CloudWatch</a> can provide download details and much more. For example, user IP addresses and geographic location can be captured when a monitored event occurs. </p> <h2 id="others">Many Websites Perform Surveillance</h2> <p> Wired Magazine published an article on a similar type of surveillance (keyloggers) on May 11, 2022: <a href='https://www.wired.com/story/leaky-forms-keyloggers-meta-tiktok-pixel-study/' target='_blank' rel='nofollow'>Thousands of Popular Websites See What You Type—Before You Hit Submit</a>. I paraphrased two sentences from that article: </p> <ul class="quote"> <li>1.8% of websites studied gathered an EU user's email address without their consent, and a staggering 2.95% logged a US user's email.</li> <li>For US users, 8.4% of sites may have been leaking data to Meta, Facebook’s parent company, and 7.4% of sites may be impacted for EU users.</li> </ul> Make a Visual Studio Code Extension 2022-03-01T00:00:00-05:00 https://mslinn.github.io/blog/2022/03/01/make-vscode-extension <p> I want to be able to move pages in my Jekyll-powered website without breaking links from the outside. The <a href='https://github.com/jekyll/jekyll-redirect-from' target='_blank' rel='nofollow'><code>jekyll-redirect-from</code></a> Jekyll plugin generates little HTML files that contain <code>http-meta-refresh</code> client-side redirects as desired. I wanted an easy way to inject the names of those redirect pages into the Jekyll front matter. Since I usually author my website using Visual Studio Code, a custom Visual Studio Code extension seems like a good way to create the redirects. </p> <h2 id="requirements">Extension Requirements</h2> <p> All the extension has to do is discover the URL path component of the page currently being edited, and write an entry into the front matter. For example, I have set up Jekyll such that given a page at <code>collections/_posts/2022/2022-02-21-jekyll-debugging.html</code>, it would deploy to <code>/blog/2022/02/21/jekyll-debugging.html</code>. The required front matter would be: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7de3ad57f29e'><button class='copyBtn' data-clipboard-target='#id7de3ad57f29e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>redirect_from: - /blog/2022/02/21/jekyll-debugging.html</pre> <p> With that in mind, if I want to move a published post to another location without breaking it, the extension should: </p> <ol> <li>Present a new menu option when a right-click on a file in the side bar, or a right-click on a file name tab in the editor.</li> <li> When the menu item is selected, obtain the relative path to the file within the project directory (for example: <code>collections/_posts/2022/2022-02-21-jekyll-debugging.html</code>) </li> <li> Convert the relative path to the deployed relative path (for example, <code>/blog/2022/02/21/jekyll-debugging.html</code>) </li> <li> Write the entry into the front matter of the file. For example: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfda460174a4b'><button class='copyBtn' data-clipboard-target='#idfda460174a4b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>redirect_from:<br> - /blog/2022/02/21/jekyll-debugging.html</pre> </li> </ol> <h2 id="background">Background</h2> The <a href='https://code.visualstudio.com/api/get-started/your-first-extension' target='_blank' rel='nofollow'>Microsoft documentation</a> describes how to write an Visual Studio Code extension. <h2 id="setup">Setup</h2> <p> Visual Studio Code extensions are written in JavaScript (actually, node.js) or TypeScript. Ensure that a version of node.js has been installed: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idaf37dabd3371'><button class='copyBtn' data-clipboard-target='#idaf37dabd3371' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>which node <span class='unselectable'>/home/mslinn/.nvm/versions/node/v17.3.1/bin/node </span></pre> <p> Ensure you have <a href='properly'>set up your global package manager</a> for node.js, then type: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf8c8f3a92843'><button class='copyBtn' data-clipboard-target='#idf8c8f3a92843' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>npm install -g <a href='https://github.com/yeoman/yo/' target='_blank' rel='nofollow'>yo</a> <a href='https://github.com/microsoft/vscode-generator-code' target='_blank' rel='nofollow'>generator-code</a> <span class='unselectable'>&nbsp; npm WARN deprecated har-validator@5.1.5: this library is no longer supported npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142 added 872 packages, and audited 873 packages in 57s 15 vulnerabilities (13 moderate, 2 high) To address issues that do not require attention, run: npm audit fix To address all issues (including breaking changes), run: npm audit fix --force Run `npm audit` for details. </span></pre> <p> The <code>yo</code> module is the culprit; lets address its vulnerabilities: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfad67a11dd51'><button class='copyBtn' data-clipboard-target='#idfad67a11dd51' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>npm install -g npm-check-updates <span class='unselectable'>added 270 packages, and audited 271 packages in 9s found 0 vulnerabilities </span></pre> <h2 id="generate">Generating the Skeleton</h2> <p> I decided to use <a href='https://www.typescriptlang.org/' target='_blank' rel='nofollow'>TypeScript</a> instead of JavaScript for the extension. TypeScript code converts to JavaScript, however it uses type inference and integrates better with some editors. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcda192cefe5b'><button class='copyBtn' data-clipboard-target='#idcda192cefe5b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mkdir redirect_generator <span class='unselectable'>$ </span>cd redirect_generator/ <span class='unselectable'>$ </span>$ yo code <span class='unselectable'>? ========================================================================== We&rsquo;re constantly looking for ways to make yo better! May we anonymously report usage statistics to improve the tool over time? More info: https://github.com/yeoman/insight &amp; http://yeoman.io ========================================================================== </span> No <span class='unselectable'>_-----_ ╭──────────────────────────╮ | | │ Welcome to the Visual │ |--(o)--| │ Studio Code Extension │ `---------´ │ generator! │ ( _´U`_ ) ╰──────────────────────────╯ /___A___\ / | ~ | __'.___.'__ ´ ` |° ´ Y ` ? What type of extension do you want to create? </span> New Extension (TypeScript) <span class='unselectable'>? What's the name of your extension? </span> redirect_generator <span class='unselectable'>? What's the identifier of your extension? </span> redirect-generator ? What's the description of your extension? Injects the URL of a redirect page into Jekyll front matter <span class='unselectable'>? Initialize a git repository? </span>Yes <span class='unselectable'>? Bundle the source code with webpack? </span>No <span class='unselectable'>? Which package manager to use? </span>npm <span class='unselectable'>Writing in /mnt/f/work/jekyll/redirect_generator/redirect-generator... create redirect-generator/.vscode/extensions.json create redirect-generator/.vscode/launch.json create redirect-generator/.vscode/settings.json create redirect-generator/.vscode/tasks.json create redirect-generator/package.json create redirect-generator/tsconfig.json create redirect-generator/.vscodeignore create redirect-generator/vsc-extension-quickstart.md create redirect-generator/README.md create redirect-generator/CHANGELOG.md create redirect-generator/src/extension.ts create redirect-generator/src/test/runTest.ts create redirect-generator/src/test/suite/extension.test.ts create redirect-generator/src/test/suite/index.ts create redirect-generator/.eslintrc.json Changes to package.json were detected. Running npm install for you to install the required dependencies. added 203 packages, and audited 204 packages in 29s found 0 vulnerabilities Your extension redirect-generator has been created! To start editing with Visual Studio Code, use the following commands: code redirect-generator Open vsc-extension-quickstart.md inside the new extension for further instructions on how to modify, test and publish your extension. For more information, also visit http://code.visualstudio.com and follow us @code. ? Do you want to open the new folder with Visual Studio Code? Open with `code` _-----_ ╭───────────────────────╮ | | │ Bye from us! │ |--(o)--| │ Chat soon. │ `---------´ │ Yeoman team │ ( _´U`_ ) │ http://yeoman.io │ /___A___\ /╰───────────────────────╯ | ~ | __'.___.'__ ´ ` |° ´ Y ` </span></pre> <p> The instructions in the web page assume the <a href='https://dictionary.cambridge.org/dictionary/english/happy-path' target='_blank' rel='nofollow'>happy path</a> is the only possibility. Instead: </p> <ol> <li>Do not work on the extension in a workspace that has any other projects in it.</li> <li>The online instructions assume that TypeScript was used, not JavaScript, so the file types are assumed to be <code>.ts</code>. </ol> <p> The instructions in <code>vsc-extension-quickstart.md</code> do not make assumptions about TypeScript. </p> <h2 id="debug">Debugging the Extension</h2> <p> I found the procedure awkward at first: </p> <ol> <li>Press <kbd>F5</kbd> to activate the first defined launcher</li> <li>Once the new debug VSCode instance settles down, click in it and type <kbd>Ctrl</kbd>-<kbd>Shift</kbd>-<kbd>P</kbd>, then type <code>redirect</code></li> <li>Any breakpoints in the original VSCode instance will trigger.</li> <li>Once the breakpoint is cleared, a message saying <b>Add redirecct from redirect_generator!</b> will appear in the debugged instance of VSCode.</li> </ol> <h2 id="anatomy">Extension Anatomy</h2> <p> I moved on to the next part of the tutorial, <a href='https://code.visualstudio.com/api/get-started/extension-anatomy' target='_blank' rel='nofollow'>Extension Anatomy</a>. </p> Iterating Slim Language Templates 2022-02-22T00:00:00-05:00 https://mslinn.github.io/blog/2022/02/22/testing-slim <video controls autoplay="1" width="100%" preload="auto" class="shadow rounded" style="margin-bottom: 1em;"> <source src="https://user-images.githubusercontent.com/485818/155521347-856ca755-cb89-4bc7-97ce-fa69d091cf7a.mp4" type="video/mp4"> Your browser does not support the video tag. Please use another browser to view this video. </video> <p> <a href='https://github.com/slim-template/slim#configuring-slim' target='_blank' rel='nofollow'>Slim</a> is a cool way to generate a potentially complex HTML block with a minimum of characters. Most often, Slim is thought of as an alternative to Ruby on Rail&rsquo;s ERB; however, it is useful in a broad range of contexts. </p> <p> The Slim Language Explorer transforms the experience of learning the Slim Language from something painfully awkward, to something pleasant. </p> <h2 id="explorer">I Have a Thing About Explorers</h2> <p> Over the years, I&rsquo;ve made about a dozen Explorers, starting with SMX Explorer in 1994 for The Internet Factory. They had the world&rsquo;s first programmable web server. I wrote the technical manual for using the programming language of their server; this was the first online publication with interactive code examples. </p> <p> <a href='/resume/history/jspexplorer/developerWorks/'>IBM published an article about JSP Explorer</a> in 2001. My <a href='/blog/index.html#Zamples'>Zamples</a> startup in 2001 was all about the world&rsquo;s first multi-lingual &ldquo;Try it!&rdquo; button, which executed the displayed code written one of many languages, some of which were pre-packaged with various popular libraries. Zamples was essentially a next-generation JSP Explorer. </p> <p> If you are a web developer, then you have seen and probably used explorers before. Ever try <a href='https://codepen.io/' target='_blank' rel='nofollow'>Code Pen</a>? I invented that technology. </p> <p> Anyway, here it is, yet another explorer, this one packaged as a git project for Ubuntu &ndash; for the Slim Language! Squint and it looks like my last name. </p> <h2 id="slim_explorer">The Slim Language Explorer</h2> <p> When you are trying to figure out how to express HTML in Slim, however, you inevitably rerun the generation process over and over. The <a href='https://github.com/mslinn/slim_explorer' target='_blank'>Slim Language Explorer</a> consists of a small Ruby program that launches Slim and displays the results of evaluating the Slim expression. </p> <h2 id="reload">Live Reload</h2> <p> The output is regenerated whenever a file is modified, created or deleted within the <code>watched</code> directory of the project. This allows you to edit the Slim expression and/or modify the YAML data, and view the updated output each time a change is made. </p> <h2 id="ruby">Embed Ruby Code</h2> <p> The template below uses a Ruby filter, shown highlighted in yellow, where Ruby code can be inserted. Everything indented after <code>Ruby:</code> is parsed as Ruby code. Gems and other code can be <code>require</code>d. Note that methods defined in a Ruby filter must be <a href='https://github.com/slim-template/slim/issues/835' target='_blank' rel='nofollow'>defined as class methods</a>, which means that when defining them, their names must be prefixed with <code>self</code>. Any <code>Module</code>s that might be <code>include</code>d would also need similar handling, or they would need to be <code>include</code>d into a <code>class</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>template.slim</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddabe80570b9f'><button class='copyBtn' data-clipboard-target='#iddabe80570b9f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class="bg_yellow">ruby:</span> require 'slugify' def <span class="bg_yellow">self.</span>padding 'padding: 0 5px 3px 5px' end def <span class="bg_yellow">self.</span>boxed(contents) "&lt;div style='border: thin solid grey; #{padding};'>#{contents}&lt;/div>" end doctype html html head title = heading body h1 = heading p = message ul li: a href="mailto:#{email}" #{name} li style="margin-top: 3px; #{padding}; background-color: #{background_color}; color: #{color};" = 'Green and white slugs ... yuck!'.slugify ==boxed 'Help, I am stuck inside this computer!'</pre> <p> The above template contains references to local variables, defined in the Ruby filter, and variables passed to it which <code>slim_explorer</code> read from a YAML file. </p> <h2 id="data">YAML Data</h2> <p> Most templates require data to inflate them. Slim Language Explorer gets data from <code>template.yaml</code>, shown below. Go ahead and change it while experimenting. </p> <div class='codeLabel unselectable' data-lt-active='false'>template.slim</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id23048fb01469'><button class='copyBtn' data-clipboard-target='#id23048fb01469' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>environment: - email: mslinn@mslinn.com - heading: Testing, 1-2-3, Testing ... - message: World peace begins with you and me. - name: Michael Slinn - color: white - background_color: green</pre> <p> The name of the top-most property (<code>environment</code>) does not matter. </p> <h2 id="install">Installation</h2> <h3 id="Ruby">Ruby Development Support</h3> <p> The official instructions for installing full Ruby <a href='https://www.ruby-lang.org/en/documentation/installation/' target='_blank' rel='nofollow'>are here</a>, although the instructions are incomplete, terse and dated. Be sure to include the development tools. For Ubuntu, this is what I recommend: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7574cd1aedc5'><button class='copyBtn' data-clipboard-target='#id7574cd1aedc5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install libruby ruby-dev</pre> <h3 id="clone">Clone slim_explorer</h3> <p> Clone the <a href='https://github.com/mslinn/slim_explorer' target='_blank'><code>slim_explorer</code> git repo</a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id827197255566'><button class='copyBtn' data-clipboard-target='#id827197255566' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git clone https://github.com/mslinn/slim_explorer.git</pre> <p> Move to the newly cloned directory: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd156e7429a4e'><button class='copyBtn' data-clipboard-target='#idd156e7429a4e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cd slim_explorer</pre> <h3 id="gems">Install Project Gems</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2dc874239d81'><button class='copyBtn' data-clipboard-target='#id2dc874239d81' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>bundle install</pre> <h2 id="repl">Slim Language REPL</h2> <p> In some sense, this is a REPL for the Slim Language. </p> <h3 id="commandMode">Command-Line Mode</h3> <p> The video above demonstrates command-line mode running in Visual Studio Code. </p> <ol> <li> To use the Slim Language Explorer, start a bash shell and type: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbb4a6170b4cb'><button class='copyBtn' data-clipboard-target='#idbb4a6170b4cb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>./slim_explorer</pre> The Slim expression(s) stored in <code>watched/template.slim</code> are evaluated and stored into <code>www/raw.htm</code> using the key and values stored in file <code>watched/scope.yaml</code>. </li> <li> Use the editor of your choice to modify any file in the <code>watched/</code> directory. </li> <li> The contents of <code>watched/template.slim</code> are re-evaluated, and <code>www/raw.html</code> is updated after each change is saved. This process continues until interrupted. </li> </ol> <h3 id="webMode">Web Server Mode</h3> <ol> <li> Start a bash shell and type: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id85763c9f0936'><button class='copyBtn' data-clipboard-target='#id85763c9f0936' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>./slim_explorer serve</pre> The Slim expression(s) stored in <code>watched/template.slim</code> are evaluated and stored into <code>www/index.html</code> using the keys and values stored in file <code>watched/scope.yaml</code>. </li> <li> Open <code>www/index.html</code> in your favorite web browser, either from the file, or at <a href='https://http://localhost:3030/index.html' target='_blank' rel='nofollow'><code>http://localhost:3030/index.html</code></a>. This is what it looks like:<br/> <div style=""> <picture> <source srcset="https://github.com/mslinn/slim_explorer/raw/master/doc/server_mode.png" type="image/webp"> <source srcset="https://github.com/mslinn/slim_explorer/raw/master/doc/server_mode.png" type="image/png"> <img src="https://github.com/mslinn/slim_explorer/raw/master/doc/server_mode.png" class=" liImg2 rounded shadow" /> </picture> </div> </li> <li> Use the editor of your choice to modify any file in the <code>watched</code> directory. </li> <li> The contents of <code>watched/template.slim</code> are re-evaluated and <code>www/index.html</code> is updated each time a file in the <code>watched</code> directory is saved, created or deleted. </li> </ol> <h2 id="video">About the Video</h2> <p> You can see the generated HTML change as the Slim expression or the YAML data is modified. </p> <p> To make the video, I installed the <a href='https://marketplace.visualstudio.com/items?itemName=tht13.html-preview-vscode' target='_blank' rel='nofollow'>Visual Studio Code HTML Preview extension</a> by Thomas Haakon Townsend, which had 1,649,338 installations. While this is a generally useful extension, it enables the instantaneous display of the generated HTML in a render pane. The extension can be installed via the command line if you like: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb68b642f94af'><button class='copyBtn' data-clipboard-target='#idb68b642f94af' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>code --install-extension tht13.html-preview-vscode@0.2.5</pre> Fun With Python Enums 2022-02-10T00:00:00-05:00 https://mslinn.github.io/blog/2022/02/10/python-3.4-enums <p> This blog post demonstrates how to define additional properties for <a href='https://docs.python.org/3/library/enum.html' target='_blank' rel='nofollow'>Python 3 enums</a>. Defining an additional property in a Python enum can provide a simple way to provide string values. The concept is then expanded to demonstrate composition, an important concept for functional programming. This post concludes with a demonstration of dynamic dispatch in Python, by further extending an enum. </p> <h2 id="enum">Adding Properties to Python Enums</h2> <p> Searching for <a href='https://www.google.com/search?q=python+enum+string+value' target='_blank' rel='nofollow'><code>python enum string value</code></a> yields some complex and arcane ways to approach the problem. </p> <p> Below is a short example of a Python enum that demonstrates a simple way to provide lower-case string values for enum constants: a new property, <code>to_s</code>, is defined. This property provides the string representation that is required. You could define other properties and methods to suit the needs of other projects. </p> <div class='codeLabel unselectable' data-lt-active='false'>cad_enums.py</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id75ce623e4df0'><button class='copyBtn' data-clipboard-target='#id75ce623e4df0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>"""Defines enums"""<br/> from enum import Enum, auto<br/> class EntityType(Enum): """Types of entities""" SITE = auto() GROUP = auto() COURSE = auto() SECTION = auto() LECTURE = auto()<br/> <span class="bg_yellow"> @property def to_s(self) -> str: """:return: lower-case name of this instance""" return self.name.lower()</span></pre> <p> Adding the following to the bottom of the program allows us to demonstrate it: </p> <div class='codeLabel unselectable' data-lt-active='false'>cad_enums.py (part 2)</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1010dca00e15'><button class='copyBtn' data-clipboard-target='#id1010dca00e15' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>if __name__ == "__main__": # Just for demonstration print("Specifying individual values:") print(f" {EntityType.SITE.value}: {EntityType.SITE.to_s}") print(f" {EntityType.GROUP.value}: {EntityType.GROUP.to_s}") print(f" {EntityType.COURSE.value}: {EntityType.COURSE.to_s}") print(f" {EntityType.SECTION.value}: {EntityType.SECTION.to_s}") print(f" {EntityType.LECTURE.value}: {EntityType.LECTURE.to_s}") print("\nIterating through all values:") for entity_type in EntityType: print(f" {entity_type.value}: {entity_type.to_s}")</pre> <p> Running the program produces this output: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id94faaf509cbf'><button class='copyBtn' data-clipboard-target='#id94faaf509cbf' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cad_enums.py <span class='unselectable'>Specifying individual values: 1: site 2: group 3: course 4: section 5: lecture Iterating through all values: 1: site 2: group 3: course 4: section 5: lecture </span></pre> <div class="right" style="font-size: 3em;">&#128513;</div> <p> Easy! </p> <h2 id="constructor">Constructing Enums</h2> <p> Enum constructors work the same as other Python class constructors. There are several ways to make a new instance of a Python enum. Let's try two ways by using the <a href='https://docs.python.org/3/tutorial/interpreter.html' target='_blank' rel='nofollow'>Python interpreter</a>. Throughout this blog post I've inserted a blank line between Python interpreter prompts for readability. </p> <div class='codeLabel unselectable' data-lt-active='false'>Python</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc13a1ad2bf15'><button class='copyBtn' data-clipboard-target='#idc13a1ad2bf15' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>python <span class='unselectable'>Python 3.9.7 (default, Sep 10 2021, 14:59:43) [GCC 11.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> </span>from cad_enums import EntityType <span class='unselectable'>>>> </span># Specify the desired enum constant value symbolically <span class='unselectable'>>>> </span><span class="bg_yellow">gtype = EntityType.GROUP</span> <span class='unselectable'>>>> </span>print(gtype) <span class='unselectable'>EntityType.GROUP </span> <span class='unselectable'>>>> </span># Specify the desired enum constant value numerically <span class='unselectable'>>>> </span><span class="bg_yellow">stype = EntityType(1)</span> <span class='unselectable'>>>> </span>print(stype) <span class='unselectable'>EntityType.SITE </span></pre> <h2 id="ordering">Enum Ordering</h2> <p> A program I am working on needs to obtain the parent <code>EntityType</code>. By 'parent' I mean the <code>EntityType</code> with the next lowest numeric value. For example, the parent of <code>EntityType.GROUP</code> is <code>EntityType.SITE</code>. We can obtain a parent enum by computing its numeric value by adding the following method to the <code>EntityType</code> class definition. </p> <div class='codeLabel unselectable' data-lt-active='false'>Python</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddb63e4778cbe'><button class='copyBtn' data-clipboard-target='#iddb63e4778cbe' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>@property def parent(self) -> <span class="bg_yellow">'EntityType'</span>: """:return: entity type of parent; site has no parent""" return EntityType(max(self.value - 1, 1))</pre> <p> The <span class="bg_yellow">return type</span> above is enclosed in quotes (<code>'EntityType'</code>) to keep Python's type checker happy, because this is a <a href='https://www.python.org/dev/peps/pep-0484/#forward-references' target='_blank' rel='nofollow'>forward reference</a>. This is a forward reference because the type is referenced before it is fully compiled. </p> <p> The complete enum class definition is now: </p> <div class='codeLabel unselectable' data-lt-active='false'>cad_enums.py</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3b554654792d'><button class='copyBtn' data-clipboard-target='#id3b554654792d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>"""Defines enums"""<br/> from enum import Enum, auto<br/> class EntityType(Enum): """Types of entities""" SITE = auto() GROUP = auto() COURSE = auto() SECTION = auto() LECTURE = auto()<br/> @property def to_s(self) -> str: """:return: lower-case name of this instance""" return self.name.lower()<br> @property def parent(self) -> 'EntityType': """:return: entity type of parent; site has no parent""" return EntityType(max(self.value - 1, 1))</pre> <p> Lets try out the new <code>parent</code> property in the Python interpreter. </p> <div class='codeLabel unselectable' data-lt-active='false'>Python</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9c83dad32be6'><button class='copyBtn' data-clipboard-target='#id9c83dad32be6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>EntityType.LECTURE.parent <span class='unselectable'>&lt;EntityType.SECTION: 4> </span> <span class='unselectable'>>>> </span>EntityType.SECTION.parent <span class='unselectable'>&lt;EntityType.COURSE: 3> </span> <span class='unselectable'>>>> </span>EntityType.COURSE.parent <span class='unselectable'>&lt;EntityType.GROUP: 2> </span> <span class='unselectable'>>>> </span>EntityType.GROUP.parent <span class='unselectable'>&lt;EntityType.SITE: 1> </span> <span class='unselectable'>>>> </span>EntityType.SITE.parent <span class='unselectable'>&lt;EntityType.SITE: 1> </span></pre> <h2 id="compose">Enum Composition</h2> <p> Like methods and properties in all other Python classes, enum methods and properties compose if they return an instance of the class. Composition is also known as <a href='https://en.wikipedia.org/wiki/Method_chaining' target='_blank' rel='nofollow'>method chaining</a>, and also can apply to class properties. Composition is an essential practice of a functional programming style. </p> <p> The <code>parent</code> property returns an instance of the <code>EntityType</code> enum class, so it can be composed with any other property or method of that class, for example the <code>to_s</code> property shown earlier. </p> <div class='codeLabel unselectable' data-lt-active='false'>Python</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id74976acda3c4'><button class='copyBtn' data-clipboard-target='#id74976acda3c4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>EntityType.LECTURE.parent.to_s <span class='unselectable'>'section' </span> <span class='unselectable'>>>> </span>EntityType.SECTION.parent.to_s <span class='unselectable'>'course' </span> <span class='unselectable'>>>> </span>EntityType.COURSE.parent.to_s <span class='unselectable'>'group' </span> <span class='unselectable'>>>> </span>EntityType.GROUP.parent.to_s <span class='unselectable'>'site' </span> <span class='unselectable'>>>> </span>EntityType.SITE.parent.to_s <span class='unselectable'>'site' </span></pre> <h2 id="dispatch">Dynamic Dispatch</h2> <p> The <a href='https://docs.python.org/3/library/typing.html#callable' target='_blank' rel='nofollow'>Python documentation</a> might lead someone to assume that writing <a href='https://en.wikipedia.org/wiki/Dynamic_dispatch' target='_blank' rel='nofollow'>dynamic dispatch</a> code is more complex than it actually is. </p> <p> To summarize the documentation, all Python classes, methods and instances are callable. <a href='https://www.tutorialsteacher.com/python/callable-method' target='_blank' rel='nofollow'><code>Callable</code> functions</a> have type <code>Callable[[InputArg1Type, InputArg2Type], ReturnType]</code>. If you do not want any type checking, write <code>Callable[..., Any]</code>. However, this is not very helpful information for dynamic dispatch. Fortunately, working with <code>Callable</code> is very simple. </p> <p class="notepaper"> You can pass around any Python class, constructor, function or method, and later provide it with the usual arguments. Invocation just works. </p> <p> Let me show you how easy it is to write dynamic dispatch code in Python, let's construct one of five classes, depending on the value of an enum. First, we need a class definition for each enum value: </p> <div class='codeLabel unselectable' data-lt-active='false'>Python</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7e2f999ec644'><button class='copyBtn' data-clipboard-target='#id7e2f999ec644' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># pylint: disable=too-few-public-methods class BaseClass(): """Demo only""" class TestLecture(BaseClass): """This constructor has type Callable[[int, str], TestLecture]""" def __init__(self, id_: int, action: str): print(f"CadLecture constructor called with id {id_} and action {action}") class TestSection(BaseClass): """This constructor has type Callable[[int, str], TestSection]""" def __init__(self, id_: int, action: str): print(f"CadSection constructor called with id {id_} and action {action}") class TestCourse(BaseClass): """This constructor has type Callable[[int, str], TestCourse]""" def __init__(self, id_: int, action: str): print(f"CadCourse constructor called with id {id_} and action {action}") class TestGroup(BaseClass): """This constructor has type Callable[[int, str], TestGroup]""" def __init__(self, id_: int, action: str): print(f"CadGroup constructor called with id {id_} and action {action}") class TestSite(BaseClass): """This constructor has type Callable[[int, str], TestSite]""" def __init__(self, id_: int, action: str): print(f"CadSite constructor called with id {id_} and action {action}")</pre> <p> Now lets add another method, called <code>construct</code>, to <code>EntityType</code> that invokes the appropriate constructor according to the value of an <code>EntityType</code> instance: </p> <div class='codeLabel unselectable' data-lt-active='false'>Python</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcc9cbbd9f8cf'><button class='copyBtn' data-clipboard-target='#idcc9cbbd9f8cf' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>@property def construct(self) -> Callable: """:return: the appropriate Callable for each enum value""" if self == EntityType.LECTURE: return TestLecture if self == EntityType.SECTION: return TestSection if self == EntityType.COURSE: return TestCourse if self == EntityType.GROUP: return TestGroup return TestSite</pre> <p class="callForWorkRHS"> Using named arguments makes your code resistant to problems that might sneak in due to parameters changing over time. </p> <p> I favor using named arguments at all times; it avoids many problems. As code evolves, arguments might be added or removed, or even reordered. </p> <p class="clear"> Let's test out dynamic dispatch in the Python interpreter. A class specific to each <code>EntityType</code> value is constructed by invoking the appropriate <code>Callable</code> and passing it named arguments <a href='https://stackoverflow.com/a/28091085/553865' target='_blank' rel='nofollow'><code>id_</code></a> and <code>action</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Python</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0d177c5c7ec3'><button class='copyBtn' data-clipboard-target='#id0d177c5c7ec3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>EntityType.LECTURE.construct(id_=55, action="gimme_lecture") <span class='unselectable'>TestLecture constructor called with id 55 and action gimme_lecture &lt;entity_types.TestLecture object at 0x7f9aac690070> </span> <span class='unselectable'>>>> </span>EntityType.SECTION.construct(id_=13, action="gimme_section") <span class='unselectable'>TestSection constructor called with id 13 and action gimme_section &lt;entity_types.TestSection object at 0x7f9aac5c1730> </span> <span class='unselectable'>>>> </span>EntityType.COURSE.construct(id_=40, action="gimme_course") <span class='unselectable'>TestCourse constructor called with id 40 and action gimme_course &lt;entity_types.TestCourse object at 0x7f9aac6900a0> </span> <span class='unselectable'>>>> </span>EntityType.GROUP.construct(id_=103, action="gimme_group") <span class='unselectable'>TestGroup constructor called with id 103 and action gimme_group &lt;entity_types.TestGroup object at 0x7f9aac4c6b20> </span> <span class='unselectable'>>>> </span>EntityType.SITE.construct(id_=1, action="gimme_site") <span class='unselectable'>TestSite constructor called with id 1 and action gimme_site &lt;entity_types.TestSite object at 0x7f9aac5c1730> </span></pre> <p> Because these factory methods return the newly created instance of the desired type, the string representation is printed on the console after the method finishes outputting its processing results, for example: <code>&lt;entity_types.TestLecture object at 0x7f9aac690070></code>. </p> <p> Using enums to construct class instances and/or invoke methods (aka dynamic dispatch) is super powerful. It rather resembles generics, actually, even though <a href='https://docs.python.org/3/library/typing.html#building-generic-types' target='_blank' rel='nofollow'>Python's support for generics</a> is still in its infancy. </p> <p> The complete Python program discussed in this post is <a href='/blog/python/entity_types.py'>here</a>. </p> Linking Directories on NTFS and Ext4 Volumes 2022-02-07T00:00:00-05:00 https://mslinn.github.io/blog/2022/02/07/wsl-volumes <p> Sometimes I need to insert some code into a program that depends on the type of format that a drive volume has. For example, today I need to either make a Windows junction to connect two directories on NTFS volumes. On the other hand, if one or both directories were on other types of volumes, I would have to connect the directories using a Linux symlink. </p> <p> All of the bash scripts shown in this blog post are meant to run in a Bash shell running on WSL or WSL2. </p> <h2 id="possible">Use Windows Junctions When Possible</h2> <p> When working on WSL, Windows junctions are more desirable than Linux hard links and symlinks because junctions are visible in Windows and also in WSL. Unlike Linux hard links, which only work within a single volume, Windows junctions can span two volumes. Linux symlinks are only visible from WSL; a symlinked directory only appears as a useless file when viewed from Windows. </p> <div style="text-align: right;"> <picture> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.webp" type="image/webp"> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.png" type="image/png"> <img src="/blog/images/wsl-volumes/windowsFileMgr.png" class="right liImg2 rounded shadow" /> </picture> </div> <p> Both directories need to be on NTFS volumes to make a Windows junction between them. Junctions are permitted within a single NTFS volume, or between two NTFS volumes. Linux symlinks can be used on all volume types, but only work properly when viewed from Linux. </p> <p> Windows junctions are shown with a small arrow icon in Windows File Manager. In the image above, the <code>curriculum</code> directory is a junction. </p> <h2 id="volumeType">Determining a Volume Type</h2> <p> I wrote the <code>volumeType</code> bash function to obtain the type of the volume that contains a file or directory. Linux <code>ext4</code> volumes have partition type <code>ext4</code>. NTFS volumes have partition type <code>9p</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id187fd28ad6f0'><button class='copyBtn' data-clipboard-target='#id187fd28ad6f0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>function volumeType { # Usually returns volume types ext4 or 9p (for NTFS) df -Th "$1" | tail -n 1 | awk '{print $2}' }</pre> <p> Here are examples of using <code>volumeType</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id01504c7384bd'><button class='copyBtn' data-clipboard-target='#id01504c7384bd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>volumeType /mnt/c # Under WSL/WSL2 this is usually NTFS <span class='unselectable'>9p </span> <span class='unselectable'>$ </span>volumeType / # For Ubuntu this defaults to ext4 <span class='unselectable'>ext4 </span></pre> <p> All of the remaining scripts on this page either return a value (indicating <code>true</code>), or they do not return anything (indicating <code>false</code>). </p> <p> Two more bash functions test if a file or directory is part of an NTFS or ext4 volume: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2cabdcae0c20'><button class='copyBtn' data-clipboard-target='#id2cabdcae0c20' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>function isNTFS { if [ "$( volumeType "$1" )" == 9p ]; then echo yes; fi } function isExt4 { if [ "$( volumeType "$1" )" == ext4 ]; then echo yes; fi }</pre> <p> Here are examples of using <code>isNTFS</code> and <code>isExt4</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9316a5b12a7d'><button class='copyBtn' data-clipboard-target='#id9316a5b12a7d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>isNTFS /mnt/c <span class='unselectable'>yes </span> <span class='unselectable'>$ </span>isExt4 /mnt/c <span class='unselectable'>$ </span>isNTFS / <span class='unselectable'>$ </span>isExt4 / <span class='unselectable'>yes </span></pre> <h2 id="junctions">Windows Junctions</h2> <p> The <code>bothOnNTFS</code> bash function indicates if both of the paths passed to it are on NTFS volumes. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9f10690d0a93'><button class='copyBtn' data-clipboard-target='#id9f10690d0a93' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>function bothOnNTFS { if [ "$( isNTFS "$1" )" ] && [ "$( isNTFS "$2" )" ]; then echo yes; fi }</pre> <p> Let's try out <code>bothOnNTFS</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddcb8d538947d'><span class='unselectable'>$ </span>bothOnNTFS /mnt/c /mnt/f <span class='unselectable'>yes </span> <span class='unselectable'>$ </span>bothOnNTFS /mnt/c /</pre> <p> <code>bothOnNTFS</code> lets us decide how to connect two directories. If they are both on NTFS volumes, we can connect them using a Windows junction; otherwise we'll need to symlink them. </p> <h2 id="connect">Connecting Via a Windows Junction or Linux Symlink</h2> <p> We could either make a Windows junction using the <a href='https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/mklink' target='_blank' rel='nofollow'><code>mklink</code></a> command, or we could make a Linux symlink using the <a href='https://man7.org/linux/man-pages/man1/ln.1.html' target='_blank' rel='nofollow'><code>ln -s</code></a> command. Notice how the order of parameters between <code>mklink</code> is the reverse of the order of the Linux <code>ln</code> command. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1dd89eee6821'><button class='copyBtn' data-clipboard-target='#id1dd89eee6821' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>if [ $( bothOnNTFS "$cadenzaCurriculum" . ) ]; then WINDOWS_PATH="$( wslpath -w "$cadenzaCurriculum/site_$TITLE" )" echo "Making Windows junction from $cadenzaCurriculum/site_$TITLE to curriculum/" cmd.exe /C mklink /j curriculum "$WINDOWS_PATH" else echo "Symlinking $cadenzaCurriculum/site_$TITLE to curriculum/" ln -s "$cadenzaCurriculum/site_$TITLE" curriculum fi</pre> When the above code ran it produced: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1de09cceb20e'>Making Windows junction from /mnt/f/work/cadenzaHome/cadenzaCurriculum/site_ScalaCourses.com to curriculum/ Junction created for curriculum <<===>> F:\work\cadenzaHome\cadenzaCurriculum\site_ScalaCourses.com</pre> Handcrafted Dynamic DNS for AWS Route53 2022-01-30T00:00:00-05:00 https://mslinn.github.io/blog/2022/01/30/ddns-route53 <p> Now that I have fiber-optic internet service in my apartment, with 500 GB/s upload and download, I thought I would save money by hosting my <a href='https://scalacourses.com' target='_blank' rel='nofollow'><code>scalacourses.com</code></a> website on an Ubuntu server that runs here, instead of AWS. My home IP address is quite stable, and only changes when the fiber modem boots up. The modem is branded as a <a href='https://support.bell.ca/internet/products/home-hub-4000-modem' target='_blank' rel='nofollow'>Bell Home Hub 4000</a>, but I believe it is actually made by Arris (formerly known as Motorola). </p> <p> It makes little sense to pay the commercial cost of dedicated dynamic DNS services (typically $55 USD / year) when it is so easy to automate, and the operational cost is less than one cent per year. </p> <p> Because I continue to use AWS Route53 for DNS, I wrote a little script that automatically checks my public IP address, and upserts a Route53 record for my home IP address whenever the IP address changes. </p> <p> The approach shown here could be used for all DNS servers that have a command-line interface, not just AWS Route53. Alternatives include <a href='https://docs.microsoft.com/en-us/cli/azure/network/dns?view=azure-cli-latest' target='_blank' rel='nofollow'>Azure DNS</a>, <a href='https://developers.cloudflare.com/cloudflare-one/tutorials/cli' target='_blank' rel='nofollow'>Cloudflare DNS</a>, <a href='https://support.dnsmadeeasy.com/support/solutions/articles/47001119947-the-ddns-shell-script' target='_blank' rel='nofollow'>DNSMadeEasy</a>, <a href='https://developer.dnsimple.com/libraries/' target='_blank' rel='nofollow'>DNSimple</a>, <a href='https://cloud.google.com/sdk/gcloud/reference/dns' target='_blank' rel='nofollow'>Google Cloud DNS</a>, and <a href='https://github.com/ultradns/dns_sprockets' target='_blank' rel='nofollow'>UltraDNS</a>. </p> <h2 id="fwd">Forwarding HTTP Requests</h2> <p> I added some entries to the modem so incoming HTTP traffic on ports 80 and 443 would be forwarded to ports 9000 and 9443 on my home server, <code>gojira</code>. </p> <div style=""> <picture> <source srcset="/blog/images/ddns/homeHubPortForward.webp" type="image/webp"> <source srcset="/blog/images/ddns/homeHubPortForward.png" type="image/png"> <img src="/blog/images/ddns/homeHubPortForward.png" class=" liImg2 rounded shadow" /> </picture> </div> <h2 id="usage">Using the Script</h2> <p> The script saves the IP address to a file, and periodically compare the saved value to the current value. Then the script modifies the Route53 record for a specified subdomain whenever the value of my public IP address changes. </p> <p> Here is the help information for the script : </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb15d414f58db'><button class='copyBtn' data-clipboard-target='#idb15d414f58db' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>dynamicDns <span class='unselectable'>dynamicDns - Maintains a dynamic DNS record in AWS Route53<br> Saves data in '/home/mslinn/.dynamicDns'<br> Syntax: dynamicDns [OPTIONS] SUB_DOMAIN DOMAIN<br> OPTIONS: -v Verbose mode<br> Example usage: dynamicDns my_subdomain my_domain.com dynamicDns -v my_subdomain my_domain.com </span></pre> <p> Here is a sample usage: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idee55f48e1782'><button class='copyBtn' data-clipboard-target='#idee55f48e1782' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>dynamicDns my_subdomain my_domain.com <span class='unselectable'>{ "ChangeInfo": { "Id": "/change/C075751811HI18SH4L8L0", "Status": "PENDING", "SubmittedAt": "2022-01-30T21:10:09.261Z", "Comment": "UPSERT a record for my_subdomain.my_domain.com" } } </span></pre> <h2 id="crontab">Invoking the Script from Crontab</h2> <p> A personal <code>crontab</code> can be modified by typing: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id59e036d6de1f'><button class='copyBtn' data-clipboard-target='#id59e036d6de1f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>crontab -e</pre> <p> I pasted in the following 2 lines into <code>crontab</code> on my Ubuntu server, running at home. These lines invoke the script via <code>crontab</code> when it boots, and every 5 minutes after that. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf3a624dc3347'><button class='copyBtn' data-clipboard-target='#idf3a624dc3347' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>@reboot /path/to/dynamicDns my_subdomain my_domain.com */5 * * * * /path/to/dynamicDns my_subdomain my_domain.com</pre> <p> That's it, <code>crontab</code> will run the script within the next 5 minutes. </p> <h2 id="source">Script Source Code</h2> <p> Here is the bash script: </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,dynamicDns' download='dynamicDns' title='Click on the file name to download the file'>dynamicDns</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="ide855c5da7067">#!/bin/bash # Author: Mike Slinn mslinn@mslinn.com # Written 2022-01-30 export SAVE_FILE_NAME="$HOME/.dynamicDns" function help &#123; echo "$( basename $0 ) - Maintains a dynamic DNS record in AWS Route53 Saves data in '$SAVE_FILE_NAME' Syntax: $( basename $0) [OPTIONS] SUB_DOMAIN DOMAIN OPTIONS: -v Verbose mode Example usage: $( basename $0) my_subdomain mydomain.com $( basename $0) -v my_subdomain mydomain.com " exit 1 &#125; function upsert &#123; export HOSTED_ZONES="$( aws route53 list-hosted-zones )" export HOSTED_ZONE_RECORD="$( jq -r ".HostedZones[] | select(.Name == \"$DOMAIN.\")" &lt;&lt;&lt; "$HOSTED_ZONES" )" export HOSTED_ZONE_RECORD_ID="$( jq -r .Id &lt;&lt;&lt; "$HOSTED_ZONE_RECORD" )" aws route53 change-resource-record-sets \ --hosted-zone-id "$HOSTED_ZONE_RECORD_ID" \ --change-batch "&#123; \"Comment\": \"UPSERT a record for $SUBDOMAIN.$DOMAIN\", \"Changes\": [&#123; \"Action\": \"UPSERT\", \"ResourceRecordSet\": &#123; \"Name\": \"$SUBDOMAIN.$DOMAIN\", \"Type\": \"A\", \"TTL\": 300, \"ResourceRecords\": [&#123; \"Value\": \"$IP\"&#125;] &#125; &#125;] &#125;" echo "$IP" > "$SAVE_FILE_NAME" &#125; if [ "$1" == -v ]; then export VERBOSE=true shift fi if [ -z "$2" ]; then help; fi set -e export SUBDOMAIN="$1" export DOMAIN="$2" export IP="$( dig +short myip.opendns.com @resolver1.opendns.com )" if [ ! -f "$SAVE_FILE_NAME" ]; then if [ "$VERBOSE" ]; then echo "Creating $SAVE_FILE_NAME"; fi upsert; elif [ $( cat "$SAVE_FILE_NAME" ) != "$IP" ]; then if [ "$VERBOSE" ]; then echo "Updating $SAVE_FILE_NAME" echo "'$IP' was not equal to '$( cat "$SAVE_FILE_NAME" )'" fi upsert; else if [ "$VERBOSE" ]; then echo "No change necessary for $SAVE_FILE_NAME"; fi fi </pre> Trimming Media Files Can Be Surprisingly Subtle 2022-01-23T00:00:00-05:00 https://mslinn.github.io/blog/2022/01/23/trimming-media <p> When I use OBS Studio to record video from Cam Link 4K and audio from Pro Tools using RME TotalMix I get really big <code>mkv</code> files. I need to be able to trim the video file so the bits before and after the good stuff are discarded. </p> <p> Lots of StackOverflow conversations revolve around trimming video files. Dozens of PC and Mac programs exist to do that task, mostly low quality and / or bothersome to use. Other solutions are overkill, for example Adobe Premiere Pro and DaVinci Resolve. </p> <p> In this post I present a bash script that reduces file size by > 80%, while preserving quality and metadata, and cropping to specified time periods. </p> <p> For any programmers who might read this: </p> <ul> <li>Some of the <code>ffmpeg</code> options this script uses are not available in older versions.</li> <li> The most important thing to know about options that might be passed to <code>ffmpeg</code> is that if you want to force a new video encoding, simply do not specify the <code>-vcodec copy</code> option. Re-encoding means that arbitrary start and end times can be specified accurately when cropping. </li> </ul> <p> <a href='https://gist.github.com/mslinn/15a1308aa2ca04a418f404e42c7f32e0' target='_blank'>This is the script:</a> </p> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/15a1308aa2ca04a418f404e42c7f32e0.js"> </script> <p> Here is a sample session, which extracts the sequence beginning at <code>00:00:25.000</code> until <code>00:02:52.000</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id68e1e2d479d3'><button class='copyBtn' data-clipboard-target='#id68e1e2d479d3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>crop 'VideoFile.mkv' 25 2:52 <span class='unselectable'>ffmpeg version 4.4-6ubuntu5 Copyright (c) 2000-2021 the FFmpeg developers built with gcc 11 (Ubuntu 11.2.0-7ubuntu1) configuration: --prefix=/usr --extra-version=6ubuntu5 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared libavutil 56. 70.100 / 56. 70.100 libavcodec 58.134.100 / 58.134.100 libavformat 58. 76.100 / 58. 76.100 libavdevice 58. 13.100 / 58. 13.100 libavfilter 7.110.100 / 7.110.100 libswscale 5. 9.100 / 5. 9.100 libswresample 3. 9.100 / 3. 9.100 libpostproc 55. 9.100 / 55. 9.100 Input #0, matroska,webm, from 'VideoFile.mkv': Metadata: ENCODER : Lavf58.29.100 Duration: 00:02:57.93, start: 0.000000, bitrate: 8137 kb/s Stream #0:0: Video: h264 (High), yuv420p(tv, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 1k tbn, 60 tbc (default) Metadata: DURATION : 00:02:57.933000000 Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp (default) Metadata: title : simple_aac_recording DURATION : 00:02:57.813000000 Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264)) Stream #0:1 -> #0:1 (copy) Press [q] to stop, [?] for help [libx264 @ 0x55fe8a8a6d40] using SAR=1/1 [libx264 @ 0x55fe8a8a6d40] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX [libx264 @ 0x55fe8a8a6d40] profile High, level 3.1, 4:2:0, 8-bit [libx264 @ 0x55fe8a8a6d40] 264 - core 160 r3011 cde9a93 - H.264/MPEG-4 AVC codec - Copyleft 2003-2020 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00 Output #0, matroska, to 'VideoFile.crop.mkv': Metadata: encoder : Lavf58.76.100 Stream #0:0: Video: h264 (H264 / 0x34363248), yuv420p(tv, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 30 fps, 1k tbn (default) Metadata: DURATION : 00:02:57.933000000 encoder : Lavc58.134.100 libx264 Side data: cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A Stream #0:1: Audio: aac (LC) ([255][0][0][0] / 0x00FF), 48000 Hz, stereo, fltp (default) Metadata: title : simple_aac_recording DURATION : 00:02:57.813000000 frame= 4410 fps= 56 q=-1.0 Lsize= 29863kB time=00:02:26.98 bitrate=1664.3kbits/s speed=1.85x video:26295kB audio:3489kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.266039% [libx264 @ 0x55fe8a8a6d40] frame I:18 Avg QP:19.94 size: 91473 [libx264 @ 0x55fe8a8a6d40] frame P:1111 Avg QP:22.35 size: 14426 [libx264 @ 0x55fe8a8a6d40] frame B:3281 Avg QP:26.70 size: 2820 [libx264 @ 0x55fe8a8a6d40] consecutive B-frames: 0.8% 0.0% 0.1% 99.1% [libx264 @ 0x55fe8a8a6d40] mb I I16..4: 9.1% 68.8% 22.1% [libx264 @ 0x55fe8a8a6d40] mb P I16..4: 0.1% 1.2% 0.4% P16..4: 41.3% 11.1% 8.7% 0.0% 0.0% skip:37.2% [libx264 @ 0x55fe8a8a6d40] mb B I16..4: 0.0% 0.2% 0.0% B16..8: 25.8% 2.6% 0.6% direct: 0.7% skip:70.0% L0:42.9% L1:51.0% BI: 6.1% [libx264 @ 0x55fe8a8a6d40] 8x8 transform intra:71.2% inter:72.8% [libx264 @ 0x55fe8a8a6d40] coded y,uvDC,uvAC intra: 81.3% 83.4% 41.2% inter: 6.8% 12.5% 0.3% [libx264 @ 0x55fe8a8a6d40] i16 v,h,dc,p: 27% 14% 15% 44% [libx264 @ 0x55fe8a8a6d40] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 15% 12% 7% 13% 9% 13% 7% 9% [libx264 @ 0x55fe8a8a6d40] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 13% 11% 10% 17% 11% 10% 6% 7% [libx264 @ 0x55fe8a8a6d40] i8c dc,h,v,p: 46% 21% 21% 11% [libx264 @ 0x55fe8a8a6d40] Weighted P-Frames: Y:0.5% UV:0.1% [libx264 @ 0x55fe8a8a6d40] ref P L0: 56.5% 10.1% 21.4% 12.0% 0.0% [libx264 @ 0x55fe8a8a6d40] ref B L0: 87.7% 8.9% 3.4% [libx264 @ 0x55fe8a8a6d40] ref B L1: 94.4% </span></pre> <p> The cropped file is quite a bit smaller than the original. I have not noticed any decrease in quality. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7e6166dc714e'><button class='copyBtn' data-clipboard-target='#id7e6166dc714e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls -AlF <span class='unselectable'>total 1546136 -rw-r--r-- 1 mslinn mslinn <span class='bg_yellow'>30579665</span> Jan 23 13:03 'VideoFile.crop.mkv' -rwxrwxrwx 1 mslinn mslinn <span class='bg_yellow'>180988555</span> Jan 22 18:31 'VideoFile.mkv' </span></pre> Windows Diskpart Cooperates With Diskmgmt 2022-01-14T00:00:00-05:00 https://mslinn.github.io/blog/2022/01/14/diskpart <p> Recently, I wanted to use some old hard drives as backup media. That meant scrubbing all the partitions off the drives, and installing new partitions, which of course would be empty. </p> <div class="right warning" style="width: 45%"> <p> Warning &ndash; Working with a command line program for system-level operations, without having backed up the system, is like walking on a tightrope without a net. </p> <p style="margin-bottom: 0"> A mistake could inadvertently wipe out a different hard drive on the computer than you intended. Without the ability to restore the system disk from backup, your computer could become inoperable. </p> </div> <p> However, I was unable to repartition one of those old drives using the GUI-based Windows <code>diskmgmt.msc</code> disk manager. The drive had soft errors. For some reason, those errors made it impossible for <code>diskmgmt.msc</code> to scan the drive. </p> <p> The more powerful Windows <a href='https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/diskpart' target='_blank' rel='nofollow'><code>diskpart</code></a>, a command line program, was able to get the job done. </p> <p> This article <a href ="#gui">ends</a> by demonstrating how you can get benefit from the <code>diskmgmt</code> GUI even when you are using the <code>diskpart</code> command line interface. This is possible because every command you type into <code>diskpart</code> causes a Windows system event to be published, and because <code>diskmgmt</code> subscribes to those events, it is able to display the results as they happen. </p> <h2 id="starting">Starting <span class="code">Diskpart</span></h2> <p> Run <code>diskpart</code> as administrator as follows: </p> <ol> <li> Press the <kbd>Windows</kbd> key. Do not hold it down, just depress it once, as you would do for any other key, and let it go. </li> <li>Type <code>diskpart</code>.</li> <li>Use the mouse or arrow keys to select <b>Run as administrator</b>.</li> </ol> <div style="text-align: center;"> <picture> <source srcset="/blog/images/diskpart/diskpartLaunch.webp" type="image/webp"> <source srcset="/blog/images/diskpart/diskpartLaunch.png" type="image/png"> <img src="/blog/images/diskpart/diskpartLaunch.png" class="center halfsize liImg2 rounded shadow" /> </picture> </div> <p> A window should open up labeled <b>Microsoft DiskPart</b>. Let's start by listing all the <code>diskpart</code> commands. </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6fdfad4fff44'><button class='copyBtn' data-clipboard-target='#id6fdfad4fff44' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>Microsoft DiskPart version 10.0.19041.964 Copyright (C) Microsoft Corporation. On computer: BEAR DISKPART> </span>help <span class='unselectable'>Microsoft DiskPart version 10.0.19041.964 ACTIVE - Mark the selected partition as active. ADD - Add a mirror to a simple volume. ASSIGN - Assign a drive letter or mount point to the selected volume. ATTRIBUTES - Manipulate volume or disk attributes. ATTACH - Attaches a virtual disk file. AUTOMOUNT - Enable and disable automatic mounting of basic volumes. BREAK - Break a mirror set. CLEAN - Clear the configuration information, or all information, off the disk. COMPACT - Attempts to reduce the physical size of the file. CONVERT - Convert between different disk formats. CREATE - Create a volume, partition or virtual disk. DELETE - Delete an object. DETAIL - Provide details about an object. DETACH - Detaches a virtual disk file. EXIT - Exit DiskPart. EXTEND - Extend a volume. EXPAND - Expands the maximum size available on a virtual disk. FILESYSTEMS - Display current and supported file systems on the volume. FORMAT - Format the volume or partition. GPT - Assign attributes to the selected GPT partition. HELP - Display a list of commands. IMPORT - Import a disk group. INACTIVE - Mark the selected partition as inactive. LIST - Display a list of objects. MERGE - Merges a child disk with its parents. ONLINE - Online an object that is currently marked as offline. OFFLINE - Offline an object that is currently marked as online. RECOVER - Refreshes the state of all disks in the selected pack. Attempts recovery on disks in the invalid pack, and resynchronizes mirrored volumes and RAID5 volumes that have stale plex or parity data. REM - Does nothing. This is used to comment scripts. REMOVE - Remove a drive letter or mount point assignment. REPAIR - Repair a RAID-5 volume with a failed member. RESCAN - Rescan the computer looking for disks and volumes. RETAIN - Place a retained partition under a simple volume. SAN - Display or set the SAN policy for the currently booted OS. SELECT - Shift the focus to an object. SETID - Change the partition type. SHRINK - Reduce the size of the selected volume. UNIQUEID - Displays or sets the GUID partition table (GPT) identifier or master boot record (MBR) signature of a disk. </span></pre> <h2 id="listing">Listing Drives, Partitions and Volumes</h2> <p> Listing the drives is generally a good first step. Let's discover the command for that: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1c1beb388ddc'><button class='copyBtn' data-clipboard-target='#id1c1beb388ddc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>list <span class='unselectable'>Microsoft DiskPart version 10.0.19041.964 DISK - Display a list of disks. For example, LIST DISK. PARTITION - Display a list of partitions on the selected disk. For example, LIST PARTITION. VOLUME - Display a list of volumes. For example, LIST VOLUME. VDISK - Displays a list of virtual disks. </span></pre> <p> OK, we can list disks, partitions, volumes and virtual disks. Let's list the disk drives. </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8ff4707463f0'><button class='copyBtn' data-clipboard-target='#id8ff4707463f0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>list disk <span class='unselectable'>Disk ### Status Size Free Dyn Gpt -------- ------------- ------- ------- --- --- Disk 0 Online 1863 GB 1024 KB * Disk 1 Online 465 GB 1024 KB Disk 2 Online 1863 GB 1024 KB * Disk 3 Online 1863 GB 0 B * Disk 5 Online 931 GB 931 GB * </span></pre> <div style=""> <picture> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.webp" type="image/webp"> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.png" type="image/png"> <img src="/blog/images/diskpart/newerTechVoyagerS3.png" class=" right" style="width: 25%; height: auto;" /> </picture> </div> <p> At this point I inserted the old 5.25" SATA drive that I wanted to repurpose into a <a href='https://www.amazon.com/NewerTech-Enclosure-Interface-NWTU3S3HD-hot-swapping/dp/B007TTQQIA' target='_blank' rel='nofollow'>NewerTech Voyager S3</a> caddy, connected to my computer via a USB 3 cable, and Windows automatically mounted it. The caddy also accepts 2.5" SATA drives, such as those commonly found in laptops. </p> <p> Now I told <code>diskpart</code> to rescan the drives, and then I listed the volumes on all disks. </p> <div class="clear"></div> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id588229cfa7a2'><button class='copyBtn' data-clipboard-target='#id588229cfa7a2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>rescan <span class='unselectable'>Please wait while DiskPart scans your configuration... DiskPart has finished scanning your configuration. DISKPART> </span>list volume <span class='unselectable'>Volume ### Ltr Label Fs Type Size Status Info ---------- --- ----------- ----- ---------- ------- --------- -------- Volume 0 D DVD-ROM 0 B No Media Volume 1 C BEAR_C NTFS Partition 1861 GB Healthy Boot Volume 2 FAT32 Partition 100 MB Healthy System Volume 3 NTFS Partition 539 MB Healthy Hidden Volume 4 NTFS Partition 450 MB Healthy Hidden Volume 5 F Work NTFS Partition 1863 GB Healthy Volume 6 E BEAR_E NTFS Partition 1863 GB Healthy <span class="bg_yellow"> Volume 8 FAT32 Partition 512 MB Healthy Hidden</span> </span></pre> <p> <code>Diskpart</code> displayed the hidden volume (#8) in the drive in the caddy. This drive has <code>readonly</code> status set, which prevents its contents from being modified or deleted. That would be good if I wanted to use this drive as an archive, but instead I want to scrub it and write new information on it. The currently existing partitions on this drive cannot be erased until <code>readonly</code> is cleared. Lets remove <code>readonly</code> status from the drive now: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide05ffab8ecdd'><button class='copyBtn' data-clipboard-target='#ide05ffab8ecdd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>select volume 8 <span class='unselectable'>Volume 8 is the selected volume. DISKPART> </span>attributes disk clear readonly <span class='unselectable'>Disk attributes cleared successfully. DISKPART> </span>rescan <span class='unselectable'>Please wait while DiskPart scans your configuration... DiskPart has finished scanning your configuration. DISKPART> </span>list volume <span class='unselectable'>Volume ### Ltr Label Fs Type Size Status Info ---------- --- ----------- ----- ---------- ------- --------- -------- Volume 0 D DVD-ROM 0 B No Media Volume 1 C BEAR_C NTFS Partition 1861 GB Healthy Boot Volume 2 FAT32 Partition 100 MB Healthy System Volume 3 NTFS Partition 539 MB Healthy Hidden Volume 4 NTFS Partition 450 MB Healthy Hidden Volume 5 F Work NTFS Partition 1863 GB Healthy Volume 6 E BEAR_E NTFS Partition 1863 GB Healthy Volume 8 FAT32 Partition 512 MB Healthy Hidden </span></pre> <h2 id="wipe">Wiping the Drive</h2> <p> Now it is time to wipe the drive clean, which removes all partitions. </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb56579a1a27d'><button class='copyBtn' data-clipboard-target='#idb56579a1a27d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>select disk 5 <span class='unselectable'>Disk 5 is now the selected disk. DISKPART> </span>list disk <span class='unselectable'>Disk ### Status Size Free Dyn Gpt -------- ------------- ------- ------- --- --- Disk 0 Online 1863 GB 1024 KB * Disk 1 Online 465 GB 1024 KB Disk 2 Online 1863 GB 1024 KB * Disk 3 Online 1863 GB 0 B * Disk 5 Online 931 GB 931 GB * DISKPART> </span>clean <span class='unselectable'>DiskPart succeeded in cleaning the disk. </span></pre> <h2 id="setRO">Archiving a Drive</h2> <p> If instead of wiping the drive, I wanted to archive the drive, I would want to set the read-only status. To do that, first select the drive as before, then type: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddd10e420a2b5'><button class='copyBtn' data-clipboard-target='#iddd10e420a2b5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>list disk <span class='unselectable'># Output as shown above </span> <span class='unselectable'>DISKPART> </span>select disk N <span class='unselectable'>Disk N is now the selected disk. </span> <span class='unselectable'>DISKPART> </span>attributes disk set readonly</pre> <p> Now the drive's contents could not accidently be erased or modified. </p> <h2 id="partition">Create A New Partition</h2> <p> We need to create a new partition that spans the entire disk on the now-empty selected drive. The <code>create</code> command can do that. Let's look at the help before using the command: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf8dca93e0e72'><button class='copyBtn' data-clipboard-target='#idf8dca93e0e72' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>create <span class='unselectable'>Microsoft DiskPart version 10.0.19041.964 PARTITION - Create a partition. VOLUME - Create a volume. VDISK - Creates a virtual disk file. DISKPART> </span>create partition <span class='unselectable'>Microsoft DiskPart version 10.0.19041.964 EFI - Create an EFI system partition. EXTENDED - Create an extended partition. LOGICAL - Create a logical drive. MSR - Create a Microsoft Reserved partition. PRIMARY - Create a primary partition. </span></pre> <p> To create a new partition that spans the entire disk on the now-empty selected drive and make it active: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd159301c29f7'><button class='copyBtn' data-clipboard-target='#idd159301c29f7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>create partition primary <span class='unselectable'>DiskPart succeeded in creating the specified partition. DISKPART> </span>select partition 1 <span class='unselectable'>Partition 1 is now the selected partition. DISKPART> </span>active <span class='unselectable'>DiskPart marked the current partition as active. </span></pre> <h2 id="format">Format A Volume</h2> <p> Let's format the entire selected drive as one volume. Like the <code>clean</code> command, the <code>format</code> command operates on the currently selected disk. First, let's look at the help: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf8c201cd663d'><button class='copyBtn' data-clipboard-target='#idf8c201cd663d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>help format <span class='unselectable'>Formats the specified volume for use with Windows. Syntax: FORMAT [[FS=<FS>] [REVISION=<X.XX>] | RECOMMENDED] [LABEL=<"label">] [UNIT=<N>] [QUICK] [COMPRESS] [OVERRIDE] [DUPLICATE] [NOWAIT] [NOERR] FS=<FS> Specifies the type of file system. If no file system is given, the default file system displayed by the FILESYSTEMS command is used. REVISION=<X.XX> Specifies the file system revision (if applicable). RECOMMENDED If specified, use the recommended file system and revision instead of the default if a recommendation exists. The recommended file system (if one exists) is displayed by the FILESYSTEMS command. LABEL=<"label"> Specifies the volume label. UNIT=<N> Overrides the default allocation unit size. Default settings are strongly recommended for general use. The default allocation unit size for a particular file system is displayed by the FILESYSTEMS command. NTFS compression is not supported for allocation unit sizes above 4096. QUICK Performs a quick format. COMPRESS NTFS only: Files created on the new volume will be compressed by default. OVERRIDE Forces the file system to dismount first if necessary. All opened handles to the volume would no longer be valid. DUPLICATE UDF Only: This flag applies to UDF format, version 2.5 or higher. This flag instructs the format operation to duplicate the file system meta-data to a second set of sectors on the disk. The duplicate meta-data is used by applications, for example repair or recovery applications. If the primary meta-data sectors are found to be corrupted, the file system meta-data will be read from the duplicate sectors. NOWAIT Forces the command to return immediately while the format process is still in progress. If NOWAIT is not specified, DiskPart will display format progress in percentage. NOERR For scripting only. When an error is encountered, DiskPart continues to process commands as if the error did not occur. Without the NOERR parameter, an error causes DiskPart to exit with an error code. A volume must be selected for this operation to succeed. Examples: FORMAT FS=NTFS LABEL="New Volume" QUICK COMPRESS FORMAT RECOMMENDED OVERRIDE </span></pre> <p> <code>Diskpart</code> automatically chooses the optimal file system for the selected drive if you do not specify it. Choices include FAT, FAT32 and NTFS. To quick format the selected drive using the optimal file system: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id57e4559efcc7'><button class='copyBtn' data-clipboard-target='#id57e4559efcc7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>format quick</pre> <p> The default is to fully format the selected drive, which is what you want if the selected drive is suspect: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id60ce46183f53'><button class='copyBtn' data-clipboard-target='#id60ce46183f53' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>format</pre> <p> You could specify multiple parameters, for example: </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id04d4c1e26b8a'><button class='copyBtn' data-clipboard-target='#id04d4c1e26b8a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>format fs=ntfs label="My Drive" quick</pre> <h2 id="assign">Assigning a Drive Letter</h2> <p> Once the drive was formatted, I assigned the selected drive the letter <code>G</code>. You do not normally need to perform this step, unless you want to <a href='https://www.groovypost.com/howto/assign-permanent-letter-removable-usb-drive-windows/' target='_blank' rel='nofollow'>define a default letter for this drive</a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id43c02a4fa89c'><button class='copyBtn' data-clipboard-target='#id43c02a4fa89c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>assign letter=G</pre> <h2 id="gui"><span class="code">Diskmgmt</span> GUI Shows Progress</h2> <p> Having a GUI continuously report the current state of the drive maintenance you are performing using a command-line interface is a good practice. </p> <p> The GUI-based Windows disk manager, <code>diskmgmt.msc</code>, can show the instantaneous progress of all <code>diskpart</code> commands, working on every drive, including creating and deleting partitions, volumes, formatting and much more. For example, formatting progress can be seen here: </p> <div style=""> <picture> <source srcset="/blog/images/diskpart/diskUI.webp" type="image/webp"> <source srcset="/blog/images/diskpart/diskUI.png" type="image/png"> <img src="/blog/images/diskpart/diskUI.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <p> Run <code>diskmgmt.msc</code> by: </p> <ol> <li>Press the <kbd>Windows</kbd> key once and let go.</li> <li>Type <code>diskmgmt</code></li> <li>Press <kbd>Enter</kbd></li> </ol> <h2 id="exit"><span class="code">Exit</span></h2> <p> To exit <code>diskpart</code>, either type <code>Exit</code> or press <kbd>Ctrl</kbd>-<kbd>C</kbd>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id60d8f523f8e8'><button class='copyBtn' data-clipboard-target='#id60d8f523f8e8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>exit</pre> <p> The drive is ready for its next assignment! </p> WSL / WSL 2 Backup and Restore 2022-01-10T00:00:00-05:00 https://mslinn.github.io/blog/2022/01/10/wsl-backup <editor-fold intro> <p> I needed to back up my WSL2 installation before <a href='https://www.microsoft.com/en-us/software-download/windows10' target='_blank' rel='nofollow'>reinstalling Windows 10</a>, as one must do every 6 months if you work your machine like a developer. It had been years since I had last refreshed this machine, and it was now bluescreening several times a day. Reboots took forever. </p> <div class="pullQuote"> <div class="right" style="font-size: 3em;">&#128513;</div> <p> This story detail the various approaches I attempted; all failed until I was able to <a href="#duplicate">duplicate</a> the original Ubuntu instance using <span class="code">LxRunOffline</span>. </p> </div> <p> I wanted to retain my WSL2 instance. When refreshing Windows you must reinstall all your programs, and you lose all the WSL/WSL2 instances. The refresh decommissions them, then moves them into a hidden directory tree, along with the rest of the stuff stored in your old Windows 10 profile directory tree. </p> <p> I also wanted to be able to replicate my WSL2 instance reliably and easily. It was very important to me to be able to administer WSL instances separately from my Windows 10 profile. </p> <p> Only recently has it become possible to work with WSL/WSL2 images on non-system drives. Most online documentation is now out of date in this regard. Running Windows off a different drive than the drive that a WSL/WSL2 client OS runs from can provide a dramatic performance boost for some applications. </p> <p> I use Dell laptops, because I love their onsite warranty price and features. However, sometimes it feels like Dell's solution to every problem is to replace the motherboard. That means Windows 10 feels a bit uncomfortable, and it wonders if it has been pirated. Refreshing Windows solves that problem, and blows away the standard WSL/WSL2 image. Again. And again. And again! Dell replaced the motherboard 4 times in 2 years for one of my laptops. </p> <p> Having the WSL/WSL2 image elsewhere in the filesystem means that reinstalling Windows is not as traumatic. </p> <p> Backing up WSL/WSL2 can be problematic. Restoring it is even more delicate. Knowing this, I decided to backup and test the restoration process before refreshing Windows and potentially losing my working system. </p> <p> I have tried several times before to make this work. As I said, it had been years since I allowed the OS in my main workstation to be refreshed. The reason I resisted doing proper maintenance was because I wanted to preserve my WSL2 Ubuntu image. Until today, I met with frustrating failures every time I attempted to use the standard tools. Today I succeeded, via new software, hence the publication of my notes. Who wants to read stories about all the things that do not work? </p> <p> Following is my experience. I began by following the directions in <a href='https://www.windowscentral.com/how-backup-windows-subsystem-linux-wsl-distribution' target='_blank' rel='nofollow'>How to back up a Windows Subsystem for Linux (WSL) distribution</a>, published 18 Feb 2021 by Windows Central. It is a nice story, but I've tried this stuff on a variety of computers, and this is not Microsoft's most robust code base as yet. Read on and I will tell you of reality as I found it. </p> </editor-fold> <editor-fold wslOptions> <h2 id="wslOptions">WSL Command-Line Options</h2> <p> First let's look at the command-line options for the <code>wsl</code> command. </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idaf8f589ce879'><button class='copyBtn' data-clipboard-target='#idaf8f589ce879' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>Microsoft Windows [Version 10.0.19044.1415] (c) Microsoft Corporation. All rights reserved. </span> <span class='unselectable'>C:\Users\Mike Slinn> </span>wsl -? <span class='unselectable'>Invalid command line option: -? Copyright (c) Microsoft Corporation. All rights reserved. Usage: wsl.exe [Argument] [Options...] [CommandLine] Arguments for running Linux binaries: If no command line is provided, wsl.exe launches the default shell. --exec, -e &lt;CommandLine> Execute the specified command without using the default Linux shell. -- Pass the remaining command line as is. Options: --cd &lt;Directory> Sets the specified directory as the current working directory. If ~ is used the Linux user&rsquo;s home path will be used. If the path begins with a / character, it will be interpreted as an absolute Linux path. Otherwise, the value must be an absolute Windows path. --distribution, -d &lt;Distro> Run the specified distribution. --user, -u &lt;UserName> Run as the specified user. Arguments for managing Windows Subsystem for Linux: --help Display usage information. --install [Options] Install additional Windows Subsystem for Linux distributions. For a list of valid distributions, use &quot;wsl --list --online&quot;. Options: --distribution, -d [Argument] Downloads and installs a distribution by name. Arguments: A valid distribution name (not case sensitive). Examples: wsl --install -d Ubuntu wsl --install --distribution Debian --set-default-version &lt;Version> Changes the default install version for new distributions. --shutdown Immediately terminates all running distributions and the WSL 2 lightweight utility virtual machine. --status Show the status of Windows Subsystem for Linux. --update [Options] If no options are specified, the WSL 2 kernel will be updated to the latest version. Options: --rollback Revert to the previous version of the WSL 2 kernel. Arguments for managing distributions in Windows Subsystem for Linux: --export &lt;Distro> &lt;FileName> Exports the distribution to a tar file. The filename can be - for standard output. --import &lt;Distro> &lt;InstallLocation> &lt;FileName> [Options] Imports the specified tar file as a new distribution. The filename can be - for standard input. Options: --version &lt;Version> Specifies the version to use for the new distribution. --list, -l [Options] Lists distributions. Options: --all List all distributions, including distributions that are currently being installed or uninstalled. --running List only distributions that are currently running. --quiet, -q Only show distribution names. --verbose, -v Show detailed information about all distributions. --online, -o Displays a list of available distributions for install with 'wsl --install'. --set-default, -s &lt;Distro> Sets the distribution as the default. --set-version &lt;Distro> &lt;Version> Changes the version of the specified distribution. --terminate, -t &lt;Distro> Terminates the specified distribution. --unregister &lt;Distro> Unregisters the distribution and deletes the root filesystem. </span></pre> </editor-fold> <editor-fold wsBackup> <h2 id="wslBackup">Backing Up With WSL</h2> <p> Now let's use the <code>wsl</code> command to back up the Ubuntu VM in WSL2. </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id39fba3969485'><button class='copyBtn' data-clipboard-target='#id39fba3969485' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --export Ubuntu ubuntuBear_2021-01-10.tar</pre> <p> Well, well, it backed up without any problem in about an hour! Color me <s>surprised</s>happy. File size was about 50 GB. </p> <p> Alright, let's try importing the image now. We'll locate the new image at <code>f:\ubuntuBear</code>. It used to be that WSL did not support moving or installing a distro to non-system drives. No longer! </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc3c5a8fd815e'><button class='copyBtn' data-clipboard-target='#idc3c5a8fd815e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --import Ubuntu f:\ubuntuBear ubuntuBear_2021-01-10.tar <span class='unselectable'>A distribution with the supplied name already exists. </span></pre> <p> The error message &ldquo;A distribution with the supplied name already exists&rdquo; makes sense because I backed up a WSL2 instance called <code>Ubuntu</code> and instance names must be unique. Let's import under the name <code>UbuntuBear</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4ad1cb2f42b6'><button class='copyBtn' data-clipboard-target='#id4ad1cb2f42b6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --import UbuntuBear f:\ubuntuBear ubuntuBear_2021-01-10.tar <span class='unselectable'>Unspecified error </span></pre> <p> Ahh, the dreaded <code>Unspecified error</code> that <code>wsl import</code> is infamous for. <a href='https://github.com/microsoft/WSL/issues/4735#issuecomment-800103777' target='_blank' rel='nofollow'>Google brought me</a> to a potential solution, <a href='https://github.com/DDoSolitary/LxRunOffline' target='_blank' rel='nofollow'><code>LxRunOffline</code></a>. </p> </editor-fold> <editor-fold lxRunOffInstall> <h2 id="LxRunOfflineInstall">Installing <span class="code">LxRunOffline</span></h2> <p> I downloaded the binaries in <a href='https://github.com/DDoSolitary/LxRunOffline/releases/download/v3.5.0/LxRunOffline-v3.5.0-msvc.zip' target='_blank' rel='nofollow'><code>LxRunOffline-v3.5.0-msvc.zip</code></a> directly from <a href='https://github.com/DDoSolitary/LxRunOffline/releases' target='_blank' rel='nofollow'>GitHub</a> into <code>C:\Program Files\LxRunOffline</code>. </p> <p> Now add <code>C:\Program Files\LxRunOffline</code> to the Windows <code>PATH</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida3b0487bc7d6'><button class='copyBtn' data-clipboard-target='#ida3b0487bc7d6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>setx PATH "%PATH%;C:\Program Files\LxRunOffline" SUCCESS: Specified value was saved. %}</pre> <p> I then ran the following in an administrative shell to register the DLL: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida530df78b25f'><button class='copyBtn' data-clipboard-target='#ida530df78b25f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>regsvr32 "C:\Program Files\LxRunOffline\LxRunOfflineShellExt.dll"</pre> </editor-fold> <editor-fold LxRunOffline> <h2 id="LxRunOffline" class="code">LxRunOffline Help Info</h2> <p> Let's progressively discover how this command-line program can be used. First I'll just type the command name, which causes the program to list its top-level actions. </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id078dc3b826d7'><button class='copyBtn' data-clipboard-target='#id078dc3b826d7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline <span class='unselectable'>[ERROR] No action is specified. Supported actions are: l, list List all installed distributions. gd, get-default Get the default distribution, which is used by bash.exe. sd, set-default Set the default distribution, which is used by bash.exe. i, install Install a new distribution. ui, uninstall Uninstall a distribution. rg, register Register an existing installation directory. ur, unregister Unregister a distribution but not delete the installation directory. m, move Move a distribution to a new directory. d, duplicate Duplicate an existing distribution in a new directory. e, export Export a distribution&quot;s filesystem to a .tar.gz file, which can be imported by the "install" command. r, run Run a command in a distribution. di, get-dir Get the installation directory of a distribution. gv, get-version Get the filesystem version of a distribution. ge, get-env Get the default environment variables of a distribution. se, set-env Set the default environment variables of a distribution. ae, add-env Add to the default environment variables of a distribution. re, remove-env Remove from the default environment variables of a distribution. gu, get-uid Get the UID of the default user of a distribution. su, set-uid Set the UID of the default user of a distribution. gk, get-kernelcmd Get the default kernel command line of a distribution. sk, set-kernelcmd Set the default kernel command line of a distribution. gf, get-flags Get some flags of a distribution. See https://docs.microsoft.com/en-us/previous-versions/windows/desktop/api/wslapi/ne-wslapi-wsl_distribution_flags for details. sf, set-flags Set some flags of a distribution. See https://docs.microsoft.com/en-us/previous-versions/windows/desktop/api/wslapi/ne-wslapi-wsl_distribution_flags for details. s, shortcut Create a shortcut to launch a distribution. ec, export-config Export configuration of a distribution to an XML file. ic, import-config Import configuration of a distribution from an XML file. sm, summary Get general information of a distribution. version Get version information about this LxRunOffline.exe. </span></pre> <p> Alright, let's get information about the <code>i</code> (<code>install</code>) option. </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id723df7673149'><button class='copyBtn' data-clipboard-target='#id723df7673149' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline i <span class='unselectable'>[ERROR] the option '-d' is required but missing Options: -n arg Name of the distribution -d arg The directory to install the distribution into. -f arg The tar file containing the root filesystem of the distribution to be installed. If a file of the same name with a .xml extension exists and "-c" isn&quot;t specified, that file will be imported as a config file. -r arg The directory in the tar file to extract. This argument is optional. -c arg The config file to use. This argument is optional. -v arg (=2) The version of filesystem to use, latest available one if not specified. -s Create a shortcut for this distribution on Desktop. </span></pre> </editor-fold> <p> I'll use the <code>-d</code> option to specify the directory to install into, as well as the <code>-f</code> and <code>-n</code> options. </p> <editor-fold LxRunOfflineImport> <h2 id="LxRunOfflineImport"><span class="code">LxRunOffline</span> Import</h2> <p> I feel brave, let's try importing for real now: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id50d80e8d2b5e'><button class='copyBtn' data-clipboard-target='#id50d80e8d2b5e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline i -d f:/ubuntuBear -n UbuntuBear -f ubuntuBear_2021-01-10.tar <span class='unselectable'>[ERROR] The distro "UbuntuBear" already exists. </span></pre> <p> Some debris remains from the failed import (remember the <code>Unspecified error</code> a moment ago?). I will delete the b0rked <code>UbuntuBear</code> instance in a moment. Let's call this newly cloned linux instance <code>UbuntuBear2</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbde4b612fffe'><button class='copyBtn' data-clipboard-target='#idbde4b612fffe' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline i -d f:/ubuntuBear -n UbuntuBear2 -f ubuntuBear_2021-01-10.tar <span class='unselectable'>[WARNING] Ignoring an unsupported file "var/lib/docker/volumes/backingFsBlockDev" of type 0060000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-17041-8893592-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10716-846285-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1899-75308257-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10151-8749190-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-18789-129803847-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-811-127664322-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12447-115564106-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-24480-65839207-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-23230-107496852-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-23230-107496852-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-24480-65839207-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-14211-5836039-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19778-110293600-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1969-34761241-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-17041-8893592-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1899-75308257-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-7642-17996-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10151-8749190-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19082-8772885-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12857-5829248-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-26692-70009588-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12447-115564106-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-22582-1181861-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-14211-5836039-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1969-34761241-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-7642-17996-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10716-846285-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-11424-57667328-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-28497-49814437-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-18789-129803847-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-2234-80045-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19082-8772885-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12857-5829248-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-22582-1181861-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-26692-70009588-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10990-5814132-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-811-127664322-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10990-5814132-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-11424-57667328-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-2234-80045-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19778-110293600-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-28497-49814437-in" of type 0010000. [WARNING] Ignoring an unsupported file "dev/random" of type 0020000. [WARNING] Ignoring an unsupported file "dev/tty" of type 0020000. [WARNING] Ignoring an unsupported file "dev/full" of type 0020000. [WARNING] Ignoring an unsupported file "dev/urandom" of type 0020000. [WARNING] Ignoring an unsupported file "dev/ptmx" of type 0020000. [WARNING] Ignoring an unsupported file "dev/zero" of type 0020000. [WARNING] Ignoring an unsupported file "dev/console" of type 0020000. [WARNING] Ignoring an unsupported file "dev/null" of type 0020000. [WARNING] Ignoring an unsupported file "dev/mapper/control" of type 0020000. [ERROR] Couldn&quot;t create the file "\\?\f:\ubuntuBear\rootfs\home\mslinn\.atom\packages\markdown-preview-plus\spec\fixtures\subdir\�cc�nt�d.md". Reason: The file exists. </span></pre> <p> Most of the warnings do not seem to be important. I do not care about Docker anyway. Also, now I know that temporary files and <code>dev</code> nodes should all be deleted before running this program. Even better, the program that created the tar should be modified to not attempt to replicate any temporary files. </p> <p> I don't understand the problem with the atom package, but I'll try to delete all of the atom settings from the tar, along with the other problematic directories, and then retry. </p> </editor-fold> <editor-fold cleanup> <h2 id="cleanup">Cleaning Up WSL</h2> <p> First let's see the VMs registered with WSL: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd1f1feb16f7f'><button class='copyBtn' data-clipboard-target='#idd1f1feb16f7f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl -l <span class='unselectable'>Windows Subsystem for Linux Distributions: Ubuntu (Default) UbuntuBear UbuntuBear2 </span></pre> <p> Let's delete the debris remaining from the failed <code>UbuntuBear</code> and <code>UbuntuBear2</code> imports: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida8471a42e546'><button class='copyBtn' data-clipboard-target='#ida8471a42e546' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --unregister UbuntuBear <span class='unselectable'>Unregistering... </span> <span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --unregister UbuntuBear2 <span class='unselectable'>Unregistering... </span></pre> <p> Attempting to delete the directory created by LxRunOffline hit a corrupted directory. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9dda5fb77530'><button class='copyBtn' data-clipboard-target='#id9dda5fb77530' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>del /s /q f:\ubuntuBear <span class='unselectable'>Deleted file - f:\ubuntuBear\rootfs\init Deleted file - f:\ubuntuBear\rootfs\lib Deleted file - f:\ubuntuBear\rootfs\lib32 Deleted file - f:\ubuntuBear\rootfs\lib64 Deleted file - f:\ubuntuBear\rootfs\libx32 Deleted file - f:\ubuntuBear\rootfs\sbin Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.wget-hsts Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.Xauthority Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zcompdump Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zcompdump-Bear-5.8 Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zcompdump-localhost-5.8 Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zshenv Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zshrc Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zshrc.pre-oh-my-zsh Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zsh_history Deleted file - f:\ubuntuBear\rootfs\home\mslinn\ancient_warmth_workspace.code-workspace Deleted file - f:\ubuntuBear\rootfs\home\mslinn\bear2.zip Deleted file - f:\ubuntuBear\rootfs\home\mslinn\bear3.zip Deleted file - f:\ubuntuBear\rootfs\home\mslinn\bearDirs.tar Deleted file - f:\ubuntuBear\rootfs\home\mslinn\dead.letter Deleted file - f:\ubuntuBear\rootfs\home\mslinn\django_bash_completion Deleted file - f:\ubuntuBear\rootfs\home\mslinn\jekyll_workspace.code-workspace Deleted file - f:\ubuntuBear\rootfs\home\mslinn\msp.txt Deleted file - f:\ubuntuBear\rootfs\home\mslinn\nodesource_setup.sh Deleted file - f:\ubuntuBear\rootfs\home\mslinn\package-lock.json Deleted file - f:\ubuntuBear\rootfs\home\mslinn\package.json Deleted file - f:\ubuntuBear\rootfs\home\mslinn\worldPeaceMusicCollective.png Deleted file - f:\ubuntuBear\rootfs\home\mslinn\worldPeaceMusicCollectiveBordered.png Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.vscode-server\extensions\wix.vscode-import-cost-2.15.0\package.json Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.vscode-server\extensions\wix.vscode-import-cost-2.15.0\pom.xml Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.vscode-server\extensions\wix.vscode-import-cost-2.15.0\README.md The file or directory is corrupted and unreadable. </span></pre> <p> &ldquo;The file or directory is corrupted and unreadable.&rdquo; That needs to be dealt with right away! I fixed the directory errors in drive F like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id099a28716311'><button class='copyBtn' data-clipboard-target='#id099a28716311' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Program Files> </span>chkdsk /F F: <span class='unselectable'>The type of the file system is NTFS. Chkdsk cannot run because the volume is in use by another process. Chkdsk may run if this volume is dismounted first. ALL OPENED HANDLES TO THIS VOLUME WOULD THEN BE INVALID. </span>y <span class='unselectable'>Chkdsk cannot dismount the volume because it is a system drive or there is an active paging file on it. Would you like to schedule this volume to be checked the next time the system restarts? (Y/N)</span> y <span class='unselectable'>This volume will be checked the next time the system restarts. </span></pre> <p> I rebooted the system, and it fixed the errors on drive F:. </p> </editor-fold> <editor-fold prune> <h2 id="prune">Pruning the tar</h2> <p> First lets back up the tar that we want to prune. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5060b0c45728'><button class='copyBtn' data-clipboard-target='#id5060b0c45728' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pushd '/mnt/c/Users/Mike Slinn/' <span class='unselectable'>$ </span>ls -alF *.tar <span class='unselectable'>-rwxr--r-- 1 mslinn mslinn 524482560 Jan 10 18:45 'bear_ubuntu_2021-01-10.tar'* -rwxr--r-- 1 mslinn mslinn 51464028160 Jan 10 21:11 'ubuntuBear_2021-01-10.tar'* </span> <span class='unselectable'>$ </span>cp ubuntuBear_2021-01-10.tar ubuntuBearPruned_2021-01-10.tar</pre> <p> Now lets try to prune out all the problematic, and unnecessary, files. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id34ba52a39a88'><button class='copyBtn' data-clipboard-target='#id34ba52a39a88' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>tar -f ubuntuBearPruned_2021-01-10.tar \ --delete dev/* \ --delete tmp/* \ --delete home/mslinn/.atom/* \ --delete var/lib/docker/* \ --delete var/sitesUbuntu/* \ --delete var/work/* \ --delete var/tmp/* <span class='unselectable'>tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.fuseoverlayfs.opaque' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.fuseoverlayfs.opaque' tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: dev/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: tmp/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: home/mslinn/.atom/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: var/lib/docker/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: var/sitesUbuntu/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: var/work/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: var/tmp/*: Not found in archive tar: Exiting with failure status due to previous errors </span></pre> <p> I think the error messages just indicate that the tar was made by <a href='https://www.freebsd.org/cgi/man.cgi?tar(1)' target='_blank' rel='nofollow'>BSD-TAR</a>, while Ubuntu uses <a href='https://www.gnu.org/software/tar/' target='_blank' rel='nofollow'>GNU-TAR</a>. I installed <code>bsdtar</code> like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb26a39f77220'><button class='copyBtn' data-clipboard-target='#idb26a39f77220' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt-get install <a href='https://packages.ubuntu.com/hirsute/libarchive-tools' target='_blank' rel='nofollow'>libarchive-tools</a> <span class='unselectable'>Reading package lists... Done Building dependency tree... Done Reading state information... Done The following NEW packages will be installed: libarchive-tools 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 57.1 kB of archives. After this operation, 207 kB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu hirsute/universe amd64 libarchive-tools amd64 3.4.3-2 [57.1 kB] Fetched 57.1 kB in 0s (167 kB/s) Selecting previously unselected package libarchive-tools. (Reading database ... 244498 files and directories currently installed.) Preparing to unpack .../libarchive-tools_3.4.3-2_amd64.deb ... Unpacking libarchive-tools (3.4.3-2) ... Setting up libarchive-tools (3.4.3-2) ... Processing triggers for man-db (2.9.4-2) ... </span></pre> <p> I <a href='https://stackoverflow.com/a/56031400/553865' target='_blank' rel='nofollow'>tried again</a> using <code>bsdtar</code>, which does not support GNU tar's <code>--delete</code> option. Unfortunately, I only got a 1KB file out, no matter what I did. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf6c5bb135aab'><button class='copyBtn' data-clipboard-target='#idf6c5bb135aab' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>bsdtar -cvf ubuntuBearPruned2_2021-01-10.tar \ --exclude 'dev/*' \ --exclude 'tmp/*' \ --exclude 'home/mslinn/.atom/*' \ --exclude 'var/lib/docker/*' \ --exclude 'var/sitesUbuntu/*' \ --exclude 'var/work/*' \ --exclude 'var/tmp/*' \ @ubuntuBear_2021-01-10.tar</pre> <p> Maybe there is a bug. Maybe there is bad documentation. I do not think I made an error. Life is short. Bugs are rampant. <a href='https://en.wikipedia.org/wiki/Illegitimi_non_carborundum' target='_blank' rel='nofollow'>Illegitimi non carborundum!</a> </p> <p> I give up on this direction. Let's try to win some other way! </p> </editor-fold> <editor-fold duplicate> <h2 id="duplicate">Success Duplicating the Ubuntu Instance</h2> <p> Let&rsquo;s try duplicating the Ubuntu instance with <code>LxRunOffline</code>. First the <code>Ubuntu</code> VM must be terminated. </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddbd78b000923'><button class='copyBtn' data-clipboard-target='#iddbd78b000923' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl -t Ubuntu <span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline duplicate -n Ubuntu -N UbuntuWsl2 -d f:\UbuntuWsl2</pre> <p> The above ran for a couple of hours and concluded without error. <code>LxRunOffline duplicate</code> automatically registers the new instance. </p> <p> The <code>F:\UbuntuWsl2</code>directory was 239 GB. That's a fair-sized VM. The directory only contains one file, called <code>ext4.vhdx</code>. </p> <p> Let's see the Ubuntu instances that are currently set: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0c76cb2e2028'><button class='copyBtn' data-clipboard-target='#id0c76cb2e2028' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --list <span class='unselectable'>Windows Subsystem for Linux Distributions: Ubuntu (Default) UbuntuWsl2 </span></pre> <div class="right" style="font-size: 3em;">&#128513;</div> <p> That is what I wanted to see! I started the new <code>UbuntuWsl2</code> Ubuntu instance like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id77a271ebef95'><button class='copyBtn' data-clipboard-target='#id77a271ebef95' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl -d UbuntuWsl2</pre> <p> To set the default Ubuntu instance I typed: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcdbc0fe6a401'><button class='copyBtn' data-clipboard-target='#idcdbc0fe6a401' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --setdefault UbuntuWsl2</pre> <p> Let's verify that worked: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idccb58dc0c58f'><button class='copyBtn' data-clipboard-target='#idccb58dc0c58f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --list <span class='unselectable'>Windows Subsystem for Linux Distributions: UbuntuWsl2 (Default) Ubuntu </span></pre> <p> I then deleted the huge tar files that I had created in earlier steps. I never needed them because I used <code>LxRunOffline duplicate</code>. </p> </editor-fold> <editor-fold USB3 SSD> <h2 id="usb3">Ubuntu on 500GB USB3 SSD</h2> <div style="text-align: right;"> <a href="https://www.amazon.com/gp/product/B08GTXVG9P" target="_blank" ><picture> <source srcset="/blog/images/wslBackup/sandiskExtreme.webp" type="image/webp"> <source srcset="/blog/images/wslBackup/sandiskExtreme.png" type="image/png"> <img src="/blog/images/wslBackup/sandiskExtreme.png" class="right liImg2 rounded shadow" /> </picture></a> </div> <p> I have a <a href='https://www.amazon.com/gp/product/B08GTXVG9P' target='_blank' rel='nofollow'>Sandisk Extreme 500GB USB3 SSD drive</a>. This tiny, light, and very portable storage device should be perfect for holding my Ubuntu development system. How cool it would be to be able to drop it into a small pocket! </p> <p> Better yet, the <a href='https://www.quora.com/Can-a-USB-3-0-external-SSD-be-faster-than-an-internal-laptop%E2%80%99s-HDD' target='_blank' rel='nofollow'>performance of portable USB3 SSD drives</a> blows away traditional hard drives. </p> <p> For example, assume that your SSD drive appears as drive <code>X</code>. Also assume that <code>LxRunOffline.exe</code> and <code>LxRunOffline.dll</code> have been copied to the root directory of the SSD drive. To register your portable Ubuntu you would simply type: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id967499704b04'><button class='copyBtn' data-clipboard-target='#id967499704b04' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\> </span>X:LxRunOffline register X:\MyUbuntu -n UbuntuMSlinn</pre> <div class="pullQuote"> You can walk up to any recently updated Windows 10 machine, plug your tiny USB3 SSD drive into a USB3 or USB3.1 port, run the registration command, and boom! you are productive with great storage performance. </div> <div class="right" style="font-size: 3em;">&#128513;</div> <h2 id="duplicate">Duplicating to SSD</h2> <p> Again, the <code>Ubuntu</code> VM must be terminated. </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7986dfb292e5'><button class='copyBtn' data-clipboard-target='#id7986dfb292e5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl -t Ubuntu <span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline duplicate -n Ubuntu -N UbuntuWsl2Extreme -d I:\UbuntuWsl2Extreme</pre> <p> The above ran for a couple of hours and concluded without error. <code>LxRunOffline duplicate</code> automatically registers the new instance. </p> <p> Let's see the Ubuntu instances that are currently set: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3572ee5a4ff1'><button class='copyBtn' data-clipboard-target='#id3572ee5a4ff1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --list <span class='unselectable'>Windows Subsystem for Linux Distributions: Ubuntu (Default) UbuntuWsl2 UbuntuWsl2Extreme </span></pre> <div class="right" style="font-size: 3em;">&#128513;</div> <h2 id="startStop">Starting and Stopping Portable VMs</h2> <p> You can start the WSL VM called <code>UbuntuMSlinn</code> by using <code>wsl -d</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7a735f6eee62'><button class='copyBtn' data-clipboard-target='#id7a735f6eee62' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\> </span>wsl -d UbuntuMSlinn</pre> <p> You can stop it by using <code>wsl -t</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2e5f33491d23'><button class='copyBtn' data-clipboard-target='#id2e5f33491d23' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\> </span>wsl -t UbuntuMSlinn</pre> </editor-fold> <editor-fold issues> <h2 id="issues">Microsoft WSL Issue Reporting</h2> <p> In the course of writing this blog post I found the <a href='https://github.com/microsoft/WSL/blob/master/CONTRIBUTING.md#8-collect-wsl-logs' target='_blank' rel='nofollow'>web page</a> for reporting a WSL issue, so that Microsoft can look at it. </editor-fold> Streaming Solo to Facebook From OBS Studio 2022-01-07T00:00:00-05:00 https://mslinn.github.io/blog/2022/01/07/streaming-facebook <p> Briefly: </p> <ol> <li>Set up Facebook for streaming</li> <li>OBS Studio initiates streaming</li> <li>Musician performs</li> <li>Helper monitors stream and moderates comments on Facebook Messenger</li> </ol> <h2 id="pageStream">Streaming From a Facebook Page</h2> <p> I prefer to stream from a specific page on Facebook, <a href='https://www.facebook.com/mslinnmusic' target='_blank' rel='nofollow'><code>facebook.com/mslinnmusic</code></a>, so my live streams can be found at <a href='https://www.facebook.com/mslinnmusic/live_videos/' target='_blank' rel='nofollow'><code>facebook.com/mslinnmusic/live_videos/</code></a>. </p> <h2 id="fbSetup">Setting Up Facebook For Streaming</h2> <p> Go to <a href='https://www.facebook.com/live/producer/' target='_blank' rel='nofollow'>Facebook Live Producer</a> and check out the various areas on screen. We'll change a few settings. </p> <div style=""> <picture> <source srcset="/blog/images/facebookStream/settingsStream.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/settingsStream.png" type="image/png"> <img src="/blog/images/facebookStream/settingsStream.png" class=" liImg2 rounded shadow" style="float: left; margin-right: 1em; width: 300px; height: auto;" /> </picture> </div> <div style=""> <picture> <source srcset="/blog/images/facebookStream/settingsViewing.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/settingsViewing.png" type="image/png"> <img src="/blog/images/facebookStream/settingsViewing.png" class=" liImg2 rounded shadow" style="float: left; margin-right: 1em; width: 300px; height: auto;" /> </picture> </div> <div style=""> <picture> <source srcset="/blog/images/facebookStream/settingsComments.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/settingsComments.png" type="image/png"> <img src="/blog/images/facebookStream/settingsComments.png" class=" liImg2 rounded shadow" style="width: 300px; height: auto;" /> </picture> </div> <div style=""> <picture> <source srcset="/blog/images/facebookStream/keyUrl.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/keyUrl.png" type="image/png"> <img src="/blog/images/facebookStream/keyUrl.png" class=" liImg2 rounded shadow" style="height: auto; width: 615px;" /> </picture> </div> <div style="float: left; margin-right: 1em; max-width: 300px"> <div style=""> <picture> <source srcset="/blog/images/facebookStream/ticketsGoLive.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/ticketsGoLive.png" type="image/png"> <img src="/blog/images/facebookStream/ticketsGoLive.png" class=" liImg2 rounded shadow" style="height: auto; width: 300px;" /> </picture> </div> I pixelated the stream key in the image to the right for security. </div> <div style=""> <picture> <source srcset="/blog/images/facebookStream/keyUrl2.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/keyUrl2.png" type="image/png"> <img src="/blog/images/facebookStream/keyUrl2.png" class=" liImg2 rounded shadow" style="width: 300px; height: auto;" /> </picture> </div> <ol> <li>Enable <b>Use a persistent stream key</b></li> <li>Copy the Facebook stream key to the clipboard.</li> <li>Enable <b>Publish as a test broadcast</b> if desired.</li> </ol> <h2 id="obsSetup">Setting Up OBS Studio For Streaming</h2> <ol> <li> Paste the Facebook persistent stream key into OBS Studio by going to <b>Settings</b> / <b>Stream</b> / <b>Facebook Live</b> / <b>Stream Key</b>.. </li> <li>Press <kbd>OK</kbd>.</li> </ol> <p> Chats from the audience will appear in Facebook Messenger. </p> <h2 id="obsSetup">Streaming</h2> <p> The Facebook streaming key is specific to the computer that is streaming. If you use more than one computer to stream from a given Facebook account, you will need to get a fresh stream key each time you stream from a different computer. </p> <p> To start streaming: </p> <ol> <li>Click on the Facebook Live Video button <div style=""> <picture> <source srcset="/blog/images/facebookStream/fbLiveVideoButton.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/fbLiveVideoButton.png" type="image/png"> <img src="/blog/images/facebookStream/fbLiveVideoButton.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> </li> <li> Wait for a couple of seconds while the Facebook Live Producer page loads any settings that you may have previously set. </li> <li> <div style=""> <picture> <source srcset="/blog/images/facebookStream/topLeft.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/topLeft.png" type="image/png"> <img src="/blog/images/facebookStream/topLeft.png" class=" liImg2 rounded shadow" style="height: auto; width: 300px;" /> </picture> </div> Fill in the title of the stream, and write a short description. </li> <li>Click on the <b>Start Streaming</b> button in OBS Studio.</li> <li> Facebook will show the stream image at the bottom right of the Facebook Live Producer web page. <div style=""> <picture> <source srcset="/blog/images/facebookStream/fbLivePreview.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/fbLivePreview.png" type="image/png"> <img src="/blog/images/facebookStream/fbLivePreview.png" class=" liImg2 rounded shadow" style="height: auto; width: 300px;" /> </picture> </div> Attention: you are not yet streaming! </li> <li> Click on the blue <b>Go live</b> button at the bottom left of the Facebook Live Producer web page. <div style=""> <picture> <source srcset="/blog/images/facebookStream/fbGoLive.webp" type="image/webp"> <source srcset="/blog/images/facebookStream/fbGoLive.png" type="image/png"> <img src="/blog/images/facebookStream/fbGoLive.png" class=" liImg2 rounded shadow" style="height: auto; width: 300px;" /> </picture> </div> You are now live.</li> <li>Check Facebook Messenger for audience feedback.</li> <li>View the stream, again, for me, this is at <a href='https://www.facebook.com/mslinnmusic/live_videos/' target='_blank' rel='nofollow'><code>https://www.facebook.com/mslinnmusic/live_videos/</code></a></li> </ol> OBS Studio Streaming Using Nvidia GTX 1660/1650 GPUs 2021-12-29T00:00:00-05:00 https://mslinn.github.io/blog/2021/12/29/obs-s1650 <p> OBS Studio running on my desktop PC was able to record audio and video at any resolution and frame rate that I desired, but whenever I tried to stream to YouTube, the video did not work. Instead of live video, just the first frame was sent; a still image. </p> <p> To help me diagnose the problem, I ran <code>UserBenchMark</code>, available for free from <a href='https://www.userbenchmark.com' target='_blank' rel='nofollow'><code>userbenchmark.com</code></a>. The benchmark showed that my PC's video card, an old EVGA GeForce GTX 760, was mismatched to the rest of the computer; it was by far the weakest link. The video card was able to drive a 4K monitor and a 1080p monitor simultaneously, for displaying static windows, but it was unsuitable for gaming and streaming live video. </p> <h2 id="recommend">OBS Studio Recommendations</h2> <p> Nvidia video cards are <a href='https://obsproject.com/forum/threads/best-gpu-for-rendering-previews-advice-needed.118869/' target='_blank' rel='nofollow'>strongly recommended by the OBS Studio developers</a>, &ldquo;preferably a Turing-based unit&rdquo;, and that the best value cards were <a href='https://gpu.userbenchmark.com/Nvidia-GTX-1660/Rating/4038' target='_blank' rel='nofollow'>GTX 1660</a> (all flavors) and GTX 1650 Super GPUs. Currently, <a href='https://www.digitaltrends.com/computing/what-to-expect-from-gpus-2022/' target='_blank' rel='nofollow'>video cards are in extremely short supply</a>, so prices for used units are higher than those for new units because new units are often not available. </p> <p> I bought a used EVGA GeForce GTX 1660 video card (model <a href='https://www.evga.com/products/Specs/GPU.aspx?pn=338d9f6e-477c-4d81-9432-64bffa3f9513' target='_blank' rel='nofollow'><code>06G-P4-1067-RX</code></a>) here in Montreal for $550 (CAD) and popped it into my computer. After installing drivers from <a href='https://https://www.nvidia.com/Download/index.aspx' target='_blank' rel='nofollow'><code>https://www.nvidia.com/Download/index.aspx</code></a>, and rebooting, OBS Studio was able to stream live without straining the PC. I also noticed that the Windows system performance was smoother, and more stable than it had been with the older video card. </p> <h2 id="fbStream">Facebook Live Stream</h2> <p> This is the <a href='https://www.facebook.com/100070935496310/videos/1163736614368165' target='_blank' rel='nofollow'>Facebook Live Stream</a> I tested with: </p> <iframe src="https://www.facebook.com/plugins/video.php?height=314&href=https%3A%2F%2Fwww.facebook.com%2Fmslinnmusic%2Fvideos%2F1163736614368165%2F&show_text=true&width=560&t=0" class="shadow rounded" width="560" height="429" style="border:none;overflow:hidden" scrolling="no" frameborder="0" allowfullscreen="true" allow="autoplay; clipboard-write; encrypted-media; picture-in-picture; web-share" allowFullScreen="true"></iframe> <h2 id="oclock">Over/Under- Clocking</h2> <p> There is no need to overclock this video card. </p> <p> Overclocking causes more power to be consumed, which means more heat must be dissipated, which means fans must work harder, which means the computer is louder. </p> <p> The fans on this video card only spin when needed, and they only spin as fast as required when needed. Fans are noisy, so the recording room is quieter when the video card is not working very hard. </p> <p> I want the computer I use for streaming to be as quiet as possible. This video card barely breathes hard as it streams live, so underclocking might be a way to further reduce power usage, and hence reduce the need for fans to spin. </p> Spring-Breezifier: Solving COVID-19 With HVAC 2021-12-22T00:00:00-05:00 https://mslinn.github.io/blog/2021/12/22/covid-hvac <p> When the COVID-19 pandemic began, the world was unprepared and no one knew how the virus was transmitted. As the medical profession lurched awkwardly into action, it gave advice that later turned out to be incorrect. As time went by, important details about how the virus is transmitted were revealed, and the previous medical advice had become politicized. The new advice has not yet been generally adopted as policy and is currently unknown to most people. </p> <div class="quoteCite shadow rounded" style="height: 230px;"> <div style=""> <picture> <source srcset="/blog/images/covidHvac/mencken.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/mencken.png" type="image/png"> <img src="/blog/images/covidHvac/mencken.png" class=" rounded" style="float:right; maxwidth; 25%; width: 130px; height: 180px; margin-top: 0;" /> </picture> </div> For every complex problem there is an answer that is clear, simple, and wrong. <br><br> &nbsp;&ndash; <a href='https://www.britannica.com/biography/H-L-Mencken' target='_blank' rel='nofollow'>H. L. Mencken</a> 1880-1956. </div> <h2 id="about">About This Article</h2> <p> This article's intent is to show how the COVID-19 pandemic could be dealt with simply, cheaply, and with a minimum of aggravation by raising awareness of the potential role of <a href='https://www.cdc.gov/infectioncontrol/guidelines/environmental/background/air.html#c3' target='_blank' rel='nofollow'>HVAC</a> (heating, ventilation and air conditioning) equipment. </p> <p> The author is an electrical engineer with no health-related or HVAC-related qualifications. However, he can read and write, ideas often come to mind when presented with new information. </p> <div class="formalNotice rounded shadow"> After initially publishing this blog post, an old friend showed me a YouTube video made by a consulting civil engineer who specializes in this topic. <a href="#youtube2">You can skip to it if you are impatient</a> and want to know in-depth technical specifics. </div> <p> All the new information referenced in this article is generally available, from qualified and reputable sources, and this article just disseminates it. </p> <p> This article concludes with actionable suggestions. </p> <h2 id="trans">COVID-19 Is Primarily Transmitted Via Aerosols</h2> <p> The most significant bit of information about COVID-19, which was not generally accepted at the beginning of the pandemic, is that it is <a href='https://www.nature.com/articles/d41586-021-00251-4' target='_blank' rel='nofollow'>mostly transmitted via aerosols</a>; that is, via small airborne droplets only a few microns in diameter. (One micron is a millionth of a meter, or one twenty-five thousandth of an inch). Aerosol particles can hang in the air for hours and travel hundreds of feet. </p> <p> The Centers for Disease Control and Prevention (CDC) officially recently officially recognized that SARS-CoV-2 (the virus that causes COVID-19) is airborne, meaning it is highly transmissible through the air. </p> <div class="quoteCite shadow rounded"> <div style="text-align: right;"> <picture> <source srcset="/blog/images/covidHvac/cdc.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/cdc.png" type="image/png"> <img src="/blog/images/covidHvac/cdc.png" class="right " style="maxwidth; 25%; width: 130px; height: auto;" /> </picture> </div> Exposure occurs in three principal ways: <ol> <li>inhalation of very fine respiratory droplets and aerosol particles</li> <li> deposition of respiratory droplets and particles on exposed mucous membranes in the mouth, nose, or eye by direct splashes and sprays </li> <li> touching mucous membranes with hands that have been soiled either directly by virus-containing respiratory fluids or indirectly by touching surfaces with virus on them. </li> </ol> <p> People release respiratory fluids during exhalation (e.g., quiet breathing, speaking, singing, exercise, coughing, sneezing) in the form of droplets across a spectrum of sizes. These droplets carry virus and transmit infection. To stay healthy, avoid these droplets. </p> <p> The largest droplets settle out of the air rapidly, within seconds to minutes. The smallest very fine droplets, and aerosol particles formed when these fine droplets rapidly dry, are small enough that they can remain suspended in the air for minutes to hours. </p> &nbsp;&ndash;From the CDC <a href='https://www.cdc.gov/coronavirus/2019-ncov/science/science-briefs/sars-cov-2-transmission.html' target='_blank' rel='nofollow'>Scientific Brief: SARS-CoV-2 Transmission</a>, Updated May 7, 2021. </div> <p> The idea that physical distancing of 6 feet (2 meters) might help keep people healthy is an example of bad science from a study <a href='https://www.businessinsider.com/6-foot-distancing-rule-is-outdated-oxford-mit-new-system-2020-8' target='_blank' rel='nofollow'>80 years ago</a> that was not debunked until recently. </p> <div class="quoteCite shadow rounded"> <div style="text-align: right;"> <picture> <source srcset="/blog/images/covidHvac/epaLogo.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/epaLogo.png" type="image/png"> <img src="/blog/images/covidHvac/epaLogo.png" class="right quartersize " /> </picture> </div> Transmission of COVID-19 from inhalation of virus in the air can occur at distances greater than six feet. Particles from an infected person can move throughout an entire room or indoor space. The particles can also linger in the air after a person has left the room – they can remain airborne for hours in some cases. <br><br> &nbsp;&ndash;From the US Environmental Protection Agency (EPA): <a href='https://www.epa.gov/coronavirus/indoor-air-and-coronavirus-covid-19' target='_blank' rel='nofollow'>Indoor Air and Coronavirus (COVID-19)</a>, web page was updated December 15, 2021. </div> <p> Distance between people is not a significant transmissibility factor. Imagine two people, back to back, one (downwind) very sick with COVID-19, and the other (healthy) upwind, and the wind is blowing at 20 mph (32 km/h). The sick person will not infect the healthy person, unless they exchange positions. Just control the air that is breathed, and all airborne viruses such as COVID-19 will be controled. </p> <div class="quoteCite shadow rounded"> <div style="text-align: right;"> <picture> <source srcset="/blog/images/covidHvac/ucsdMedicine.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/ucsdMedicine.png" type="image/png"> <img src="/blog/images/covidHvac/ucsdMedicine.png" class="right " style="maxwidth; 25%; width: 130px; height: auto;" /> </picture> </div> What we learned during the pandemic is that aerosols were one of the main drivers in spreading the COVID-19 virus and that their importance in the transmission of many other respiratory pathogens has been systematically underappreciated. <br><br> &nbsp;&ndash; Dr. Robert Schooley, Professor in the Department of Medicine at the University of California San Diego (November 22, 2021), quoted from <a href='https://ucsdnews.ucsd.edu/pressrelease/covid-gets-airborne' target='_blank' rel='nofollow'>COVID Gets Airborne &ndash; UC San Diego develops computer model to aid understanding of how viruses travel through the air</a> </div> <h2 id="virus">Controlling Virus Infiltration</h2> <p> We would do well to remember that the COVID-19 virus causes sickness. Without exposure to the virus people would not get sick. The risk that a person might catch the disease is directly related to the number of viruses inhaled per unit of time. For example, the more viruses someone inhales in an hour, the more likely they will get sick. </p> <p> <a href='https://www.statnews.com/2020/04/14/how-much-of-the-coronavirus-does-it-take-to-make-you-sick/' target='_blank' rel='nofollow'>The amount of virus necessary to make a person sick is called the infectious dose.</a> All the countermeasures that have been used (hand washing, physical distancing, masks, curfews, etc.) are intended to reduce the number of viruses people are exposed to per unit time. Pandemic policy treats countermeasures as proxies for virus transmission vectors. By not updating pandemic policy as new information becomes available, we loose the ability to control the spread of the virus. </p> <h2 id="hepa">HEPA Filters</h2> <p> HEPA filters on HVAC equipment are designed to remove these aerosols. They come in a huge variety of sizes, shapes and air flow capacities. If you could just breathe the pure air emitted from a fan that had a HEPA filter, you would not get sick from any airborne virus. Any number of people could gather safely, if each of them received an adequate supply of pure air in their face at all times. </p> <div class="quoteCite shadow rounded"> <div style="text-align: right;"> <picture> <source srcset="/blog/images/covidHvac/epaLogo.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/epaLogo.png" type="image/png"> <img src="/blog/images/covidHvac/epaLogo.png" class="right quartersize " /> </picture> </div> HEPA is a type of pleated mechanical air filter. It is an acronym for “high efficiency particulate air filter” (as officially defined by the U.S. Dept. of Energy). This type of air filter can theoretically remove at least 99.97% of dust, pollen, mold, bacteria, and any airborne particles with a size of 0.3 microns (µm). <br><br> &nbsp;&ndash;<a href='https://www.epa.gov/indoor-air-quality-iaq/what-hepa-filter-1' target='_blank' rel='nofollow'>US Environment Protection Agency</a> </div> <p> HEPA filters are inexpensive and are readily available. They are not new. In fact, <a href='https://www.apcfilters.com/the-history-of-hepa-filters/' target='_blank' rel='nofollow'>HEPA filter technology was created in the 1940s</a> by the US Army Chemical Corps and National Defense Research Committee as part of the Manhattan Project. HEPA technology was declassified after World War II and became available for commercial and personal use. HEPA filters play a key role in the research and development of modern pharmaceuticals, aerospace engineering, and computer chip manufacturing. </p> <div class="quoteCite shadow rounded"> HEPA air cleaners, which remain little-used in Canadian hospitals, are a cheap and easy way to reduce risk from airborne pathogens. <br><br> &nbsp;&ndash;From Nature Magazine, October 6, 2021: <a href='https://www.nature.com/articles/d41586-021-02669-2' target='_blank' rel='nofollow'>Real-world data show that filters clean COVID-causing virus from air &ndash; An inexpensive type of portable filter efficiently screened SARS-CoV-2 and other disease-causing organisms from hospital air.</a> </div> <p> It is easy to attach a HEPA filter to a fan, or to replace an existing HVAC filter with a HEPA filter. </p> <div style=""> <a href="https://youtu.be/kH5APw_SLUU?t=106" target="_blank" ><picture> <source srcset="/blog/images/covidHvac/hepaFan.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/hepaFan.png" type="image/png"> <img src="/blog/images/covidHvac/hepaFan.png" title="Dr. Jeffrey E. Terrell, director of the Michigan Sinus Center, demonstrates how to build an air purifier with a HEPA filter for about $25 with parts from your local hardware store." class=" fullsize liImg2 rounded shadow" alt="Dr. Jeffrey E. Terrell, director of the Michigan Sinus Center, demonstrates how to build an air purifier with a HEPA filter for about $25 with parts from your local hardware store." /> </picture></a> <figcaption class="fullsize" style="width: 100%; text-align: center;"> <a href="https://youtu.be/kH5APw_SLUU?t=106" target="_blank" > Dr. Jeffrey E. Terrell, director of the Michigan Sinus Center, demonstrates how to build an air purifier with a HEPA filter for about $25 with parts from your local hardware store. </a> </figcaption> </figure> </div> <p> One of the comments on the above video, from Rick Rude in 2014, was: </p> <div class="quoteCite shadow rounded"> Better to pull air thru the filter, if you push air through the filter dust that builds up will get blown off and go back into the room. Also, if the fan is pulling air through, the filter will stick to the fan and won't need as much tape to hold it on. Always handle used filters with care because handling them rough will release concentrated dust back into your air. If possible, always take air cleaners or vacuum cleaners outside to change the filters or bags. </div> <h2 id="youtube2">How to Identify and Rectify Poorly Ventilated Indoor Spaces Using Engineering Controls</h2> <p> <a href='https://www.linkedin.com/in/elfstrom/' target='_blank' rel='nofollow'>David Elfstrom, P. Eng.</a>, is a independent civil engineer based in Simcoe, Ontario who consults on the efficient use of energy in buildings. During the pandemic he has been drawing attention to the importance of ventilation, filtration, and overall indoor environmental quality. Earlier this year Mr. Elfstrom completed a ventilation and air pathways assessment of an apartment building in outbreak for a public health unit. This presentation was part of Passive Buildings Canada's 2021 Annual General Meeting. </p> <style>.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }</style><div class='embed-container'> <iframe title="YouTube video player" width="640" height="390" src="//www.youtube.com/embed/9J96F32tv1s" frameborder="0" allowfullscreen></iframe></div> <p style="margin-top: 1em;"> Mr. Elfstrom also co-authored the <a href='https://docs.google.com/document/d/17tKk8Da8tnchtnp9ZRe7fPazGAmXtvoA-n4GZcY0_fQ/edit' target='_blank' rel='nofollow'>Masks4Canada Room Ventilation/Filtration Guide and Tip Sheet</a>. </p> <h2 id="hvac">N95, KF94, FFP2, and 9152 Masks and Their Cousins</h2> <p> If you need to move about in a crowd, or enter a space that contained people a few hours ago, you need to wear a properly fitting face mask that filters out the tiny aerosol particles that contain the COVID-19 virus. <a href='https://www.travelawaits.com/2559161/n95-vs-kn95-vs-kf94-masks-for-travel/' target='_blank' rel='nofollow'>N95 masks and their cousins</a> (KN95, KF94, etc.) do the job because they filter particles as small as 0.3 microns. This is similar to the particle size filtered by HEPA filters. Caution: KN95 is a self-reported test standard, and lacks strict government regulation by China, resulting in many underperforming and often flat-out fake masks. </p> <p> The US National Personal Protective Technology Laboratory (NPPTL), which is part of the US National Institute for Occupational Safety and Health (NIOSH), which itself is part of the CDC, evaluated various masks and published the results as <a href='https://www.cdc.gov/niosh/npptl/respirators/testing/NonNIOSHresults.html' target='_blank' rel='nofollow'>NPPTL Respirator Assessments to Support the COVID-19 Response</a>. </p> <p> The following types of masks do <b>not</b> reliably filter tiny aerosols, so they do not adequately protect you from COVID-19 variants such as Omicron: </p> <ol> <li> <a href='https://www.theguardian.com/commentisfree/2021/dec/27/best-masks-covid-tests-cloth-surgical-respirators' target='_blank' rel='nofollow'>Surgical masks</a> (what most people wear these days) </li> <li>Masks with activated charcoal</li> <li>Vented masks</li> <li>Cloth masks</li> <li>Gaiters</li> <li>Ill-fitting masks that do not cover and seal the mouth and nose</li> </ol> <iframe width="690" height="388" src="https://www.youtube.com/embed/WE5Uo3F2TdU" frameborder="0" allowfullscreen class="rounded shadow liImg"></iframe> <h2 id="other">Inoculations and Pills</h2> <div style="text-align: right;"> <a href="https://www.reuters.com/business/healthcare-pharmaceuticals/us-fda-set-authorize-pfizer-merck-covid-19-pills-this-week-bloomberg-news-2021-12-21/" target="_blank" ><picture> <source srcset="/blog/images/covidHvac/covidPills.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/covidPills.png" type="image/png"> <img src="/blog/images/covidHvac/covidPills.png" title="COVID-19 pills" class="right quartersize liImg2 rounded shadow" alt="COVID-19 pills" /> </picture></a> </div> <p> Inoculations, including boosters, have proven to be very helpful. Pills from Pfizer Inc. and Merck & Co. for people sick with COVID-19 will also be very helpful once they become available. However, pills will only help sick people get better, they will not prevent getting sick. </p> <p> Until everyone in the entire world has somehow acquired an effective level of antibodies, COVID-19 will continue to mutate. </p> <h2 id="go">The Only Solution Available Today</h2> <p> Inoculations and masks are good, but they are not perfect preventative measures against COVID-19. Protection against infection is not 100%. At present, the only preventative measure against COVID-19 that would allow people to safely mingle in person would be to guarantee continuous streams of pure air directed individually at each person's face. </p> <p> This measure would also prevent the spread of <a href='https://bmcinfectdis.biomedcentral.com/articles/10.1186/s12879-019-3707-y' target='_blank' rel='nofollow'>all other diseases and their variants that are primarily spread via aerosols</a>, including coronoviruses, colds, flus, tuberculosis, MERS-CoV, measles, ebola and chickenpox. Pollen, dust and other particulates would also be removed, so athsma sufffers would feel relief from extended periods breathing pure air. </p> <p> The solution to living with COVID-19 is simple, safe, inexpensive and is not disruptive: </p> <div class="formalNotice rounded shadow"> <h2 class="centered" style="margin: 0; padding: 0">Live your social life with a pure breeze in your face.</h2> <div style=""> <picture> <source srcset="/blog/images/covidHvac/breeze-dandelion.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/breeze-dandelion.png" type="image/png"> <img src="/blog/images/covidHvac/breeze-dandelion.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <div style="clear:both; font-size: 0"></div> </div> <p> Life as we knew it before the pandemic started 2 years ago could resume, mostly. </p> <h2 id="hvac">HVAC Upgrades Are Key</h2> <p> The engineering group that encompasses HVAC is ASHRAE. ASHRAE was formed as the American Society of Heating, Refrigerating and Air-Conditioning Engineers by the merger in 1959 of American Society of Heating and Air-Conditioning Engineers (ASHAE), founded in 1894 and The American Society of Refrigerating Engineers (ASRE), founded in 1904. </p> <p> ASHRAE offers <a href='https://www.ashrae.org/file%20library/technical%20resources/covid-19/core-recommendations-for-reducing-airborne-infectious-aerosol-exposure.pdf' target='_blank' rel='nofollow'>Core Recommendations for Reducing Airborne Infectious Aerosol Exposure</a>. <a href='https://www.iso-aire.com/blog/what-are-the-differences-between-a-merv-13-and-a-hepa-filter' target='_blank' rel='nofollow'>MERV 13 or better</a> levels of performance are recommended, and HEPA filters surpass that specifation. In fact, all HEPA filters have a rating of a MERV 17 or higher. </p> <h2 id="entre">Entrepreneurs</h2> <p> This represents an opportunity for entrepreneurs. </p> <h3 id="restos">Restaurants</h2> <p> Imagine a restaurant that stays open with near-normal seating capacity because each table is equipped with a fan and ducting that blows a gentle breeze of pure air directly into the face of every patron. The air would recirculate within the restaurant, so there would be no need to upgrade the existing HVAC system because the HEPA filters located at each table would continuously clean the air within the restaurant. Those filters would need to be cleaned and disinfected daily. The wait staff would all wear N95 or similar masks, however each cashier could instead enjoy their own gentle breeeze of pure air. </p> <p> Cost to equip each seat in the restaurant could be less than $50. For do-it-yourself owners, the cost could approach $10 per seat. For example, a 100-seat restaurant might be able to retrofit 75 of those seats at a cost of less than $3750. They would never have to close again due to any pandemic caused by airborne aerosols... provided, of course, that local regulations recognized this approach. </p> <h3 id="stores">Retail Stores and Office Buildings</h2> <p> Enclosing sales clerks in stores behind clear plastic walls is wrong because it decreases air circulation. Instead, each those staff members should have a dedicated ducted fan that blows a gentle breeze of pure air into their face at all times. </p> <p> Similarly, queues of people awaiting their turn at checking out should have fans blowing pure air at their heads. The air would recirculate within the store, there would be no need to upgrade the existing HVAC system because the HEPA filters would continuously clean the air within the building. </p> <h2 id="domain">Spring-Breezifier Is Offered Into the Public Domain</h2> <p> I offer this idea to the world; I am an idea machine so I cannot act on most of them. Just for fun, let's call this idea the <i>Spring-Breezifier</i>. </p> <p> Designing and building Spring-Breezifiers seems like it might be a good high school or youth group project. HVAC installers in particular should be able to make short work of this idea; go ahead and make lots of money providing pure airflows by building custom Spring-Breezifiers for your customers, we all thank you! </p> <p> If anyone would like to talk to me about their Spring-Breezifier project, I would be happy to speak with them. </p> <h2 id="next">Next Steps</h2> <ol> <li> Medical HVAC specialists could suggest the necessary cubic feet per minute of air required per person sufficient to guarantee a continuous stream of pure air, and provide guidelines for designing ductwork to direct that air effectively. </li> <li> Build and test prototypes. Most of the effort will be in designing and building the ducting and/or the scaffolding. Adding inner baffles would raise the cost and weight, lower the noise and make the airflow less turbulent. Perhaps Spring-Breezifiers could be built entirely from off-the-shelf parts for some or even most installations. Large-format 3D printers for special circumstances might be helpful. </li> <li>Propose the inclusion of pure air streams into public spaces as a matter of public health policy.</li> </ol> Recording Solo With OBS Studio 2021-11-15T00:00:00-05:00 https://mslinn.github.io/blog/2021/11/15/obs-web <p> Making video recordings of yourself solo is a challenge in many ways. Not only do you need to start and stop recordings, but you also need to hear and see yourself as you appear in the video, and either adjust settings or adjust yourself. </p> <p> For me, one of the biggest issues is starting and stopping the recording without moving out of position. The space where I sit in order to play has a camera, lights and microphones, and all of that cannot be right in front of the computer that records the audio and video. I need another viewing surface, and a means of controlling the recording software. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/recordingSpace.webp" type="image/webp"> <source srcset="/blog/images/obs/recordingSpace.png" type="image/png"> <img src="/blog/images/obs/recordingSpace.png" class="center liImg2 rounded shadow" /> </picture> </div> <p> The software I usually need to run in order to make music videos includes RME TotalMix, Pro Tools, and OBS Studio. Another problem that solo recording artists face is that these programs must all run at the same time, and all of them require lots of real estate on your computer screen. </p> <h2 id="remote">RME TotalMix Remote and Avid EUCON</h2> <p> I have experimented with various ways of controlling the software, including controlling the recording computer from a touchscreen laptop, a first-generation iPad mini, an Android phone, and MIDI devices. <a href='https://www.rme-usa.com/totalmix-fx-remote.html' target='_blank' rel='nofollow'>RME TotalMix Remote</a>, <a href='https://cdn-www.avid.com/-/media/avid/files/products-pdf/artist-control/eucon_application_setup_v2_6.pdf' target='_blank' rel='nofollow'>Pro Tools EUCON</a> and <a href='https://avid.secure.force.com/pkb/articles/en_US/User_Guide/EuControl-Product-Guides' target='_blank' rel='nofollow'>Pro Tools EUControl</a> all work well on every device I have tried, for example: </p> <ul> <li>RME TotalMix Remote on my first-generation iPad mini</li> <li>RME TotalMix Remote on a touchscreen laptop</li> <li>Pro Tools EUControl and EUCON Client on touchscreen laptop</li> <li>Avid Control on the iPad mini</li> </ul> <p> It is also possible to use MIDI controllers that have knobs, slider and buttons to control arbitrary software using <a href='https://www.bome.com/products/miditranslator' target='_blank' rel='nofollow'>Bome MIDI Translator Pro</a>. That software is complex, but very capable. MIDI devices generally require cables, and setting them up can be time-consuming, and that setup is specific for each MIDI controller you might want to try. <a href='https://www.google.com/search?q=midi+over+wifi' target='_blank' rel='nofollow'>MIDI over WiFi</a> is interesting, but requires extra setup. Maybe I will dig into this topic one day. </p> <h2 id="ipad_mini">iPad Mini Is Too Small</h2> <p> The iPad mini works well, but it is awkward to run TotalMix Remote and Avid Control together on such a tiny device. I can run TotalMix remote on the iPad mini, and Avid Control on my Android phone, although the phone's screen is too small to be useful. </p> <p> Using a second laptop is better because multiple programs can be visible at the same time, and I don't have to think about which device to pick up in order to make an adjustment or start/stop recording. </p> <p> Perhaps using the <a href='https://support.apple.com/en-ca/HT207582' target='_blank' rel='nofollow'>iPadOS multitasking feature</a> on an iPad Pro might work as well as running several programs on a laptop, but an iPad Pro is as expensive as a laptop, and laptops are more versatile. </p> <h2 id="obs-different">OBS Studio Control Options</h2> <p> Two simple ways that I experimented with in order to control OBS Studio from the recording space were abandoned: </p> <ol> <li> Plugging in a second keyboard on the recording computer and using OBS Studio hotkeys to start and stop recording (difficult to see the computer screen, and the hot keys only work if OBS Studio is in the foreground.)</li> <li> Using a <a href='https://www.dell.com/en-us/shop/dell-24-touch-monitor-p2418ht/apd/210-alcs/monitors-monitor-accessories' target='_blank' rel='nofollow'>touch-sensitive computer monitor</a> on the recording computer, and mounting the monitor on a swivel arm (OBS Studio's buttons are too small to work reliably, half the time the wrong button is pushed). <div class="videoWrapper2 shadow" style="margin-top: 1em"> <iframe width="1267" height="722" src="https://www.youtube.com/embed/I346wSqhobw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> </div> </li> </ol> <p> <code>Obs-web</code> is a better way, because it provides a live web page that can be accessed from anywhere in your local area network. However, you need some technical ability because there is no user-friendly way to set up <code>obs-web</code>. Once setup, operation is very simple, however. </p> <p> <code>Obs-web</code> requires an OBS Studio plugin called <code>obs-websocket</code>. </p> <div class="formalNotice rounded shadow"> <div style="text-align: right;"> <picture> <source srcset="/blog/images/obs/obsws_logo_new.webp" type="image/webp"> <source srcset="/blog/images/obs/obsws_logo_new.png" type="image/png"> <img src="/blog/images/obs/obsws_logo_new.png" class="right quartersize " /> </picture> </div> <i>From <a href='https://discord.com/channels/715691013825364120/715692236901187674/911388251791708190' target='_blank' rel='nofollow'>Discord #announcements (obs-websocket)</a></i><br> @everyone Hey there, update for you regarding obs-websocket. We've moved to the OBS Project! <https://github.com/obsproject/obs-websocket> This is exciting news. It means that obs-websocket is on the roadmap to be included as a default plugin in OBS Studio.<br><br> I'll share our current roadmap, which may change at any point:<br><br> <b>**Til the end of 2021:**</b> Release obs-websocket version 5.0.0 as an independent plugin, officially beginning the transition period from 4.x to 5.x.<br><br> <b>**A few months into 2022:**</b> Merge obs-websocket as a submodule to obs-studio, and include it by default in the next major OBS release. This will overwrite obs-websocket 4.x, officially deprecating those versions. <br><br> As for our financial contributors over at Open Collective, we're working to get all of that data transferred to the OBS Project's page, where we will have a project dedicated to obs-websocket. Palakis and I are looking forward to obs-websocket's future in OBS. We will be keeping our roles as lead maintainers of the plugin. </div> <h2 id="obs-web">Installing Obs-web</h2> <p> Using a mobile device or another computer to control OBS Studio is possible using an OBS Studio plugin called <a href='https://github.com/obsproject/obs-websocket' target='_blank' rel='nofollow'>OBS-websocket</a>, and a client called <a href='https://github.com/Niek/obs-web' target='_blank' rel='nofollow'>obs-web</a>. Some technical experience with computers is required for this approach, but it works quite well. </p> <ol> <li> OBS-websocket was installed easily on the computer that is running OBS Studio by using the provided installer program. Once installed, the OBS Studio Tools menu had a new item called <b>Websockets Server Settings</b>. <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/obs-websocket-menu-addition.webp" type="image/webp"> <source srcset="/blog/images/obs/obs-websocket-menu-addition.png" type="image/png"> <img src="/blog/images/obs/obs-websocket-menu-addition.png" class="center liImg2 rounded shadow" /> </picture> </div> </li> <li> Clicking on the new OBS Studio Tools menu allows you to configure OBS-websocket. Notice that I disabled <b>Enable authentication</b>. <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/obs-websocket-config.webp" type="image/webp"> <source srcset="/blog/images/obs/obs-websocket-config.png" type="image/png"> <img src="/blog/images/obs/obs-websocket-config.png" class="center liImg2 rounded shadow" /> </picture> </div> </li> <li> On the same computer, download OBS-web from <a href='https://github.com/Niek/obs-web' target='_blank' rel='nofollow'>this page</a> and click on <b>Download latest build here</b>. <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/obs-web-download.webp" type="image/webp"> <source srcset="/blog/images/obs/obs-web-download.png" type="image/png"> <img src="/blog/images/obs/obs-web-download.png" class="center halfsize liImg2 rounded shadow" /> </picture> </div> </li> <li> I extracted the contents of the download, provided as a zip file, to <code>E:\media\obs-web-gh-pages</code>. </li> <li> I used Python 3 to start a small web server on port 4321 that serves the OBS-web files like this: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id390065ec4b64'><button class='copyBtn' data-clipboard-target='#id390065ec4b64' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn></span>python -m http.server \ --directory E:\media\obs-web-gh-pages \ 4321 <span class='unselectable'>Serving HTTP on :: port 4321 (http://[::]:4321/) ... ::1 - - [15/Nov/2021 10:49:52] "GET / HTTP/1.1" 304 - ::1 - - [15/Nov/2021 10:49:53] "GET /service-worker.js HTTP/1.1" 304 - </span></pre> The above incantation works on Windows, Mac and Linux. You only need to adjust the directory that the OBS-web files are stored in. </li> </ol> <h2 id="usingObsWeb">Using Obs-web</h2> <p> There are 2 small web servers now running on the computer with OBS Studio: the Python web server I started on the command line, and the OBS-web plugin for OBS Studio. This is how all the various bits of software work together: </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/obs-web-flow.webp" type="image/webp"> <source srcset="/blog/images/obs/obs-web-flow.png" type="image/png"> <img src="/blog/images/obs/obs-web-flow.png" class="center liImg2 rounded shadow" /> </picture> </div> <p> The computer running OBS Studio had IP address <code>192.168.1.77</code>. Knowing the IP address of that computer allows it and its programs to be accessed from other computers in the same network. <a href='https://www.tp-link.com/us/support/faq/838/' target='_blank' rel='nofollow'>Discover your computer's IP address.</a> </p> <!-- <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/obs-web-not-connected.webp" type="image/webp"> <source srcset="/blog/images/obs/obs-web-not-connected.png" type="image/png"> <img src="/blog/images/obs/obs-web-not-connected.png" class="center liImg2 rounded shadow" /> </picture> </div> <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/obs-web-not-connected-small.webp" type="image/webp"> <source srcset="/blog/images/obs/obs-web-not-connected-small.png" type="image/png"> <img src="/blog/images/obs/obs-web-not-connected-small.png" class="center liImg2 rounded shadow" /> </picture> </div> <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/obs-web-connecting.webp" type="image/webp"> <source srcset="/blog/images/obs/obs-web-connecting.png" type="image/png"> <img src="/blog/images/obs/obs-web-connecting.png" class="center liImg2 rounded shadow" /> </picture> </div> --> <p> To control OBS Studio from any computer or device in my local area network, I just need to point a web browser on the device to the <code>http://192.168.1.77:4321/#192.168.1.77:4444</code>. The buttons displayed on the web browser can be clicked on using a mouse, or by a finger if the device is touch-senstive. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/obs/obs-web-connected.webp" type="image/webp"> <source srcset="/blog/images/obs/obs-web-connected.png" type="image/png"> <img src="/blog/images/obs/obs-web-connected.png" class="center liImg2 rounded shadow" /> </picture> </div> <p> The 3 scenes I defined in OBS Studio are shown as blue and green buttons labeled <b>Default</b>, <b>Sony ILCE-7SM3</b> and <b>ProTools</b>. I can click on the red button labeled <b>Start recording</b>. Works really well. </p> Using an HDMI Splitter with OBS Studio 2021-11-13T00:00:00-05:00 https://mslinn.github.io/blog/2021/11/13/hdmi-splitter <p> <a href='/blog/2021/11/12/external-video-monitor.html'>A previous blog post</a> discussed the benefits of using an external monitor attached to your camera. HDMI is the most common OBS Studio is a wonderful tool for aggregating media into a real-time stream, and/or making a permanent recording. If you use an <a href='https://www.amazon.com/gp/product/B07XCZC6SP' target='_blank' rel='nofollow'>HDMI splitter</a>, the video from the camera can be viewed on a large monitor right in front of you, and the video can also be sent to the computer that is running OBS Studio. </p> <div style="text-align: center;"> <a href="https://www.amazon.com/gp/product/B07XCZC6SP" target="_blank" ><picture> <source srcset="/blog/media/videoSplitter.webp" type="image/webp"> <source srcset="/blog/media/videoSplitter.png" type="image/png"> <img src="/blog/media/videoSplitter.png" title="A 4K Video Splitter" class="center halfsize " alt="A 4K Video Splitter" /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://www.amazon.com/gp/product/B07XCZC6SP" target="_blank" > A 4K Video Splitter </a> </figcaption> </figure> </div> <p> The video splitter shown works with video resolutions up to 4K @ 60 Hz, which includes 4K @ 30 Hz and all 1080p video variants. When used in tandem with an HDMI to USB converter, for example a <a href='https://www.elgato.com/en/cam-link-4k' target='_blank' rel='nofollow'>CamLink 4K</a>, the HDMI video is converted into a video stream that OBS Studio can accept, either 1080p up to 60 Hz or 4K at 30 Hz. </p> <div style="text-align: center;"> <a href="https://www.elgato.com/en/cam-link-4k" target="_blank" ><picture> <source srcset="/blog/media/camlink-packaging.webp" type="image/webp"> <source srcset="/blog/media/camlink-packaging.png" type="image/png"> <img src="/blog/media/camlink-packaging.png" title="This HDMI to USB 3 adaptor is powered via USB.<br>No additional power is required." class="center halfsize " alt="This HDMI to USB 3 adaptor is powered via USB.<br>No additional power is required." /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://www.elgato.com/en/cam-link-4k" target="_blank" > This HDMI to USB 3 adaptor is powered via USB.<br>No additional power is required. </a> </figcaption> </figure> </div> <p> Here is a conceptual diagram of the signal flow between the camera, video splitter, CamLink and the computer running OBS Studio: </p> <div style="text-align: center;"> <picture> <source srcset="/blog/media/video-monitor-and-camlink.webp" type="image/webp"> <source srcset="/blog/media/video-monitor-and-camlink.png" type="image/png"> <img src="/blog/media/video-monitor-and-camlink.png" title="Signal paths with HDMI splitter and CamLink." class="center liImg2 rounded shadow" alt="Signal paths with HDMI splitter and CamLink." /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> Signal paths with HDMI splitter and CamLink. </figcaption> </figure> </div> <p> I have had the video splitter only one day so far, and it has worked well. The splitter takes about 8 seconds after being plugged in before it generates video output. Check out this video saved from OBS Studio using the above configuration: </p> <div class="videoWrapper2 shadow" style="margin-bottom: 1em"> <iframe width="1267" height="722" src="https://www.youtube.com/embed/eGYhGHl3xV0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> </div> <p> The video signal is noticably degraded by passing throught the video splitter. Compare the above video quality to the video below, made with the same equipment but without the video splitter: </p> <div class="videoWrapper2 shadow"> <iframe width="1267" height="722" src="https://www.youtube.com/embed/lkNe7gBP4YE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> </div> External Video Monitors For Cameras 2021-11-12T00:00:00-05:00 https://mslinn.github.io/blog/2021/11/12/external-video-monitor <p> When you are making a video recording of yourself, and no-one is helping you, the first problem you are likely to encounter is the need to see yourself. There are many reasons for needing to see yourself, including the need to ensure that: </p> <ul> <li>The camera is pointing at the right spot.</li> <li>The zoom level is correct.</li> <li>The camera is focused on the right area.</li> <li>The depth of field is correct.</li> <li>You look the way you want.</li> </ul> <h2 id="flip">Flip Monitors</h2> <p> Some cameras have a small flip-out viewscreen that can be seen from the front of the camera. That is only useful if you are very close to the camera, which means that most of the above reasons are unsatisfied. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/media/HR_G7X_MARKII_3QBACKLCD_CL.0.webp" type="image/webp"> <source srcset="/blog/media/HR_G7X_MARKII_3QBACKLCD_CL.0.png" type="image/png"> <img src="/blog/media/HR_G7X_MARKII_3QBACKLCD_CL.0.png" title="Canon Powershot G7x Mark II, showing its flip-up viewscreen" class="center halfsize liImg2 rounded shadow" alt="Canon Powershot G7x Mark II, showing its flip-up viewscreen" /> </picture> <figcaption class="halfsize" style="width: 100%; text-align: center;"> Canon Powershot G7x Mark II, showing its flip-up viewscreen </figcaption> </figure> </div> <p> You need a larger monitor, and it would be best if that monitor was located in the direction that you will be looking while making the recording. Quite often, this means the monitor should be directly under or over the camera. </p> <h2 id="overkill">Overkill</h2> <p> If you search online for solutions to this problem, you might get the impression that you need to spend a lot of money on external camera monitors that are 5" or 8" wide. Many articles and videos imply that you might even want to build a cage around the camera to hold both the monitor and extra batteries. This can cost hundreds or even thousands of dollars, but there is no need to go that way if you are shooting video indoors, for example, in a music studio. Not only that, but an 8" wide monitor is too small to be useful when working solo. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/media/atomos.webp" type="image/webp"> <source srcset="/blog/media/atomos.png" type="image/png"> <img src="/blog/media/atomos.png" title="Atomos Ninja V attached to a camera" class="center halfsize liImg2 rounded shadow" alt="Atomos Ninja V attached to a camera" /> </picture> <figcaption class="halfsize" style="width: 100%; text-align: center;"> Atomos Ninja V attached to a camera </figcaption> </figure> </div> <h2 id="computer">Computer Monitors</h2> <p> You can instead attach a computer monitor or a good quality TV to your camera. They are much larger, and some computer monitors and TVs provide powered USB ports for devices such as the camera, a music stand light, etc. This reduces the amount of wires and power plugs necessary. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/media/externalMonitor.webp" type="image/webp"> <source srcset="/blog/media/externalMonitor.png" type="image/png"> <img src="/blog/media/externalMonitor.png" title="A computer monitor attached to a camera via HDMI" class="center liImg2 rounded shadow" alt="A computer monitor attached to a camera via HDMI" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> A computer monitor attached to a camera via HDMI </figcaption> </figure> </div> <p> Most cameras provide a micro HDMI port, and most computer monitors accept fullsize HDMI connections. An <a href='https://www.amazon.com/AmazonBasics-High-Speed-Micro-HDMI-HDMI-Cable/dp/B014I8TZXW' target='_blank' rel='nofollow'>micro HDMI to fullsize HDMI adaptor cable</a> is all you need to connect the camera to the computer monitor. Note that most cameras do not provide audio on the HDMI output port. </p> <div style="text-align: center;"> <a href="https://www.amazon.com/AmazonBasics-High-Speed-Micro-HDMI-HDMI-Cable/dp/B014I8TZXW" target="_blank" ><picture> <source srcset="/blog/media/microHdmiAdapter.webp" type="image/webp"> <source srcset="/blog/media/microHdmiAdapter.png" type="image/png"> <img src="/blog/media/microHdmiAdapter.png" title="Micro-HDMI and HDMI connectors" class="center quartersize liImg2 rounded shadow" style="padding-left: 1em; padding-right: 1em; padding-top: 1em;" alt="Micro-HDMI and HDMI connectors" /> </picture></a> <figcaption class="quartersize" style="width: 100%; text-align: center;"> <a href="https://www.amazon.com/AmazonBasics-High-Speed-Micro-HDMI-HDMI-Cable/dp/B014I8TZXW" target="_blank" > Micro-HDMI and HDMI connectors </a> </figcaption> </figure> </div> <h2 id="usb">USB Ports</h2> <p> The least expensive computer monitors do not have USB ports, and some USB ports provide more power than others. Read monitor specifications before you buy. </p> <p> For this to work, the computer monitor needs to have <a href='https://en.wikipedia.org/wiki/USB_hardware' target='_blank' rel='nofollow'>USB type A ports</a> as shown below, either on the back of the monitor, or on the side. There is no need to connect the USB type B port to anything if you just want to provide power via USB type A connectors: </p> <div style="text-align: center;"> <picture> <source srcset="/blog/media/monitor-usb.webp" type="image/webp"> <source srcset="/blog/media/monitor-usb.png" type="image/png"> <img src="/blog/media/monitor-usb.png" title="USB ports on the back of a video monitor" class="center liImg2 rounded shadow" alt="USB ports on the back of a video monitor" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> USB ports on the back of a video monitor </figcaption> </figure> </div> <p> The USB ports should provide at least 1 amp <i>each</i>. Some USB devices require more power than that. Some manufacturers do not publish the power provided by their computer monitor's USB ports, so try with your actual USB devices before you buy, or at least have the option to return the monitor if the USB ports do not supply enough power. </p> <h2 id="mySetup">My Setup</h2> <p> I use the <a href='https://support.d-imaging.sony.co.jp/app/iemobile/en/' target='_blank' rel='nofollow'>Sony Imaging Edge Remote</a> camera app for my Android phone to control my <a href='https://www.sony.ca/en/electronics/interchangeable-lens-cameras/ilce-7m3-body-kit' target='_blank' rel='nofollow'>Sony Alpha 7 Mark iii camera</a> remotely. With that software I can adjust aperture and other parameters. I prefer to use the <a href='https://www.sony.ca/en/electronics/interchangeable-lens-cameras-tripods-remotes/rmt-p1bt' target='_blank' rel='nofollow'>Sony Wireless Remote Commander</a> to control recording because I can feel the physical controls, so I can operate it without looking. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/media/video-monitor-for-camera.webp" type="image/webp"> <source srcset="/blog/media/video-monitor-for-camera.png" type="image/png"> <img src="/blog/media/video-monitor-for-camera.png" title="Video monitor and power for camera" class="center liImg2 rounded shadow" alt="Video monitor and power for camera" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> Video monitor and power for camera </figcaption> </figure> </div> <p> I have several monitors that provide USB connections, including a BenQ BL3201PH 32" 4K Monitor, an ASUS ProArt Display PA248Q and an ASUS PA238Q. I use the PA238Q for monitoring the camera. The monitor is not attached to a computer, and I locate it as required for the scene I am shooting. </p> <div style="text-align: center;"> <a href="https://www.asus.com/Displays-Desktops/Monitors/ProArt/ProArt-Display-PA248Q/" target="_blank" ><picture> <source srcset="/blog/media/ASUS_PA248Q.webp" type="image/webp"> <source srcset="/blog/media/ASUS_PA248Q.png" type="image/png"> <img src="/blog/media/ASUS_PA248Q.png" title="ASUS ProArt Display PA248Q video monitor<br>with 4 USB 3 type A connectors on the side" class="center halfsize liImg2 rounded shadow" alt="ASUS ProArt Display PA248Q video monitor<br>with 4 USB 3 type A connectors on the side" /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://www.asus.com/Displays-Desktops/Monitors/ProArt/ProArt-Display-PA248Q/" target="_blank" > ASUS ProArt Display PA248Q video monitor<br>with 4 USB 3 type A connectors on the side </a> </figcaption> </figure> </div> <div style="text-align: center;"> <a href="https://www.asus.com/Displays-Desktops/Monitors/ProArt/ProArt-Display-PA238Q/" target="_blank" ><picture> <source srcset="/blog/media/ASUS_PA238Q.webp" type="image/webp"> <source srcset="/blog/media/ASUS_PA238Q.png" type="image/png"> <img src="/blog/media/ASUS_PA238Q.png" title="ASUS ProArt Display PA238Q video monitor<br> has 2 USB 2 type A connectors on the side, and 2 more underneath" class="center halfsize liImg2 rounded shadow" alt="ASUS ProArt Display PA238Q video monitor<br> has 2 USB 2 type A connectors on the side, and 2 more underneath" /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://www.asus.com/Displays-Desktops/Monitors/ProArt/ProArt-Display-PA238Q/" target="_blank" > ASUS ProArt Display PA238Q video monitor<br> has 2 USB 2 type A connectors on the side, and 2 more underneath </a> </figcaption> </figure> </div> <p> The UBS ports on my ASUS ProArt Display PA238Q provide enough power to run my Sony Alpha 7 Mark iii camera, which is nice, but not enough to run my <a href='https://www.amazon.com/gp/product/B07XCZC6SP' target='_blank' rel='nofollow'>HDMI splitter</a>. This just means that I must use a dedicated power supply for the HDMI splitter. I will talk about how I use the HDMI splitter in another blog post. </p> Sending DAW Output to OBS Studio Using RME TotalMix 2021-11-08T00:00:00-05:00 https://mslinn.github.io/blog/2021/11/08/totalmix-daw-obs <p> <a href='https://www.rme-usa.com/totalmix-fx.html' target='_blank' rel='nofollow'>RME TotalMix</a> provides routing and mixing functions in software for RME audio interfaces. I use an <a href='https://archiv.rme-audio.de/en/products/fireface_ufx.php' target='_blank' rel='nofollow'>RME UFX</a> on my main <a href='https://en.wikipedia.org/wiki/Digital_audio_workstation' target='_blank' rel='nofollow'>DAW</a>, and an <a href='https://archiv.rme-audio.de/en/products/fireface_uc.php' target='_blank' rel='nofollow'>RME UC</a> for mobile recordings. TotalMix works with all RME audio interfaces, including the popular <a href='https://rme-audio.de/babyface-pro-fs.html' target='_blank' rel='nofollow'>BabyFace</a>. </p> <p> This blog post briefly outlines how to capture a live performance on video, using your studio-quality microphones while enjoying the real-time effects and mixing capability of your favorite DAW software. You can also use this setup to make a high quality video capture of karaoke, that is, singing along or playing along with pre-recorded music. The recorded music could either be mixed using your DAW, or by providing it to OBS Studio as a Media Source. </p> <p> Nothing described in this blog post is specific to any particular DAW software; I mostly use Pro Tools and Ableton Live, but Cakewalk, etc would work just as well with these instructions. I tested this with Windows 10. Apparently TotalMix on Mac does not mute some inputs properly when loopback is enabled, or something like that; I am unclear exactly what the problem is. </p> <p> No changes to the system hardware or software are necessary to send DAW output to OBS Studio if you use an RME audio interface, most especially no physical or virtual cables are necessary. </p> <div class="formalNotice shadow rounded"> <h2 id="but" style="margin-top: 0">What If I Do Not Have an RME Audio Interface?</h2> If you have an audio interface made by another manufacturer then you will probably need to use a virtual audio cable, such as <a href='https://vb-audio.com/Cable/' target='_blank' rel='nofollow'>VB-Cable or HIFI-CABLE & ASIO-Bridge</a>. Both programs are available for Windows and Mac. Online help for those programs are available <a href='https://forum.vb-audio.com/index.php?sid=94158a3f6725fd801b7b52ea044c6ccb' target='_blank' rel='nofollow'>on the VB-Audio forums</a>. </div> <p> The way for TotalMix to route the DAW output channels to another stereo input is called ‘loopback’. <a href='https://www.youtube.com/watch?v=m-Zeruz-9Zk' target='_blank' rel='nofollow'>This YouTube video</a> shows how to configure TotalMix to use loopback, but unfortunately the video stops short of actually showing how to work with the routed audio. Read on and I will tell you the rest of the story. It is quick and easy! </p> <h2 id="signals">Signal Paths</h2> <p> The following diagram shows the important signal paths for this setup. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/totalmixObs/rmeLoopbackSignalPaths.webp" type="image/webp"> <source srcset="/blog/images/totalmixObs/rmeLoopbackSignalPaths.png" type="image/png"> <img src="/blog/images/totalmixObs/rmeLoopbackSignalPaths.png" class="center liImg2 rounded shadow" /> </picture> </div> <h2 id="steps">Step by Step</h2> <p> As shown in the video, set up an unused TotalMix output which will be sent to OBS Studio. Let’s use the AES output for that. The steps are: </p> <ol> <li> Open the AES tools (click on the little wrench icon) and enable loopback, like this: <div style="text-align: center;"> <picture> <source srcset="/blog/images/totalmixObs/totalMixAes.webp" type="image/webp"> <source srcset="/blog/images/totalmixObs/totalMixAes.png" type="image/png"> <img src="/blog/images/totalmixObs/totalMixAes.png" class="center liImg2 rounded shadow" /> </picture> </div> Unfortunately, TotalMix will not show loopback signal at AES input. (Hey, RME, please do something about this!) </li> <li style="clear: both"> In OBS Studio, define a new Audio Input Capture source that captures the output of the TotalMix AES loopback. First press the <b>+</b> but1ton, then click on <b>Audio Input Capture</b>: <div style="text-align: center;"> <picture> <source srcset="/blog/images/totalmixObs/totalMixAesDefine.webp" type="image/webp"> <source srcset="/blog/images/totalmixObs/totalMixAesDefine.png" type="image/png"> <img src="/blog/images/totalmixObs/totalMixAesDefine.png" class="center liImg2 rounded shadow" /> </picture> </div> </li> <li style="clear: both"> I called the new OBS Studio source <b>AES</b>: <div style="text-align: center;"> <picture> <source srcset="/blog/images/totalmixObs/totalMixAesDefineName.webp" type="image/webp"> <source srcset="/blog/images/totalmixObs/totalMixAesDefineName.png" type="image/png"> <img src="/blog/images/totalmixObs/totalMixAesDefineName.png" class="center liImg2 rounded shadow" /> </picture> </div> </li> <li style="clear: both"> To complete the definition of the new OBS Studio source, select the input source whose name starts with <b>AES (RME</b>: <div style="text-align: center;"> <picture> <source srcset="/blog/images/totalmixObs/obsStudioAesInput.webp" type="image/webp"> <source srcset="/blog/images/totalmixObs/obsStudioAesInput.png" type="image/png"> <img src="/blog/images/totalmixObs/obsStudioAesInput.png" class="center liImg2 rounded shadow" /> </picture> </div> </li> <li style="clear: both"> Set the mix levels in OBS Studio. <div style="text-align: center;"> <picture> <source srcset="/blog/images/totalmixObs/obsStudioMixLevels.webp" type="image/webp"> <source srcset="/blog/images/totalmixObs/obsStudioMixLevels.png" type="image/png"> <img src="/blog/images/totalmixObs/obsStudioMixLevels.png" class="center liImg2 rounded shadow" /> </picture> </div> </li> </ol> <p> That is all, you are done! The processed audio from your DAW will now be mixed with the other audio and video streams you set up. I use a <a href='https://www.elgato.com/en/cam-link-4k' target='_blank' rel='nofollow'>Camlink 4K</a> to stream video coming from my <a href='https://www.sony.ca/en/electronics/interchangeable-lens-cameras/ilce-7m3-body-kit' target='_blank' rel='nofollow'>Sony Alpha 7 Mark iii camera</a>. OBS Studio mixes all the audio and video streams, and the combined live media stream can be sent to Instagram, YouTube, Facebook etc, and/or it can be saved as an mp4 or mkv. I prefer to save as mkv. </p> <h2 id="hot">Hot Key</h2> <p> BTW, I set the hot key for recording the mix using OBS Studio to <kbd>Ctrl</kbd>-<kbd>Shift</kbd>-<kbd>Z</kbd>. You can find these settings at <b>File</b> / <b>Settings</b> / <b>HotKeys</b>: </p> <div style=""> <picture> <source srcset="/blog/images/totalmixObs/obsStudioHotKeys.webp" type="image/webp"> <source srcset="/blog/images/totalmixObs/obsStudioHotKeys.png" type="image/png"> <img src="/blog/images/totalmixObs/obsStudioHotKeys.png" class=" liImg2 rounded shadow" /> </picture> </div> <h2 id="testing">Test Recording</h2> <p> Here is a quick test recording that I made using the above setup, of an extemporaneous composition, while writing this blog post. Notice the echo and reverb effects on the audio; those were added by Pro Tools. I wore headphones so I could listen to the audio without worrying about feedback or adding extra echo. </p> <div class="videoWrapper2 shadow"> <iframe width="1267" height="722" src="https://www.youtube.com/embed/lkNe7gBP4YE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> </div> Extracting Audio from an MP4 as 32-bit WAV 2021-11-04T00:00:00-04:00 https://mslinn.github.io/blog/2021/11/04/mp4-to-wav <p> My <a href='https://www.sony.ca/en/electronics/interchangeable-lens-cameras/ilce-7m3-body-kit' target='_blank' rel='nofollow'>Sony Alpha 7 Mark iii camera</a> creates mp4 files with good quality stereo audio. I wanted to extract the audio to a 32-bit wav file, so I could work on it further in Pro Tools. Here is a bash script I wrote to do that: </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,mp4ToWav' download='mp4ToWav' title='Click on the file name to download the file'>mp4ToWav</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id087c4733e240">#!/bin/bash # $1 Input file path function help &#123; if [ "$1" ]; then printf "$1\n\n"; fi echo "$(basename $0) - Extract audio stream from an mp4 file and save as 32-bit wav Usage: $(basename $0) filename " exit 1 &#125; if [ -z "$1" ]; then help "Error: no media file name specified"; fi if [ ! -f "$1" ]; then help "Error: '$1' not found"; fi filename="$( basename -- "$1" )" path="$( dirname "$1" )" extension="$&#123;filename##*.&#125;" filename="$&#123;filename%.*&#125;" ffmpeg \ -i "$1" \ -vn \ -acodec pcm_f32le \ -ar 44100 \ -ac 2 \ "$path/$filename.wav" </pre> <p> This is a sample usage: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idae0f4a2e4501'><button class='copyBtn' data-clipboard-target='#idae0f4a2e4501' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mp4ToWav "Video Files/Descending C to G djembe" ffmpeg version 4.3.2-0+deb11u1ubuntu1 Copyright (c) 2000-2021 the FFmpeg developers built with gcc 10 (Ubuntu 10.2.1-20ubuntu1) configuration: --prefix=/usr --extra-version=0+deb11u1ubuntu1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared libavutil 56. 51.100 / 56. 51.100 libavcodec 58. 91.100 / 58. 91.100 libavformat 58. 45.100 / 58. 45.100 libavdevice 58. 10.100 / 58. 10.100 libavfilter 7. 85.100 / 7. 85.100 libavresample 4. 0. 0 / 4. 0. 0 libswscale 5. 7.100 / 5. 7.100 libswresample 3. 7.100 / 3. 7.100 libpostproc 55. 7.100 / 55. 7.100 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x55fc3d8d1f80] st: 0 edit list: 1 Missing key frame while searching for timestamp: 1001 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x55fc3d8d1f80] st: 0 edit list 1 Cannot find an index entry before timestamp: 1001. Guessed Channel Layout for Input Stream #0.1 : stereo Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Video Files/Descending C to G djembe.mp4': Metadata: major_brand : XAVC minor_version : 16785407 compatible_brands: XAVCmp42iso2 creation_time : 2021-10-31T19:00:25.000000Z Duration: 00:09:06.55, start: 0.000000, bitrate: 51575 kb/s Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709/bt709/iec61966-2-4), 1920x1080 [SAR 1:1 DAR 16:9], 49492 kb/s, 59.94 fps, 59.94 tbr, 60k tbn, 119.88 tbc (default) Metadata: creation_time : 2021-10-31T19:00:25.000000Z handler_name : Video Media Handler encoder : AVC Coding Stream #0:1(und): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, stereo, s16, 1536 kb/s (default) Metadata: creation_time : 2021-10-31T19:00:25.000000Z handler_name : Sound Media Handler Stream #0:2(und): Data: none (rtmd / 0x646D7472), 491 kb/s (default) Metadata: creation_time : 2021-10-31T19:00:25.000000Z handler_name : Timed Metadata Media Handler timecode : 07:09:43:54 Stream mapping: Stream #0:1 -> #0:0 (pcm_s16be (native) -> pcm_f32le (native)) Press [q] to stop, [?] for help Output #0, wav, to 'Video Files/Descending C to G djembe.wav': Metadata: major_brand : XAVC minor_version : 16785407 compatible_brands: XAVCmp42iso2 ISFT : Lavf58.45.100 Stream #0:0(und): Audio: pcm_f32le ([3][0][0][0] / 0x0003), 44100 Hz, stereo, flt, 2822 kb/s (default) Metadata: creation_time : 2021-10-31T19:00:25.000000Z handler_name : Sound Media Handler encoder : Lavc58.91.100 pcm_f32le size= 188304kB time=00:09:06.55 bitrate=2822.4kbits/s speed= 153x video:0kB audio:188304kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000059%</pre> <p> The original mp4 was 3.4GB, and the output wav was 188MB. </p> Sony Alpha 7 Mark iii Camera Media Encodings 2021-11-03T00:00:00-04:00 https://mslinn.github.io/blog/2021/11/03/sony-a7iii-encodings <style> dt, dd { font-style: normal } </style> <p> The documentation for the <a href='https://www.sony.ca/en/electronics/interchangeable-lens-cameras/ilce-7m3-body-kit' target='_blank' rel='nofollow'>Sony Alpha 7 Mark iii camera</a> does not properly describe the differences in encodings between the various video formats available. The settings trade off quality versus file size. I want to know which settings to use for various purposes. To this end, I made a short video clip for each camera setting and examined its properties. This blog post details my findings. </p> <div style=""> <a href="https://www.sony.ca/en/electronics/interchangeable-lens-cameras/ilce-7m3-body-kit" target="_blank" ><picture> <source srcset="/blog/images/sonyA7iii.webp" type="image/webp"> <source srcset="/blog/images/sonyA7iii.png" type="image/png"> <img src="/blog/images/sonyA7iii.png" class=" liImg2 rounded shadow" /> </picture></a> </div> <p> Sony engineers have no doubt chosen these settings after careful deliberation and testing. Unfortunately, product documentation does not discuss when these settings should be used. </p> <p> To be specific, after working through all of this material, I still do not know when should I use the various formats. All I know for sure is that higher bit rate settings make larger video clips, and that there might be a quality difference. The degree to which the quality difference is perceptible is unknown. I would like guidelines in order to make intelligent decisions. Here are the specific formats that need to be rationalized: </p> <p style="margin-left: 2em"> <code>4K_30p_100M</code> vs. <code>4K_30p_60M</code><br> <code>4K_24p_100M</code> vs. <code>4K_24p_60M</code><br> <code>HD_120p_100M</code> vs. <code>HD_120p_60M</code><br> <code>HD_60p_50M</code> vs. <code>HD_60p_25M</code><br> <code>HD_30p_50M</code> vs. <code>HD_30p_16M</code> </p> <h2 id="background">Background</h2> <p> Here are a few key aspects from the Sony product specifications, to which I have added comments <i>in italics</i>: </p> <dl> <dt>Recording Format</dt> <dd> XAVC S, AVCHD format Ver. 2.0 compliant.<br> <i> The documentation is rather terse; the above means that two recording formats are available: XAVC, a proprietary Sony media format, and AVCHD, described <a href="#avchd">later in this document</a>. <a href='https://en.wikipedia.org/wiki/XAVC' target='_blank' rel='nofollow'>Wikipedia</a> says &ldquo;XAVC supports resolutions up to 3840 × 2160, uses MP4 as the container format, and uses either AAC or LPCM for the audio.&rdquo; </i> </dd> <dt>Video Compression</dt> <dd> XAVC S: MPEG-4 AVC/H.264, AVCHD: MPEG-4 AVC/H.264<br> <i>Again, the above means that the camera supports two types of video compression: XAVC S and AVCHD.</i> </dd> <dt>Audio Recording Format</dt> <dd> XAVC S: LPCM 2ch, AVCHD: Dolby® Digital (AC-3) 2ch, Dolby® Digital Stereo Creator.<br> <i>I understand this to mean that the camera supports 3 recording formats: <a href='https://en.wikipedia.org/wiki/Dolby_Digital' target='_blank' rel='nofollow'>Dolby AC-3</a> (a lossy format), AVCHD, and Dolby® Digital Stereo Creator (which I did not find).</i> <dd> <dt>Clean HDMI Output</dt> <dd> <i>This is important when streaming (via HDMI).</i><br> 3840 x 2160 (30p),<br> 3840 x 2160 (25p),<br> 3840 x 2160 (24p),<br> 1920 x 1080 (60p),<br> 1920 x 1080 (60i),<br> 1920 x 1080 (50p),<br> 1920 x 1080 (50i),<br> 1920 x 1080 (24p),<br> YCbCr 4:2:2 8bit / RGB 8bit.<br> <i>25p (25 frames per second) is noticably less smooth than 50p or 60p. I think that the extra resolution of 4K is not as noticable as the choppy video that it introduces, so my choice for streaming with this camera is 1920 x 1080 (60p). I am unclear about "YCbCr 4:2:2 8bit / RGB 8bit"; does that apply to 3840 x 2160 (25p), or to all the settings, or is it a separate setting?</i> </dd> <dt>Color Space</dt> <dd> xvYCC standard (x.v.Colour when connected via HDMI cable) compatible with TRILUMINOS Color.<br> <i>This means that color is somewhat degraded when streamed via HDMI. I would like to understand how much it is degraded.</i> </dd> </dl> <h2 id="mediaDump">Script to Examine Media Format</h2> <p> I wrote this bash script to examine video clips recorded at different settings. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,mediaDump' download='mediaDump' title='Click on the file name to download the file'>mediaDump</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id219da9a6e68b">#!/bin/bash function help &#123; if [ "$1" ]; then printf "$1\n\n"; fi echo "$(basename $0) - Dump information about a media file Usage: $(basename $0) filename " exit 1 &#125; if [ -z "$1" ]; then help "Error: no media file name specified"; fi ffprobe \ -hide_banner \ -loglevel fatal \ -show_error \ -show_format \ -show_streams \ -show_private_data \ -print_format json \ "$1" </pre> <h3 id="mediaDumpExample">Using the Script</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idea1d88d0a04d'><button class='copyBtn' data-clipboard-target='#idea1d88d0a04d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mediaDump 4K_24p_60M.mp4 <span class='unselectable'>{ "streams": [ { "index": 0, "codec_name": "h264", "codec_long_name": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10", "profile": "High", "codec_type": "video", "codec_time_base": "1001/48000", "codec_tag_string": "avc1", "codec_tag": "0x31637661", "width": 3840, "height": 2160, "coded_width": 3840, "coded_height": 2160, "closed_captions": 0, "has_b_frames": 1, "sample_aspect_ratio": "1:1", "display_aspect_ratio": "16:9", "pix_fmt": "yuv420p", "level": 51, "color_range": "tv", "color_space": "bt709", "color_transfer": "iec61966-2-4", "color_primaries": "bt709", "chroma_location": "left", "refs": 1, "is_avc": "true", "nal_length_size": "4", "r_frame_rate": "24000/1001", "avg_frame_rate": "24000/1001", "time_base": "1/24000", "start_pts": 0, "start_time": "0.000000", "duration_ts": 108108, "duration": "4.504500", "bit_rate": "53533087", "bits_per_raw_sample": "8", "nb_frames": "108", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0 }, "tags": { "creation_time": "2021-11-03T17:02:12.000000Z", "language": "und", "handler_name": "Video Media Handler", "encoder": "AVC Coding" } }, { "index": 1, "codec_name": "pcm_s16be", "codec_long_name": "PCM signed 16-bit big-endian", "codec_type": "audio", "codec_time_base": "1/48000", "codec_tag_string": "twos", "codec_tag": "0x736f7774", "sample_fmt": "s16", "sample_rate": "48000", "channels": 2, "bits_per_sample": 16, "r_frame_rate": "0/0", "avg_frame_rate": "0/0", "time_base": "1/48000", "start_pts": 0, "start_time": "0.000000", "duration_ts": 216240, "duration": "4.505000", "bit_rate": "1536000", "nb_frames": "216240", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0 }, "tags": { "creation_time": "2021-11-03T17:02:12.000000Z", "language": "und", "handler_name": "Sound Media Handler" } }, { "index": 2, "codec_type": "data", "codec_tag_string": "rtmd", "codec_tag": "0x646d7472", "r_frame_rate": "0/0", "avg_frame_rate": "0/0", "time_base": "1/24000", "start_pts": 0, "start_time": "0.000000", "duration_ts": 108108, "duration": "4.504500", "bit_rate": "196411", "nb_frames": "108", "disposition": { "default": 1, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0 }, "tags": { "creation_time": "2021-11-03T17:02:12.000000Z", "language": "und", "handler_name": "Timed Metadata Media Handler", "timecode": "07:28:14:16" } } ], "format": { "filename": "4K_24p_60M.mp4", "nb_streams": 3, "nb_programs": 0, "format_name": "mov,mp4,m4a,3gp,3g2,mj2", "format_long_name": "QuickTime / MOV", "start_time": "0.000000", "duration": "4.505000", "size": "33559173", "bit_rate": "59594535", "probe_score": 100, "tags": { "major_brand": "XAVC", "minor_version": "16785407", "compatible_brands": "XAVCmp42iso2", "creation_time": "2021-11-03T17:02:12.000000Z" } } } </span></pre> <p> The following attributes, which were of interest to me, had the same values for all the video clips I tested: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idae5a2b689fd6'><button class='copyBtn' data-clipboard-target='#idae5a2b689fd6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>streams[0].codec_name: h264 streams[0].codec_long_name: H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 streams[1].codec_name: pcm_s16be streams[1].codec_long_name: PCM signed 16-bit big-endian streams[1].sample_rate: 48000 format.format_long_name: QuickTime / MOV</pre> <p> In other words, the audio codec (<code>pcm_s16be</code>) and video format (<code>QuickTime / MOV</code>) were the same for all recording settings. The audio for the sample video clips was always recorded as 16-bits at 48 KHz, which is DVD quality. The only significant differences between the video clips are the bit rate of the H.264 codec and the video resolution. </p> <p> I wrote another bash script called <code>mediaSummary</code> that simply output the details of interest. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,mediaSummary' download='mediaSummary' title='Click on the file name to download the file'>mediaSummary</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idfae23e4e5471">#!/bin/bash function help &#123; if [ "$1" ]; then printf "$1\n\n"; fi echo "$(basename $0) - Dump selected information about a media file Usage: $(basename $0) filename " exit 1 &#125; if [ -z "$1" ]; then help "Error: no media file name specified"; fi LC_NUMERIC=en_US.utf8 JSON="$( mediaDump "$1" )" STREAM0="$( jq '.streams[0]' &lt;&lt;&lt; "$JSON" )" codec_bit_rate0="$( jq -r .bit_rate &lt;&lt;&lt; "$STREAM0" )" codec_bit_rate0_formatted="$( printf "%'d" $codec_bit_rate0 )" echo "codec_bit_rate: $codec_bit_rate0_formatted"</pre> <h2 id="output">Media Formats Tested</h2> <p> I used the following menu sequence to set the camera to various media settings: </p> <h3 id="4k_settings">4K Video Settings</h3> <p> 4K video files are much larger than HD video files, and require significantly more processing power for post-production. </p> <p> MENU → Movie1 → File Format → XAVC S 4K<br> MENU → Movie1 → Record Setting → 30p 100M, 30p 60M, 24p 100M and 24p 60M. </p> <h3 id="4k_details">4K Video Format Details</h3> <p> All of the 4K formats had the same resolution: 3840 x 2160 pixels. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idaa504d47f343'><button class='copyBtn' data-clipboard-target='#idaa504d47f343' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mediaSummary 4K_24p_100M.mp4 codec_bit_rate: 95,404,278</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc6ea5d062803'><button class='copyBtn' data-clipboard-target='#idc6ea5d062803' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mediaSummary 4K_24p_60M.mp4 codec_bit_rate: 53,533,087</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id20539a7bed52'><button class='copyBtn' data-clipboard-target='#id20539a7bed52' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mediaSummary 4K_30p_100M.mp4 codec_bit_rate: 96,398,171</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide87fce63a289'><button class='copyBtn' data-clipboard-target='#ide87fce63a289' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mediaSummary 4K_30p_60M.mp4 codec_bit_rate: 56,046,286</pre> <h3 id="hd">HD Video Settings</h3> <p> HD, also known as 1080p, is the most common video resolution on the internet today: 1920 x 1280 pixels. </p> <p> MENU → Movie1 → File Format → XAVC S HD<br> MENU → Movie1 → Record Setting → 60p 50M / 60p 25M, 30p 50M, 30p 16M, 24p 50M, 120p 100M and 120p 60M </p> <h3 id="hd_details">HD Video Format Details</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3449aa074c49'><button class='copyBtn' data-clipboard-target='#id3449aa074c49' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ mediaSummary HD_120p_100M.mp4 codec_bit_rate: 95,430,311</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6fd65ae5e74a'><button class='copyBtn' data-clipboard-target='#id6fd65ae5e74a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ mediaSummary HD_120p_60M.mp4 codec_bit_rate: 56,137,995</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide7604b8eacb8'><button class='copyBtn' data-clipboard-target='#ide7604b8eacb8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ mediaSummary HD_60p_50M.mp4 codec_bit_rate: 47,991,727</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfe5e32ac0987'><button class='copyBtn' data-clipboard-target='#idfe5e32ac0987' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ mediaSummary HD_60p_25M.mp4 codec_bit_rate: 24,176,270</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd698c7e7f5d8'><button class='copyBtn' data-clipboard-target='#idd698c7e7f5d8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ mediaSummary HD_30p_50M.mp4 codec_bit_rate: 46,589,694</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1ff645750305'><button class='copyBtn' data-clipboard-target='#id1ff645750305' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ mediaSummary HD_30p_16M.mp4 codec_bit_rate: 15,483,830</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd0392b30eee9'><button class='copyBtn' data-clipboard-target='#idd0392b30eee9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ mediaSummary HD_24p_50M.mp4 codec_bit_rate: 47,529,910</pre> <h3 id="avchd">AVCHD Video Format</h3> <p class="warning"> Warning! <a href='https://www.easefab.com/avchd-tips/import-mts-to-davinci-resolve.html' target='_blank' rel='nofollow'>Davinci Resolve has issues with this format.</a> </p> <p> Because I use Davinci Resolve I did not test AVCHD. Wikipedia says: </p> <p class="quote"> Developed jointly by Sony and Panasonic, the format was introduced in 2006 primarily for use in high definition consumer camcorders.<br><br> For video compression, AVCHD uses the H.264/MPEG-4 AVC standard, supporting a variety of standard, high definition, and stereoscopic (3D) video resolutions. For audio compression, it supports both Dolby AC-3 (Dolby Digital) and uncompressed linear PCM audio. Stereo and multichannel surround (5.1) are both supported. <br><br> This format is compatible with Blu-ray. </p> <p> To enable AVCHD, use this menu sequence: </p> <p> MENU → Movie1 → File Format → AVCHD </p> <p> Jonny Elwyn has a good article entitled <a href='https://www.premiumbeat.com/blog/avchd-editing-workflow/' target='_blank' rel='nofollow'>Should Editors Transcode AVCHD to ProRes in Premiere?</a> </p> Disappointing Scala 3 Installation Experience 2021-05-19T00:00:00-04:00 https://mslinn.github.io/blog/2021/05/19/installing-scala-3.0 <p> I run <a href='https://scalacourses.com' target='_blank' rel='nofollow'>ScalaCourses.com</a>, an online Scala training web site. After years of hype, Scala 3 is now available. </p> <h2 id="prime">Scala 3: Not Yet Ready for Production</h2> <p> <a href='https://www.infoq.com/news/2021/03/scala3/' target='_blank' rel='nofollow'>Scala 3</a> was eight years in the making. You would never know that from the horrible installation process and the disappointing installation instructions. </p> <p> It is going to take quite a while before <a href='https://scalatimes.com/d374aea433' target='_blank' rel='nofollow'>Scala 3</a>, which was known as Dotty before it was released, can be trusted in production. According to the <a href='https://github.com/lampepfl/dotty' target='_blank' rel='nofollow'>Dotty GitHub project</a>, the only published future milestone is <a href='https://github.com/lampepfl/dotty/milestones' target='_blank' rel='nofollow'>v3.1.0, which has no due date</a>. Given that Scala 3 uses an entirely new build process, and an entirely new (and nonstandard) installation process, and that the internals of the Scala compiler were almost completely replaced, I doubt that version will be stable enough for use on production projects. </p> <h2 id="choices">Installation Choices</h2> <p> Installation cholices include: </p> <ul> <li>Command-line Scala has limited use cases, but is nice to have around for occassional experimentation.</li> <li> <a href='https://www.scala-lang.org/blog/2021/04/08/scala-3-in-sbt.html' target='_blank' rel='nofollow'>SBT</a> (which features an enhanced Scala REPL) is very helpful for interactively developing code, as well as for building and testing. </li> <li> <a href='https://www.jetbrains.com/help/idea/discover-intellij-idea-for-scala.html' target='_blank' rel='nofollow'>IntelliJ</a> provides the best Scala coding productivity and code quality. </li> <li> <a href='https://shunsvineyard.info/2020/11/20/setting-up-vs-code-for-scala-development-on-wsl/' target='_blank' rel='nofollow'>VSCode</a> has been playing catch-up but is not full-featured yet. </li> </ul> <h2 id="install">Installation Transcript</h2> <p> The following is the transcript of how I installed command-line Scala on Ubuntu 20.10 running under WSL2. </p> <h3 class="numbered" id="remove_scala2">Remove Scala 2</h3> <p> This step is not required. Scala 2 and Scala 3 can easily co-exist on the same system because their names are different. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc245680769b2'><button class='copyBtn' data-clipboard-target='#idc245680769b2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt remove scala <span class='unselectable'>Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be REMOVED: scala 0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded. After this operation, 666 MB disk space will be freed. Do you want to continue? [Y/n] (Reading database ... 236024 files and directories currently installed.) Removing scala (2.13.4-400) ... Processing triggers for man-db (2.9.3-2) ... </span></pre> <h3 class="numbered" id="cs">Install Coursier</h3> <p> Coursier is a multithreaded downloader for project dependencies, and it now also downloads Scala 3. SBT uses coursier internally. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idaa5b8488d335'><button class='copyBtn' data-clipboard-target='#idaa5b8488d335' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -fLo cs https://git.io/coursier-cli-"$(uname | tr LD ld)" <span class='unselectable'>&nbsp; % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 144 100 144 0 0 285 0 --:--:-- --:--:-- --:--:-- 6000 100 57.1M 100 57.1M 0 0 3656k 0 0:00:15 0:00:15 --:--:-- 4092k </span> <span class='unselectable'>$ </span>mv cs ~/.local/bin/ <span class='unselectable'>$ </span>chmod a+x ~/.local/bin/cs <span class='unselectable'>$ </span>cs install cs <span class='unselectable'>https://repo1.maven.org/maven2/io/get-coursier/apps/maven-metadata.xml 100.0% [##########] 1.8 KiB (8.5 KiB / s) https://repo1.maven.org/maven2/io/get-coursier/coursier-cli_2.12/maven-metadata.xml No new update since 2021-03-23 14:35:16 Wrote cs Warning: /home/mslinn/.local/share/coursier/bin is not in your PATH To fix that, add the following line to ~/.bashrc export PATH="$PATH:/home/mslinn/.local/share/coursier/bin" </span> <span class='unselectable'>$ </span> export PATH="$PATH:/home/mslinn/.local/share/coursier/bin"</pre> <p> Now let's test Coursier: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3e0bc5fcc49f'><button class='copyBtn' data-clipboard-target='#id3e0bc5fcc49f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cs <span class='unselectable'>Coursier 2.0.16 Usage: cs [options] [command] [command-options] Available commands: bootstrap, channel, complete, fetch, get, install, java, java-home, launch, list, publish, resolve, setup, uninstall, update, search Type cs command --help for help on an individual command </span></pre> <p> This is the Coursier help message for the <code>install</code> subcommand: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3cd2f852e081'><button class='copyBtn' data-clipboard-target='#id3cd2f852e081' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cs install --help <span class='unselectable'>Command: install Usage: cs install --cache &lt;string?&gt; Cache directory (defaults to environment variable COURSIER_CACHE, or ~/.cache/coursier/v1 on Linux and ~/Library/Caches/Coursier/v1 on Mac) --mode | -m &lt;offline|update-changing|update|missing|force&gt; Download mode (default: missing, that is fetch things missing from cache) --ttl | -l &lt;duration&gt; TTL duration (e.g. &quot;24 hours&quot;) --parallel | -n &lt;int&gt; Maximum number of parallel downloads (default: 6) --checksum &lt;checksum1,checksum2,...&gt; Checksum types to check - end with none to allow for no checksum validation if no checksum is available, example: SHA-256,SHA-1,none --retry-count &lt;int&gt; Retry limit for Checksum error when fetching a file --cache-file-artifacts | --cfa &lt;bool&gt; Flag that specifies if a local artifact should be cached. --follow-http-to-https-redirect &lt;bool&gt; Whether to follow http to https redirections --credentials &lt;host(realm) user:pass|host user:pass&gt; Credentials to be used when fetching metadata or artifacts. Specify multiple times to pass multiple credentials. Alternatively, use the COURSIER_CREDENTIALS environment variable --credential-file &lt;string*&gt; Path to credential files to read credentials from --use-env-credentials &lt;bool&gt; Whether to read credentials from COURSIER_CREDENTIALS (env) or coursier.credentials (Java property), along those passed with --credentials and --credential-file --quiet | -q &lt;counter&gt; Quiet output --verbose | -v &lt;counter&gt; Increase verbosity (specify several times to increase more) --progress | -P &lt;bool&gt; Force display of progress bars --log-changing &lt;bool&gt; Log changing artifacts --log-channel-version | --log-index-version | --log-jvm-index-version &lt;bool&gt; Log app channel or JVM index version --graalvm-home &lt;string?&gt; --graalvm-option &lt;string*&gt; --graalvm-default-version &lt;string?&gt; --install-dir | --dir &lt;string?&gt; --install-platform &lt;string?&gt; Platform for prebuilt binaries (e.g. &quot;x86_64-pc-linux&quot;, &quot;x86_64-apple-darwin&quot;, &quot;x86_64-pc-win32&quot;) --install-prefer-prebuilt &lt;bool&gt; --only-prebuilt &lt;bool&gt; Require prebuilt artifacts for native applications, don&#39;t try to build native executable ourselves --repository | -r &lt;maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|ivy:$pattern&gt; Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots) --default-repositories &lt;bool&gt; --proguarded &lt;bool?&gt; --channel &lt;org:name&gt; Channel for apps --default-channels &lt;bool&gt; Add default channels --contrib &lt;bool&gt; Add contrib channel --file-channels &lt;bool&gt; Add channels read from the configuration directory --jvm &lt;string?&gt; --jvm-dir &lt;string?&gt; --system-jvm &lt;bool?&gt; --local-only &lt;bool&gt; --update &lt;bool&gt; --jvm-index &lt;string?&gt; --repository | -r &lt;maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|scala-integration|scala-nightlies|ivy:$pattern|jitpack|clojars|jcenter|apache:$repo&gt; Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots) --no-default &lt;bool&gt; Do not add default repositories (~/.ivy2/local, and Central) --sbt-plugin-hack &lt;bool&gt; Modify names in Maven repository paths for sbt plugins --drop-info-attr &lt;bool&gt; Drop module attributes starting with &#39;info.&#39; - these are sometimes used by projects built with sbt --channel &lt;org:name&gt; Channel for apps --default-channels &lt;bool&gt; Add default channels --contrib &lt;bool&gt; Add contrib channel --file-channels &lt;bool&gt; Add channels read from the configuration directory --repository | -r &lt;maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|scala-integration|scala-nightlies|ivy:$pattern|jitpack|clojars|jcenter|apache:$repo&gt; Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots) --no-default &lt;bool&gt; Do not add default repositories (~/.ivy2/local, and Central) --sbt-plugin-hack &lt;bool&gt; Modify names in Maven repository paths for sbt plugins --drop-info-attr &lt;bool&gt; Drop module attributes starting with &#39;info.&#39; - these are sometimes used by projects built with sbt --channel &lt;org:name&gt; Channel for apps --default-channels &lt;bool&gt; Add default channels --contrib &lt;bool&gt; Add contrib channel --file-channels &lt;bool&gt; Add channels read from the configuration directory --env &lt;bool&gt; --disable-env | --disable &lt;bool&gt; --setup &lt;bool&gt; --user-home &lt;string?&gt; --add-channel &lt;string*&gt; (deprecated) --force | -f &lt;bool&gt; </span></pre> <h3 class="numbered" id="s3">Install Scala 3</h3> <p> The Scala 3 compiler and REPL are separate programs: <code>scala3-compiler</code> and <code>scala3-repl</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id03ed3066730b'><button class='copyBtn' data-clipboard-target='#id03ed3066730b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cs install scala3-compiler <span class='unselectable'>https://repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.0.0/scala3-compiler_3-3.0.0.pom 100.0% [##########] 4.8 KiB (79.7 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.0.0/tasty-core_3-3.0.0.pom 100.0% [##########] 3.5 KiB (69.5 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.0.0/scala3-library_3-3.0.0.pom 100.0% [##########] 3.6 KiB (53.9 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.0.0/scala3-interfaces-3.0.0.pom 100.0% [##########] 3.4 KiB (65.9 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.0.0/scala3-interfaces-3.0.0.jar 100.0% [##########] 3.4 KiB (113.9 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.0.0/tasty-core_3-3.0.0.jar 100.0% [##########] 71.9 KiB (192.7 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.0.0/scala3-library_3-3.0.0.jar 100.0% [##########] 1.1 MiB (1.8 MiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.0.0/scala3-compiler_3-3.0.0.jar 100.0% [##########] 14.7 MiB (3.7 MiB / s) Wrote scala3-compiler </span> <span class='unselectable'>$ </span>cs install scala3-repl <span class='unselectable'>https://repo1.maven.org/maven2/io/get-coursier/apps/maven-metadata.xml No new update since 2021-05-14 04:42:19 Wrote scala3-repl </span></pre> <h3 class="numbered" id="repl3">Run Scala 3 REPL</h3> <p> The part is easy! </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcc8e9d989f96'><button class='copyBtn' data-clipboard-target='#idcc8e9d989f96' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>scala3-repl --version <span class='unselectable'>Scala code runner version 3.0.0 -- Copyright 2002-2021, LAMP/EPFL </span> <span class='unselectable'>$ </span>scala3-repl <span class='unselectable'>scala&gt; </span></pre> <h2 id="sbt">Easily Run Scala REPL With SBT</h2> <p> If you do not mind directories called <code>project/</code> and <code>target/</code> being created in your current directory, and you have already <a href='https://www.scala-sbt.org/download.html' target='_blank' rel='nofollow'>installed sbt</a>, you can get a REPL powered by Scala 3 like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id86a5de572361'><button class='copyBtn' data-clipboard-target='#id86a5de572361' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sbt "-Dsbt.version=1.5.2" ++3.0.0! console <span class='unselectable'>[info] welcome to sbt 1.5.2 (Ubuntu Java 11.0.11) [info] loading global plugins from /home/mslinn/.sbt/1.0/plugins [info] loading project definition from /var/work/ancientWarmth/ancientWarmth/project [info] set current project to ancientwarmth (in build file:/var/work/ancientWarmth/ancientWarmth/) [info] Forcing Scala version to 3.0.0 on all projects. [info] Reapplying settings... [info] set current project to ancientwarmth (in build file:/var/work/ancientWarmth/ancientWarmth/) [info] Updating [info] Resolved dependencies [info] Updating https://repo1.maven.org/maven2/org/scala-lang/scaladoc_3/3.0.0/scaladoc_3-3.0.0.pom 100.0% [##########] 6.1 KiB (82.6 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-tasty-inspector_3/3.0.0/scala3-tasty-inspector_3-3.0.0.pom 100.0% [##########] 3.6 KiB (80.8 KiB / s) [info] Resolved dependencies [info] Fetching artifacts of [info] Fetched artifacts of [info] Fetching artifacts of https://repo1.maven.org/maven2/org/scala-lang/scala3-tasty-inspector_3/3.0.0/scala3-tasty-inspector_3-3.0.0.jar 100.0% [##########] 16.6 KiB (338.1 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scaladoc_3/3.0.0/scaladoc_3-3.0.0.jar 100.0% [##########] 1.5 MiB (3.1 MiB / s) [info] Fetched artifacts of scala&gt; </span></pre> <p> Thanks to <a href='https://twitter.com/renghenKornel/status/1395684928440791040' target='_blank' rel='nofollow'>@renghen</a> for this tip. </p> <h2 id="sc">ScalaCourses</h2> <p> If you want to learn how to work effectively with Scala for functional and object-oriented programming, <a href='https://scalacourses.com' target='_blank' rel='nofollow'>ScalaCourses.com</a> is your best option. The course material is suitable for Scala 2 and Scala 3. Visit ScalaCourses.com to learn how to become a proficient Scala programmer. </p> OCI / Docker / AWS Lambda / Django / Buildah / podman 2021-04-29T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/29/buildah-podman-python-lambda <style> body { counter-reset: pcounter; } p.count:before { counter-increment: pcounter; content: counter(pcounter) ")\A0"; } </style> <editor-fold goal> <p> This blog post is a work in progress. Some of it may be incorrect, and some thoughts might lead nowhere. I am publicly posting it in this state so I can discuss it with others. This post will be improved as information becomes available. </p> <h2 id="goal">Goal</h2> <div style="text-align: right;"> <a href="https://podman.io" target="_blank" ><picture> <source srcset="/blog/images/buildahPodman/podman-logo-crop.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/podman-logo-crop.png" type="image/png"> <img src="/blog/images/buildahPodman/podman-logo-crop.png" class="right liImg2 rounded shadow" style="padding: 1em; height: 191px; width: auto;" /> </picture></a> </div> <div style="text-align: right;"> <a href="https://buildah.io/" target="_blank" ><picture> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.png" type="image/png"> <img src="/blog/images/buildahPodman/buildah-logo-crop.png" class="right liImg2 rounded shadow" style="padding: 0.73em; height: 191px; width: auto;" /> </picture></a> </div> <p> <a href='/blog/2021/04/28/buildah-podman.html'>As previously discussed</a>, Buildah is a drop-in replacement for using <code>docker build</code> and a <code>Dockerfile</code>. Buildah’s <code>build-using-dockerfile</code>, or <code>bud</code> argument makes it behave just like <code>docker build</code> does. </p> <p> The goal of this blog post is to use Buildah / <code>podman</code> to create an Open Container Initiative (OCI) container image with a Django app, including the Python 3.8 runtime installed. The Django app will start when the container is created. The code for the Django app will be stored on the local machine where its source code can be edited, and it will be mapped into the container from the host system. Changes made to the code from the host system will be immediately visible inside the container. </p> <h2 id="todo">TODO</h2> <p class="count"> Background: AWS publishes <a href='https://docs.aws.amazon.com/lambda/latest/dg/python-image.html' target='_blank' rel='nofollow'>Deploying Python with an AWS base image</a>, but that does not discuss running or testing. <a href='https://docs.aws.amazon.com/lambda/latest/dg/getting-started-create-function.html' target='_blank' rel='nofollow'>Create a Lambda function with the console</a> is a more complete article, but is focused on using the web browser console, using Docker, and Node.js. So many differences from the desired goal make the articles difficult to translate to AWS CLI, Buildah / <code>podman</code> and Python. </p> <p class="count"> Talk about the <a href='https://github.com/aws/aws-lambda-runtime-interface-emulator' target='_blank' rel='nofollow'>AWS Lambda Runtime Interface Emulator</a>, compare and contrast with the <a href='https://pypi.org/project/awslambdaric/' target='_blank' rel='nofollow'>AWS Lambda Python Runtime Interface Client</a>. </p> <p class="count"> Compare these <a href='https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html' target='_blank' rel='nofollow'>AWS Lambda Runtimes</a> with other, equivalant runtimes. </p> <p class="count"> OCI images are swapped in when AWS Lambda is invoked. Do larger images cost more to use? If so, discuss. </p> </editor-fold> <editor-fold main> <h2 id="main">Deploy Python Lambda function with Container Image</h2> <p> Consider this <code>Dockerfile</code>, which launches a Python 3.8 command-line application in a manner compatible with AWS Lambda: </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,Dockerfile' download='Dockerfile' title='Click on the file name to download the file'>Dockerfile</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id4dc29bb169ef">FROM public.ecr.aws/lambda/python:3.8 COPY app.py ./ CMD ["app.handler"] </pre> <p> Following is a small Python app called <code>app.py</code>, which will be launched by the <code>Dockerfile</code>. The Python app can be run as an AWS Lambda program because it implements the <code>handler</code> entry point. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,app.py' download='app.py' title='Click on the file name to download the file'>app.py</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id3175fef419da">import sys def handler(event, context): return f"Hello from AWS Lambda using Python &#123;sys.version&#125;!" </pre> </editor-fold> <editor-fold build_hello> <h2 id="build">Build image</h2> <p> Buildah builds the image, just the same way that Docker would: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide5d8b8f5754c'><button class='copyBtn' data-clipboard-target='#ide5d8b8f5754c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah bud -t hello . <span class='unselectable'>STEP 1: FROM public.ecr.aws/lambda/python:3.8 Getting image source signatures Copying blob 03ac043af787 skipped: already exists Copying blob 420e64b38334 done Copying blob ff259f25b075 done Copying blob 3ff716981d54 done Copying blob 6b6e623a48a8 done Copying blob 9aa8f1e66d54 done Copying config 67dc3a2a54 done Writing manifest to image destination Storing signatures STEP 2: COPY app.py ./ STEP 3: CMD ["app.handler"] STEP 4: COMMIT hello Getting image source signatures Copying blob 683073d39306 skipped: already exists Copying blob 658871a69e1f skipped: already exists Copying blob 6fa16f35d11e skipped: already exists Copying blob d6fa53d6caa6 skipped: already exists Copying blob 61c062506436 skipped: already exists Copying blob 1c1d66a5fd95 skipped: already exists Copying blob 33af9dc6463a done Copying config 98862dfd20 done Writing manifest to image destination Storing signatures --&gt; 98862dfd208 98862dfd2087152ee821553d6cb1c033e735af06e5f11c814bcc9300fb65584e </span></pre> </editor-fold> <editor-fold deploy_local> <h2 id="deploy_local">Test Lambda function Locally</h2> <p> Before calling the Lambda API from a local container, first run the container. Containers default to running in the foreground, but the <code>-d</code> option causes a container to be run as a background process. This container is given the name <code>hello</code>, the external HTTP endpoint at 9000 is mapped to internal port 8080, and the latest version of the <code>hello</code> lambda function is run in the container. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id06a9a5fb394c'><button class='copyBtn' data-clipboard-target='#id06a9a5fb394c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman run \ -d \ --name hello \ -p 9000:8080 \ hello:latest <span class='unselectable'>d4d296e4c91d01c98d312e3f79599dca53990d95218e94bbdfbbac6a43cde9e8 </span></pre> <p> Call the local version of the Lambda API: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd2853bec1805'><button class='copyBtn' data-clipboard-target='#idd2853bec1805' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl \ -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" \ -d '{}' <span class='unselectable'>"Hello from AWS Lambda using Python 3.8.9 (default, Apr 20 2021, 13:58:54) \n[GCC 7.3.1 20180712 (Red Hat 7.3.1-12)]!" </span></pre> <p> Stop the container called <code>hello</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9610f3b39d3b'><button class='copyBtn' data-clipboard-target='#id9610f3b39d3b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman stop hello <span class='unselectable'>96cc1b1ed92368a1165d6a6ad0b1e5544d4ac751b64e94df33bf2322e6d7b30c </span></pre> </editor-fold> <editor-fold create_repo> <h2 id="podman_tag">Create AWS ECR Repository</h2> <p> AWS provides a registry for OCI-compatible image repositories called the <a href='https://aws.amazon.com/ecr/' target='_blank' rel='nofollow'>AWS Elastic Container Registry (ECR)</a>. </p> <editor-fold create_repo_help> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idec84965bbf2d'><button class='copyBtn' data-clipboard-target='#idec84965bbf2d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ecr create-repository help CREATE-REPOSITORY() CREATE-REPOSITORY()<br/> NAME create-repository -<br/> DESCRIPTION Creates a repository. For more information, see Amazon ECR Repositories in the Amazon Elastic Container Registry User Guide .<br/> See also: AWS API Documentation<br/> See &#39;aws help&#39; for descriptions of global parameters.<br/> SYNOPSIS create-repository --repository-name &lt;value&gt; [--tags &lt;value&gt;] [--image-tag-mutability &lt;value&gt;] [--image-scanning-configuration &lt;value&gt;] [--cli-input-json &lt;value&gt;] [--generate-cli-skeleton &lt;value&gt;]<br/> OPTIONS --repository-name (string) The name to use for the repository. The repository name may be spec- ified on its own (such as nginx-web-app ) or it can be prepended with a namespace to group the repository into a category (such as project-a/nginx-web-app ).<br/> --tags (list) The metadata that you apply to the repository to help you categorize and organize them. Each tag consists of a key and an optional value, both of which you define. Tag keys can have a maximum character length of 128 characters, and tag values can have a maximum length of 256 characters.<br/> (structure) The metadata that you apply to a resource to help you categorize and organize them. Each tag consists of a key and an optional value, both of which you define. Tag keys can have a maximum character length of 128 characters, and tag values can have a maximum length of 256 characters.<br/> Key -&gt; (string) One part of a key-value pair that make up a tag. A key is a general label that acts like a category for more specific tag values.<br/> Value -&gt; (string) The optional part of a key-value pair that make up a tag. A value acts as a descriptor within a tag category (key).<br/> Shorthand Syntax:<br/> Key=string,Value=string ...<br/> JSON Syntax:<br/> [ { &quot;Key&quot;: &quot;string&quot;, &quot;Value&quot;: &quot;string&quot; } ... ]<br/> --image-tag-mutability (string) The tag mutability setting for the repository. If this parameter is omitted, the default setting of MUTABLE will be used which will al- low image tags to be overwritten. If IMMUTABLE is specified, all im- age tags within the repository will be immutable which will prevent them from being overwritten.<br/> Possible values:<br/> o MUTABLE<br/> o IMMUTABLE<br/> --image-scanning-configuration (structure) The image scanning configuration for the repository. This setting determines whether images are scanned for known vulnerabilities af- ter being pushed to the repository.<br/> scanOnPush -&gt; (boolean) The setting that determines whether images are scanned after be- ing pushed to a repository. If set to true , images will be scanned after being pushed. If this parameter is not specified, it will default to false and images will not be scanned unless a scan is manually started with the StartImageScan API.<br/> Shorthand Syntax:<br/> scanOnPush=boolean<br/> JSON Syntax:<br/> { &quot;scanOnPush&quot;: true|false }<br/> --cli-input-json (string) Performs service operation based on the JSON string provided. The JSON string follows the format provided by --gen- erate-cli-skeleton. If other arguments are provided on the command line, the CLI values will override the JSON-provided values. It is not possible to pass arbitrary binary values using a JSON-provided value as the string will be taken literally.<br/> --generate-cli-skeleton (string) Prints a JSON skeleton to standard output without sending an API request. If provided with no value or the value input, prints a sample input JSON that can be used as an argument for --cli-input-json. If provided with the value output, it validates the command inputs and returns a sample output JSON for that command.<br/> See &#39;aws help&#39; for descriptions of global parameters.<br/> EXAMPLES Example 1: To create a repository<br/> The following create-repository example creates a repository inside the specified namespace in the default registry for an account.<br/> aws ecr create-repository \ --repository-name project-a/nginx-web-app<br/> Output:<br/> { &quot;repository&quot;: { &quot;registryId&quot;: &quot;123456789012&quot;, &quot;repositoryName&quot;: &quot;sample-repo&quot;, &quot;repositoryArn&quot;: &quot;arn:aws:ecr:us-west-2:123456789012:repository/project-a/nginx-web-app&quot; } }<br/> For more information, see Creating a Repository in the Amazon ECR User Guide.<br/> Example 2: To create a repository configured with image tag immutabil- ity<br/> The following create-repository example creates a repository configured for tag immutability in the default registry for an account.<br/> aws ecr create-repository \ --repository-name sample-repo \ --image-tag-mutability IMMUTABLE<br/> Output:<br/> { &quot;repository&quot;: { &quot;registryId&quot;: &quot;123456789012&quot;, &quot;repositoryName&quot;: &quot;sample-repo&quot;, &quot;repositoryArn&quot;: &quot;arn:aws:ecr:us-west-2:123456789012:repository/sample-repo&quot;, &quot;imageTagMutability&quot;: &quot;IMMUTABLE&quot; } }<br/> For more information, see Image Tag Mutability in the Amazon ECR User Guide.<br/> Example 3: To create a repository configured with a scanning configura- tion<br/> The following create-repository example creates a repository configured to perform a vulnerability scan on image push in the default registry for an account.<br/> aws ecr create-repository \ --repository-name sample-repo \ --image-scanning-configuration scanOnPush=true<br/> Output:<br/> { &quot;repository&quot;: { &quot;registryId&quot;: &quot;123456789012&quot;, &quot;repositoryName&quot;: &quot;sample-repo&quot;, &quot;repositoryArn&quot;: &quot;arn:aws:ecr:us-west-2:123456789012:repository/sample-repo&quot;, &quot;imageScanningConfiguration&quot;: { &quot;scanOnPush&quot;: true } } }<br/> For more information, see Image Scanning in the Amazon ECR User Guide.<br/> OUTPUT repository -&gt; (structure) The repository that was created.<br/> repositoryArn -&gt; (string) The Amazon Resource Name (ARN) that identifies the repository. The ARN contains the arn:aws:ecr namespace, followed by the re- gion of the repository, AWS account ID of the repository owner, repository namespace, and repository name. For example, arn:aws:ecr:region:012345678910:repository/test .<br/> registryId -&gt; (string) The AWS account ID associated with the registry that contains the repository.<br/> repositoryName -&gt; (string) The name of the repository.<br/> repositoryUri -&gt; (string) The URI for the repository. You can use this URI for Docker push or pull operations.<br/> createdAt -&gt; (timestamp) The date and time, in JavaScript date format, when the reposi- tory was created.<br/> imageTagMutability -&gt; (string) The tag mutability setting for the repository.<br/> imageScanningConfiguration -&gt; (structure) The image scanning configuration for a repository.<br/> scanOnPush -&gt; (boolean) The setting that determines whether images are scanned after being pushed to a repository. If set to true , images will be scanned after being pushed. If this parameter is not speci- fied, it will default to false and images will not be scanned unless a scan is manually started with the StartImageScan API.<br/> <br/> CREATE-REPOSITORY()</pre> </editor-fold> <p> The following creates an AWS ECR image repository in called <code>hello</code> within the <code>test</code> namespace. <a href='https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-scanning.html' target='_blank' rel='nofollow'>Images are scanned</a> for known vulnerabilities after they are pushed to the repository. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9cfffffeccc2'><button class='copyBtn' data-clipboard-target='#id9cfffffeccc2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ecr create-repository \ --repository-name test/hello \ --image-scanning-configuration scanOnPush=true <span class='unselectable'>{ "repository": { "repositoryArn": "arn:aws:ecr:us-east-1:031372724784:repository/test/hello", "registryId": "031372724784", "repositoryName": "test/hello", "repositoryUri": "031372724784.dkr.ecr.us-east-1.amazonaws.com/test/hello", "createdAt": 1620232146.0, "imageTagMutability": "MUTABLE", "imageScanningConfiguration": { "scanOnPush": true } } } </span></pre> </editor-fold> <editor-fold podman_tag> <h2 id="podman_tag">Tag Image</h2> <p class="quote"> <b><code>podman tag</code></b> &ndash; Assigns a new image name to an existing image. A full name refers to the entire image name, including the optional tag after the <code>:</code>. If there is no tag provided, then podman will default to latest for both the image and the target-name. &nbsp; &ndash; From <code>man podman-tag</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5bc59b93e8d8'><button class='copyBtn' data-clipboard-target='#id5bc59b93e8d8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>IMAGE_NAME=hello <span class='unselectable'>$ </span>IMAGE_VERSION=0.1 <span class='unselectable'>$ </span>podman tag $IMAGE_NAME:$IMAGE_VERSION \ $REGISTRY/$IMAGE_NAME:$IMAGE_VERSION <span class='unselectable'>$ </span>podman images <span class='unselectable'>REPOSITORY TAG IMAGE ID CREATED SIZE localhost/hello 0.1 98862dfd2087 39 minutes ago 622 MB 752246127823.dkr.ecr.us-east-1.amazonaws.com/hello latest 98862dfd2087 39 minutes ago 622 MB public.ecr.aws/lambda/python 3.8 67dc3a2a54fb 25 hours ago 622 MB 752246127823.dkr.ecr.us-east-1.amazonaws.com/ancientwarmth latest 5d18ea34fc30 28 hours ago 2.03 GB localhost/ancientwarmth latest 5d18ea34fc30 28 hours ago 2.03 GB &lt;none&gt; &lt;none&gt; 40ef32b39cf4 5 days ago 622 MB docker.io/library/amazonlinux latest 53ef897d731f 5 days ago 170 MB docker.io/amazon/aws-lambda-python 3.8 e12ea62c5582 9 days ago 622 MB docker.io/library/alpine latest 6dbb9cc54074 2 weeks ago 5.88 MB docker.io/lambci/lambda build-python3.8 714c659c9f6f 3 months ago 2.03 GB </span></pre> </editor-fold> <editor-fold push_ecr> <h2 id="push">Push Image to ECR</h2> <p> <code>Podman</code> will use the IAM credentials for the <code>dev</code> <a href='https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html' target='_blank' rel='nofollow'>profile</a> in <code>~/.aws/credentials</code> to log into that AWS account: </p> <div class='codeLabel unselectable' data-lt-active='false'>~/.aws/credentials</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida91691140c8c'><button class='copyBtn' data-clipboard-target='#ida91691140c8c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[default] aws_access_key_id = ******************** aws_secret_access_key = **************************************** region = us-east-1<br> [dev] aws_access_key_id = ******************** aws_secret_access_key = **************************************** region = us-east-1</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfbf68aed3c05'><button class='copyBtn' data-clipboard-target='#idfbf68aed3c05' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>export AWS_PROFILE=dev <span class='unselectable'>$ </span>AWS_ACCOUNT="$( aws sts get-caller-identity \ --query Account \ --output text )" <span class='unselectable'>$ </span>AWS_REGION="$( aws configure get region )" <span class='unselectable'>$ </span>REGISTRY="$AWS_ACCOUNT.dkr.ecr.$AWS_REGION.amazonaws.com" <span class='unselectable'>$ </span>aws ecr get-login-password \ --region "$AWS_REGION" | \ podman login \ --password-stdin \ --username AWS \ "$REGISTRY" <span class='unselectable'>Login Succeeded! </span></pre> <p> Now that <code>podman</code> is logged into AWS, use <code>podman</code> push the image to AWS ECR: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4adf2d664e86'><button class='copyBtn' data-clipboard-target='#id4adf2d664e86' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman push test/$IMAGE_NAME \ $REGISTRY/$IMAGE_NAME:$IMAGE_VERSION <span class='unselectable'>Getting image source signatures Copying blob 692590faf2d1 [--------------------------------------] 8.0b / 8.2MiB Copying blob 397718cff58d [--------------------------------------] 8.0b / 206.2MiB Copying blob 9ca787b1c91c [--------------------------------------] 8.0b / 93.1MiB Copying blob ef26f5221b79 [--------------------------------------] 8.0b / 196.7MiB Copying blob 0a3f69c27a89 [--------------------------------------] 8.0b / 316.4MiB Copying blob 5b3cbb76df75 [--------------------------------------] 8.0b / 1.1GiB Copying blob e9cad39831b0 [--------------------------------------] 8.0b / 3.5KiB Error: Error copying image to the remote destination: Error writing blob: Error initiating layer upload to /v2/ancientwarmth/blobs/uploads/ in 752246127823.dkr.ecr.us-east-1.amazonaws.com: name unknown: The repository with name 'hello' does not exist in the registry with id '752246127823' </span></pre> <p> The results of an <a href='https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ecr/describe-image-scan-findings.html' target='_blank' rel='nofollow'>image scan</a> for the new repository can be retrieved as follows: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6a5683bd975c'><button class='copyBtn' data-clipboard-target='#id6a5683bd975c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ecr describe-image-scan-findings \ --repository-name test/hello \ --image-id imageTag=tag_name</pre> </editor-fold> <editor-fold aw> </editor-fold> <editor-fold buildah_python> <h2 id="buildah_python">Deploy Python Lambda function with Container Image</h2> <p> <code>Podman</code> can invoke the app using an OCI container with Amazon Linux 2 and Python 3.8: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id24af0f6d2e65'><button class='copyBtn' data-clipboard-target='#id24af0f6d2e65' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman container run -ti \ public.ecr.aws/lambda/python:3.8 \ blog/docker/podman/app.py <span class='unselectable'>Trying to pull public.ecr.aws/lambda/python:3.8... Getting image source signatures Copying blob 1de4740de1c2 done Copying blob 03ac043af787 done Copying blob 2e2bb77ae2dc done Copying blob 842c9dce67e8 done Copying blob df513d38f4d9 done Copying blob 031c6369fb2b done Copying config e12ea62c55 done Writing manifest to image destination Storing signatures time="2021-05-02T23:38:30.971" level=info msg="exec '/var/runtime/bootstrap' (cwd=/var/task, handler=)" </span></pre> </editor-fold> Docker, OCI Images, Buildah and podman 2021-04-28T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/28/buildah-podman <editor-fold intro> <p> There are many ways to create and run Docker-compatible images. Docker is probably the worst option, mostly because it runs as a daemon, and all *nix daemons run with <code>root</code> privileges. Also, the <code>docker-ce</code> package lists <code>iptables</code> as a dependency, which needs <code>systemd</code> to be running normally, and WSL2 only partially supports <code>systemd</code>. </p> <p> <a href='https://www.capitalone.com/tech/cloud/container-runtime/' target='_blank' rel='nofollow'>A Comprehensive Container Runtime Comparison</a> provides helpful background information and an interesting historical viewpoint. </p> <h2 id="oci">Open Container Initiative (OCI)</h2> <div style=""> <a href="https://opencontainers.org/" target="_blank" ><picture> <source srcset="/blog/images/buildahPodman/oci_logo.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/oci_logo.png" type="image/png"> <img src="/blog/images/buildahPodman/oci_logo.png" class=" fullsize liImg2 rounded shadow" /> </picture></a> </div> <p> The latest evolution of Docker-compatible images, <a href='https://github.com/opencontainers/image-spec' target='_blank' rel='nofollow'>OCI image format</a> (not to be confused with <a href='https://www.oracle.com/ca-en/cloud/' target='_blank' rel='nofollow'>Oracle Cloud Infrastructure</a>), is compatible with: </p> <ul style="column-count: 2;"> <li><a href='https://aws.amazon.com/lambda/' target='_blank' rel='nofollow'>AWS Lambda</a></li> <li><a href='https://azure.microsoft.com/en-us/services/functions/' target='_blank' rel='nofollow'>Azure Functions</a></li> <li><a href='https://azure.microsoft.com/en-us/services/kubernetes-service/' target='_blank' rel='nofollow'>Azure Kubernetes Service</a></li> <li><a href='https://buildah.io/' target='_blank' rel='nofollow'>Buildah</a></li> <li><a href='https://buildpacks.io/' target='_blank' rel='nofollow'>Cloud Native Buildpacks</a></li> <li><a href='https://circleci.com/' target='_blank' rel='nofollow'>CircleCI</a></li> <li><a href='https://www.docker.com/' target='_blank' rel='nofollow'>Docker</a></li> <li><a href='https://dokku.com/' target='_blank' rel='nofollow'>Dokku</a></li> <li><a href='https://gitlab.com' target='_blank' rel='nofollow'>GitLab</a></li> <li><a href='https://cloud.google.com/container-registry/docs/image-formats' target='_blank' rel='nofollow'>Google Cloud</a></li> <li><a href='https://heroku.com' target='_blank' rel='nofollow'>Heroku</a></li> <li><a href='https://containerjournal.com/topics/container-management/what-is-knative-and-what-can-it-do-for-you/' target='_blank' rel='nofollow'>Knative</a></li> <li><a href='https://kubernetes.io/' target='_blank' rel='nofollow'>Kubernetes</a></li> <li><a href='https://podman.io/' target='_blank' rel='nofollow'><code>podman</code></a></li> <li><a href='https://github.com/containers/skopeo' target='_blank' rel='nofollow'><code>skopeo</code></a></li> <li><a href='https://spring.io/guides/topicals/spring-boot-docker/' target='_blank' rel='nofollow'>Spring Boot</a></li> <li><a href='https://cloud.google.com/tekton' target='_blank' rel='nofollow'>Tekton</a></li> </ul> <p> Supported OCI formats include: </p> <ul style="column-count: 2;"> <li>Docker containers schema 1</li> <li>Docker containers schema 2</li> <li>Pods (groups of containers)</li> <li>Images</li> <li>Volumes</li> </ul> <h2 id="three">Buildah, podman and skopeo</h2> <p> This blog post discusses 3 related open source projects from RedHat / IBM that provide an alternative to Docker: Buildah, <code>podman</code> and <code>skopeo</code>. These 3 projects share a common source code base, and are daemonless tools for managing Open Container Initiative (OCI) images. </p> <p> Paraphrasing the reasons expressed in <a href='https://developers.redhat.com/blog/2019/02/21/podman-and-buildah-for-docker-users/' target='_blank' rel='nofollow'>Podman and Buildah for Docker Users</a> for using <code>podman</code> instead of Docker, wherever <code>podman</code> is mentioned, read &ldquo;<code>podman</code>, Buildah and <code>skopeo</code>&rdquo;: </p> <p class="quoteCite" cite="From &ldquo;Podman and Buildah for Docker Users&rdquo;"> The Podman approach is simply to directly interact with the image registry, with the container and image storage, and with the Linux kernel through the <code>runC</code> container runtime process (not a daemon).<br><br> Running Podman as a normal user means that Podman will, by default, store images and containers in the user’s home directory. Podman uses a repository in the user’s home directory: <code>~/.local/share/containers</code> (instead of <code>/var/lib/docker</code>).<br><br> Despite the new locations for the local repositories, the images created by Docker and Podman are compatible with the OCI standard. Podman can push to and pull from popular container registries like Quay.io and Docker hub, as well as private registries. </p> <h2 id="buildah_vs_podman">Buildah vs. podman</h2> <p> <code>Podman</code> can build OCI containers interactively or in batch mode. You can either build using a <code>Dockerfile</code> using <code>podman build</code> (batch mode), or you can interactively run a container, make changes to the running image, and then <code>podman commit</code> those changes to a new image tag. </p> <p> Buildah was written before <code>podman</code>. Some of Buildah's source code for creating and managing container images was ported to <code>podman</code>. The <code>podman build</code> command is a subset of Buildah&rsquo;s functionality. </p> <p> <p> However, apparently the differences between the two programs are important: </p> <p class="quote"> Buildah builds OCI images. Confusingly, <code>podman build</code> can also be used to build Docker images also, but it’s incredibly slow and used up a lot of disk space by using the <code>vfs</code> storage driver by default. <code>buildah bud</code> (‘build using Dockerfile’) was much faster for me, and uses the overlay storage driver. <br><br> &nbsp; &ndash; From <a href='https://zwischenzugs.com/page/3/' target='_blank' rel='nofollow'>Goodbye Docker: Purging is Such Sweet Sorrow</a> by Ian Miell. </p> </editor-fold> <editor-fold podman> <h2 id="podman">podman</h2> <div style=""> <a href="https://podman.io" target="_blank" ><picture> <source srcset="/blog/images/buildahPodman/podman-logo.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/podman-logo.png" type="image/png"> <img src="/blog/images/buildahPodman/podman-logo.png" class=" liImg2 rounded shadow" style="padding: 1em" /> </picture></a> </div> <p> <code>Podman</code> supports developing, managing, and running OCI Containers on Linux systems, including WSL, without requiring <code>root</code> privilege. </p> <div class='codeLabel unselectable' data-lt-active='false'>shell Installation on Ubuntu</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id99a7ab54aff1'><button class='copyBtn' data-clipboard-target='#id99a7ab54aff1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install buildah podman skopeo</pre> <div class="pullQuote"> Podman commands are very nearly the same as Docker’s. </div> <p> Because <code>podman</code> is a drop-in replacement for <code>docker</code>, the following alias enables the <code>docker</code> command to invoke <code>podman</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>~/.bash_aliases</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id45be05874069'><button class='copyBtn' data-clipboard-target='#id45be05874069' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>alias docker=podman</pre> <p> As described in <a href='https://www.vultr.com/docs/how-to-install-and-use-podman-on-ubuntu-20-04' target='_blank' rel='nofollow'>How to Install and Use Podman on Ubuntu 20.04</a>, I added <code>'registry.access.redhat.com'</code> to the list of <code>registries</code> in <code>/etc/containers/registries.conf</code>. I also added <a href='https://gallery.ecr.aws/' target='_blank' rel='nofollow'><code>'gallery.ecr.aws'</code></a> and <a href='https://cloud.google.com/container-registry/docs/pushing-and-pulling#add-registry' target='_blank' rel='nofollow'><code>'gcr.io'</code></a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/containers/registries.conf</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id941d5fd200c4'><button class='copyBtn' data-clipboard-target='#id941d5fd200c4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[registries.search] registries = ['docker.io', 'gallery.ecr.aws', 'gcr.io', 'quay.io', 'registry.access.redhat.com']</pre> </editor-fold> <editor-fold help> <h3 id="podmanHelp"><span class="code">podman</span> Help</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8b3eb9dc1323'><button class='copyBtn' data-clipboard-target='#id8b3eb9dc1323' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman --help <span class='unselectable'>Manage pods, containers and images<br/> Usage: podman [flags] podman [command]<br/> Available Commands: attach Attach to a running container auto-update Auto update containers according to their auto-update policy build Build an image using instructions from Containerfiles commit Create new image based on the changed container container Manage containers cp Copy files/folders between a container and the local filesystem create Create but do not start a container diff Display the changes to the object&#39;s file system events Show podman events exec Run a process in a running container export Export container&#39;s filesystem contents as a tar archive generate Generate structured data based on containers and pods. healthcheck Manage health checks on containers help Help about any command history Show history of a specified image image Manage images images List images in local storage import Import a tarball to create a filesystem image info Display podman system information init Initialize one or more containers inspect Display the configuration of object denoted by ID kill Kill one or more running containers with a specific signal load Load an image from container archive login Login to a container registry logout Logout of a container registry logs Fetch the logs of one or more containers manifest Manipulate manifest lists and image indexes mount Mount a working container&#39;s root filesystem network Manage networks pause Pause all the processes in one or more containers play Play a pod and its containers from a structured file. pod Manage pods port List port mappings or a specific mapping for the container ps List containers pull Pull an image from a registry push Push an image to a specified destination restart Restart one or more containers rm Remove one or more containers rmi Removes one or more images from local storage run Run a command in a new container save Save image to an archive search Search registry for image start Start one or more containers stats Display a live stream of container resource usage statistics stop Stop one or more containers system Manage podman tag Add an additional name to a local image top Display the running processes of a container unmount Unmounts working container&#39;s root filesystem unpause Unpause the processes in one or more containers unshare Run a command in a modified user namespace untag Remove a name from a local image version Display the Podman Version Information volume Manage volumes wait Block on one or more containers<br/> Flags: --cgroup-manager string Cgroup manager to use (&quot;cgroupfs&quot;|&quot;systemd&quot;) (default &quot;cgroupfs&quot;) --cni-config-dir string Path of the configuration directory for CNI networks --conmon string Path of the conmon binary -c, --connection string Connection to use for remote Podman service --events-backend string Events backend to use (&quot;file&quot;|&quot;journald&quot;|&quot;none&quot;) (default &quot;file&quot;) --help Help for podman --hooks-dir strings Set the OCI hooks directory path (may be set multiple times) (default [/usr/share/containers/oci/hooks.d]) --identity string path to SSH identity file, (CONTAINER_SSHKEY) --log-level string Log messages above specified level (debug, info, warn, error, fatal, panic) (default &quot;error&quot;) --namespace string Set the libpod namespace, used to create separate views of the containers and pods on the system --network-cmd-path string Path to the command for configuring the network -r, --remote Access remote Podman service (default false) --root string Path to the root directory in which data, including images, is stored --runroot string Path to the &#39;run directory&#39; where all state information is stored --runtime string Path to the OCI-compatible binary used to run containers, default is /usr/bin/runc --storage-driver string Select which storage driver is used to manage storage of images and containers (default is overlay) --storage-opt stringArray Used to pass an option to the storage driver --syslog Output logging information to syslog as well as the console (default false) --tmpdir string Path to the tmp directory for libpod state content. Note: use the environment variable &#39;TMPDIR&#39; to change the temporary storage location for container images, &#39;/var/tmp&#39;. --url string URL to access Podman service (CONTAINER_HOST) (default &quot;unix:/home/mslinn/.docker/run/podman/podman.sock&quot;) -v, --version Version of Podman<br/> Use &quot;podman [command] --help&quot; for more information about a command. </span></pre> </editor-fold> <editor-fold padman_info> <h3 id="padman_info"><span class="code">podman info</span></h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id061e3e15351f'><button class='copyBtn' data-clipboard-target='#id061e3e15351f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman info <span class='unselectable'>host: arch: amd64 buildahVersion: 1.15.2 cgroupVersion: v1 conmon: package: &#39;conmon: /usr/libexec/podman/conmon&#39; path: /usr/libexec/podman/conmon version: &#39;conmon version 2.0.20, commit: unknown&#39; cpus: 8 distribution: distribution: ubuntu version: &quot;20.10&quot; eventLogger: file hostname: Bear idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 kernel: 5.4.72-microsoft-standard-WSL2 linkmode: dynamic memFree: 897724416 memTotal: 6231638016 ociRuntime: name: runc package: &#39;containerd.io: /usr/bin/runc&#39; path: /usr/bin/runc version: |- runc version 1.0.0-rc93 commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec spec: 1.0.2-dev go: go1.13.15 libseccomp: 2.5.1 os: linux remoteSocket: path: /home/mslinn/.docker/run/podman/podman.sock rootless: true slirp4netns: executable: /bin/slirp4netns package: Unknown version: |- slirp4netns version 1.0.1 commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4 libslirp: 4.3.1 swapFree: 0 swapTotal: 0 uptime: 306h 23m 6.37s (Approximately 12.75 days) registries: search: - quay.io - docker.io - gallery.ecr.aws - registry.access.redhat.com store: configFile: /home/mslinn/.config/containers/storage.conf containerStore: number: 8 paused: 0 running: 0 stopped: 8 graphDriverName: overlay graphOptions: overlay.mount_program: Executable: /bin/fuse-overlayfs Package: Unknown Version: |- fusermount3 version: 3.9.3 fuse-overlayfs: version 1.0.0 FUSE library version 3.9.3 using FUSE kernel interface version 7.31 graphRoot: /home/mslinn/.local/share/containers/storage graphStatus: Backing Filesystem: extfs Native Overlay Diff: &quot;false&quot; Supports d_type: &quot;true&quot; Using metacopy: &quot;false&quot; imageStore: number: 4 runRoot: /home/mslinn/.docker/run/containers volumePath: /home/mslinn/.local/share/containers/storage/volumes version: APIVersion: 1 Built: 0 BuiltTime: Wed Dec 31 19:00:00 1969 GitCommit: &quot;&quot; GoVersion: go1.14.7 OsArch: linux/amd64 Version: 2.0.6 </span></pre> </editor-fold> <editor-fold padman_container_help> <h3 id="padman_container_help"><span class="code">podman container</span> Help</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id960a903880f7'><button class='copyBtn' data-clipboard-target='#id960a903880f7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>man podman-container <span class='unselectable'>podman-container(1) General Commands Manual podman-container(1)<br/> NAME podman-container - Manage containers<br/> SYNOPSIS podman container subcommand<br/> DESCRIPTION The container command allows you to manage containers<br/> COMMANDS &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9516;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9516;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488; &#9474;Command &#9474; Man Page &#9474; Description &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;attach &#9474; podman-attach(1) &#9474; Attach to a running container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;checkpoint &#9474; podman-container-checkpoint(1) &#9474; Checkpoints one or more running &#9474; &#9474; &#9474; &#9474; containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;cleanup &#9474; podman-container-cleanup(1) &#9474; Cleanup the container&#39;s network &#9474; &#9474; &#9474; &#9474; and mountpoints. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;commit &#9474; podman-commit(1) &#9474; Create new image based on the &#9474; &#9474; &#9474; &#9474; changed container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;cp &#9474; podman-cp(1) &#9474; Copy files/folders between a &#9474; &#9474; &#9474; &#9474; container and the local &#9474; &#9474; &#9474; &#9474; filesystem. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;create &#9474; podman-create(1) &#9474; Create a new container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;diff &#9474; podman-diff(1) &#9474; Inspect changes on a container or &#9474; &#9474; &#9474; &#9474; image&#39;s filesystem. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;exec &#9474; podman-exec(1) &#9474; Execute a command in a running &#9474; &#9474; &#9474; &#9474; container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;exists &#9474; podman-container-exists(1) &#9474; Check if a container exists in &#9474; &#9474; &#9474; &#9474; local storage &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;export &#9474; podman-export(1) &#9474; Export a container&#39;s filesystem &#9474; &#9474; &#9474; &#9474; contents as a tar archive. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;init &#9474; podman-init(1) &#9474; Initialize a container &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;inspect &#9474; podman-inspect(1) &#9474; Display a container or image&#39;s &#9474; &#9474; &#9474; &#9474; configuration. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;kill &#9474; podman-kill(1) &#9474; Kill the main process in one or &#9474; &#9474; &#9474; &#9474; more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;list &#9474; podman-ps(1) &#9474; List the containers on the &#9474; &#9474; &#9474; &#9474; system.(alias ls) &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;logs &#9474; podman-logs(1) &#9474; Display the logs of a container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;mount &#9474; podman-mount(1) &#9474; Mount a working container&#39;s root &#9474; &#9474; &#9474; &#9474; filesystem. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;pause &#9474; podman-pause(1) &#9474; Pause one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;port &#9474; podman-port(1) &#9474; List port mappings for the &#9474; &#9474; &#9474; &#9474; container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;prune &#9474; podman-container-prune(1) &#9474; Remove all stopped containers &#9474; &#9474; &#9474; &#9474; from local storage. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;restart &#9474; podman-restart(1) &#9474; Restart one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;restore &#9474; podman-container-restore(1) &#9474; Restores one or more containers &#9474; &#9474; &#9474; &#9474; from a checkpoint. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;rm &#9474; podman-rm(1) &#9474; Remove one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;run &#9474; podman-run(1) &#9474; Run a command in a container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;runlabel &#9474; podman-container-runlabel(1) &#9474; Executes a command as described &#9474; &#9474; &#9474; &#9474; by a container image label. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;start &#9474; podman-start(1) &#9474; Starts one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;stats &#9474; podman-stats(1) &#9474; Display a live stream of one or &#9474; &#9474; &#9474; &#9474; more container&#39;s resource usage &#9474; &#9474; &#9474; &#9474; statistics. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;stop &#9474; podman-stop(1) &#9474; Stop one or more running &#9474; &#9474; &#9474; &#9474; containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;top &#9474; podman-top(1) &#9474; Display the running processes of &#9474; &#9474; &#9474; &#9474; a container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;unmount &#9474; podman-unmount(1) &#9474; Unmount a working container&#39;s &#9474; &#9474; &#9474; &#9474; root filesystem.(Alias unmount) &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;unpause &#9474; podman-unpause(1) &#9474; Unpause one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;wait &#9474; podman-wait(1) &#9474; Wait on one or more containers to &#9474; &#9474; &#9474; &#9474; stop and print their exit codes. &#9474; &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9524;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9524;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;<br/> SEE ALSO podman, podman-exec, podman-run<br/> podman-container(1) </span></pre> </editor-fold> <editor-fold podman_run_help> <h3 id="podman_help">Podman Run Help</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5dd1d1778a5c'><button class='copyBtn' data-clipboard-target='#id5dd1d1778a5c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman container run --help <span class='unselectable'>Run a command in a new container<br/> Description: Runs a command in a new container from the given image<br/> Usage: podman container run [flags] IMAGE [COMMAND [ARG...]]<br/> Examples: podman container run imageID ls -alF /etc podman container run --network=host imageID dnf -y install java podman container run --volume /var/hostdir:/var/ctrdir -i -t fedora /bin/bash<br/> Flags: --add-host strings Add a custom host-to-IP mapping (host:ip) (default []) --annotation strings Add annotations to container (key:value) -a, --attach strings Attach to STDIN, STDOUT or STDERR --authfile string Path of the authentication file. Use REGISTRY_AUTH_FILE environment variable to override --blkio-weight string Block IO weight (relative weight) accepts a weight value between 10 and 1000. --blkio-weight-device DEVICE_NAME:WEIGHT Block IO weight (relative device weight, format: DEVICE_NAME:WEIGHT) --cap-add strings Add capabilities to the container --cap-drop strings Drop capabilities from the container --cgroup-parent string Optional parent cgroup for the container --cgroupns string cgroup namespace to use --cgroups string control container cgroup configuration (&quot;enabled&quot;|&quot;disabled&quot;|&quot;no-conmon&quot;) (default &quot;enabled&quot;) --cidfile string Write the container ID to the file --conmon-pidfile string Path to the file that will receive the PID of conmon --cpu-period uint Limit the CPU CFS (Completely Fair Scheduler) period --cpu-quota int Limit the CPU CFS (Completely Fair Scheduler) quota --cpu-rt-period uint Limit the CPU real-time period in microseconds --cpu-rt-runtime int Limit the CPU real-time runtime in microseconds --cpu-shares uint CPU shares (relative weight) --cpus float Number of CPUs. The default is 0.000 which means no limit --cpuset-cpus string CPUs in which to allow execution (0-3, 0,1) --cpuset-mems string Memory nodes (MEMs) in which to allow execution (0-3, 0,1). Only effective on NUMA systems. -d, --detach Run container in background and print container ID --detach-keys [a-Z] Override the key sequence for detaching a container. Format is a single character [a-Z] or a comma separated sequence of `ctrl-&lt;value&gt;`, where `&lt;value&gt;` is one of: `a-cf`, `@`, `^`, `[`, `\`, `]`, `^` or `_` (default &quot;ctrl-p,ctrl-q&quot;) --device strings Add a host device to the container --device-cgroup-rule strings Add a rule to the cgroup allowed devices list --device-read-bps strings Limit read rate (bytes per second) from a device (e.g. --device-read-bps=/dev/sda:1mb) --device-read-iops strings Limit read rate (IO per second) from a device (e.g. --device-read-iops=/dev/sda:1000) --device-write-bps strings Limit write rate (bytes per second) to a device (e.g. --device-write-bps=/dev/sda:1mb) --device-write-iops strings Limit write rate (IO per second) to a device (e.g. --device-write-iops=/dev/sda:1000) --disable-content-trust This is a Docker specific option and is a NOOP --dns strings Set custom DNS servers --dns-opt strings Set custom DNS options --dns-search strings Set custom DNS search domains --entrypoint string Overwrite the default ENTRYPOINT of the image -e, --env stringArray Set environment variables in container (default [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,TERM=xterm]) --env-file strings Read in a file of environment variables --env-host Use all current host environment variables in container --expose strings Expose a port or a range of ports --gidmap strings GID map to use for the user namespace --group-add strings Add additional groups to join --health-cmd string set a healthcheck command for the container (&#39;none&#39; disables the existing healthcheck) --health-interval string set an interval for the healthchecks (a value of disable results in no automatic timer setup) (default &quot;30s&quot;) --health-retries uint the number of retries allowed before a healthcheck is considered to be unhealthy (default 3) --health-start-period string the initialization time needed for a container to bootstrap (default &quot;0s&quot;) --health-timeout string the maximum time allowed to complete the healthcheck before an interval is considered failed (default &quot;30s&quot;) -h, --hostname string Set container hostname --http-proxy Set proxy environment variables in the container based on the host proxy vars (default true) --image-volume string Tells podman how to handle the builtin image volumes (&quot;bind&quot;|&quot;tmpfs&quot;|&quot;ignore&quot;) (default &quot;bind&quot;) --init Run an init binary inside the container that forwards signals and reaps processes --init-path string Path to the container-init binary -i, --interactive Keep STDIN open even if not attached --ip string Specify a static IPv4 address for the container --ipc string IPC namespace to use --kernel-memory &lt;number&gt;[&lt;unit&gt;] Kernel memory limit (format: &lt;number&gt;[&lt;unit&gt;], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes)) -l, --label stringArray Set metadata on container --label-file strings Read in a line delimited file of labels --log-driver string Logging driver for the container --log-opt strings Logging driver options --mac-address string Container MAC address (e.g. 92:d0:c6:0a:29:33) -m, --memory &lt;number&gt;[&lt;unit&gt;] Memory limit (format: &lt;number&gt;[&lt;unit&gt;], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes)) --memory-reservation &lt;number&gt;[&lt;unit&gt;] Memory soft limit (format: &lt;number&gt;[&lt;unit&gt;], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes)) --memory-swap string Swap limit equal to memory plus swap: &#39;-1&#39; to enable unlimited swap --memory-swappiness int Tune container memory swappiness (0 to 100, or -1 for system default) (default -1) --mount stringArray Attach a filesystem mount to the container --name string Assign a name to the container --network string Connect a container to a network (default &quot;slirp4netns&quot;) --no-healthcheck Disable healthchecks on container --no-hosts Do not create /etc/hosts within the container, instead use the version from the image --oom-kill-disable Disable OOM Killer --oom-score-adj int Tune the host&#39;s OOM preferences (-1000 to 1000) --pid string PID namespace to use --pids-limit int Tune container pids limit (set 0 for unlimited, -1 for server defaults) --pod string Run container in an existing pod --pod-id-file string Read the pod ID from the file --privileged Give extended privileges to container -p, --publish strings Publish a container&#39;s port, or a range of ports, to the host (default []) -P, --publish-all Publish all exposed ports to random ports on the host interface --pull string Pull image before creating (&quot;always&quot;|&quot;missing&quot;|&quot;never&quot;) (default &quot;missing&quot;) -q, --quiet Suppress output information when pulling images --read-only Make containers root filesystem read-only --read-only-tmpfs When running containers in read-only mode mount a read-write tmpfs on /run, /tmp and /var/tmp (default true) --replace If a container with the same name exists, replace it --restart string Restart policy to apply when a container exits (&quot;always&quot;|&quot;no&quot;|&quot;on-failure&quot;) --rm Remove container (and pod if created) after exit --rmi Remove container image unless used by other containers --rootfs The first argument is not an image but the rootfs to the exploded container --seccomp-policy string Policy for selecting a seccomp profile (experimental) (default &quot;default&quot;) --security-opt stringArray Security Options --shm-size &lt;number&gt;[&lt;unit&gt;] Size of /dev/shm (format: &lt;number&gt;[&lt;unit&gt;], where unit = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes)) (default &quot;65536k&quot;) --sig-proxy Proxy received signals to the process (default true) --stop-signal string Signal to stop a container. Default is SIGTERM --stop-timeout uint Timeout (in seconds) to stop a container. Default is 10 (default 10) --subgidname string Name of range listed in /etc/subgid for use in user namespace --subuidname string Name of range listed in /etc/subuid for use in user namespace --sysctl strings Sysctl options --systemd string Run container in systemd mode (&quot;true&quot;|&quot;false&quot;|&quot;always&quot;) (default &quot;true&quot;) --tmpfs tmpfs Mount a temporary filesystem (tmpfs) into a container -t, --tty Allocate a pseudo-TTY for container --uidmap strings UID map to use for the user namespace --ulimit strings Ulimit options -u, --user string Username or UID (format: &lt;name|uid&gt;[:&lt;group|gid&gt;]) --userns string User namespace to use --uts string UTS namespace to use -v, --volume stringArray Bind mount a volume into the container --volumes-from strings Mount volumes from the specified container(s) -w, --workdir string Working directory inside the container </span></pre> </editor-fold> <editor-fold podman_run> <h2 id="podman_run">podman run</h2> <p> From <a href='https://chariotsolutions.com/blog/post/building-and-deploying-lambdas-from-a-docker-container/' target='_blank' rel='nofollow'>Building and Deploying Lambdas from a Docker Container</a> by Keith Gregory: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idccd5dce25820'><button class='copyBtn' data-clipboard-target='#idccd5dce25820' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman run \ -it \ --entrypoint /bin/bash \ --rm \ -v /tmp:/mnt \ amazon/aws-lambda-python:3.8 <span class='unselectable'>Trying to pull quay.io/amazon/aws-lambda-python:3.8... Requesting bear token: invalid status code from registry 405 (Method Not Allowed) Trying to pull docker.io/amazon/aws-lambda-python:3.8... Getting image source signatures Copying blob df513d38f4d9 skipped: already exists Copying blob 2e2bb77ae2dc skipped: already exists Copying blob 031c6369fb2b skipped: already exists Copying blob 03ac043af787 skipped: already exists Copying blob 842c9dce67e8 skipped: already exists Copying blob 1de4740de1c2 [--------------------------------------] 0.0b / 0.0b Copying config e12ea62c55 done Writing manifest to image destination Storing signatures bash-4.2# </span>pwd <span class='unselectable'>/var/task </span></pre> <h2 id="cleanup">Cleaning Up a Container</h2> <p> <code>podman container cleanup</code> is a good command to know about. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id06b986a64536'><button class='copyBtn' data-clipboard-target='#id06b986a64536' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman container cleanup --help <span class='unselectable'>Cleanup network and mountpoints of one or more containers Description: podman container cleanup Cleans up mount points and network stacks on one or more containers from the host. The container name or ID can be used. This command is used internally when running containers, but can also be used if container cleanup has failed when a container exits. Usage: podman container cleanup [options] CONTAINER [CONTAINER...] Examples: podman container cleanup --latest podman container cleanup ctrID1 ctrID2 ctrID3 podman container cleanup --all Options: -a, --all Cleans up all containers --exec string Clean up the given exec session instead of the container -l, --latest Act on the latest container podman is aware of Not supported with the "--remote" flag --rm After cleanup, remove the container entirely --rmi After cleanup, remove the image entirely </span></pre> </editor-fold> <editor-fold buildah> <h2 id="builah">Buildah</h2> <div style=""> <a href="https://buildah.io/" target="_blank" ><picture> <source srcset="/blog/images/buildahPodman/buildah-logo.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/buildah-logo.png" type="image/png"> <img src="/blog/images/buildahPodman/buildah-logo.png" class=" liImg2 rounded shadow" style="padding: 1em" /> </picture></a> </div> <p> <a href='https://buildah.io/' target='_blank' rel='nofollow'>Buildah</a> is a drop-in replacement for using <code>docker build</code> and a <code>Dockerfile</code>. </p> <div class="quote"> Where Buildah really shines is in its native commands, which you can use to interact with container builds. Rather than using <code>build-using-dockerfile/bud</code> for each build, Buildah has commands to actually interact with the temporary container created during the build process. (Docker uses temporary, or intermediate containers, too, but you don’t really interact with them while the image is being built.) <br><br> Unlike <code>docker build</code>, Buildah doesn’t commit changes to a layer automatically for every instruction in the <code>Dockerfile</code> &ndash; it builds everything from top to bottom, every time. On the positive side, this means non-cached builds (for example, those you would do with automation or build pipelines) end up being somewhat faster than their Docker build counterparts, especially if there are many instructions. <br><br> &nbsp; &ndash; From <a href='https://opensource.com/article/18/6/getting-started-buildah' target='_blank' rel='nofollow'>Getting started with Buildah.</a>, published by <code>opensource.com</code> </div> <p> Some key Buildah subcommands: </p> <dl> <dt class="code">buildah bud</dt> <dd>Buildah’s <code>build-using-dockerfile</code>, or <code>bud</code> argument makes it behave just like <code>docker build</code> does.</dd> <dt class="code">buildah from</dt> <dd>Build up a container root filesystem from an image or from scratch.</dd> <dt class="code">buildah config</dt> <dd>Adjust defaults in the image's configuration blob.</dd> <dt class="code">buildah run</dt> <dd> <code>buildah run</code> is for running commands that build a container image. This is similar to <code>RUN</code> in a <code>Dockerfile</code>, and unlike <code>docker run</code>. </dd> <dt class="code">buildah commit</dt> <dd>Commit changes to the container to a new image.</dd> <dt class="code">buildah push</dt> <dd>Push images to registries (such a Quay) or a local <code>dockerd</code> instance.</dd> <dt class="code"></dt> <dd></dd> <dt class="code"></dt> <dd></dd> <dt class="code"></dt> <dd></dd> </dl> <h3 id="buildahHelp">Buildah Help</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1b63d8f64a08'><button class='copyBtn' data-clipboard-target='#id1b63d8f64a08' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah -h <span class='unselectable'>A tool that facilitates building OCI images<br/> Usage: buildah [flags] buildah [command]<br/> Available Commands: add Add content to the container build-using-dockerfile Build an image using instructions in a Dockerfile commit Create an image from a working container config Update image configuration settings containers List working containers and their base images copy Copy content into the container from Create a working container based on an image help Help about any command images List images in local storage info Display Buildah system information inspect Inspect the configuration of a container or image login Login to a container registry logout Logout of a container registry manifest Manipulate manifest lists and image indexes mount Mount a working container&#39;s root filesystem pull Pull an image from the specified location push Push an image to a specified destination rename Rename a container rm Remove one or more working containers rmi Remove one or more images from local storage run Run a command inside of the container tag Add an additional name to a local image umount Unmount the root file system of the specified working containers unshare Run a command in a modified user namespace version Display the Buildah version information<br/> Flags: -h, --help help for buildah --log-level string The log level to be used. Either &quot;debug&quot;, &quot;info&quot;, &quot;warn&quot; or &quot;error&quot;. (default &quot;error&quot;) --registries-conf string path to registries.conf file (not usually used) --registries-conf-dir string path to registries.conf.d directory (not usually used) --root string storage root dir (default &quot;/var/lib/containers/storage&quot;) --runroot string storage state dir (default &quot;/var/run/containers/storage&quot;) --storage-driver string storage-driver --storage-opt strings storage driver option --userns-gid-map ctrID:hostID:length default ctrID:hostID:length GID mapping to use --userns-uid-map ctrID:hostID:length default ctrID:hostID:length UID mapping to use -v, --version version for buildah<br/> Use &quot;buildah [command] --help&quot; for more information about a command. </span></pre> </editor-fold> <editor-fold buildah_use> <h3 id="buildahUse">Buildah / <span class="code">Dockerfile</span> Compatibility</h3> <div style=""> <picture> <source srcset="/blog/images/buildahPodman/whales.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/whales.png" type="image/png"> <img src="/blog/images/buildahPodman/whales.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <p> <a href='https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/building_running_and_managing_containers/building-container-images-with-buildah_porting-containers-to-systemd-using-podman' target='_blank' rel='nofollow'>Buildah</a> can create an image from a Dockerfile by typing: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc69d304fd0d2'><button class='copyBtn' data-clipboard-target='#idc69d304fd0d2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah bud -t hello .</pre> <p> &hellip;instead of: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idde674f4e0afe'><button class='copyBtn' data-clipboard-target='#idde674f4e0afe' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker build -t hello .</pre> <p> Buildah can create an image called <code>hello</code> from the <code>Dockerfile</code> and the Python app by typing: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id450a9886fe8e'><button class='copyBtn' data-clipboard-target='#id450a9886fe8e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah bud -t hello . <span class='unselectable'>STEP 1: FROM public.ecr.aws/lambda/python:3.8 Getting image source signatures Copying blob 1de4740de1c2 done Copying blob 2e2bb77ae2dc done Copying blob df513d38f4d9 done Copying blob 03ac043af787 done Copying blob 031c6369fb2b done Copying blob 842c9dce67e8 done Copying config e12ea62c55 done Writing manifest to image destination Storing signatures STEP 2: COPY app.py ./ STEP 3: CMD [&quot;app.handler&quot;] STEP 4: COMMIT hello Getting image source signatures Copying blob 109f575f8e6a skipped: already exists Copying blob ff64b4f854ad skipped: already exists Copying blob dd66ad8702f4 skipped: already exists Copying blob d6fa53d6caa6 skipped: already exists Copying blob 80166c3283e5 skipped: already exists Copying blob 61f74564c3aa skipped: already exists Copying blob d95ebdc79761 done Copying config 40ef32b39c done Writing manifest to image destination Storing signatures --&gt; 40ef32b39cf 40ef32b39cf4ffd3d2e4e3426bec4a5ea168524f7f3fcfe863a378abd9794270 </span></pre> <p> Once the build is complete, the new image can be displayed with the <code>buildah images</code> command: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfec5dda5000c'><button class='copyBtn' data-clipboard-target='#idfec5dda5000c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah images <span class='unselectable'>REPOSITORY TAG IMAGE ID CREATED SIZE localhost/hello latest 40ef32b39cf4 56 seconds ago 622 MB </span></pre> <p> The new image, tagged <code>hello:latest</code>, can be pushed to a remote image registry. This is easily accomplished with the <code>buildah push</code> command. </p> </editor-fold> <editor-fold buildah_push> <h3 id="buildah_push"><span class="code">buildah push</span> Help</h3> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id55cc57949e3d'><button class='copyBtn' data-clipboard-target='#id55cc57949e3d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>man buildah-push <span class='unselectable'>buildah-push(1) General Commands Manual buildah-push(1)<br/> NAME buildah-push - Push an image from local storage to elsewhere.<br/> SYNOPSIS buildah push [options] image [destination]<br/> DESCRIPTION Pushes an image from local storage to a specified destination, decompressing and recompessing layers as needed.<br/> imageID Image stored in local container/storage<br/> DESTINATION The DESTINATION is a location to store container images. If omitted, the source image parameter will be reused as destination.<br/> The Image &quot;DESTINATION&quot; uses a &quot;transport&quot;:&quot;details&quot; format. Multiple transports are supported:<br/> dir:path An existing local directory path storing the manifest, layer tarballs and signatures as individual files. This is a non-standardized format, primarily useful for debugging or noninvasive container inspection.<br/> docker://docker-reference An image in a registry implementing the &quot;Docker Registry HTTP API V2&quot;. By default, uses the authorization state in $XDG\_RUNTIME\_DIR/containers/auth.json, which is set using (buildah login). If the authorization state is not found there, $HOME/.docker/config.json is checked, which is set using (docker login). If docker-reference does not include a registry name, the image will be pushed to a registry running on local&#8208; host.<br/> docker-archive:path[:docker-reference] An image is stored in the docker save formatted file. docker-reference is only used when creating such a file, and it must not contain a digest.<br/> docker-daemon:docker-reference An image _dockerreference stored in the docker daemon internal storage. If _dockerreference does not begin with a valid registry name (a domain name containing &quot;.&quot; or the reserved name &quot;localhost&quot;) then the default registry name &quot;docker.io&quot; will be prepended. _dockerreference must contain either a tag or a digest. Alternatively, when reading images, the format can also be docker-daemon:algo:digest (an image ID).<br/> oci:path:tag An image tag in a directory compliant with &quot;Open Container Image Layout Specification&quot; at path.<br/> oci-archive:path:tag An image tag in a tar archive compliant with &quot;Open Container Image Layout Specification&quot; at path.<br/> If the transport part of DESTINATION is omitted, &quot;docker://&quot; is assumed.<br/> OPTIONS --authfile path<br/> Path of the authentication file. Default is ${XDG_RUNTIME_DIR}/containers/auth.json, which is set using buildah lo&#8208; gin. If the authorization state is not found there, $HOME/.docker/config.json is checked, which is set using docker login.<br/> --cert-dir path<br/> Use certificates at path (*.crt, *.cert, *.key) to connect to the registry. Default certificates directory is /etc/containers/certs.d.<br/> --creds creds<br/> The [username[:password]] to use to authenticate with the registry if required. If one or both values are not sup&#8208; plied, a command line prompt will appear and the value can be entered. The password is entered without echo.<br/> --digestfile Digestfile<br/> After copying the image, write the digest of the resulting image to the file.<br/> --disable-compression, -D<br/> Don&#39;t compress copies of filesystem layers which will be pushed.<br/> --encryption-key key<br/> The [protocol:keyfile] specifies the encryption protocol, which can be JWE (RFC7516), PGP (RFC4880), and PKCS7 (RFC2315) and the key material required for image encryption. For instance, jwe:/path/to/key.pem or pgp:admin@exam&#8208; ple.com or pkcs7:/path/to/x509-file.<br/> --format, -f<br/> Manifest Type (oci, v2s1, or v2s2) to use when saving image to directory using the &#39;dir:&#39; transport (default is manifest type of source)<br/> --quiet, -q<br/> When writing the output image, suppress progress output.<br/> --remove-signatures<br/> Don&#39;t copy signatures when pushing images.<br/> --sign-by fingerprint<br/> Sign the pushed image using the GPG key that matches the specified fingerprint.<br/> --tls-verify bool-value<br/> Require HTTPS and verify certificates when talking to container registries (defaults to true)<br/> EXAMPLE This example pushes the image specified by the imageID to a local directory in docker format.<br/> # buildah push imageID dir:/path/to/image<br/> This example pushes the image specified by the imageID to a local directory in oci format.<br/> # buildah push imageID oci:/path/to/layout:image:tag<br/> This example pushes the image specified by the imageID to a tar archive in oci format.<br/> # buildah push imageID oci-archive:/path/to/archive:image:tag<br/> This example pushes the image specified by the imageID to a container registry named registry.example.com.<br/> # buildah push imageID docker://registry.example.com/repository:tag<br/> This example pushes the image specified by the imageID to a container registry named registry.example.com and saves the digest in the specified digestfile.<br/> # buildah push --digestfile=/tmp/mydigest imageID docker://registry.example.com/repository:tag<br/> This example works like docker push, assuming registry.example.com/my_image is a local image.<br/> # buildah push registry.example.com/my_image<br/> This example pushes the image specified by the imageID to a private container registry named registry.example.com with authentication from /tmp/auths/myauths.json.<br/> # buildah push --authfile /tmp/auths/myauths.json imageID docker://registry.example.com/repository:tag<br/> This example pushes the image specified by the imageID and puts into the local docker container store.<br/> # buildah push imageID docker-daemon:image:tag<br/> This example pushes the image specified by the imageID and puts it into the registry on the localhost while turning off tls verification. # buildah push --tls-verify=false imageID docker://localhost:5000/my-imageID<br/> This example pushes the image specified by the imageID and puts it into the registry on the localhost using creden&#8208; tials and certificates for authentication. # buildah push --cert-dir /auth --tls-verify=true --creds=username:password imageID docker://local&#8208; host:5000/my-imageID<br/> ENVIRONMENT BUILD_REGISTRY_SOURCES<br/> BUILD_REGISTRY_SOURCES, if set, is treated as a JSON object which contains lists of registry names under the keys insecureRegistries, blockedRegistries, and allowedRegistries.<br/> When pushing an image to a registry, if the portion of the destination image name that corresponds to a registry is compared to the items in the blockedRegistries list, and if it matches any of them, the push attempt is denied. If there are registries in the allowedRegistries list, and the portion of the name that corresponds to the registry is not in the list, the push attempt is denied.<br/> TMPDIR The TMPDIR environment variable allows the user to specify where temporary files are stored while pulling and pushing images. Defaults to &#39;/var/tmp&#39;.<br/> FILES registries.conf (/etc/containers/registries.conf)<br/> registries.conf is the configuration file which specifies which container registries should be consulted when com&#8208; pleting image names which do not include a registry or domain portion.<br/> policy.json (/etc/containers/policy.json)<br/> Signature policy file. This defines the trust policy for container images. Controls which container registries can be used for image, and whether or not the tool should trust the images.<br/> SEE ALSO buildah(1), buildah-login(1), containers-policy.json(5), docker-login(1), containers-registries.conf(5)<br/> buildah June 2017 buildah-push(1) </span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3cf7e099c3a2'><button class='copyBtn' data-clipboard-target='#id3cf7e099c3a2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah run \ --entrypoint /var/lang/bin/pip \ --rm \ --user "$(id -u):$(id -g)" \ -v "$(pwd):/mnt" \ amazon/aws-lambda-python:3.8 \ install --target /mnt/build --upgrade psycopg2-binary</pre> </editor-fold> <editor-fold how_to> <div style="text-align: right;"> <a href="https://www.scholastic.ca/books/view/how-to-speak-dolphin" target="_blank" ><picture> <source srcset="/blog/images/buildahPodman/howToSpeakDolphin.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/howToSpeakDolphin.png" type="image/png"> <img src="/blog/images/buildahPodman/howToSpeakDolphin.png" class="right liImg2 rounded shadow" style="width: 25%; height: auto;" /> </picture></a> </div> <h2 id="howto">How To</h2> <p> The following was inspired by <a href='https://github.com/groda/big_data/blob/master/docker_for_beginners.md#recap-images-and-containers' target='_blank' rel='nofollow'>Recap: images and containers</a> from <b>Docker for beginners</b>. The equivalent commands for Docker alternatives are shown. </p> <h3 class="clear" id="ver">Check software version</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id52577a8689ca'><span class='unselectable'>$ </span>docker -v<br><span class='unselectable'>Docker version 20.10.2, build 20.10.2-0ubuntu1~20.10.1 </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb8af13792674'><span class='unselectable'>$ </span>buildah -v<br><span class='unselectable'>buildah version 1.15.2 (image-spec 1.0.1, runtime-spec 1.0.2-dev) </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id41d0bc74cfa0'><span class='unselectable'>$ </span>podman -v<br><span class='unselectable'>podman version 2.0.6 </span></pre> <div style="text-align: right;"> <picture> <source srcset="/blog/images/buildahPodman/aws_linux.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/aws_linux.png" type="image/png"> <img src="/blog/images/buildahPodman/aws_linux.png" class="right " style="width: 25%; height: auto; margin-bottom: 1em;" /> </picture> </div> <h3 id="dlimg">Download the Amazon Linux 2 image</h3> <p> AWS Lambda functions run under Amazon Linux. </p> <p> Each of these 3 commands does a very similar task, downloading a specific image. Docker uses different subdirectories for images than Buildah and <code>podman</code> do. </p> <div class="clear"> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id33c096e468c2'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker pull amazonlinux <span class='unselectable'>Using default tag: latest latest: Pulling from library/amazonlinux 3c2c91c7c431: Pull complete Digest: sha256:06b9e2433e4e563e1d75bc8c71d32b76dc49a2841e9253746eefc8ca40b80b5e Status: Downloaded newer image for amazonlinux:latest docker.io/library/amazonlinux:latest </span></pre> </div> <p> Buildah works without complaint. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id97dd7fd96ccc'><span class='unselectable'>$ </span>buildah pull amazonlinux <span class='unselectable'>53ef897d731f9a5673c083d0e86d7911f85d6e63bb2be2346b17bdbacdc58637 </span></pre> <p> <code>podman</code> seems to hiccup and then complete successfully. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4b96394cd3d9'><span class='unselectable'>$ </span>podman pull amazonlinux <span class='unselectable'>Trying to pull quay.io/amazonlinux... error parsing HTTP 404 response body: invalid character &#39;&lt;&#39; looking for beginning of value: &quot;&lt;!DOCTYPE HTML PUBLIC \&quot;-//W3C//DTD HTML 3.2 Final//EN\&quot;&gt;\n&lt;title&gt;404 Not Found&lt;/title&gt;\n&lt;h1&gt;Not Found&lt;/h1&gt;\n&lt;p&gt;The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.&lt;/p&gt;\n&quot; Trying to pull docker.io/library/amazonlinux... Getting image source signatures Copying blob 3c2c91c7c431 [--------------------------------------] 0.0b / 0.0b Copying config 53ef897d73 done Writing manifest to image destination Storing signatures 53ef897d731f9a5673c083d0e86d7911f85d6e63bb2be2346b17bdbacdc58637 </span></pre> <div style="text-align: right;"> <picture> <source srcset="/blog/images/buildahPodman/bash-logo.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/bash-logo.png" type="image/png"> <img src="/blog/images/buildahPodman/bash-logo.png" class="right liImg2 rounded shadow" style="width: 35%; height: auto; margin-bottom: 1em;" /> </picture> </div> <h3 id="echo">Run a Bash Command in an OCI Container</h3> <p> Again, <code>Docker</code> must be run as root for this operation, this represents an unnecessary security risk. </p> <div class="clear"> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id82fa754315f0'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker container run amazonlinux echo 'Hello World!' <span class='unselectable'>Hello World! </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idec1fffa1faed'><span class='unselectable'>$ </span>podman container run amazonlinux <a href='https://opensource.com/article/18/6/linux-version#how-to-find-the-linux-kernel-version' target='_blank' rel='nofollow'>cat /etc/os-release</a> <span class='unselectable'>VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" </span></pre> </div> <div style="text-align: right;"> <picture> <source srcset="/blog/images/buildahPodman/kodak-carousel-projector.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/kodak-carousel-projector.png" type="image/png"> <img src="/blog/images/buildahPodman/kodak-carousel-projector.png" class="right liImg2 rounded shadow" style="width: 35%; height: auto; margin-bottom: 1em;" /> </picture> </div> <h3 id="localImages">Show All Locally Available Images</h3> <p> Again, <code>Docker</code> must be run as root for this operation, this represents an unnecessary security risk. </p> <div class="clear"><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd159f3d95999'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker images <span class='unselectable'>REPOSITORY TAG IMAGE ID CREATED SIZE amazonlinux latest 53ef897d731f 21 hours ago 163MB </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id24ea45209b40'><span class='unselectable'>$ </span>podman images <span class='unselectable'>REPOSITORY TAG IMAGE ID CREATED SIZE localhost/hello latest 40ef32b39cf4 5 hours ago 622 MB docker.io/library/amazonlinux latest 53ef897d731f 21 hours ago 170 MB </span></pre> </div> <div style="text-align: right;"> <picture> <source srcset="/blog/images/buildahPodman/containerShip.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/containerShip.png" type="image/png"> <img src="/blog/images/buildahPodman/containerShip.png" class="right liImg2 rounded shadow" style="width: 35%; height: auto; margin-bottom: 1em;" /> </picture> </div> <h3 id="listCont">List OCI Containers</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0ed307d8cc84'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker container ls<br><span class='unselectable'>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7205d1c08d1d'><span class='unselectable'>$ </span>podman container ls<br><span class='unselectable'>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES </span></pre> <h3 id="listcont4">View All OCI Containers (Running or Not)</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide67a5bbc1ee9'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker container ls -a <span class='unselectable'>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 250f56d9aced amazonlinux "echo 'Hello World!'" 14 minutes ago Exited (0) 14 minutes ago competent_einstein </span></pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd0f02118bea6'><span class='unselectable'>$ </span>podman container ls -a <span class='unselectable'>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0f8203e9d3b8 docker.io/library/amazonlinux:latest echo Hello world! 36 minutes ago Exited (0) 36 minutes ago beautiful_mestorf 14282ace8978 docker.io/library/amazonlinux:latest echo Hello world! 36 minutes ago Exited (0) 36 minutes ago beautiful_goldwasser 1b9a8db52fb9 docker.io/library/alpine:latest echo Hello World! About an hour ago Exited (0) About an hour ago zealous_easley 6444ee144488 docker.io/library/amazonlinux:latest echo Hello World! 12 minutes ago Exited (0) 12 minutes ago frosty_ritchie 7444122cbc59 docker.io/library/alpine:latest cat /etc/motd About an hour ago Exited (0) About an hour ago elated_sammet aef84973d6ad docker.io/library/amazonlinux:latest echo Hello world! About an hour ago Exited (0) About an hour ago lucid_sinoussi e210f74bc209 docker.io/library/amazonlinux:latest cat /etc/motd About an hour ago Exited (0) About an hour ago jovial_borg </span></pre> <h3 id="listCont3">List Running OCI containers</h3> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idabbb2b9bdc10'><span class='unselectable'>$ </span><span class="bg_yellow">sudo</span> docker container ps -a <span class='unselectable'>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 250f56d9aced amazonlinux "echo 'Hello World!'" 5 minutes ago Exited (0) 5 minutes ago competent_einstein </span></pre> <p> <code>podman</code> has a problem with the <code>container ps</code> sub-subcommand. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id662054bf8de7'><span class='unselectable'>$ </span>podman container ps -a <span class='unselectable'>Error: unrecognized command `podman container ps` Try 'podman container --help' for more information. </span></pre> <h3 id="buildah_push_use"><span class="code">buildah push</span> to Docker Daemon</h3> <div style=""> <picture> <source srcset="/blog/images/buildahPodman/containerShipTugboat.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/containerShipTugboat.png" type="image/png"> <img src="/blog/images/buildahPodman/containerShipTugboat.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcca228161e19'><button class='copyBtn' data-clipboard-target='#idcca228161e19' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah push hello:latest docker-daemon:hello:latest <span class='unselectable'>Getting image source signatures Copying blob sha256:72fcdba8cff9f105a61370d930d7f184702eeea634ac986da0105d8422a17028 247.02 MiB / 247.02 MiB [==================================================] 2s Copying blob sha256:e567905cf805891b514af250400cc75db3cb47d61219750e0db047c5308bd916 144.75 MiB / 144.75 MiB [==================================================] 1s Copying config sha256:6d54bef73e638f2e2dd8b7bf1c4dfa26e7ed1188f1113ee787893e23151ff3ff 1.59 KiB / 1.59 KiB [======================================================] 0s Writing manifest to image destination Storing signatures </span> <span class='unselectable'>$ </span>buildah images | head -n2 <span class='unselectable'>REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/hello latest 6d54bef73e63 2 minutes ago 398 MB </span> <span class='unselectable'>$ </span>buildah run -t hello:latest <span class='unselectable'>Hello, world! </span></pre> <h3 id="buildah_rmi">Delete an OCI Image</h3> <div style=""> <picture> <source srcset="/blog/images/buildahPodman/containerSky.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/containerSky.png" type="image/png"> <img src="/blog/images/buildahPodman/containerSky.png" class=" fullsize liImg2 rounded shadow" /> </picture> </div> <p> Delete an OCI image in Buildah's <code>~/.local/share/container</code> directory with the <code>rmi</code> subcommand: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1cc16e8ffd10'><button class='copyBtn' data-clipboard-target='#id1cc16e8ffd10' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah rmi e12ea62c5582 <span class='unselectable'>e12ea62c5582f91a2228e3e284ea957f2df4f1cdb150fd2c189ef8f11d7633ce </span></pre> </editor-fold> Stack Overflow Culture: Zero-Sum, Authoritarian and Hormonally Imbalanced 2021-04-18T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/18/so-culture <p> This blog post describes some problems that significantly impact <a href='https://www.crunchbase.com/organization/stack-overflow' target='_blank' rel='nofollow'>Stack Overflow</a> users, and offers suggestions for improvement. If you are unfamiliar with Stack Overflow, or would like to read a summary of what this blog post is based on, Kevin Workman wrote a terrific summary in March 2019 entitled &ldquo;<a href='https://happycoding.io/blog/stack-overflow-culture-wars' target='_blank' rel='nofollow'>The Stack Overflow Culture Wars</a>&rdquo;. Very little has changed since then. </p> <div style="text-align: center;"> <a href="https://www.stackoverflow.com" target="_blank" ><picture> <source srcset="/blog/images/stackOverflow/stackoverflowLogo.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/stackoverflowLogo.png" type="image/png"> <img src="/blog/images/stackOverflow/stackoverflowLogo.png" class="center halfsize liImg2 rounded shadow" style="padding: 20px;" /> </picture></a> </div> <fold-article intro> <h2 id="intro">Introduction</h2> <p> <a href='https://www.stackoverflow.com' target='_blank' rel='nofollow'>Stack Overflow</a> is the premier website world-wide for programmers to help each other by asking and answering questions. It has a defined protocol for this type of interaction, however new user on-boarding is often ineffective, so newcomers are not properly informed, and old-timers often do not exhibit appropriate people skills. The <a href='https://games.greggman.com/game/done-with-stackoverflow/' target='_blank' rel='nofollow'>protocol is somewhat misguided</a> and appropriate tools for better interaction are not provided. </p> <p> As a result, this website has developed a well-documented reputation for being “<a href='https://codeblog.jonskeet.uk/2018/03/17/stack-overflow-culture/' target='_blank' rel='nofollow'>a valuable resource, but a scary place to contribute due to potential hostility.</a>” </p> <div class="pullQuote"> Sometimes, loving something means caring enough to admit that it has a problem.<br><br> &nbsp; &ndash; <a href='https://stackoverflow.blog/2018/04/26/stack-overflow-isnt-very-welcoming-its-time-for-that-to-change/' target='_blank' rel='nofollow'>Jay Hanlon</a>, writing about Stack Overflow when he was EVP of Culture and Experience. </div> </fold-article> <fold-article problem> <div class="clear quote"> Stack Overflow suffers from militant moderators who close and delete reasonable submissions and answers due to Draconian rules.<br><br> &nbsp; &ndash; <code>sleavey</code> commenting on a Hacker News thread entitled <a href='https://news.ycombinator.com/item?id=16610353' target='_blank' rel='nofollow'>Stack Overflow Culture</a>. </div> </fold-article> <fold-article top> <h2 id="topdown">Change Starts At the Top</h2> <h3 id="Chandrasekar">Prashanth Chandrasekar, CEO</h3> <p> Prashanth Chandrasekar became CEO of Stack Overflow in September, 2019. <a href='https://www.intercom.com/blog/podcasts/prashanth-chandrasekar-on-writing-the-script-of-the-future/' target='_blank' rel='nofollow'>Inside Intercom interviewed Mr. Chandrasekar</a> in August 2020. This puff piece made no mention of any cultural problems. The focus was on the brilliance of Stack Overflow&rsquo;s technology. If Mr. Chandrasekar has a vision for how to guide social change, he did not mention it. Instead, in the article and his publications since then he only speaks publicly about <a href='https://stackoverflow.blog/author/pchandrasekar/' target='_blank' rel='nofollow'>transitioning Stack Overflow to a product-led SaaS company</a>. </p> <p> Shortly after becoming CEO, Mr. Chandrasekar published &ldquo;<a href='https://stackoverflow.blog/2020/01/21/scripting-the-future-of-stack-2020-plans-vision/' target='_blank' rel='nofollow'>Scripting the Future of Stack Overflow</a>, in which he wrote: </p> <p class="quote"> We learned that we needed much better channels to listen to our moderators and community members. We have not evolved the existing channels of engagement for power users in our community, like Meta, or articulated how we intended to make improvements going forward. This has caused friction as our user base and business have rapidly grown. We acknowledge these issues, apologize for our mistakes, and have plans for improving in the future. </p> <p> Later in the article, he mentioned improvements to the code of conduct, a survey and establishing a moderator council. More than 2 years later, none of this has made the slightest difference in Stack Overflow's culture. </p> <h3 id="pathak">Mihir Pathak — EVP, Strategy & Transformation</h3> <p> Prior to his employment at Stack Overflow, Mr. Pathak was a derivatives strategist McKinsey &amp; Company, a <a href='https://www.mckinsey.com/business-functions/organization/our-insights/the-four-building-blocks--of-change#' target='_blank' rel='nofollow'>world-renouned change management company</a>. That is to say, although Mr. Pathak worked at McKinsey, he was not there to assist other companies make structural changes; instead, he was responsible for pricing methodologies and hedging techniques underlying financial derivative products and options trading strategies &ndash; a heads-down money manager. </p> <p> <a href='https://stackoverflow.com/company/leadership/mihir-pathak' target='_blank' rel='nofollow'>Mr. Pathak&rsquo;s page at StackOverflow</a> is no longer available. <a href='https://webcache.googleusercontent.com/search?q=cache:1eKtOpLW2VoJ:https://stackoverflow.com/company/leadership/mihir-pathak' target='_blank' rel='nofollow'>Google cached it</a>. No announcement has been made about his departure or if there will be a replacement. </p> <p> Mr. Pathak's job description is still visible at <a href='https://www.themuse.com/profiles/stackoverflow' target='_blank' rel='nofollow'><code>themuse.com</code></a>: </p> <p class="quote"> Mihir is responsible for the long-term business strategy of Stack Overflow, which includes forming partnerships with like-minded organizations and understanding how to best serve the needs of future developers. </p> <p> Mr. Pathak was employed from September 2016 at Stack Overflow as a business development executive, not a change management champion. </p> <h3 id="dietrich">Teresa Dietrich, Chief Product and Technology Officer</h3> <p> In January 2020 <a href='https://www.crunchbase.com/person/teresa-dietrich' target='_blank' rel='nofollow'>Teresa Dietrich</a> was made Chief Product and Technology Officer. She also came from McKinsey &amp; Company, where she was Global Head of Product &amp; Engineering. Two months after she took the job she wrote &ldquo;<a href='https://stackoverflow.blog/2020/02/25/sharing-our-first-quarter-2020-community-roadmap/' target='_blank' rel='nofollow'>Sharing our first quarter 2020 community roadmap</a>&rdquo;. That rather bland article did not seem to indicate any recognition of serious problems within the Stack Overflow culture, and I could not find any publicly available results of the studies that were mentioned. </p> <p> 16 months after Ms. Dietrich started her job, I do not see any cultural change. Has Ms. Dietrich been tasked with leading such a change? If so: </p> <ul> <li>Does she have board-level support?</li> <li>Does she have what she needs to make significant cultural changes?</li> <li> Why has she not made any public acknowledgement of a cultural problem? One possible answer is that her boss, Mr. Chandrasekar, has not done so. </li> <li> Is Ms. Dietrich the right person for the job? This is not a purely technical problem, it is a social problem. I believe that women in general possess more highly developed social skills, but the skills necessary to climb to the top are not the skills required to make this type of cultural shift. </li> </ul> <div class="pullQuote"> Are Mr. Chandrasekar and Ms. Dietrich part of the cultural problem, or part of the solution? </div> </fold-article> <fold-article women> </fold-article> <fold-article me> <h2 id="me">Where Am I Coming From?</h2> <p> I have no agenda, no investment in the status quo, nothing to prove, no contacts at the company, I am not an undercover journalist, and I am not a competitor or investor. I am just Joe User... and I am not shy when I believe I have something that I think needs to be heard. </p> <p> If I mispeak, please tell me. If I missed something, again please tell me. If there is a bigger picture I would like to learn about it. </p> <p> I have used Stack Overflow and its <a href='https://stackexchange.com/sites' target='_blank' rel='nofollow'>sibling websites</a> for more than 10 years. Until a few weeks ago, I contributed a few answers here and there, and otherwise did not spend much time on the sites. All contributors are volunteers, so the only reasons to contribute are for prestige, social bonding and altruism. </p> <p> For almost 30 days, in my spare time, I helped people who asked questions on Stack Overflow. I am writing this blog post because although I enjoy helping others, the enjoyment I experienced while doing so within the Stack Overflow environment was overshadowed by the regressive behavior tolerated and even enforced by other contributors. I should also say that I did encounter certain other contributors with whom interaction was very pleasant. However, interactions with alpha contributors with the highest scores seemed more often than not to be quite unpleasant. Since I first published this blog post, I have mostly not interacted on the site. I remain open to contributing to improvements in the culture. </p> <p> The next image is provided so readers know that I am able to effectively participate in the current Stack Overflow website. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/stackOverflow/stackOverflow.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/stackOverflow.png" type="image/png"> <img src="/blog/images/stackOverflow/stackOverflow.png" title="The volume of accepted and upvoted answers put me in the top 0.14% of Stack Overflow answerers in under 30 days." class="center halfsize liImg2 rounded shadow" alt="The volume of accepted and upvoted answers put me in the top 0.14% of Stack Overflow answerers in under 30 days." /> </picture> <figcaption class="halfsize" style="width: 100%; text-align: center;"> The volume of accepted and upvoted answers put me in the top 0.14% of Stack Overflow answerers in under 30 days. </figcaption> </figure> </div> <p> Over 100 of the answers I offered in a 30-day period were accepted as the preferred answer. In fact, most of my answers were selected as the preferred answer. That means many more alternative answers were not accepted. </p> <p> After losing, sometimes a contributor will delete their post. I have done it myself, when the winning post was significantly better by all measures. </p> <p> Some of the other contributors who had provided alternative answers that were not selected clearly felt they had lost a competition. This structure, and others that I describe below, define a system designed to generate hostility. Stack Overflow, as currently implemented, <a href='https://en.wikipedia.org/wiki/Dominance_hierarchy' target='_blank' rel='nofollow'>promotes dominance behavior</a>, which for most primates (other than <a href='https://www.scientificamerican.com/article/bonobo-sex-and-society-2006-06/' target='_blank' rel='nofollow'>bonobos</a>) is patriarchal. </p> </fold-article> <fold-article gamify> <div style="text-align: right;"> <a href="http://localhost:4001/blog/2021/04/18/so-culture.html" target="_blank" ><picture> <source srcset="/blog/images/stackOverflow/codinghorror-app-icon.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/codinghorror-app-icon.png" type="image/png"> <img src="/blog/images/stackOverflow/codinghorror-app-icon.png" class="right liImg2 rounded shadow" /> </picture></a> </div> <h2 id="gamification">Gamification</h2> <p> Stack Overflow's success has be in part due to its successful <a href='https://en.wikipedia.org/wiki/Gamification' target='_blank' rel='nofollow'>gamification</a> of the interaction between questioners and answerers. Gamification is powerful and addictive. Unfortunately, the model chosen resembles FPS (<a href='https://en.wikipedia.org/wiki/First-person_shooter' target='_blank' rel='nofollow'>first-person shooter</a>) games, instead of co-operative games. </p> <p> Jeff Atwood, one of the two original authors of Stack Overflow, wrote an article in 2011 entitled <a href='https://blog.codinghorror.com/the-gamification/' target='_blank' rel='nofollow'>The Gamification</a>, in which he writes: </p> <div class="quote"> Gaming elements are not tacked on to the Stack Exchange Q&A engine, but a natural and essential element of the design. </div> <p> Just below that sentence, Mr. Atwood shows a screenshot from an FPS video game: </p> <div style="text-align: center;"> <a href="https://blog.codinghorror.com/the-gamification/" target="_blank" ><picture> <source srcset="/blog/images/stackOverflow/fps.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/fps.png" type="image/png"> <img src="/blog/images/stackOverflow/fps.png" title="FPS Game screenshot from 'The Gamification'." class="center halfsize liImg2 rounded shadow" alt="FPS Game screenshot from 'The Gamification'." /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://blog.codinghorror.com/the-gamification/" target="_blank" > FPS Game screenshot from 'The Gamification'. </a> </figcaption> </figure> </div> <div class="quote"> I haven't ever quite come out and said it this way, but … I played a lot of Counter-Strike from 1998 to 2001, and Stack Overflow is in many ways my personal Counter-Strike.<br><br> &nbsp; &ndash; Jeff Atwood, from &ldquo;The Gamification&rdquo; </div> <p> FPS games are structured so the player only wins by killing others. This is an entirely different motivational structure than a scenario where a person only wins if others succeed. </p> </fold-article> <fold-article terms> <h2 id="terms">Terminology</h2> <p> I use a few non-standard terms in this blog post: </p> <dl> <dt>Questioner</dt> <dd>One who provides a question</dd> <dt>Answerer</dt> <dd>One who provides an answer</dd> <dt>Downvoter</dt> <dd>One who downvotes another person's contribution</dd> <dt>Downvotee</dt> <dd>One who has their contribution downvoted</dd> </dl> </fold-article> <fold-article downvote> <h2 id="dialog">Dialog Improves Most Questions</h2> <p> A small percentage of questions asked on Stack Overflow are unambiguous, contain all the necessary information, and are phrased well enough to be understood. For these questions, answers can be posted without any interaction between questioner and potential answerers. </p> <p> However, most questions involve a dialog between potential answerers and the questioner. In the dialog, the question is refined, and the questioner's code and any other relevant data is elicited and provided. The tools provided for such a dialog are unfortunately problematic. </p> <p> The only two mechanisms for interaction between questioner and potential answerers are comments and answers. Comments have severe limitations that greatly reduce their effectiveness for eliciting information from a questioner: </p> <ul> <li>Comments must be very short</li> <li>Comments cannot be formatted properly</li> <li>Comments cannot be edited for more than a few minutes</li> </ul> <p> This means that answerers who are trying to explain something to the questioner to elicit more information, or guide the questioner towards understanding their problem better, must resort to posting an incomplete answer. Posting an incomplete answer is risky; other potential answerers can attack the answer by downvoting it. </p> <div class="pullQuote"> Downvotes typically last forever </div> </fold-article> <fold-article incentives> <h2 class="clear" id="broken">Some Incentives Promote Hostility</h2> <p> Many long-time users have completely objectified other users, and act as if Stack Overflow is a video game. Points are accumulated, and at any given time there are a finite number of questions to answer. A person's reputation on Stack Overflow is represented by a single number, which is the number of points they have accumulated. This single number is the structural source of many problems. A more nuanced reputation score would be a giant step forward. </p> <p> Many of these long-term answerers have come to view their participation on Stack Overflow as a zero-sum competition; they can only win (that is, have their answer accepted) if everyone else loses (that is, no-one else provides an answer that is accepted). </p> <p> Some answerers employ intimidation order to suppress competition. Downvotes are often used in the same way against other answerers as a <a href='https://en.wikipedia.org/wiki/Brushback_pitch' target='_blank' rel='nofollow'>brush back pitch</a> in baseball. </p> <div class="pullQuote"> Downvoting has no negative consequences for the downvoter </div> <p> Standard operating procedure for competitively-minded answerers is to downvote answers from others at every opportunity. There is no risk in downvoting: </p> <ul> <li>Downvotes are untraceable; there is no public record of who downvoted or when a downvote was cast.</li> <li>Downvotes are free to downvoters; this encourages liberal downvoting.</li> </ul> <p> This scenario incentivizes competitively-minded answerers to strafe the competition with downvotes at every opportunity. </p> <div class="pullQuote"> If downvotes cost the downvoter the same number of points as they penalize the downvotee, then downvotes would become much rarer. </div> <p> Downvotes typically last forever. Yes, a downvoter could theoretically reverse a downvote, but it is awkward for them to find their old downvotes, and there is no incentive to do so. </p> <p> Questions from newcomers are also frequently downvoted, without discussion, or along with hostile remarks. That leaves a permanent impression, and tends to select for new members who are comfortable with dominance-based hostility. This is a self-perpetuating, and highly toxic, social order. </p> <div style="text-align: center;"> <a href="https://www.focusforhealth.org/how-toxic-masculinity-harms-men-and-society-as-a-whole/" target="_blank" ><picture> <source srcset="/blog/images/stackOverflow/toxicMasulinity.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/toxicMasulinity.png" type="image/png"> <img src="/blog/images/stackOverflow/toxicMasulinity.png" title="Toxic Masculinity Harms Men and Society As A Whole, from Focus for Health" class="center halfsize liImg2 rounded shadow" alt="Toxic Masculinity Harms Men and Society As A Whole, from Focus for Health" /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://www.focusforhealth.org/how-toxic-masculinity-harms-men-and-society-as-a-whole/" target="_blank" > Toxic Masculinity Harms Men and Society As A Whole, from Focus for Health </a> </figcaption> </figure> </div> </fold-article> <fold-article suggest_onboard> <div style="text-align: right;"> <picture> <source srcset="/blog/images/stackOverflow/stackOverflowHelp.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/stackOverflowHelp.png" type="image/png"> <img src="/blog/images/stackOverflow/stackOverflowHelp.png" class="right quartersize liImg2 rounded shadow" /> </picture> </div> <h2 id="onboard">Suggestion: Gamified Onboarding</h2> <p> The only information for new users that is directly accessible from the Stack Overflow front page, is the Help Center, under the question mark icon. It is obvious from the often very polite and tentative inquiries made by new users when they ask their first question that they never noticed that information, or if they did it did not seem relevant. ... and then those new users are mercilessly slammed. </p> <p> New users should be presented with a <a href='https://www.appcues.com/blog/the-5-best-user-onboarding-experiences' target='_blank' rel='nofollow'>short instructional question and answer-style introduction</a>, where information is provided on how to be a good Stack Overflow user. This should happen before they get the opportunity to post their first question. Different levels of users should be explained. Although Stack Overflow is all-English, the onboarding should be multilingual. </p> </fold-article> <fold-article suggest_multilingual> <h2 id="multling">Suggestion: Multilingual Support</h2> <p> A high percentage of users do not speak English very well. They really struggle, and tolerance is low on Stack Overflow for bad English. Other sites, for example Facebook and LinkedIn, have a translation facility built right in. I think Facebook did a particularly good job. Why not do something similar on StackOverflow.com? English would be the official language, but everyone world-wide would be able to interact much more effectively. </p> <div class="pullQuote"> This site is multilingual.<br/> It is not that hard to do! </div> <p> Machine translation is really quite good. I have it on this site. What to view this site in one of over 100 languages? Just <a href='#body'>go to the top of this page</a> and click on this pull-down menu labeled Select Language: </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/stackOverflow/selectLanguage.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/selectLanguage.png" type="image/png"> <img src="/blog/images/stackOverflow/selectLanguage.png" class="center halfsize " /> </picture> </div> <p> You will then see the list of languages that you can view this website in: </p> <div style=""> <picture> <source srcset="/blog/images/stackOverflow/selectLanguages.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/selectLanguages.png" type="image/png"> <img src="/blog/images/stackOverflow/selectLanguages.png" class=" liImg2 rounded shadow" /> </picture> </div> <p> Morally speaking, not to provide a multilingual site discriminates against non-English-speaking people. </p> </fold-article> <fold-article suggest_rep> <h2 id="elicitation">Suggestion: More Nuanced Reputation Score</h2> <p> Instead of a single metric, answerers should be rated along several dimensions. Economists and psychologists both use multidimensional diagrams. The following diagram represents data in 4 dimensions. More dimensions can easily be shown in this type of diagram. </p> <div style=""> <picture> <source srcset="/blog/images/stackOverflow/multiDimentionalPlot.webp" type="image/webp"> <source srcset="/blog/images/stackOverflow/multiDimentionalPlot.png" type="image/png"> <img src="/blog/images/stackOverflow/multiDimentionalPlot.png" title="Multi-dimensional data can easily be visualized by outlined shapes" class=" fullsize liImg2 rounded shadow" alt="Multi-dimensional data can easily be visualized by outlined shapes" /> </picture> <figcaption class="fullsize" style="width: 100%; text-align: center;"> Multi-dimensional data can easily be visualized by outlined shapes </figcaption> </figure> </div> <p> Instead of "bigger is better" (a single number indicating status, with the high score indicating alpha status), more information would allow for more detail, so a fuzzy diagram would show little interaction, while a highly detailed and intricate design would indicate a lot of participation. Some answerers might be stronger regarding some metrics, while being weaker on other metrics. </p> <p> Instead of displaying a person's score, the shape of their participation would be shown. Some people prefer to be seen as well-rounded, others prefer to be the best on selected aspects and ignore the others. One size does not fit all. </p> <p> People would start to give pet names for various shapes. Jokes would be made about the shapes. </p> <p> HR personnel would start to hire teams based on how well these shapes meshed together. Money will be made by timely entrepreneurs because these shapes will quickly be adopted industry-wide. Some will aspire to change their shape. </p> <p> An entire industry will spring up servicing those who wish to modify their shape. </p> </fold-article> <fold-article suggest_active> <h2 id="restrict_down">Suggestion: Encourage and Highlight Elicitation</h2> <p> <a href='https://en.wikipedia.org/wiki/Elicitation_technique' target='_blank' rel='nofollow'>Elicitation</a> is a difficult skill to master. The current high-scorers have no incentives to employ elicitation, and they act as if on a campaign to eradicate it. </p> <div class="pullQuote"> Introduce an Elicitation Phase </div> <p> A button should be introduced that lets everyone who visits the question page that a potential answerer would like to elicit information. While at least one such button is enabled, no downvotes are possible relating to the question, and the question cannot be closed or moved to another forum by anyone, regardless of their privilege level. This elicitation mode times out, but not suddenly or unexpectedly. Instead, it backs off, rather like the Ethernet back-off algorithm for collision resolution used in random access MAC protocols. Both the potential answerer and the questioner are given cues that they have a question or a response waiting, and after a period of inactivity the special status ends. I leave the details of the timing undefined for others to discuss. </p> </fold-article> <fold-article suggest_crowd> <h2 class="restrict_down" id="crowd">Suggestion: Restrict Downvoting</h2> <p> Downvoting needs to incorporate: </p> <ul> <li>Accountability (no more anonymous downvoting)</li> <li>Cost (no more drive-by shootings without consequences)</li> <li>Time window (vote after the dust settles, not during the elicitation phase)</li> <li>Public displays of user profiles should prominently display that user's downvotes and upvotes, with links</li> <li>Aggregate statistics on user profiles of their percentage downvotes and upvotes, trends (absolute and relative), etc.</li> </ul> </fold-article> <fold-article suggest_crowd> </fold-article> Serverless E-Commerce 2021-04-14T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/14/serverless-ecommerce <style> h2.numbered:before { color: darkgreen; content: "Option " counter(h2counter) ":\A0\A0\A0"; } </style> <editor-fold intro> <p> As readers of this blog know, I have been <a href='/django/index.html'>chronicling my adventure into Python-powered e-commerce</a> for several months. I have been focusing on Django in general, and <a href='https://django-oscar.readthedocs.io' target='_blank' rel='nofollow'>Django-Oscar</a> in particular. Webapps made with this technology are almost exclusively run on dedicated real or virtual machines. <a href='https://www.cloudflare.com/learning/serverless/what-is-serverless/' target='_blank' rel='nofollow'>Serverless</a> computing is a method of providing backend services on an as-used basis. AWS Lambda is the best-known example of serverless computing, and it combines nicely with a CDN like AWS CloudFront. </p> <p> This blog post discusses 3 goals for an e-commerce system. Two goals are provided by the technology behind <a href='https://martinfowler.com/articles/serverless.html' target='_blank' rel='nofollow'>serverless webapps</a>: </p> <ol> <li>Enormous and instantaneous scalability.</li> <li>Pay-as-you-go without an up-front cost commitment.</li> </ol> <p> I have one more goal: very low latency for online shoppers. </p> </editor-fold> <editor-fold big_picture> <h2 id="big">The Big Picture</h2> <p class="quote"> AWS Lambda consists of two main parts: the Lambda service which manages the execution requests, and the Amazon Linux micro virtual machines provisioned using AWS Firecracker, which actually runs the code. <br><br> A Firecracker VM is started the first time a given Lambda function receives an execution request (the so-called “Cold Start”), and as soon as the VM starts, it begins to poll the Lambda service for messages. When the VM receives a message, it runs your function code handler, passing the received message JSON to the function as the event object. <br><br> Thus every time the Lambda service receives a Lambda execution request, it checks if there is a Firecracker microVM available to manage the execution request. If so, it delivers the message to the VM to be executed. <br><br> In contrast, if no available Firecracker VM is found, it starts a new VM to manage the message. Each VM executes one message at a time, so if a lot of concurrent requests are sent to the Lambda service, for example due to a traffic spike received by an API gateway, several new Firecracker VMs will be started to manage the requests and the average latency of the requests will be higher since each VM takes roughly a second to start. <br><br> &nbsp; &ndash; From <a href='https://www.proud2becloud.com/how-to-run-any-programming-language-on-aws-lambda-custom-runtimes/' target='_blank' rel='nofollow'>How to run any programming language on AWS Lambda: Custom Runtimes</a> by Matteo Moroni. </p> </editor-fold> <editor-fold lambdalimits> <h2 id="lambdalimits">AWS Lambda Limits</h2> <p> AWS Lambda programs have access to considerable resources, enough for most e-commerce stores. The AWS Lambda runtime environment has the following limitations, some of which can be improved upon with some work: </p> <ul> <li>The disk space (ephemeral) is limited to 512 MB.</li> <li>The default deployment package size is 50 MB.</li> <li>The memory range is from 128 to 3008 MB.</li> <li>The maximum execution timeout for a function is 15 minutes.</li> <li>Request and response (synchronous calls) body payload size can be up to 6 MB.</li> <li>Event request (asynchronous calls) body can be up to 128 KB.</li> </ul> </editor-fold> <editor-fold cf> <div style="text-align: right;"> <picture> <source srcset="/blog/images/django/clouds.webp" type="image/webp"> <source srcset="/blog/images/django/clouds.png" type="image/png"> <img src="/blog/images/django/clouds.png" class="right quartersize liImg2 rounded shadow" /> </picture> </div> <h2 id="cf">CloudFront</h2> <p> <a href='https://forum.djangoproject.com/u/briancaffey/summary' target='_blank' rel='nofollow'>Brian Caffey</a> wrote <a href='https://forum.djangoproject.com/t/building-a-django-application-on-aws-with-cloud-development-kit-cdk/2830' target='_blank' rel='nofollow'>Building a Django application on AWS with Cloud Development Kit (CDK)</a>. The website Mr. Caffey's article discusses does not use Lambda, instead his website is always running. So, this option is quite informative and well-thought-out, but it is AWS-specific and does not discuss serverless architecture. </p> <p> For me, the most interesting part about Mr. Caffey's article is it mentions using 3 origins with AWS CloudFront: (1) an origin for ALB (for hosting the Django API), (2) an origin for the S3 website (static Vue.js site), and (3) an S3 origin for Django assets. Mr. Caffey does not say why he used 3 origins, but feeding one CloudFront distribution from multiple origins would mean that all of their content would appear on the same Internet subdomain. </p> <p> This means that the <a href='https://www.w3.org/TR/2020/SPSD-cors-20200602/#resource-preflight-requests' target='_blank' rel='nofollow'>extra HTTP handshaking required for certain CORS</a> (cross-origin HTTP requests) requests between subdomains would be avoided; specifically, there would be no need for pre-flight requests. This would make the website seem noticeably faster if users did lots of content editing and/or transactions with the website. My own pet project has users creating and modifying content, and purchasing product, so taking the requirement for the CORS handshakes away would be a win, plus the end user's web browser could reuse the origin HTTP connection, speeding up even non-cacheable requests. </p> <p> Tamás Sallai wrote <a href='https://advancedweb.hu/how-to-route-to-multiple-origins-with-cloudfront/' target='_blank' rel='nofollow'>How to route to multiple origins with CloudFront &mdash; Set up path-based routing with Terraform</a>. Mr. Sallai <a href='https://advancedweb.hu/' target='_blank' rel='nofollow'>is a prolific writer</a>! </p> </editor-fold> <editor-fold edge> <div style="text-align: right;"> <picture> <source srcset="/blog/images/django/lifeOnTheEdge.webp" type="image/webp"> <source srcset="/blog/images/django/lifeOnTheEdge.png" type="image/png"> <img src="/blog/images/django/lifeOnTheEdge.png" class="right quartersize liImg2 rounded shadow" /> </picture> </div> <h2 id="edge">Edge Computing</h2> <p> Performing computations and serving assets from a <a href='https://aws.amazon.com/cloudfront/features/' target='_blank' rel='nofollow'>nearby point of presence</a> minimizes latency for end users. E-commerce customers much prefer online stores that respond quickly. Edge computing can deliver that experience world-wide, and developers can deploy their work from wherever they are. </p> <p> <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-at-the-edge.html' target='_blank' rel='nofollow'>AWS Lambda@Edge</a> (<a href='https://aws.amazon.com/lambda/edge/' target='_blank' rel='nofollow'>console</a>) runs the Lambda computation in one of 13 regional AWS points of presence, one hop removed from the CloudFront edge locations, or at least in the same availability zone at the CloudFront point of presence. Distributed database issues would need to be addressed before significant benefits would accrue from this implementing this decentralized architecture. Unfortunately, Lambda@Edge has some significant restrictions that prevent it from running nontrivial Django apps. </p> <h3>Lambda@Edge Restrictions</h3> <p>From <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-requirements-limits.html' target='_blank' rel='nofollow'>requirements and restrictions on using Lambda functions with CloudFront</a>, it is apparent that it is not possible to run non-trivial Django apps securely at the edge with good performance.</p> <ul> <li>You can add triggers only for functions in the US East (N. Virginia) Region.</li> <li>You can’t configure your Lambda function to access resources inside your VPC.</li> <li>AWS Lambda environment variables are not supported.</li> <li>Lambda functions with AWS Lambda layers are not supported.</li> <li>Using AWS X-Ray is not supported.</li> <li>AWS Lambda reserved concurrency and provisioned concurrency are not supported.</li> <li>Lambda functions defined as container images are not supported.</li> </ul> <p> Until such time as Lambda@Edge removes the above restrictions, Django webapps will continue to be deployed as centralized webapps, which means that ultra-low latency is not possible world-wide. </p> <h3 id="cf_fns">CloudFront Functions</h3> <p> <a href='https://aws.amazon.com/blogs/aws/introducing-cloudfront-functions-run-your-code-at-the-edge-with-low-latency-at-any-scale/' target='_blank' rel='nofollow'>CloudFront Functions</a> are closer to the user, but have even more restrictions than Lambda@Edge. Alas, CloudFront Functions do not seem likely to be able to support significant computation any time soon. </p> <div style=""> <a href="https://aws.amazon.com/blogs/aws/introducing-cloudfront-functions-run-your-code-at-the-edge-with-low-latency-at-any-scale/" target="_blank" ><picture> <source srcset="/blog/images/serverlessEcommerce/cloudfront-functions-only-lambda-egde-1024x413.webp" type="image/webp"> <source srcset="/blog/images/serverlessEcommerce/cloudfront-functions-only-lambda-egde-1024x413.png" type="image/png"> <img src="/blog/images/serverlessEcommerce/cloudfront-functions-only-lambda-egde-1024x413.png" title="From &ldquo;Introducing CloudFront Functions – Run Your Code at the Edge with Low Latency at Any Scale&rdquo;" class=" liImg2 rounded shadow" alt="From &ldquo;Introducing CloudFront Functions – Run Your Code at the Edge with Low Latency at Any Scale&rdquo;" /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://aws.amazon.com/blogs/aws/introducing-cloudfront-functions-run-your-code-at-the-edge-with-low-latency-at-any-scale/" target="_blank" > From &ldquo;Introducing CloudFront Functions – Run Your Code at the Edge with Low Latency at Any Scale&rdquo; </a> </figcaption> </figure> </div> </editor-fold> <editor-fold iac> <h2 id="vendor">Infrastructure as Code (IaC)</h2> <div class="quote"> For anything bigger than a toy cloud application, Infrastructure as Code (IaC) is table stakes. You’d be hard-pressed to find someone managing anything of scale who thinks letting folks point and click in the console is the optimal route. <br><br>&nbsp; &ndash; From <a href='https://acloudguru.com/blog/engineering/cloudformation-terraform-or-cdk-guide-to-iac-on-aws' target='_blank' rel='nofollow'>CloudFormation, Terraform, or CDK? A guide to IaC on AWS</a> by Jared Short, published by <code>acloudguru.com</code>. </div> <div class="quote"> <a href='https://www.hashicorp.com/products/terraform' target='_blank' rel='nofollow'>Terraform</a>, AWS CloudFormation, Packer, Pulumi, and GeoEngineer are the most popular tools in the category "Infrastructure Build Tools". <br> &nbsp; &ndash; <a href='https://stackshare.io/infrastructure-build-tools' target='_blank' rel='nofollow'>from Stackshare.io</a> </div> </editor-fold> <editor-fold infographic> <h2 id="infographic">Infographic: Lambda Framework Comparison</h2> <p> Yan Cui at Lumigo.io made <a href='https://lumigo.io/aws-lambda-deployment/' target='_blank' rel='nofollow'>this terrific infographic</a>, which compares 9 serverless application frameworks and infrastructure management tools according to opinionatedness and customizability. This article discusses some of those technologies. </p> <div style=""> <a href="https://lumigo.io/aws-lambda-deployment/" target="_blank" ><picture> <source srcset="/blog/images/django/lumigoComparison.webp" type="image/webp"> <source srcset="/blog/images/django/lumigoComparison.png" type="image/png"> <img src="/blog/images/django/lumigoComparison.png" title="From 'AWS Lambda Deployment Frameworks', by Yan Cui at lumigo.io" class=" fullsize liImg2 rounded shadow" alt="From 'AWS Lambda Deployment Frameworks', by Yan Cui at lumigo.io" /> </picture></a> <figcaption class="fullsize" style="width: 100%; text-align: center;"> <a href="https://lumigo.io/aws-lambda-deployment/" target="_blank" > From 'AWS Lambda Deployment Frameworks', by Yan Cui at lumigo.io </a> </figcaption> </figure> </div> <p> The trade-off between customizability and opinionatedness is that highly customizable frameworks require more code to do things that opinionated frameworks do more succinctly. On the other hand, very opinionated frameworks are more limited in their abilities. A classic example of an opinionated framework is Ruby on Rails, which is specifically designed for master/detail applications. Other types of applications should use a different framework, or no framework at all. </p> <p> Two of the technologies on the above infographic are Zappa and Terraform, both of which I discuss in this blog post. Zappa is rather opinionated, while Terraform is very customizable. </p> </editor-fold> <editor-fold cdk> <h2 class="numbered" id="cdk">AWS Cloud Development Kit (CDK)</h2> <p> AWS CDK provides a programmatic interface for modeling and provisioning cloud resources. Languages supported include Java, JavaScript, .NET, Node.js, Python and Typescript. </p> <p> Even if AWS is not directly the service provider, awareness of the <a href='https://aws.amazon.com/cdk/' target='_blank' rel='nofollow'>AWS CDK</a> is important because some other options, for example the <a href='https://aws.amazon.com/blogs/developer/introducing-the-cloud-development-kit-for-terraform-preview/' target='_blank' rel='nofollow'>Cloud Development Kit for Terraform</a> (cdktf), are based on <a href='https://github.com/aws/aws-cdk' target='_blank' rel='nofollow'>AWS CDK</a>. </p> </editor-fold> <editor-fold chalice> <h2 class="numbered" id="chalice">Chalice &ndash; Serverless Django on AWS</h2> <p> <a href='https://aws.github.io/chalice/' target='_blank' rel='nofollow'>Chalice</a> is an AWS open-source project that has good traction. This Python serverless microframework for AWS allows applications that use Amazon API Gateway and AWS Lambda to be quickly created and deployed. </p> <p> The name and logo of this project are suggestive of the Holy Grail. I found the thinly veiled references to Christianity to be off-putting. Religious references have no place in a professional environment. Programmers who work with this project have religious icons, words and phrases continuously presented to them, and they must write words that are strongly identified with Christian doctrine for them to write software. This is forced <a href='https://www.vocabulary.com/dictionary/indoctrination' target='_blank' rel='nofollow'>indoctrination</a>. </p> </editor-fold> <editor-fold zappa> <h2 class="numbered" id="zappa">Django w/ Zappa &amp; AWS Lambda</h2> <div style="text-align: right;"> <picture> <source srcset="/blog/images/django/zappa_400x400.webp" type="image/webp"> <source srcset="/blog/images/django/zappa_400x400.png" type="image/png"> <img src="/blog/images/django/zappa_400x400.png" title="Don't eat the yellow snow" class="right quartersize liImg2 rounded shadow" alt="Don't eat the yellow snow" /> </picture> </div> <p> <a href='https://github.com/zappa/Zappa' target='_blank' rel='nofollow'>Zappa</a> is a popular library for serverless web hosting of Python webapps. Zappa allows Python WSGI webapps like Django to run on AWS Lambda instead of from within a container like AWS EC2. I am particularly interested in using Zappa to package and run <code>django-oscar</code> for AWS Lambda and CloudFront. </p> <p> Zappa can perform two primary functions: </p> <ol> <li> <b>Packaging</b> &ndash; Zappa can build a Django webapp into an AWS Lambda package. The package can be delivered via other mechanisms, for example mechanisms that are not even Python aware. </li> <li> <b>Deploying</b> &ndash; Zappa can deploy and Django webapp to AWS Lambda, and configure several AWS services to feed events to the Django webapp. </li> </ol> <div class="quote"> Zappa does not provide a means to define additional resources as part of the overall infrastructure. It is also somewhat rigid in how it defines certain resources which can lead to friction when incorporating Zappa within organizations with more rigid requirements on cloud resource management. With Zappa, you are better off allowing it to manage all the pieces needed for your web application on its own and manage other resources with a separate tool such as stacker or Terraform. <br><br> &hellip; or use Zappa's <code>package</code> command to create an archive that is ready for upload to lambda and utilize the other helpful functions the project provides for use after code is deployed. <br><br> &nbsp; &ndash; from <a href='https://www.jbssolutions.com/resources/blog/evolution-maintainable-lambda-development-pt-2/' target='_blank' rel='nofollow'>The Evolution of Maintainable Lambda Development Pt 2</a> by JBS Custom Software Solutions. </div> <p> The Zappa documentation is excellent. The project has some rough edges, but the new regime coming on board seem competent and fired up. They have some work ahead to set things straight, but the technical path seems clear. </p> <p> I think this project deserves special attention. Lots of moldy issues and PRs need to be processed, which a small team could get done fairly quickly. The project might also benefit from someone to hone the messaging. I opened an <a href='https://github.com/zappa/Zappa/issues/968' target='_blank' rel='nofollow'>issue on the Zappa GitHub microsite</a> to discuss this. </p> <p> This seminal project has been around several years, and other well-known projects that have been developed since Zappa was first released have acknowledged that Zappa provided inspiration. Time to brush it up and set it straight again; its best days lie ahead! </p> <p> Edgar Roman wrote this helpful document: <a href='https://romandc.com/zappa-django-guide/' target='_blank' rel='nofollow'>Guide to using Django with Zappa</a>. </p> <p> I've messing around with Zappa, will report back. </p> <h3 id="videos">Videos</h3> <p> <a href='https://www.google.com/search?client=firefox-b-d&q=aws+zappa+django+video' target='_blank' rel='nofollow'>Videos of Zappa exist</a>. </p> <ul> <li> This video has got all the right technologies mixed together for me: <a href='https://www.youtube.com/watch?v=Gf0vpJQZeBI' target='_blank' rel='nofollow'>Serverless Deployment of a Django Project with AWS Lambda, Zappa, S3 and PostgreSQL</a>. </li> </ul> </editor-fold> <editor-fold option_djambda> <h2 class="numbered" id="djambda">Djambda / AWS Lambda / Terraform</h2> <p> Terraform does not impose a runtime dependency unless the realtime orchestration features are used. </p> <p> <a href='https://github.com/netsome/djambda' target='_blank' rel='nofollow'>Djambda</a> is an example project setting up Django application in AWS Lambda managed by Terraform. I intend to play with it and write up my experience right here Real Soon Now. </p> <p> This project uses GitHub Actions to create environments for the master branch and pull requests. I wonder if this project can be used without GitHub actions? </p> <div class="quote"> [Terraform] does not provide an abstraction layer for the AWS, Azure, or Google Cloud. It does that deliberately, as you should embrace all aspects when using cloud - not extract a common denominator from the services delivered by the cloud provider. <br><br> &nbsp; &ndash; From <a href='https://awsmaniac.com/aws-cdk-why-not-terraform/' target='_blank' rel='nofollow'>AWS CDK? Why not Terraform?</a> by Wojciech Gawroński. </div> </editor-fold> <editor-fold option_serverless> <h2 class="numbered" id="serverless">Serverless Framework with WSGI</h2> <p> The <a href='https://www.serverless.com/plugins/serverless-wsgi' target='_blank' rel='nofollow'>docs</a> describe Serverless WSGI as: </p> <div class="quote"> Serverless plugin to deploy WSGI applications (Flask/Django/Pyramid etc.) and bundle Python packages. </div> <p> I am concerned that the Serverless architecture requires an <a href='https://www.serverless.com/pricing/fair-use-policy/' target='_blank' rel='nofollow'>ongoing runtime dependency</a> on the viability and good will of Serverless, Inc. Any hiccup on their part will immediately be felt by all their users. It would make me nervous to base daily operational infrastructure on this. </p> </editor-fold> <editor-fold poor-trade> <h3 id="bintray">Bintray and JCenter Went <i>Poof!</i></h3> <p> I do not want to rely upon online services from a software tool vendor to run my builds. The Scala community is still recovering from Bintray and JCenter shutting down. I had dozens of Scala libraries on Bintray. I do not plan to migrate them, they are gone from public access. </p> <div class="quote"> On February 3, 2021, JFrog announced that they will be shutting down Bintray and JCenter. A complete shutdown is planned for February 2022. <br><br> &ndash; <a href='https://blog.gradle.org/jcenter-shutdown' target='_blank' rel='nofollow'>JCenter shutdown impact on Gradle builds</a> </div> <h3 id="free">Trading Autonomy for Minimal Convenience is a Poor Trade</h3> <p> Remember that free products are usually subject to change or termination without notice. Examples abound of many companies whose free (and non-free) products suddenly ceased. There is no need to assume this type of vulnerability, so I block my metaphoric ears to the siren sound that tempts trusting souls into assuming unnecessary dependencies, and I chose tooling that is completely under my control. </p> <div class="quote"> <h2>What happens if I exceed the fair use policy?</h2> <p> <i>From the <a href='https://www.serverless.com/pricing/' target='_blank' rel='nofollow'>Serverless Pricing and Terms page</a>.</i> </p> We want to offer a lot of value for free so you can get your idea off the ground, before introducing any infrastructure cost. The intent of the fair use policy is to ensure that we can provide a high quality of service without incurring significant infrastructure costs. The great majority of users will fall well within range of the typical usage guidelines. While we reserve the right to throttle services if usage exceeds the fair use policy, we do not intend to do so as long as we can deliver a high quality of service without significant infrastructure costs. <br><br> If you anticipate your project will exceed these guidelines, please contact our support team. We’ll work with you on a plan which scales well. </div> </editor-fold> <h2 id="apprunner">AWS AppRunner</h2> <p> AWS just announced <a href='https://aws.amazon.com/apprunner/' target='_blank' rel='nofollow'>AppRunner</a>. I wonder how suitable it is... </p> Merging a Remote File with a Local File 2021-04-12T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/12/merging-remote-file <p> Today I am once again re-installing WSL2 on one of my laptops. Seems that a Windows 10 installation&rsquo;s half-life is measured in months, after which time a reset is required. The reset preserves data, but not installed programs and not the WSL setup. </p> <p> When I set up an OS I often use a pre-existing system&rsquo;s files as templates for the new system&rsquo;s files. </p> <h2 id="meld">Meld</h2> <p> <a href='https://meldmerge.org/' target='_blank' rel='nofollow'>Meld</a> is a fantastic, F/OSS file and directory merge tool. 2-way and 3-way merges are supported. Meld uses X Windows for its user interface. <a href='https://opticos.github.io/gwsl/' target='_blank' rel='nofollow'>GWSL</a> makes it easy to run X apps on WSL and WSL2. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id05df73d511a1'><button class='copyBtn' data-clipboard-target='#id05df73d511a1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install meld</pre> <p> Merging a remote file with a local file using Meld is easy once you know how. Unless the remote file system is mounted locally, Meld cannot be used to modify <i>remote</i> files and directories, just <i>local</i> files and directories. </p> <p> Following is the incantation I used to display my local <code>.profile</code> and interactively merge it with my profile on an Ubuntu Linux machine called <code>gojira</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd06ceefe65ff'><button class='copyBtn' data-clipboard-target='#idd06ceefe65ff' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>meld ~/.profile &lt;(ssh mslinn@gojira cat .profile)&</pre> <p> The above runs <code>ssh</code> in a subshell, logs in as <code>mslinn</code> to the machine called <code>gojira</code> and then displays the contents of <code>.profile</code> on <code>gojira</code>. Meld compares the output of <code>cat</code> with the local copy of <code>~/.profile</code>, and displays the differences: </p> <div style=""> <picture> <source srcset="/blog/images/mergeRemote/meld.webp" type="image/webp"> <source srcset="/blog/images/mergeRemote/meld.png" type="image/png"> <img src="/blog/images/mergeRemote/meld.png" class=" liImg2 rounded shadow" /> </picture> </div> <div class="right" style="font-size: 3em;">&#128513;</div> <p> Meld makes it easy to reconcile file versions. </p> Visual Studio Code Workspace Settings 2021-04-11T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/11/svcode-workspace-settings <p> For me, the killer feature that Visual Studio Code is how it integrates the Windows user interface with working on WSL and WSL2. Programs residing on the active WSL OS image execute natively on that OS, while VSCode continues to run as a native Windows application. This is possible because VSCode installs a proxy on the target OS. The proxy does the bidding of the Windows executable. </p> <p> Getting a project to execute on the target OS instead of the host OS can be tricky. I have found that using a workspace to hold a collection of VSCode projects is very helpful, because the definition of the collection also defines how they are handled. </p> <p> WSL projects have different types of VSCode workspace entries than Windows entries do. They are easy to recognize and change once you know what to look for. The two possibile types of VSCode workspace project entries in a <code>.workspace</code> file are: </p> <ul> <li><b>WSL Project</b> &mdash; <code>"uri": "vscode-remote://wsl+ubuntu/path/to/vscode/project"</code></li> <li><b>Windows Project</b> &mdash; <code>"path": "C:\\path\\to\\vscode\\project"</code></li> </ul> <p> The following VSCode workspace file has both types of entries. For me, this is an error; I only want WSL projects. My task is to change the yellow highlighted Windows project and make it look like the other WSL projects. </p> <div class='codeLabel unselectable' data-lt-active='false'>aw.workspace.code_workspace</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id16f7ca4cce13'>{ "folders": [ { "uri": "vscode-remote://wsl+ubuntu/var/sitesUbuntu/www.ancientwarmth.com" }, { "uri": "vscode-remote://wsl+ubuntu/var/work/django/django" }, { "uri": "vscode-remote://wsl+ubuntu/var/work/django/oscar" }, { "uri": "vscode-remote://wsl+ubuntu/var/work/ancientWarmth/ancientWarmth" }, { <span class="bg_yellow">"path": "../../var/work/django/main"</span> } ], "remoteAuthority": "wsl+Ubuntu", "settings": { "liveServer.settings.multiRootWorkspaceName": "www.mslinn.com", "python.pythonPath": "/var/work/django/oscar/bin/python", "git.ignoreLimitWarning": true, "sqltools.connections": [ { "previewLimit": 50, "server": "localhost", "port": 5432, "driver": "PostgreSQL", "name": "Ancient Warmth on Camille", "database": "ancient_warmth", "username": "postgres", "password": "hithere" } ] } }</pre> <p> All I need to do is change this entry: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idedbd8d4d9592'><span class="bg_yellow">"path": "../..</span>/var/work/django/main"</pre> <p>To:</p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0f93baeeb4c7'><span class="bg_yellow">"uri": "vscode-remote://wsl+ubuntu</span>/var/var/work/django/main"</pre> <div class="right" style="font-size: 3em;">&#128513;</div> <p> The modified entry will cause VSCode to launch the project from WSL, instead of Windows. </p> Replicating a Git Directory Tree 2021-04-10T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/10/git-tree <p> Whenever I set up an operating system for one of my computers one of the tedious tasks that must be performed is to replicate the git repositories. </p> <p> It is a bad idea to attempt to copy an entire git repository between computers, because the <code>.git</code> directories within them can quite large. So large, in fact, that it might much more time to copy than re-cloning. I think the reason is that copying the entire git repo actually means copying the same information twice: first the <code>.git</code> hidden directory, complete with all the history for the project, and then again for the files in the currently checked out branch. Git repos store the entire development history of the project in their <code>.git</code> directories, so they are often much larger than the actual code that is checked out at any given time. </p> <p> I have several trees of git repositories, grouped into subdirectories. Here is a sanitized depiction of one of my git directory trees: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9ebc7afc0dd8'>├── cadenzaHome │   ├── cadenzaAssets │   ├── cadenzaCode │   │   ├── cadenzaClient │   │   ├── cadenzaCourseCode │   │   ├── cadenzaDependencies │   │   ├── cadenzaLibs │   │   ├── cadenzaServer │   │   ├── cadenzaServerNext │   │   └── cadenzaSupport │   ├── cadenzaCreative │   │   └── cadenzaCreativeTemplates │   ├── cadenzaCreativeBackup │   └── cadenzaCurriculum ├── django │   ├── django │   ├── django-oscar │   ├── frobshop │   ├── main │   └── oscar ├── jekyll │   ├── jekyllTemplate │   └── jekyll-flexible-include-plugin</pre> <p> Some git repos are forks, and I defined <code>upstream</code> git remotes for them, in addition to the usual <code>origin</code> remote. </p> <p> This morning I found myself facing the boring task of doing this manually once again. Instead, I wrote this script, which scans a git directory tree and writes out a script that clones the repos in the tree, and adds <code>upstream</code> remotes as required. Directories containing a file called <code>.ignore</code> are ignored. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,gitUrls' download='gitUrls' title='Click on the file name to download the file'>gitUrls</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="ida9fbeed70e2f">#!/bin/bash function help &#123; printf "$1Replicates tree of git repos " exit 1 &#125; function doOne &#123; cd "$CLONE_DIR" > /dev/null PROJECT_DIR="$( basename "$CLONE_DIR" )" # Might have been renamed after cloning # echo "CLONE_DIR: $CLONE_DIR" # echo "PROJECT_DIR: $PROJECT_DIR" ORIGIN_URL="$( git config --local remote.origin.url )" CLONE_DIR_PARENT="$( realpath "$CLONE_DIR/.." )" echo "mkdir -p '$CLONE_DIR_PARENT'" echo "pushd '$CLONE_DIR_PARENT' > /dev/null" echo "git clone '$ORIGIN_URL'" UPSTREAM_URL="$( git config --local remote.upstream.url )" if [ "$UPSTREAM_URL" ]; then if [ "$ORIGIN_URL" != "no_push" ]; then echo "cd \"$PROJECT_DIR\"" echo "git remote add upstream '$UPSTREAM_URL'" fi fi echo "popd > /dev/null" GIT_DIR_NAME="$( basename "$PWD" )" if [ "$GIT_DIR_NAME" != "$PROJECT_DIR" ]; then echo "# Git project directory was renamed, renaming this copy to match original directory structure" echo "mv \"$GIT_DIR_NAME\" \"$PROJECT_DIR\"" fi echo &#125; if [ -z "$1" ]; then help "Error: Please specify the subdirectory to traverse.\n\n"; fi BASE="$1" DIRS="$( find $BASE -type d \( -execdir test -e &#123;&#125;/.ignore \; -prune \) -o \( -execdir test -d &#123;&#125;/.git \; -prune -print \) )" for DIR in $DIRS; do export CLONE_DIR="$( realpath "$DIR" )" doOne done </pre> <p> Here is the output generated for the above directory tree: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id343ba7b09c17'><span class='unselectable'>$ </span>gitUrls $work <span class='unselectable'>mkdir -p '/var/work/cadenzaHome/cadenzaCreative' pushd '/var/work/cadenzaHome/cadenzaCreative' > /dev/null git clone 'git@github.com:mslinn/cadenzaCreativeTemplates.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome' pushd '/var/work/cadenzaHome' > /dev/null git clone 'git@github.com:mslinn/cadenzaAssets.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaSupport' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaSupport' > /dev/null git clone 'git@github.com:mslinn/dottyTemplate.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' > /dev/null git clone 'git@github.com:mslinn/scalacourses-play-utils.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' > /dev/null git clone 'git@github.com:mslinn/scalacourses-utils.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaLibs' > /dev/null git clone 'git@github.com:mslinn/scalacourses-slick-utils.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode' pushd '/var/work/cadenzaHome/cadenzaCode' > /dev/null git clone 'git@bitbucket.org:mslinn/cadenzaserver.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaCourseCode/ScalaCourses.com/group_scalaCore' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaCourseCode/ScalaCourses.com/group_scalaCore' > /dev/null git clone 'ssh://git@bitbucket.org/mslinn/course_scala_intro_code.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode/cadenzaCourseCode/ScalaCourses.com/group_scalaCore' pushd '/var/work/cadenzaHome/cadenzaCode/cadenzaCourseCode/ScalaCourses.com/group_scalaCore' > /dev/null git clone 'git@bitbucket.org:mslinn/course_scala_intermediate_code.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome/cadenzaCode' pushd '/var/work/cadenzaHome/cadenzaCode' > /dev/null git clone 'git@github.com:mslinn/cadenzaClient.git' popd > /dev/null mkdir -p '/var/work/cadenzaHome' pushd '/var/work/cadenzaHome' > /dev/null git clone 'git@github.com:mslinn/cadenzaCurriculum.git' popd > /dev/null mkdir -p '/var/work' pushd '/var/work' > /dev/null git clone 'git@github.com:mslinn/jekyllTemplate.git' popd > /dev/null mkdir -p '/var/work/django' pushd '/var/work/django' > /dev/null git clone 'git@github.com:mslinn/django-oscar.git' cd "django-oscar" git remote add upstream 'git@github.com:django-oscar/django-oscar.git' popd > /dev/null mkdir -p '/var/work/django' pushd '/var/work/django' > /dev/null git clone 'git@github.com:mslinn/frobshop.git' popd > /dev/null mkdir -p '/var/work/django' pushd '/var/work/django' > /dev/null git clone 'git@github.com:mslinn/django.git' cd "django" git remote add upstream 'git@github.com:django/django.git' popd > /dev/null mkdir -p '/var/work/jekyll' pushd '/var/work/jekyll' > /dev/null git clone 'git@github.com:mslinn/jekyll-flexible-include-plugin.git' cd "jekyll-flexible-include-plugin" git remote add upstream 'https://idiomdrottning.org/jekyll-include-absolute-plugin' popd > /dev/null mkdir -p '/var/work/jekyll' pushd '/var/work/jekyll' > /dev/null git clone 'git@github.com:mslinn/jekyllTemplate.git' popd > /dev/null </span></pre> <div class="right" style="font-size: 3em;">&#128513;</div> <p> Now all I had to do was paste the above bash commands into a terminal on the new system, and a short time later the git repositories were set up the way I needed them. </p> A Python Virtual Environment For Every Project 2021-04-09T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/09/python-venvs <p> Python virtual environments are cheap to make and use. I have adopted the habit of making a Python virtual environment (<i>venv</i>) for each significant Python project, plus a default venv for trivial Python work. </p> <p> Dedicating a venv for each Python project means that dependencies for any given Python project do not impact the dependencies for any other Python projects. Things just work better. </p> <editor-fold free> <h2 id="free">VEnvs are Nearly Free</h2> <p> The cost of a venv is virtually free. This is because by default, the executable images are linked, so they do not require much storage space. The <code>ls</code> command below shows that the <code>python</code> program in the <code>aw</code> venv is linked to <code>/usr/bin/python3.8</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id83e9e5c60289'><button class='copyBtn' data-clipboard-target='#id83e9e5c60289' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls -go ~/venv/aw/bin/python <span class='unselectable'>lrwxrwxrwx 1 18 Apr 9 06:01 <span class="bg_yellow">/home/mslinn/venv/aw/bin/python -> /usr/bin/python3.8</span> </span></pre> </editor-fold> <editor-fold std> <h2 id="standard">Standard Procedure For Creating a VEnv</h2> <p> I name each venv the same as my python project. My projects are stored under the directory pointed to by <code>$work</code>. </p> <p> My standard procedure when making a Python project called <code>$work/blah</code> is to also create a venv for it at <code>~/venv/blah</code>. A bash alias could be defined called <code>blah</code> that activates the venv and <code>cd</code>s into the project directory: </p> <div class='codeLabel unselectable' data-lt-active='false'>~/.bash_aliases</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0d9ce2114469'><button class='copyBtn' data-clipboard-target='#id0d9ce2114469' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>alias blah="source ~/venv/blah/bin/activate; cd $work/blah"</pre> <div class="right" style="font-size: 3em;">&#128513;</div> <p> Now you could type <code>blah</code> at a shell prompt and you would be working on that project. Boom! </p> </editor-fold> <editor-fold create_script> <h2 id="create_script" class="clear">Script For Creating a VEnv</h2> <p> Here is a bash script that creates the venv and changes <code>~/.bashrc</code> and <code>~/.bash_aliases</code> for you. It assumes that you keep your projects under <code>$work</code>. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,newVenv' download='newVenv' title='Click on the file name to download the file'>newVenv</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id64fd3fe1ad26">#!/bin/bash function help &#123; echo -e "$1$(basename $0) - Create a Python virtual environment with a given name. Usage: $(basename $0) venv_name The new virtual environment will be created under ~/venv/. If a project directory called \$work/venv_name exists before this script runs, then a bash alias is created named after the venv." exit 1 &#125; if [ -z `which virtualenv` ]; then sudo apt install virtualenv; fi if [ -z "$1" ]; then help "Please specify a name for the virtual environment.\n\n"; fi if [ "$1" == -h ]; then help; fi VENV="$1" shift mkdir -p "$HOME/venv" cd "$HOME/venv" virtualenv "$VENV" DIR="$HOME/venv/$VENV" echo "source $DIR/bin/activate" >> $HOME/.bashrc echo echo "Activation for "$VENV" in future shells was appended to $HOME/.bashrc" echo "To activate the "$VENV" venv in this shell right now, type: source ~/venv/$VENV/bin/activate" if [ "$work" ] &amp;&amp; [ -d "$work/$VENV" ]; then echo "alias $VENV='source $DIR/bin/activate; cd $work/$VENV'" >> $HOME/.bash_aliases echo "An alias called $VENV for future shells was appended to $HOME/.bash_aliases" echo "To define the alias in this shell right now, type: alias $VENV='source $DIR/bin/activate; cd $work/$VENV'" else echo "To define an alias, type something like this: alias $VENV=\"source $DIR/bin/activate; cd $work/$VENV\"" fi </pre> <p> This is the help message for the script: </p> <div class='codeLabel unselectable' data-lt-active='false'>newVenv help message</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id39090d35d4e6'><button class='copyBtn' data-clipboard-target='#id39090d35d4e6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>newVenv -h <span class='unselectable'>newVenv - Create a Python virtual environment with a given name. Usage: newVenv venv_name The new virtual environment will be created under ~/venv/. If a project directory called $work/venv_name exists before this script runs, then a bash alias is created named after the venv. </span></pre> <p> Let's use the script to create a venv called <code>aw</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>newVenv help message</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id02338155b43d'><button class='copyBtn' data-clipboard-target='#id02338155b43d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>newVenv aw <span class='unselectable'>created virtual environment CPython3.8.6.final.0-64 in 528ms creator CPython3Posix(dest=/home/mslinn/venv/aw, clear=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/mslinn/.local/share/virtualenv) added seed packages: pip==20.1.1, pkg_resources==0.0.0, setuptools==44.0.0, wheel==0.34.2 activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator </span></pre> </editor-fold> <editor-fold use> <h2 id="use">Script for Using a VEnv</h2> <p> Here is a script that can display the available Python virtual environments, and optionally activates one them. It does not use bash aliases. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,use' download='use' title='Click on the file name to download the file'>use</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id8193480a44e6">#!/bin/bash function help &#123; echo "Usage:" for f in $HOME/venv/*; do if [ -d "$f" ]; then echo " . $(basename $0) $(basename $f)"; fi done return 2 &#125; unset PV if [ "$1" == -h ]; then help elif [ "$1" ]; then PV="$1" else PV="default" fi if [ "$PV" ]; then DIR="$HOME/venv/$PV" if [ ! -d "$DIR" ]; then echo "Error: $DIR does not exist." return 1 fi if [ ! -f "$DIR/bin/python" ]; then echo "Error: No Python virtual environment is installed in $DIR" return 1 fi echo "Setting Python virtual environment to $DIR" source "$DIR/bin/activate" fi </pre> <p> Here are examples of using the script to change virtual environments: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc2dcc13bb675'><span class='unselectable'>$ </span>. use -h <span class='unselectable'>Usage: . bash aw . bash default </span> <span class='unselectable'>$ </span>. use <span class='unselectable'>Setting Python virtual environment to /home/mslinn/venv/default </span> <span class='unselectable'>(default) $ </span>. use aw <span class='unselectable'>Setting Python virtual environment to /home/mslinn/venv/aw </span> <span class='unselectable'>(aw) $ </span></pre> <p> Notice that the last command above changed the shell prompt, in that <code>(aw)</code> was prepended to the normal prompt. To cause all future shells to use this virtual environment by default, the script adds a line to <code>~/.bashrc</code> that looks like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf91b08373b23'><button class='copyBtn' data-clipboard-target='#idf91b08373b23' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo "source ~/venv/aw/bin/activate" &gt;&gt; ~/.bashrc</pre> <p> At this point the virtual environment just contained executable images for Python. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id25e4e136fc13'><span class='unselectable'>$ </span>ls ~/venv/aw/** <span class='unselectable'>~/venv/aw/pyvenv.cfg ~/venv/aw/bin: activate activate.ps1 chardetect distro easy_install pip pip3.8 python3.8 wheel3 activate.csh activate.xsh chardetect-3.8 distro-3.8 easy_install-3.8 pip-3.8 python wheel activate.fish activate_this.py chardetect3 distro3 easy_install3 pip3 python3 wheel-3.8 ~/venv/aw/lib: python3.8 </span></pre> </editor-fold> <editor-fold deactivate> <h2 id="deactivate">Deactivate a VEnv</h2> <p> Stop using venvs with `deactivate`: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfabe336a351a'><span class='unselectable'>(aw) $ </span>deactivate <span class='unselectable'>$ </span></pre> </editor-fold> <editor-fold alias2> <h2 id="alias2">Activate With an Alias</h2> <p> Once again we can use a bash alias, this time to invoke the <code>use</code> script. We can call the alias <code>use</code>, because bash aliases have precedence over bash scripts. This alias removes the need to type <code>.</code> or <code>source</code> before the script name (which you know is <code>use</code>, if you have been following along). </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id72e88cd4bc41'><span class='unselectable'>$ </span>alias use="source use" <span class='unselectable'>$ </span>use <span class='unselectable'>(default) $ </span>use aw <span class='unselectable'>(aw) $ </span></pre> <p> You can add the alias to <code>bash_aliases</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idda31e33c8dae'><span class='unselectable'>$ </span>echo 'alias use="source use"' >> ~/.bash_aliases</pre> <div class="right" style="font-size: 3em;">&#128513;</div> </editor-fold> <editor-fold virt> <h2 id="virt">Directory-Locked Python Virtualization</h2> <p> After setting up a Python virtual environment, a quick examination of the <code>pip</code> script shows that it is hard-coded to the directory that it was made for: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4a7b5bb45f30'><button class='copyBtn' data-clipboard-target='#id4a7b5bb45f30' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>head -n 1 ~/venv/aw/bin/pip <span class='unselectable'><span class="bg_yellow">#!/home/mslinn/venv/aw/bin/python</span> </span></pre> <p> For virtualized environments, such as Docker, this means that a Python virtual environment created without Docker can only be used within a Docker image if the path to it is the same from within the Docker image as when it was created. </p> </editor-fold> <h2 id="updating">Updating Python</h2> <p> To update the version of Python in a venv, just run the same command that you used to create the venv in the first place. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id163bbeb9c2aa'><button class='copyBtn' data-clipboard-target='#id163bbeb9c2aa' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>virtualenv ~/venv/aw</pre> <h2 id="further">For Further Reading</h2> <p> <a href='https://mitelman.engineering/blog/python-best-practice/automating-python-best-practices-for-a-new-project/' target='_blank' rel='nofollow'>Python Best Practices for a New Project in 2021</a> </p> <editor-fold summary> <h2 id="summary">Summary</h2> <ul> <li>Demonstrated how to make an alias for working with Python virtual environments (<i>venvs</i>) that are coupled with Python projects.</li> <li>The <code>newVenv</code> bash script was demonstrated for making new Python virtual environments.</li> <li>The <code>use</code> bash source script was demonstrated for activating a venv.</li> <li>Deactivating the current venv was demonstrated using the <code>deactivate</code> command, provided with every venv.</li> <li>The <code>use</code> alias for <code>source use</code> was demonstrated for more conveniently selecting a venv.</li> <li>Locked directories mean that Python virtual environments should normally only be created in the same environment they are intended to be used.</li> </ul> </editor-fold> Escaping HTML on Clipboard From a Windows Hot Key via WSL 2021-04-03T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/03/escape-html-clipboard <p> I frequently show HTML source code when I write. That HTML must be escaped prior to displaying it on a web page. </p> <h2 id="bash_script">Script to Apply HTML Escape to Clipboard</h2> <p> This bash script applies an HTML escape conversion to the contents of the system clipboard. If you have WSL on your machine, you could store it on the WSL file system, for example in <code>~/.local/bin/escapeHtml</code>. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,escapeHtml' download='escapeHtml' title='Click on the file name to download the file'>escapeHtml</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idb1dc265c3ab8">#!/bin/bash # SPDX-License-Identifier: Apache-2.0 function help &#123; echo "$(basename $0) - Escapes HTML with entities. Reads from STDIN or pipe, or converts the clipboard. Result is copied to the clipboard. " exit 1 &#125; function filesAreLinked &#123; "$1" -ef "$2" &#125; function checkDependencies &#123; if [ -z `which recode` ]; then yes | sudo apt install recode; fi # See https://github.com/sindresorhus/clipboard-cli if [ -z `which clipboard` ]; then if [ "$( filesAreLinked /bin/npm /usr/local/lib/node_modules/npm/bin/npm-cli.js )" ]; then # No nodejs venv sudo -H npm install --global clipboard-cli else # nodejs venv npm install --global clipboard-cli fi fi &#125; if [ "$1" == -h ]; then help; fi checkDependencies if [ -t 0 ]; then # Not reading from a terminal HTML="$( clipboard )" else # Reading from stdin or pipe # See https://stackoverflow.com/a/32365596/553865 HTML=$(cat; echo x) HTML=$&#123;HTML%x&#125; # Strip the trailing x fi RESULT="$( recode utf8..html &lt;&lt;&lt; "$HTML" )" echo "$RESULT" | sed "s/&amp;#13;//g" | sed "s/'/\&amp;#39;/g" | clipboard echo "$( echo "$RESULT" | wc -l ) lines have been placed on the clipboard." </pre> <h2 id="use">Using the Script</h2> <ol> <li>Select some text in any document or anywhere that text can be selected.</li> <li>Run <code>escapeHtml</code> on the same machine. If you have Windows with WSL you can run the script there, or run it in native Windows, does not matter.</li> <li>Paste escaped HTML into your target document.</li> </ol> <h2 id="trigger">Hot Key Trigger</h2> <p> Trigger the script with a hot key via your OS's facilities. This section just discusses how native Windows hot keys can be used to trigger this script running in WSL. </p> <ol> <li>Right-click in a folder</li> <li>Select <b>New</b> / <b>Shortcut</b></li> <li> For <b>Type the location of the item</b>, type:<br><br> <div class='codeLabel unselectable' data-lt-active='false'>Type the location of item</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idef33763d0a5c'><button class='copyBtn' data-clipboard-target='#idef33763d0a5c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>%windir%\System32\wsl.exe ~/.local/bin/escapeHtml/escapeHtml</pre> </li> <li>Click <b>Change icon</b> and select a retro icon for this shortcut.</li> <li>Click on <b>Apply</b>.</li> <li>Click on <b>Next</b>.</li> <li> When prompted for <b>Type a name for this shortcut</b>, save as <code>HTML Escape Clip to Clip</code>. </li> <li>Click on <b>Finish</b>.</li> <li>Right-click on the new shortcut and click in <b>Shortcut key</b></li> <li>I used <kbd>Ctrl</kbd>+<kbd>Alt</kbd>+<kbd>A</kbd>.</li> <li>Click on <b>OK</b>.</li> </ol> <div class="right" style="font-size: 3em;">&#128513;</div> <p> You can now quickly copy HTML from any source to the clipboard, apply an HTML escape conversion to the clipboard contents, and then paste the escaped HTML into an editor. </p> Microsoft Visual Studio Code Notes 2021-03-22T00:00:00-04:00 https://mslinn.github.io/blog/2021/03/22/vscode-notes <h2 id="keys">Useful Default Key Bindings</h2> <dl> <dt><kbd>Ctrl</kbd>+<kbd>K</kbd> &nbsp; <kbd>Ctrl</kbd>+<kbd>S</kbd></dt> <dd> <b>Display / edit the <a href='https://code.visualstudio.com/docs/getstarted/keybindings' target='_blank' rel='nofollow'>Keyboard Shortcuts</a> definitions.</b><br> You can filter the keybindings by pressing <kbd>Alt</kbd>+<kbd>K</kbd> or clicking the icon of the little keyboard at the top right of the Keyboard Shortcut page, then press the keys that you want to see the key binding for. The keyboard icon starts keystroke recording mode. Recording mode is sticky; each time you revisit the Keyboard Shortcuts tab you can just press the keys you are interested in to see their bindings. Step by step: <ol> <li> When you press the <kbd>Ctrl</kbd> key you will see <code>"ctrl"</code> displayed, and recording mode continues to listen to what you type. Don't do this right now, but FYI, if you toggle keystroke recording mode now, and then remove the quotes around <code>"ctrl"</code>, you will see a sorted list of all the key chords bound to <kbd>Ctrl</kbd>. </li> <li> Next, when you add the <kbd>Shift</kbd> key to the key chord, you then see <code>"ctrl+shift"</code> displayed. Don't do this right now, but FYI, if you toggle keystroke recording mode now, and then remove the quotes, you will see a sorted list of all the key chords bound to <kbd>Ctrl</kbd>+<kbd>Shift</kbd>. </li> <li>Finally, adding the <kbd>=</kbd> key to the key chord shows all the commands bound to <kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>=</kbd>.</li> </ol> </dd> <dt><kbd>Ctrl</kbd>+<kbd>K</kbd>+<kbd>0</kbd> (zero)</dt> <dd>Completely fold the active editor contents.</dd> <dt><kbd>Ctrl</kbd>+<kbd>K</kbd>+<kbd>1</kbd> (one)</dt> <dd>Fold level 1 the active editor contents.</dd> <dt><kbd>Ctrl</kbd>+<kbd>K</kbd>+<kbd>2</kbd></dt> <dd>Fold level 2 the active editor contents.</dd> <dt><kbd>Ctrl</kbd>+<kbd>B</kbd></dt> <dd>Toggle side bar visibility.</dd> <dt><kbd>Ctrl</kbd>+<kbd>P</kbd></dt> <dd> Show names of recently opened tabs, which might contain files to edit, or might be VSCode settings, or VSCode key bindings, etc. Click on a tab name to open it. This key binding is bound to <b>Go to File</b>, which is slightly logical but a not a good descriptive name. </dd> <dt><kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>P</kbd></dt> <dd>Open the <a href='https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette' target='_blank' rel='nofollow'>Command Palette</a>.</dd> <dt><kbd>Ctrl</kbd>+<kbd>L</kbd> &nbsp; <kbd>G</kbd></dt> <dd> Open the active (currently edited) file on GitHub in the default web browser. Requires the <a href='https://marketplace.visualstudio.com/items?itemName=sysoev.vscode-open-in-github' target='_blank' rel='nofollow'>Open in GitHub</a> extension. </dd> <dt><kbd>Ctrl</kbd>+<kbd>,</kbd> (comma)</dt> <dd>Open the <a href='https://code.visualstudio.com/docs/getstarted/settings' target='_blank' rel='nofollow'>Settings</a> tab.</dd> <dt><kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>T</kbd></dt> <dd>Reopen the most recently closed editor tab.</dd> <dt><kbd>Alt</kbd>+<kbd>Shift</kbd>+<kbd>F</kbd></dt> <dd>Format the current file.</dd> </dl> <h2 id="reveal">Annoying Side Bar Auto Reveal</h2> <h3>Problem</h3> <p> When editing a file, if you <kbd>Ctrl</kbd>+<kbd>click</kbd> on a function, method or class name defined in a dependency, the dependency's folder will be expanded in the side bar. Some dependencies are deeply nested, which means that the side bar expands quite a lot. In order to close the folders in the side bar it is necessary to go all the way back to the top of that folder, which is annoying and wastes time. </p> <h3>Solution</h3> <p> In settings, look for <b>Explorer: Auto Reveal</b>, which controls whether the explorer should automatically reveal and select files when opening them. </p> <p> To do that, bring up settings with <kbd>Ctrl</kbd>+<kbd>,</kbd> (comma), and then type <code>reveal ex</code> into the filter. </p> <p> The default value is <code>true</code>. Change the value to <code>false</code>. </p> <h3>Bonus: Reveal Active File in Side Bar</h3> <p> To scroll to a file that you are editing in the list of files in the side bar, right-click on the file's tab, then select <b>Reveal in side bar</b>. </p> <p> Even better, define a keyboard shortcut to do this: </p> <ol> <li> Bring up the <b>Keyboard Shortcuts</b> definitions by typing <kbd>Ctrl</kbd>+<kbd>K</kbd> &nbsp; <kbd>Ctrl</kbd>+<kbd>S</kbd>. </li> <li>Type <code>reveal side</code> into the search bar.</li> <li>Double-click on <b>File: Reveal Active File in Side Bar</b>.</li> <li> For my desired key shortcut, I pressed <kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>Alt</kbd>+<kbd>R</kbd>, then pressed <kbd>Enter</kbd>. </li> </ol> <h2 id="plugins">Plugins</h2> <ul> <li><a href='https://mitelman.engineering/blog/python-best-practice/automating-python-best-practices-for-a-new-project/#code-analysis-with-flake8-linter' target='_blank' rel='nofollow'>Flake 8</a></li> </ul> Command-Line AWS Utilities 2021-03-22T00:00:00-04:00 https://mslinn.github.io/blog/2021/03/22/command-line-aws-utilities <editor-fold intro> <p> Here are some command-line utilities I have written for AWS. They are dependent on <a href='https://aws.amazon.com/cli/' target='_blank' rel='nofollow'>aws cli</a>. You can <a href='/mslinn_aws.tar'>download all of these utilities</a> in tar format. Extract them into the current directory like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id637f84313a7e'><button class='copyBtn' data-clipboard-target='#id637f84313a7e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>tar xf mslinn_aws.tar</pre> </editor-fold> <editor-fold awsCfInvalidate> <h2 id="awsCfInvalidate"><span class="code">awsCfInvalidate</span></h2> <p> Given a CloudFront distribution ID, invalidate the distribution. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,awsCfInvalidate' download='awsCfInvalidate' title='Click on the file name to download the file'>awsCfInvalidate</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id503ef918ebbe">#!/bin/bash function help &#123; printf "$1$(basename $0) - Invalidate the CloudFront distribution for the given ID. If no distribution with the given ID exists, the empty string is returned and the return code is 2. A message is printed asynchronously to the console when the invalidation completes. Syntax: $(basename $0) distId Syntax: awsCfS3Dist www.mslinn.com | $(basename $0) " exit 1 &#125; function waitForInvalidation &#123; echo "Waiting for invalidation $2 to complete." aws cloudfront wait invalidation-completed \ --distribution-id "$1" \ --id "$2" echo "Invalidation $2 has completed." &#125; if [ "$1" == -h ]; then help; fi if [ "$1" ]; then DIST_ID="$1" shift elif [ ! -t 0 ]; then read -r DIST_ID fi if [ -z "$DIST_ID" ]; then help 'Error: No CloudFront distribution ID was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi JSON="$( aws cloudfront create-invalidation \ --distribution-id "$DIST_ID" \ --paths "/*" )" INVALIDATION_ID="$( jq -r .Invalidation.Id &lt;&lt;&lt; "$JSON" )" waitForInvalidation "$DIST_ID" "$INVALIDATION_ID" &amp; </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide3a6a420f710'><button class='copyBtn' data-clipboard-target='#ide3a6a420f710' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfInvalidate E2P5S6OYKQNB6B <span class='unselectable'>Waiting for invalidation IFOPKECU4YYHD to complete. </span> <span class='unselectable'><i>... do other things ...</i> </span> <span class='unselectable'>$ </span><span class='unselectable'>Invalidation IFOPKECU4YYHD has completed. </span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2d9e7a42ddb1'><button class='copyBtn' data-clipboard-target='#id2d9e7a42ddb1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfS3Dist www.mslinn.com | awsCfInvalidate <span class='unselectable'>Waiting for invalidation IFOPKECU4YYHD to complete. </span> <span class='unselectable'><i>... do other things ...</i> </span> <span class='unselectable'>$ </span><span class='unselectable'>Invalidation IFOPKECU4YYHD has completed. </span></pre> </editor-fold> <editor-fold awsCfS3Dist> <h2 id="awsCfS3Dist"><span class="code">awsCfS3Dist</span></h2> <p> Given an S3 bucket name, return the CloudFront distribution JSON. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,awsCfS3Dist' download='awsCfS3Dist' title='Click on the file name to download the file'>awsCfS3Dist</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id5059a5660985">#!/bin/bash function help &#123; printf "$1$(basename $0) - Obtain the CloudFront distribution JSON for an S3 bucket. If no S3 bucket with the given name exists, the empty string is returned and the return code is 2. Syntax: $(basename $0) bucketName Syntax: echo bucketName | $(basename $0) " exit 1 &#125; if [ "$1" == -h ]; then help; fi if [ "$1" ]; then BUCKET_NAME="$1" shift elif [ ! -t 0 ]; then read -r BUCKET_NAME fi if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi if [ "$( aws s3api head-bucket --bucket $BUCKET_NAME 2> >(grep -i 'Not Found') )" ]; then >&amp;2 echo "Error: Bucket $BUCKET_NAME does not exist." exit 2 fi DIST_ID="$( awsCfS3DistId "$BUCKET_NAME" )" if [ -z "$DIST_ID" ]; then exit 2; fi aws cloudfront get-distribution-config --id "$DIST_ID" </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id17e6ab480de1'><button class='copyBtn' data-clipboard-target='#id17e6ab480de1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfS3Dist www.mslinn.com <span class='unselectable'>{ "ETag": "E1DIZUSLMOLXKP", "DistributionConfig": { "CallerReference": "1454487160038", "Aliases": { "Quantity": 2, "Items": [ "www.mslinn.com", "mslinn.com" ] }, "DefaultRootObject": "index.html", "Origins": { "Quantity": 1, "Items": [ { "Id": "S3-www.mslinn.com", "DomainName": "www.mslinn.com.s3-website-us-east-1.amazonaws.com", "OriginPath": "", "CustomHeaders": { "Quantity": 0 }, "CustomOriginConfig": { "HTTPPort": 80, "HTTPSPort": 443, "OriginProtocolPolicy": "http-only", "OriginSslProtocols": { "Quantity": 3, "Items": [ "TLSv1", "TLSv1.1", "TLSv1.2" ] }, "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5 }, "ConnectionAttempts": 3, "ConnectionTimeout": 10 } ] }, "OriginGroups": { "Quantity": 0 }, "DefaultCacheBehavior": { "TargetOriginId": "S3-www.mslinn.com", "TrustedSigners": { "Enabled": false, "Quantity": 0 }, "ViewerProtocolPolicy": "redirect-to-https", "AllowedMethods": { "Quantity": 2, "Items": [ "HEAD", "GET" ], "CachedMethods": { "Quantity": 2, "Items": [ "HEAD", "GET" ] } }, "SmoothStreaming": false, "Compress": true, "LambdaFunctionAssociations": { "Quantity": 0 }, "FieldLevelEncryptionId": "", "CachePolicyId": "658327ea-f89d-4fab-a63d-7e88639e58f6" }, "CacheBehaviors": { "Quantity": 0 }, "CustomErrorResponses": { "Quantity": 2, "Items": [ { "ErrorCode": 403, "ResponsePagePath": "", "ResponseCode": "", "ErrorCachingMinTTL": 60 }, { "ErrorCode": 404, "ResponsePagePath": "", "ResponseCode": "", "ErrorCachingMinTTL": 60 } ] }, "Comment": "", "Logging": { "Enabled": false, "IncludeCookies": false, "Bucket": "", "Prefix": "" }, "PriceClass": "PriceClass_All", "Enabled": true, "ViewerCertificate": { "ACMCertificateArn": "arn:aws:acm:us-east-1:031372724784:certificate/2be42926-829c-4db9-be7d-a72e951256d4", "SSLSupportMethod": "sni-only", "MinimumProtocolVersion": "TLSv1", "Certificate": "arn:aws:acm:us-east-1:031372724784:certificate/2be42926-829c-4db9-be7d-a72e951256d4", "CertificateSource": "acm" }, "Restrictions": { "GeoRestriction": { "RestrictionType": "none", "Quantity": 0 } }, "WebACLId": "", "HttpVersion": "http1.1", "IsIPV6Enabled": false } } </span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9451236fc259'><button class='copyBtn' data-clipboard-target='#id9451236fc259' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo www.mslinn.com | awsCfS3Dist <span class='unselectable'>{ "ETag": "E1DIZUSLMOLXKP", "DistributionConfig": { "CallerReference": "1454487160038", "Aliases": { "Quantity": 2, "Items": [ "www.mslinn.com", "mslinn.com" ] }, "DefaultRootObject": "index.html", "Origins": { "Quantity": 1, "Items": [ { "Id": "S3-www.mslinn.com", "DomainName": "www.mslinn.com.s3-website-us-east-1.amazonaws.com", "OriginPath": "", "CustomHeaders": { "Quantity": 0 }, "CustomOriginConfig": { "HTTPPort": 80, "HTTPSPort": 443, "OriginProtocolPolicy": "http-only", "OriginSslProtocols": { "Quantity": 3, "Items": [ "TLSv1", "TLSv1.1", "TLSv1.2" ] }, "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5 }, "ConnectionAttempts": 3, "ConnectionTimeout": 10 } ] }, "OriginGroups": { "Quantity": 0 }, "DefaultCacheBehavior": { "TargetOriginId": "S3-www.mslinn.com", "TrustedSigners": { "Enabled": false, "Quantity": 0 }, "ViewerProtocolPolicy": "redirect-to-https", "AllowedMethods": { "Quantity": 2, "Items": [ "HEAD", "GET" ], "CachedMethods": { "Quantity": 2, "Items": [ "HEAD", "GET" ] } }, "SmoothStreaming": false, "Compress": true, "LambdaFunctionAssociations": { "Quantity": 0 }, "FieldLevelEncryptionId": "", "CachePolicyId": "658327ea-f89d-4fab-a63d-7e88639e58f6" }, "CacheBehaviors": { "Quantity": 0 }, "CustomErrorResponses": { "Quantity": 2, "Items": [ { "ErrorCode": 403, "ResponsePagePath": "", "ResponseCode": "", "ErrorCachingMinTTL": 60 }, { "ErrorCode": 404, "ResponsePagePath": "", "ResponseCode": "", "ErrorCachingMinTTL": 60 } ] }, "Comment": "", "Logging": { "Enabled": false, "IncludeCookies": false, "Bucket": "", "Prefix": "" }, "PriceClass": "PriceClass_All", "Enabled": true, "ViewerCertificate": { "ACMCertificateArn": "arn:aws:acm:us-east-1:031372724784:certificate/2be42926-829c-4db9-be7d-a72e951256d4", "SSLSupportMethod": "sni-only", "MinimumProtocolVersion": "TLSv1", "Certificate": "arn:aws:acm:us-east-1:031372724784:certificate/2be42926-829c-4db9-be7d-a72e951256d4", "CertificateSource": "acm" }, "Restrictions": { "GeoRestriction": { "RestrictionType": "none", "Quantity": 0 } }, "WebACLId": "", "HttpVersion": "http1.1", "IsIPV6Enabled": false } } </span></pre> </editor-fold> <editor-fold awsCfS3DistId> <h2 id="awsCfS3DistId"><span class="code">awsCfS3DistId</span></h2> <p> Given an S3 bucket name, return the CloudFront distribution ID. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,awsCfS3DistId' download='awsCfS3DistId' title='Click on the file name to download the file'>awsCfS3DistId</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id29463d81c345">#!/bin/bash function help &#123; printf "$1$(basename $0) - Obtain the CloudFront distribution ID for an S3 bucket. If no S3 bucket with the given name exists, the empty string is returned and the return code is 2. Syntax: $(basename $0) bucketName Syntax: echo bucketName | $(basename $0) " exit 1 &#125; if [ "$1" == -h ]; then help; fi if [ "$1" ]; then BUCKET_NAME="$1" shift elif [ ! -t 0 ]; then read -r BUCKET_NAME fi if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi if [ "$( aws s3api head-bucket --bucket $BUCKET_NAME 2> >(grep -i 'Not Found') )" ]; then >&amp;2 echo "Error: Bucket $BUCKET_NAME does not exist." exit 2 fi DIST_ID="$( aws cloudfront list-distributions \ --query "DistributionList.Items[*].&#123;id:Id,origin:Origins.Items[0].Id&#125;[?origin=='S3-$BUCKET_NAME'].id" \ --output text )" if [ -z "$DIST_ID" ]; then exit 2; fi echo "$DIST_ID" </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1a79bcc17e55'><button class='copyBtn' data-clipboard-target='#id1a79bcc17e55' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfS3DistId www.mslinn.com <span class='unselectable'>E2P5S6OYKQNB6B </span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id219cf72fd3ea'><button class='copyBtn' data-clipboard-target='#id219cf72fd3ea' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo www.mslinn.com | awsCfS3DistId <span class='unselectable'>E2P5S6OYKQNB6B </span></pre> </editor-fold> <editor-fold awsCfS3MakeDist> <h2 id="awsCfS3MakeDist"><span class="code">awsCfS3MakeDist</span></h2> <p> Creates a CloudFront distribution for the given bucket name. Returns the new distribution's ID. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,awsCfS3MakeDist' download='awsCfS3MakeDist' title='Click on the file name to download the file'>awsCfS3MakeDist</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id439c17953702">#!/bin/bash function help &#123; printf "$1$(basename $0) - Make a new CloudFront distribution for the given S3 bucket name. Returns the new distribution's ID. Syntax: $(basename $0) bucketName Syntax: echo bucketName | $(basename $0) " exit 1 &#125; function doesDistributionExist &#123; DIST_ID="$( awsCfS3Dist "$BUCKET_NAME" )" if [ "$DIST_ID" ]; then echo true; fi &#125; function createDist &#123; read -r -d '' NEW_DIST_JSON &lt;&lt;EOF &#123; "CallerReference": "$BUCKET_NAME", "Aliases": &#123; "Quantity": 0 &#125;, "DefaultRootObject": "index.html", "Origins": &#123; "Quantity": 1, "Items": [ &#123; "Id": "$BUCKET_NAME", "DomainName": "$BUCKET_NAME.s3.amazonaws.com", "S3OriginConfig": &#123; "OriginAccessIdentity": "" &#125; &#125; ] &#125;, "DefaultCacheBehavior": &#123; "TargetOriginId": "$BUCKET_NAME", "ForwardedValues": &#123; "QueryString": true, "Cookies": &#123; "Forward": "none" &#125; &#125;, "TrustedSigners": &#123; "Enabled": false, "Quantity": 0 &#125;, "ViewerProtocolPolicy": "redirect-to-https", "MinTTL": 3600 &#125;, "CacheBehaviors": &#123; "Quantity": 0 &#125;, "Comment": "", "Logging": &#123; "Enabled": false, "IncludeCookies": true, "Bucket": "", "Prefix": "" &#125;, "PriceClass": "PriceClass_All", "Enabled": true &#125; EOF NEW_DIST_RESULT_JSON = "$( aws cloudfront create-distribution --distribution-config "$NEW_DIST_JSON" )" DISTRIBUTION_ID="$( jq -r '.Distribution.Id' &lt;&lt;&lt; "$NEW_DIST_RESULT_JSON" )" echo "$DISTRIBUTION_ID" &#125; if [ "$1" == -h ]; then help; fi if [ -t 0 ]; then if [ -z "$1" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi BUCKET_NAME="$1" shift else read -r BUCKET_NAME fi if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi if [ "$( aws s3api head-bucket --bucket $BUCKET_NAME 2> >(grep -i 'Not Found') )" ]; then >&amp;2 echo "Error: Bucket $BUCKET_NAME does not exist." exit 2 fi if [ "$(doesDistributionExist)" ]; then >&amp;2 echo "Error: a CloudFront distibution already exists for S3 bucket $BUCKET_NAME" exit 3 fi createDist </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd558c2b4e7aa'><button class='copyBtn' data-clipboard-target='#idd558c2b4e7aa' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfS3MakeDist my_bucket <span class='unselectable'>E2P5S6OYKQNB6B </span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9d02cf39f121'><button class='copyBtn' data-clipboard-target='#id9d02cf39f121' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo my_bucket | awsCfS3MakeDist <span class='unselectable'>E2P5S6OYKQNB6B </span></pre> </editor-fold> <editor-fold awsS3Mb> <h2 id="awsS3Mb"><span class="code">awsS3Mb</span></h2> <p> Make a new S3 bucket with the given name in the default AWS region. If the <code>--public-read</code> option is provided, set the ACL to <code>public-read</code> </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,awsS3Mb' download='awsS3Mb' title='Click on the file name to download the file'>awsS3Mb</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id3beb2574a3c8">#!/bin/bash function help &#123; printf "$1$(basename $0) - Make a new S3 bucket with the given name in the default AWS region. Syntax: $(basename $0) bucketName [OPTIONS] Syntax: echo bucketName | $(basename $0) [OPTIONS] Options are: --public-read Set bucket ACL to public-read " exit 1 &#125; if [ "$1" == -h ]; then help; fi if [ "$1" == "--public-read" ]; then ACL="public-read" shift fi if [ "$1" ]; then BUCKET_NAME="$1" shift elif [ ! -t 0 ]; then read -r BUCKET_NAME fi if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi aws s3 mb s3://$BUCKET_NAME if [ "$ACL" ]; then aws s3api put-bucket-acl --bucket $BUCKET_NAME --acl $ACL fi </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6a44bfe33c62'><button class='copyBtn' data-clipboard-target='#id6a44bfe33c62' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsS3Mb my_bucket</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3bd1a510a1bd'><button class='copyBtn' data-clipboard-target='#id3bd1a510a1bd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsS3Mb my_bucket --public-read</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide96a4f3afdfd'><button class='copyBtn' data-clipboard-target='#ide96a4f3afdfd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo my_bucket | awsS3Mb --public-read</pre> </editor-fold> <editor-fold awsS3Website> <h2 id="awsS3Website"><span class="code">awsS3Website</span></h2> <p> Enable an S3 bucket to be a website. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,awsS3Website' download='awsS3Website' title='Click on the file name to download the file'>awsS3Website</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id71d0c525bd7c">#!/bin/bash function help &#123; printf "$1$(basename $0) - Enable an S3 bucket to be a website. Syntax: $(basename $0) bucketName Syntax: echo bucketName | $(basename $0) " exit 1 &#125; if [ "$1" == -h ]; then help; fi if [ "$1" ]; then BUCKET_NAME="$1" shift elif [ ! -t 0 ]; then read -r BUCKET_NAME fi if [ -z "$BUCKET_NAME" ]; then help 'Error: No S3 bucket name was specified.\n\n'; fi if [ "$1" ]; then help 'Error: Too many arguments provided.\n\n'; fi aws s3 website s3://$BUCKET_NAME \ --index-document index.html \ --error-document 404.html </pre> <p> Example usages: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0262614574cd'><button class='copyBtn' data-clipboard-target='#id0262614574cd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsS3Website my_bucket</pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide2cf61d95eb5'><button class='copyBtn' data-clipboard-target='#ide2cf61d95eb5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo my_bucket | awsS3Website</pre> </editor-fold> CORS on AWS S3 and Cloudfront 2021-03-21T00:00:00-04:00 https://mslinn.github.io/blog/2021/03/21/cors-aws <editor-fold intro> <p> This post shows how to enable CORS on an AWS S3 bucket with AWS CLI, then modify the bucket&rsquo;s CloudFront distribution. In preparing this blog post, I found that the <a href='https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html' target='_blank' rel='nofollow'>AWS S3 CORS documentation</a> needs to be read in conjunction with how <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/header-caching.html#header-caching-web-cors' target='_blank' rel='nofollow'>AWS CloudFront can be configured to handle CORS</a>. </p> <p> I used one origin for testing. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd378756cbe2f'><button class='copyBtn' data-clipboard-target='#idd378756cbe2f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ORIGIN=ancientwarmth.com <span class='unselectable'>$ </span>JSON_FILE=cors.json</pre> <p> The CORS configuration for the AWS S3 bucket will be stored in the file pointed to by <code>JSON_FILE</code>. </p> </editor-fold> <editor-fold defS3Cors> <h2 id="defS3Cors">Define the AWS S3 Bucket CORS Configuration</h2> <p> This configuration (in JSON format) contains 1 rule: </p> <ol> <li>Allow <code>GET</code> HTTP methods from anywhere.</li> </ol> <div class='codeLabel unselectable' data-lt-active='false'>cors.json</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc264cae28afb'><button class='copyBtn' data-clipboard-target='#idc264cae28afb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{ "CORSRules": [ { "AllowedHeaders": [], "AllowedMethods": [ "GET" ], "AllowedOrigins": [ "*" ], "ExposeHeaders": [] } ] }</pre> <p> You can read about CORS configuration in the <a href='https://docs.aws.amazon.com/AmazonS3/latest/userguide/ManageCorsUsing.html#cors-example-1' target='_blank' rel='nofollow'>AWS documentation</a>. </p> </editor-fold> <editor-fold setS3Cors> <h2 id="setS3Cors">Set the AWS S3 Bucket CORS Configuration</h2> <p> It is easy to set the CORS configuration for an AWS S3 bucket using AWS CLI&rsquo;s <a href='https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-cors.html' target='_blank' rel='nofollow'><code>aws s3api put-bucket-cors</code> subcommand</a>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6ed7635f1d59'><button class='copyBtn' data-clipboard-target='#id6ed7635f1d59' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>BUCKET=assets.ancientwarmth.com <span class='unselectable'>$ </span>aws s3api put-bucket-cors \ --bucket $BUCKET \ --cors-configuration "file://$JSON_FILE"</pre> </editor-fold> <editor-fold testS3Cors> <h2 id="testS3Cors">Test the AWS S3 Bucket CORS Configuration</h2> <p> Now it is time to test the S3 bucket&rsquo;s CORS configuration using <code>curl</code>. I defined a bash function to peform the test to save typing. You can use it by first copy/pasting the code below into a shell prompt, then calling the function with the proper arguments, as shown. The function requires 3 arguments: the request origin, the URL of an asset in an AWS S3 bucket, and an HTTP method (which must be in UPPPER CASE). </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide61c9cd25b15'><button class='copyBtn' data-clipboard-target='#ide61c9cd25b15' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>function testCors { if [ -z "$1" ]; then echo "Error: No origin was provided"; exit 1; fi if [ -z "$2" ]; then echo "Error: No URL to test was provided"; exit 1; fi if [ "$3" ]; then METHOD="$3"; else METHOD=GET; fi curl -I -X OPTIONS \ --no-progress-meter \ -H "Origin: $1" \ -H "Access-Control-Request-Method: $METHOD" \ "$2" 2>&1 | \ grep --color=never 'Access-Control' }</pre> <p> The JSON file for testing CORS was <code><a href='https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/index.html#path-argument-type' target='_blank' rel='nofollow'>s3://</a>$BUCKET/testCors.json</code>: </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,testCors.json' download='testCors.json' title='Click on the file name to download the file'>testCors.json</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="ide555b6a39e6d">&#123; "key1": "value1", "key2": "value2" &#125; </pre> <p> We will know if CORS is set up properly by receiving a header containing <code>Access-Control-Allow-Origin: *</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4d0409d8b5d0'><button class='copyBtn' data-clipboard-target='#id4d0409d8b5d0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>URL="https://s3.amazonaws.com/$BUCKET/testCors.json" <span class='unselectable'>$ </span>testCors $ORIGIN $URL GET <span class='unselectable'><span class="bg_yellow">Access-Control-Allow-Origin: *</span> Access-Control-Allow-Methods: GET Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method </span></pre> <p> The origin worked when the bucket is accessed via a <code>GET</code> method sent to its <code>s3.amazonaws.com</code> DNS alias (yay!). </p> <p> CORScanner (<a href="/blog/2021/03/20/cors.html#corscanner">discussed in a previous blog post</a>) reported no issues: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc471ecaa02fa'><button class='copyBtn' data-clipboard-target='#idc471ecaa02fa' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cors -u s3.amazonaws.com/assets.ancientwarmth.com/testCors.json <span class='unselectable'>Starting CORS scan... Finished CORS scanning... </span></pre> </editor-fold> <editor-fold cf> <h2 id="cf">CloudFront</h2> <p> I have not worked through the process of using AWS CLI to obtain a JSON object describing the distribution, and then changing some properties and writing it back. So until that happy day comes, here are 2 screen shots of the <a href='https://console.aws.amazon.com/cloudfront/home' target='_blank' rel='nofollow'>AWS CloudFront web console</a> showing the settings. The first screen shot shows the <b>Behaviors</b> tab of the top-level details of the <code>assets.ancientwarmth.com</code> CloudFront distribution. </p> <div style=""> <picture> <source srcset="/blog/images/aws/cfBehaviorCors0.webp" type="image/webp"> <source srcset="/blog/images/aws/cfBehaviorCors0.png" type="image/png"> <img src="/blog/images/aws/cfBehaviorCors0.png" title="CloudFront / Edit Distribution / Behaviors <br> About to click on <b>Edit</b> (default behavior)" class=" liImg2 rounded shadow" alt="CloudFront / Edit Distribution / Behaviors <br> About to click on <b>Edit</b> (default behavior)" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> CloudFront / Edit Distribution / Behaviors <br> About to click on <b>Edit</b> (default behavior) </figcaption> </figure> </div> <p> My application does not require users to upload anything, so everything in the S3 bucket is truly static. Thus I have no need to <code>PUT</code>, <code>POST</code> or <code>DELETE</code> HTTP methods for the AWS S3 content. I have not seen a good explanation of why enabling <code>OPTIONS</code> HTTP methods is necessary, but every person on Stack Overflow who got CORS to work with AWS S3 says this was necessary. With that in mind, I set the following for the next screen shot: </p> <ul> <li><b>Viewer Protocol Policy:</b> <code>Redirect HTTP to HTTPS</code></li> <li><b>Allowed HTTP Methods:</b> <code>GET, HEAD, OPTIONS</code></li> <li><b>Cached HTTP Methods:</b> Enable <code>OPTIONS</code></li> <li><b>Use a cache policy and origin request policy:</b> (default is Use legacy cache settings, which is usually undesirable)</li> <li><b>Cache Policy:</b> <code>Managed-CachingOptimized</code></li> <li><b>Origin Request Policy:</b> <code>Managed-CORS-S3Origin</code></li> </ul> <div style=""> <picture> <source srcset="/blog/images/aws/cfBehaviorCors1.webp" type="image/webp"> <source srcset="/blog/images/aws/cfBehaviorCors1.png" type="image/png"> <img src="/blog/images/aws/cfBehaviorCors1.png" title="Editing default CloudFront distribution behavior" class=" liImg2 rounded shadow" alt="Editing default CloudFront distribution behavior" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> Editing default CloudFront distribution behavior </figcaption> </figure> </div> <h3 id="cfManagedCorsS3OriginPolicy">Managed CORS S3 Origin Poligy</h3> <p> AWS CloudFront's <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/using-managed-origin-request-policies.html' target='_blank' rel='nofollow'>managed origin request policy</a> called <code>Managed-CORS-S3Origin</code> includes the headers that enable cross-origin resource sharing (CORS) requests when the origin is an Amazon S3 bucket. This policy's settings are: </p> <ul> <li><b>Query strings included in origin requests</b>: None</li> <li><b>Headers included in origin requests</b>: <ul> <li><code>Origin</code></li> <li><code>Access-Control-Request-Headers</code></li> <li><code>Access-Control-Request-Method</code></li> </ul> </li> <li><b>Cookies included in origin requests</b>: None</li> </ul> <div style=""> <picture> <source srcset="/blog/images/aws/cfManagedCorsS3OriginPolicy.webp" type="image/webp"> <source srcset="/blog/images/aws/cfManagedCorsS3OriginPolicy.png" type="image/png"> <img src="/blog/images/aws/cfManagedCorsS3OriginPolicy.png" class=" liImg2 rounded shadow" /> </picture> </div> </editor-fold> <editor-fold wait> <h2 id="wait">Wait or Invalidate</h2> <p> Whenever you make a configuration change to a CloudFront distribution, or the contents change, the distributed assets will not reflect those changes until the next CloudFront invalidation. Automatic invalidations take 20 minutes. You can invalidate manually for near-instant gratification. I use my <a href='/blog/2021/03/22/command-line-aws-utilities.html#awsCfInvalidate'>AWS command-line utilities</a> to invalidate manually: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id72a963b14db1'><button class='copyBtn' data-clipboard-target='#id72a963b14db1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>awsCfS3DistId $BUCKET | awsCfInvalidate</pre> <p> Now the grand finale: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb3ee34fe2b8f'><button class='copyBtn' data-clipboard-target='#idb3ee34fe2b8f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>testCors $ORIGIN $URL GET <span class='unselectable'>Access-Control-Allow-Origin: * Access-Control-Allow-Methods: GET Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method </span></pre> <div class="right" style="font-size: 3em;">&#128513;</div> <p> The presence of the <code>Access-Control-Allow-Origin</code> header indicates that CORS allowed the data file to be transferred from the content server (AWS S3/CloudFront) to the origin server (the command line). </p> </editor-fold> Cross-Origin Resource Sharing (CORS) 2021-03-20T00:00:00-04:00 https://mslinn.github.io/blog/2021/03/20/cors <editor-fold intro> <p> Many have tried to explain CORS, but most have not provided a clear explanation. I am going to try, then I will refer to explanations by others, who also provide examples. </p> </editor-fold> <editor-fold origin> <h2 id="origin">Origin and Origin Server</h2> <p> A website is delivered to web browsers from an <i>origin server</i>, or <i>origin</i> for short. The origin server is principally responsible for generating web pages. </p> <p> An origin is a combination of 3 things: </p> <ol> <li>A scheme (<code>http</code>, <code>https</code>, etc.)</li> <li>A (sub)domain, for example <code>localhost</code>, <code>blah.com</code> or <code>assets.blah.com</code>.</li> <li>A port, for example 80, 443, 8000, etc.</li> </ol> <p> All three things must match in order for two URLs to be considered to be from the same origin. For example: </p> <table class="table"> <tr> <th>URL 1</th> <th>URL 2</th> <th>Same Origin?</th> </tr> <tr> <td><code>http://blah.com</code></td> <td><code>http<span class="bg_yellow">s</span>://blah.com</code></td> <th>No</th> </tr> <tr> <td><code>https://blah.com</code></td> <td><code>https://<span class="bg_yellow">assets.</span>blah.com</code></td> <th>No</th> </tr> <tr> <td><code>https://blah.com</code></td> <td><code>https://blah.com<span class="bg_yellow">/path/page.html</span></code></td> <th>Yes</th> </tr> </table> </editor-fold> <editor-fold contentServer> <h2 id="contentServer">Content Server</h2> <p> In this article, I use the term <i>content server</i> to refer to sources of online information other than the origin server. Resources referenced by a web page, such as images, JavaScript, CSS, and data might be provided by the origin server, or they might come from a content server. </p> <p> Because every server has by definition a different origin, content servers always have a different origin than the origin server. Static resources (resources that do not change) are often served by <i>content delivery networks</i> (CDNs), which are also content servers. </p> <p> The Cross-Origin Resource Sharing (CORS) standard controls if a web page can load resources from content servers. Content servers are in charge of their content; they decide which origin servers they wish to co-operate with. When CORS support is properly configured, content servers include HTTP headers into their responses that tell a web browser if those resources may be read by the web page being constructed. </p> <p> Data is a special type of resource. CORS restricts how data is exchanged between the web page delivered to the web browser from the origin server and content servers. In particular, JSON and XML data communicated to and from content servers requires CORS authorization. Furthermore, requests (from the web browser) that send JSON, XML and other data formats to content servers also require CORS authorization. </p> <div class="pullQuote">Content servers are in charge of their content; they decide which origin servers they wish to co-operate with.</div> </editor-fold> <editor-fold ctype> <h2 id="ctype">Content-Type Header</h2> <p> The <a href='https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type' target='_blank' rel='nofollow'><code>Content-Type</code> header</a> is used to indicate the <a href='https://developer.mozilla.org/en-US/docs/Glossary/MIME_type' target='_blank' rel='nofollow'><code>media type</code></a> of the resource. The old name <i>MIME type</i> has been replaced by <i>media type</i>. <a href='https://www.iana.org/assignments/media-types/media-types.xhtml' target='_blank' rel='nofollow'>Here is a list of media types.</a> </p> <p> Media types with names that start with <code>application</code> require CORS authentication if they are delivered from content servers, for example <code>application/json</code> and <code>application/javascript</code>. </p> <p> As well, a few media types with names that start with <code>text</code> require CORS authentication if they are delivered from content servers, for example <code>text/xml</code> and <code>text/xml-external-parsed-entity</code>. </p> </editor-fold> <editor-fold resources> <h2 id="resources">Further Reading</h2> <h3 id="Kosaka">Mariko Kosaka</h3> <p> Mariko Kosaka has written an easy-to-understand article describing CORS, and provides a simple but effective working Express website for demonstration. </p> <div class="quote"> The same-origin policy tells the browser to block cross-origin requests. When you want to get a public resource from a different origin, the resource-providing server needs to tell the browser &lsquo;This origin where the request is coming from can access my resource&rsquo;. The browser remembers that and allows cross-origin resource sharing. <br><br> <span style="font-style: normal"> &nbsp; &ndash; <a href='https://web.dev/cross-origin-resource-sharing/' target='_blank' rel='nofollow'>Mariko Kosaka</a></span> </div> </editor-fold> <editor-fold gilling> <h3 id="Gilling">Derric Gilling and MDN</h3> <p> Derric Gilling has written a more in-depth yet very approachable article describing CORS. I've paraphrased his quoting of the Mozilla Developer Network documentation into the following: </p> <div class="quote"> CORS is a security mechanism that allows a web page from one domain or Origin to access a resource with a different domain (a cross-domain request). CORS is a relaxation of the same-origin policy implemented in modern browsers. Without features like CORS, websites are restricted to accessing resources from the same origin through what is known as same-origin policy. <br><br> Any CORS request has to be preflighted if:<br> <ul> <li>It uses methods other than <code>GET</code>, <code>HEAD</code> or <code>POST</code>.</li> <li> If POST is used to send request data with a <code>Content-Type</code> other than <code>application/x-www-form-urlencoded</code>, <code>multipart/form-data</code>, or <code>text/plain</code>. Examples: <ul> <li> A <code>POST</code> request sends an XML payload to the server; this requires the <code>Content-Type</code> header is set either to <code>application/xml</code> or <code>text/xml</code>. </li> <li> A website makes an AJAX call that <code>POST</code>s JSON data to a REST API, this requires the <code>Content-Type</code> header is set to <code>application/json</code>. </li> </ul> </li> </ul> <span style="font-style: normal">&nbsp; &ndash; <a href='https://www.moesif.com/blog/technical/cors/Authoritative-Guide-to-CORS-Cross-Origin-Resource-Sharing-for-REST-APIs/#how-cors-works/' target='_blank' rel='nofollow'>Derric Gilling</a> <br> &nbsp; &ndash; <a href='https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#preflighted_requests' target='_blank' rel='nofollow'>Mozilla Developer Network</a> </span> </div> </editor-fold> <editor-fold preflight> <h3 id="preflight">Preflight Requests</h3> <p> CORS preflight requests effectively double the latency of user requests for <a href='https://developer.mozilla.org/en-US/docs/Glossary/CRUD' target='_blank' rel='nofollow'>CRUD actions</a>. Client-side and server-side caching can help reduce this overhead for many circumstances. In <a href='https://www.mslinn.com/blog/2021/04/14/serverless-ecommerce.html#cf' target='_blank' rel='nofollow'>another blog post</a> I discuss how to use a CDN with multiple origin servers to completely eliminate the need for preflight requests. </p> <p> For additional background, please see: </p> <ul> <li><a href='https://www.rehanvdm.com/serverless/cloudfront-reverse-proxy-api-gateway-to-prevent-cors/index.html' target='_blank' rel='nofollow'>CloudFront reverse proxy API Gateway to prevent CORS</a> by Rehan van der Merwe</li> <li><a href='https://httptoolkit.tech/blog/cache-your-cors/' target='_blank' rel='nofollow'>Cache your CORS, for performance & profit</a> by Tim Perry</li> </ul> </editor-fold> <editor-fold keycdn> <h3 id="KeyCDN">KeyCDN</h3> <p> KeyCDN has an even more in-depth yet still very approachable <a href='https://www.keycdn.com/support/cors' target='_blank' rel='nofollow'>article describing CORS</a>. </p> </editor-fold> <editor-fold corscanner> <h2 id="corscanner">CORScanner</h2> <p> <a href='https://github.com/chenjj/CORScanner' target='_blank' rel='nofollow'>CORScanner</a> is a popular tool for detecting CORS misconfiguration. It is a Python module that can be executed as a shell command. Install CORScanner like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8d894b3c7872'><button class='copyBtn' data-clipboard-target='#id8d894b3c7872' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pip install cors</pre> <p> The above adds a new executable called <code>cors</code> in the same directory where your <code>python</code> command resides. <p> <p> The <code>cors</code> documentation <a href='https://www.merriam-webster.com/thesaurus/conflate#verb' target='_blank' rel='nofollow'>conflates</a> the words URL and origin. Everywhere the word <code>URL</code> appears in the documentation, the word <code>origin</code> should be assumed. </p> </editor-fold> <editor-fold scannEx> <h3 id="scannEx">Example: Check Domain</h3> <p>Use the <code>-u</code> option to specify an origin to test:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbdf9a08512b5'><button class='copyBtn' data-clipboard-target='#idbdf9a08512b5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cors -u api.github.com <span class='unselectable'>Starting CORS scan... Finished CORS scanning... </span></pre> <p> To enable more debug info, use the <code>-v</code> option more than once. We can see that specifying <code>https</code> restricts testing to that <code>scheme</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbd9aceaf8a1c'><button class='copyBtn' data-clipboard-target='#idbd9aceaf8a1c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cors -vv -u https://api.github.com <span class='unselectable'>Starting CORS scan... 2021-03-21 09:55:58 INFO Start checking reflect_origin for https://api.github.com 2021-03-21 09:55:58 INFO nothing found for {url: https://api.github.com, origin: https://evil.com, type: reflect_origin} 2021-03-21 09:55:58 INFO Start checking prefix_match for https://api.github.com 2021-03-21 09:55:58 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com.evil.com, type: prefix_match} 2021-03-21 09:55:58 INFO Start checking suffix_match for https://api.github.com 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://evilgithub.com, type: suffix_match} 2021-03-21 09:55:59 INFO Start checking trust_null for https://api.github.com 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: null, type: trust_null} 2021-03-21 09:55:59 INFO Start checking include_match for https://api.github.com 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://ithub.com, type: include_match} 2021-03-21 09:55:59 INFO Start checking not_escape_dot for https://api.github.com 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://api.githubacom, type: not_escape_dot} 2021-03-21 09:55:59 INFO Start checking custom_third_parties for https://api.github.com 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://whatever.github.io, type: custom_third_parties} 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: http://jsbin.com, type: custom_third_parties} 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://codepen.io, type: custom_third_parties} 2021-03-21 09:55:59 INFO nothing found for {url: https://api.github.com, origin: https://jsfiddle.net, type: custom_third_parties} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: http://www.webdevout.net, type: custom_third_parties} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://repl.it, type: custom_third_parties} 2021-03-21 09:56:00 INFO Start checking special_characters_bypass for https://api.github.com 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com_.evil.com, type: special_characters_bypass} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com-.evil.com, type: special_characters_bypass} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&quot;.evil.com, type: special_characters_bypass} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com{.evil.com, type: special_characters_bypass} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com}.evil.com, type: special_characters_bypass} 2021-03-21 09:56:00 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com^.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%60.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com!.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com~.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com`.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com;.evil.com, type: special_characters_bypass} 2021-03-21 09:56:01 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com|.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&apos;.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com(.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com).evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com*.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com,.evil.com, type: special_characters_bypass} 2021-03-21 09:56:02 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com$.evil.com, type: special_characters_bypass} 2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com=.evil.com, type: special_characters_bypass} 2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%0b.evil.com, type: special_characters_bypass} 2021-03-21 09:56:03 INFO Start checking trust_any_subdomain for https://api.github.com 2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: https://evil.api.github.com, type: trust_any_subdomain} 2021-03-21 09:56:03 INFO Start checking https_trust_http for https://api.github.com 2021-03-21 09:56:03 INFO nothing found for {url: https://api.github.com, origin: http://api.github.com, type: https_trust_http} Finished CORS scanning... </span></pre> </editor-fold> <editor-fold scannEx2> <h3 id="scannEx2">Example: Check Origin</h3> <p> To check CORS misconfigurations of an origin: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3db49d8fdd44'><button class='copyBtn' data-clipboard-target='#id3db49d8fdd44' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cors -vvu https://api.github.com/users/mslinn/repos <span class='unselectable'>Starting CORS scan... 2021-03-21 10:08:49 INFO Start checking reflect_origin for https://api.github.com 2021-03-21 10:08:49 INFO nothing found for {url: https://api.github.com, origin: https://evil.com, type: reflect_origin} 2021-03-21 10:08:49 INFO Start checking prefix_match for https://api.github.com 2021-03-21 10:08:49 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com.evil.com, type: prefix_match} 2021-03-21 10:08:49 INFO Start checking suffix_match for https://api.github.com 2021-03-21 10:08:49 INFO nothing found for {url: https://api.github.com, origin: https://evilgithub.com, type: suffix_match} 2021-03-21 10:08:49 INFO Start checking trust_null for https://api.github.com 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: null, type: trust_null} 2021-03-21 10:08:50 INFO Start checking include_match for https://api.github.com 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://ithub.com, type: include_match} 2021-03-21 10:08:50 INFO Start checking not_escape_dot for https://api.github.com 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://api.githubacom, type: not_escape_dot} 2021-03-21 10:08:50 INFO Start checking custom_third_parties for https://api.github.com 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://whatever.github.io, type: custom_third_parties} 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: http://jsbin.com, type: custom_third_parties} 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://codepen.io, type: custom_third_parties} 2021-03-21 10:08:50 INFO nothing found for {url: https://api.github.com, origin: https://jsfiddle.net, type: custom_third_parties} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: http://www.webdevout.net, type: custom_third_parties} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://repl.it, type: custom_third_parties} 2021-03-21 10:08:51 INFO Start checking special_characters_bypass for https://api.github.com 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com_.evil.com, type: special_characters_bypass} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com-.evil.com, type: special_characters_bypass} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&quot;.evil.com, type: special_characters_bypass} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com{.evil.com, type: special_characters_bypass} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com}.evil.com, type: special_characters_bypass} 2021-03-21 10:08:51 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com^.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%60.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com!.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com~.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com`.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com;.evil.com, type: special_characters_bypass} 2021-03-21 10:08:52 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com|.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&apos;.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com(.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com).evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com*.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com,.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com$.evil.com, type: special_characters_bypass} 2021-03-21 10:08:53 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com=.evil.com, type: special_characters_bypass} 2021-03-21 10:08:54 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:08:54 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%0b.evil.com, type: special_characters_bypass} 2021-03-21 10:08:54 INFO Start checking trust_any_subdomain for https://api.github.com 2021-03-21 10:08:54 INFO nothing found for {url: https://api.github.com, origin: https://evil.api.github.com, type: trust_any_subdomain} 2021-03-21 10:08:54 INFO Start checking https_trust_http for https://api.github.com 2021-03-21 10:08:54 INFO nothing found for {url: https://api.github.com, origin: http://api.github.com, type: https_trust_http} Finished CORS scanning... </span></pre> <p> If a <code>scheme</code> is not specified, then both <code>http</code> and <code>https</code> are tested: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ided1fb270202d'><button class='copyBtn' data-clipboard-target='#ided1fb270202d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cors -vvu api.github.com/users/mslinn/repos <span class='unselectable'>Starting CORS scan... 2021-03-21 10:03:30 INFO Start checking reflect_origin for http://api.github.com 2021-03-21 10:03:30 INFO Start checking reflect_origin for https://api.github.com 2021-03-21 10:03:30 INFO nothing found for {url: https://api.github.com, origin: https://evil.com, type: reflect_origin}2021-03-21 10:03:30 INFO Start checking prefix_match for https://api.github.com 2021-03-21 10:03:30 INFO nothing found for {url: http://api.github.com, origin: http://evil.com, type: reflect_origin} 2021-03-21 10:03:30 INFO Start checking prefix_match for http://api.github.com 2021-03-21 10:03:30 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com.evil.com, type: prefix_match} 2021-03-21 10:03:30 INFO Start checking suffix_match for https://api.github.com 2021-03-21 10:03:30 INFO nothing found for {url: https://api.github.com, origin: https://evilgithub.com, type: suffix_match} 2021-03-21 10:03:30 INFO Start checking trust_null for https://api.github.com 2021-03-21 10:03:30 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com.evil.com, type: prefix_match} 2021-03-21 10:03:30 INFO Start checking suffix_match for http://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: null, type: trust_null} 2021-03-21 10:03:31 INFO Start checking include_match for https://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: http://api.github.com, origin: http://evilgithub.com, type: suffix_match} 2021-03-21 10:03:31 INFO Start checking trust_null for http://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://ithub.com, type: include_match}2021-03-21 10:03:31 INFO Start checking not_escape_dot for https://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://api.githubacom, type: not_escape_dot} 2021-03-21 10:03:31 INFO Start checking custom_third_parties for https://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: http://api.github.com, origin: null, type: trust_null} 2021-03-21 10:03:31 INFO Start checking include_match for http://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://whatever.github.io, type: custom_third_parties} 2021-03-21 10:03:31 INFO nothing found for {url: http://api.github.com, origin: http://ithub.com, type: include_match} 2021-03-21 10:03:31 INFO Start checking not_escape_dot for http://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: http://jsbin.com, type: custom_third_parties} 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://codepen.io, type: custom_third_parties} 2021-03-21 10:03:31 INFO nothing found for {url: http://api.github.com, origin: http://api.githubacom, type: not_escape_dot} 2021-03-21 10:03:31 INFO Start checking custom_third_parties for http://api.github.com 2021-03-21 10:03:31 INFO nothing found for {url: https://api.github.com, origin: https://jsfiddle.net, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: https://whatever.github.io, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: http://www.webdevout.net, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://repl.it, type: custom_third_parties} 2021-03-21 10:03:32 INFO Start checking special_characters_bypass for https://api.github.com 2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: http://jsbin.com, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com_.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com-.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: https://codepen.io, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&quot;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com{.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: https://jsfiddle.net, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com}.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:03:32 INFO nothing found for {url: http://api.github.com, origin: http://www.webdevout.net, type: custom_third_parties} 2021-03-21 10:03:32 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com^.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: https://repl.it, type: custom_third_parties} 2021-03-21 10:03:33 INFO Start checking special_characters_bypass for http://api.github.com 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%60.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com!.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com_.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com~.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com`.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com-.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com&quot;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com|.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com{.evil.com, type: special_characters_bypass} 2021-03-21 10:03:33 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com&qpos;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com(.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com}.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com).evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com*.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com,.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com^.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com$.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com=.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com%60.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com!.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO nothing found for {url: https://api.github.com, origin: https://api.github.com%0b.evil.com, type: special_characters_bypass} 2021-03-21 10:03:34 INFO Start checking trust_any_subdomain for https://api.github.com 2021-03-21 10:03:35 INFO nothing found for {url: https://api.github.com, origin: https://evil.api.github.com, type: trust_any_subdomain} 2021-03-21 10:03:35 INFO Start checking https_trust_http for https://api.github.com 2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com~.evil.com, type: special_characters_bypass} 2021-03-21 10:03:35 INFO nothing found for {url: https://api.github.com, origin: http://api.github.com, type: https_trust_http} 2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com`.evil.com, type: special_characters_bypass} 2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com|.evil.com, type: special_characters_bypass} 2021-03-21 10:03:35 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com&.evil.com, type: special_characters_bypass} 2021-03-21 10:03:36 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com&apos;.evil.com, type: special_characters_bypass} 2021-03-21 10:03:36 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com(.evil.com, type: special_characters_bypass} 2021-03-21 10:03:36 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com).evil.com, type: special_characters_bypass} 2021-03-21 10:03:36 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com*.evil.com, type: special_characters_bypass} 2021-03-21 10:03:37 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com,.evil.com, type: special_characters_bypass} 2021-03-21 10:03:38 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com$.evil.com, type: special_characters_bypass} 2021-03-21 10:03:38 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com=.evil.com, type: special_characters_bypass} 2021-03-21 10:03:38 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com+.evil.com, type: special_characters_bypass} 2021-03-21 10:03:38 INFO nothing found for {url: http://api.github.com, origin: http://api.github.com%0b.evil.com, type: special_characters_bypass} 2021-03-21 10:03:38 INFO Start checking trust_any_subdomain for http://api.github.com 2021-03-21 10:03:39 INFO nothing found for {url: http://api.github.com, origin: http://evil.api.github.com, type: trust_any_subdomain} Finished CORS scanning... </span></pre> </editor-fold> AWS S3 and CloudFront SSL 2021-03-19T00:00:00-04:00 https://mslinn.github.io/blog/2021/03/19/aws-ssl <style> .sslScenario { border: thin solid grey; border-radius: 4px; padding: 8px; } </style> <p> SSL certificates need to match the domain they are served from. </p> <p> AWS uses one of several SSL certificates, depending on the <a href='https://docs.aws.amazon.com/AmazonS3/latest/dev-retired/UsingBucket.html#access-bucket-intro' target='_blank' rel='nofollow'>domain that an asset is requested from</a>. </p> <ul> <li> AWS S3 applies an SSL certificate for <code>https</code> requests. The SSL certificate chosen depends on the bucket endpoint used: <code>s3.amazonaws.com</code>, <code>*.s3.amazonaws.com</code>, or <code>s3.<i>region</i>.amazonaws.com</code>. </li> <li> AWS CloudFront will apply your custom SSL certificate (for example, a wildcard certificate such as <code>*.ancientwarmth.com</code>) for <code>https</code> requests to the CNAME for that distribution, otherwise it will apply the wildcard SSL certificate for <code>*.cloudfront.net</code>. </li> </ul> <h2 id="examples">Example <span class="code">SSL</span> URLs</h2> <p> My AWS S3 bucket called <code>assets.ancientwarmth.com</code> is served via a CloudFront distribution with URL <code>d1bci9l8cjf24o.cloudfront.net</code> that applies a wildcard SSL certificate for <code>*.ancientwarmth.com</code> that I created using <a href='https://aws.amazon.com/certificate-manager/' target='_blank' rel='nofollow'>AWS Certificate Manager</a>. I defined a CNAME called <code>assets.ancientwarmth.com</code> for that same CloudFront distribution using Route 53. </p> <p> All of the following URLs can be used to access my content, providing the SSL certificate matches the requested domain. </p> <p class="sslScenario"> <b>URL:</b> <code>https://d1bci9l8cjf24o.cloudfront.net</code><br> <b>Origin Type:</b> CloudFront distribution<br> <b>SSL certificate origin:</b> <code>*.cloudfront.net</code><br> <b>Valid SSL certificate?</b> Yes. </p> <p class="sslScenario"> <b>URL:</b> <code>https://assets.ancientwarmth.com</code><br> <b>Origin Type:</b> CloudFront distribution<br> <b>SSL certificate origin:</b> <code>*.ancientwarmth.com</code><br> <b>Valid SSL certificate?</b> Yes. (I created this wildcard certificate using Route 53.) </p> <p class="sslScenario"> <b>S3 path-style URL:</b> <code>https://s3.us-east-1.amazonaws.com/assets.ancientwarmth.com</code><br> <b>Origin Type:</b> S3 bucket<br> <b>SSL certificate origin:</b> <code>s3.us-east-1.amazonaws.com</code><br> <b>Valid SSL certificate?</b> Yes. </p> <p class="sslScenario"> <b>S3 dot URL:</b> <code>https://assets.ancientwarmth.com.s3.amazonaws.com</code><br> <b>Origin Type:</b> S3 bucket<br> <b>SSL certificate origin:</b> <code>*.s3.amazonaws.com</code><br> <b>Valid SSL certificate?</b> No, does not match URL (wildcards only match one subdomain). </p> <p class="sslScenario"> <b>S3 dot Region URL:</b> <code>https://assets.ancientwarmth.com.s3.us-east-1.amazonaws.com</code><br> <b>Origin Type:</b> S3 bucket<br> <b>SSL certificate origin:</b> <code>s3.amazonaws.com</code><br> <b>Valid SSL certificate?</b> No, does not match URL. </p> <h2 id="curl">Testing with <span class="code">curl</span></h2> <p> <code>Curl</code> is often used to test SSL requests. In the following <code>curl</code> commands, the <a href='https://curl.se/docs/manpage.html#-I' target='_blank' rel='nofollow'><code>-I</code> option</a> just fetches the headers, and the <a href='https://curl.se/docs/manpage.html#-v' target='_blank' rel='nofollow'><code>-v</code> option</a> provides verbose output. You can see the SSL certificate negotation. </p> <p> Fetching an asset from a CloudFront distribution using the AWS <code>*.cloudfront.net</code> wildcard SSL certificate: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida9843d30cd2b'><button class='copyBtn' data-clipboard-target='#ida9843d30cd2b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -Iv \ https://d1bci9l8cjf24o.cloudfront.net/js/jquery.modal.min.js <span class='unselectable'>* Trying 52.85.149.22:443... * TCP_NODELAY set * Connected to d1bci9l8cjf24o.cloudfront.net (52.85.149.22) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=*.cloudfront.net * start date: Feb 22 00:00:00 2021 GMT * expire date: Feb 21 23:59:59 2022 GMT * subjectAltName: host "d1bci9l8cjf24o.cloudfront.net" matched cert's "*.cloudfront.net" * issuer: C=US; O=DigiCert Inc; CN=DigiCert Global CA G2 * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x5585dcfaf7e0) > HEAD /js/jquery.modal.min.js HTTP/2 > Host: d1bci9l8cjf24o.cloudfront.net > user-agent: curl/7.68.0 > accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS == 128)! < HTTP/2 200 HTTP/2 200 < content-type: application/javascript content-type: application/javascript < content-length: 4953 content-length: 4953 < date: Sat, 20 Mar 2021 14:11:34 GMT date: Sat, 20 Mar 2021 14:11:34 GMT < last-modified: Sat, 20 Mar 2021 03:14:08 GMT last-modified: Sat, 20 Mar 2021 03:14:08 GMT < etag: "c8f50397e0560719c62a35318f413e16" etag: "c8f50397e0560719c62a35318f413e16" < accept-ranges: bytes accept-ranges: bytes < server: AmazonS3 server: AmazonS3 < x-cache: Miss from cloudfront x-cache: Miss from cloudfront < via: 1.1 0712e4ad4264127dfcb76a114b130495.cloudfront.net (CloudFront) via: 1.1 0712e4ad4264127dfcb76a114b130495.cloudfront.net (CloudFront) < x-amz-cf-pop: IAD89-C3 x-amz-cf-pop: IAD89-C3 < x-amz-cf-id: hWrjwajqqkI9-rJnK1BSQqkX9DPXIlZJLfa28UaIeze7taBP5kqMNg== x-amz-cf-id: hWrjwajqqkI9-rJnK1BSQqkX9DPXIlZJLfa28UaIeze7taBP5kqMNg== < * Connection #0 to host d1bci9l8cjf24o.cloudfront.net left intact </span></pre> <p> Fetching an asset from a CloudFront distribution using my <code>*.ancientwarmth.com</code> wildcard SSL certificate: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3f0900fff536'><button class='copyBtn' data-clipboard-target='#id3f0900fff536' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -Iv \ https://assets.ancientwarmth.com/js/jquery.modal.min.js <span class='unselectable'>modal.min.js> https://assets.ancientwarmth.com/js/jquery.modal.min.js * Trying 13.226.36.16:443... * TCP_NODELAY set * Connected to assets.ancientwarmth.com (13.226.36.16) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=*.ancientwarmth.com * start date: Mar 14 00:00:00 2021 GMT * expire date: Apr 12 23:59:59 2022 GMT * subjectAltName: host "assets.ancientwarmth.com" matched cert's "*.ancientwarmth.com" * issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x5599720157e0) > GET /js/jquery.modal.min.js HTTP/2 > Host: assets.ancientwarmth.com > user-agent: curl/7.68.0 > accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS == 128)! < HTTP/2 200 < content-type: application/javascript < content-length: 4953 < date: Sat, 20 Mar 2021 12:34:25 GMT < last-modified: Sat, 20 Mar 2021 03:14:08 GMT < etag: "c8f50397e0560719c62a35318f413e16" < accept-ranges: bytes < server: AmazonS3 < x-cache: Hit from cloudfront < via: 1.1 4e3df844337032b56b8434990b0f76ca.cloudfront.net (CloudFront) < x-amz-cf-pop: EWR53-C2 < x-amz-cf-id: 17Dxn6QqtK6JfkJwFnESVYsG-Cbzu6H-sOTWcGDpznGcpjIZbhJDRA== < age: 5195 < * Connection #0 to host assets.ancientwarmth.com left intact </span></pre> <p> Fetching an asset from an S3 bucket using an AWS SSL certificate for all S3 buckets in region <code>us-east-1</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id185630113039'><button class='copyBtn' data-clipboard-target='#id185630113039' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -Iv \ https://s3.us-east-1.amazonaws.com/assets.ancientwarmth.com/js/jquery.modal.min.js <span class='unselectable'>* Trying 52.216.24.46:443... * TCP_NODELAY set * Connected to s3.us-east-1.amazonaws.com (52.216.24.46) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=s3.amazonaws.com * start date: Aug 4 00:00:00 2020 GMT * expire date: Aug 9 12:00:00 2021 GMT * subjectAltName: host "s3.us-east-1.amazonaws.com" matched cert's "s3.us-east-1.amazonaws.com" * issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2 * SSL certificate verify ok. > GET /assets.ancientwarmth.com/js/jquery.modal.min.js HTTP/1.1 > Host: s3.us-east-1.amazonaws.com > User-Agent: curl/7.68.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < x-amz-id-2: xIXUHy7YBpjZaF+cpGoSAwNvC5+NrmM5pmJM8nInI6weEkbht350xSPC9+yOBJrGs9GY0hn2V7Y= < x-amz-request-id: JM2K8HR109JNMMB1 < Date: Sat, 20 Mar 2021 14:03:56 GMT < Last-Modified: Sat, 20 Mar 2021 03:14:08 GMT < ETag: "c8f50397e0560719c62a35318f413e16" < Accept-Ranges: bytes < Content-Type: application/javascript < Content-Length: 4953 < Server: AmazonS3 < * Connection #0 to host s3.us-east-1.amazonaws.com left intact </span></pre> Pretty JSON Reduces Errors and Fatigue 2021-02-23T00:00:00-05:00 https://mslinn.github.io/blog/2021/02/23/pretty-json <p> I've been using <a href='https://stedolan.github.io/jq/' target='_blank' rel='nofollow'>jq</a> to format my JSON for years. It is easy to format a JSON document, just pass it through <code>jq</code> without any options or arguments. Notice, however, that a lot of vertical space is wasted using this formatting: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbdaab7d118b0'><button class='copyBtn' data-clipboard-target='#idbdaab7d118b0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ jq < blog/colors.json { "colors": [ { "color": "black", "hex": "#000", "rgb": [ 0, 0, 0 ] }, { "color": "red", "hex": "#f00", "rgb": [ 255, 0, 0 ] }, { "color": "yellow", "hex": "#ff0", "rgb": [ 255, 255, 0 ] }, { "color": "green", "hex": "#0f0", "rgb": [ 0, 255, 0 ] }, { "color": "cyan", "hex": "#0ff", "rgb": [ 0, 255, 255 ] }, { "color": "blue", "hex": "#00f", "rgb": [ 0, 0, 255 ] }, { "color": "magenta", "hex": "#f0f", "rgb": [ 255, 0, 255 ] }, { "color": "white", "hex": "#fff", "rgb": [ 255, 255, 255 ] } ] }</pre> <p> After reading <a href='http://www.ohler.com/dev/pretty.html' target='_blank' rel='nofollow'>The Pretty JSON Revolution</a> I decided to try the program the article mentioned, <code>oj</code>. <code>oj</code> is a Go program. Here is how I compiled it on Ubuntu: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idef1fb5db1cf2'><button class='copyBtn' data-clipboard-target='#idef1fb5db1cf2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install golang-go <span class='unselectable'>$ </span>go get github.com/ohler55/ojg <span class='unselectable'>$ </span>go get github.com/ohler55/ojg/cmd/oj</pre> <p> By default, compiled go projects are placed in the <code>~/go/bin/</code> directory. Here is how I added that directory the the <code>PATH</code>, and made an alias for invoking the program with the proper options for maximum prettiness: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4240cef4f1d8'><button class='copyBtn' data-clipboard-target='#id4240cef4f1d8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo "$HOME/go/bin/:$PATH" >> ~/.bashrc <span class='unselectable'>$ </span>echo "alias pprint='oj -i 2 -s -p 80.3'" >> ~/.bash_aliases <span class='unselectable'>$ </span>source ~/.bashrc</pre> <p> Pretty-printing the JSON in <code>colors.json</code> with <code>oj</code> is easy: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcb978986a0fe'><button class='copyBtn' data-clipboard-target='#idcb978986a0fe' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pprint colors.json <span class='unselectable'>{ "colors": [ {"color": "black", "hex": "#000", "rgb": [0, 0, 0]}, {"color": "red", "hex": "#f00", "rgb": [255, 0, 0]}, {"color": "yellow", "hex": "#ff0", "rgb": [255, 255, 0]}, {"color": "green", "hex": "#0f0", "rgb": [0, 255, 0]}, {"color": "cyan", "hex": "#0ff", "rgb": [0, 255, 255]}, {"color": "blue", "hex": "#00f", "rgb": [0, 0, 255]}, {"color": "magenta", "hex": "#f0f", "rgb": [255, 0, 255]}, {"color": "white", "hex": "#fff", "rgb": [255, 255, 255]} ] } </span></pre> <p> I like it! </p> <style>.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }</style><div class='embed-container'> <iframe title="YouTube video player" width="640" height="390" src="//www.youtube.com/embed/34wJt3pRY0w" frameborder="0" allowfullscreen></iframe></div> <p style="text-align: center"> <i>Give it to Mikey. He won't eat it. He hates everything!</i> </p> <p> I will continue to use <a href='https://stedolan.github.io/jq/manual/' target='_blank' rel='nofollow'><code>jq</code> for queries</a>, but I'll use <code>oj</code> for pretty-printing from now on. </p> JavaScript Named Arguments and Class Constructors 2021-02-11T00:00:00-05:00 https://mslinn.github.io/blog/2021/02/11/javascript-named-arguments <p> Named arguments make a program safe from errors caused by changes to method arguments. JavaScript named arguments can appear in any order. Default values for parameters allow an API to evolve gracefully without runtime errors. </p> <p> Building on the article entitled <a href='https://afontcu.medium.com/cool-javascript-9-named-arguments-functions-that-get-and-return-objects-337b6f8cfa07' target='_blank' rel='nofollow'>Cool JavaScript 9: Named arguments — Functions that get and return Objects</a>, this article shows how JavaScript class constructors can use named arguments, optionally define default values for parameters, and conveniently inflate new class instances from JSON. </p> <p> In this article I use Node.js for convenience, however the code shown will run in all modern web browsers. </p> <h2 id="stdArgs">JavaScript Class Definition Encapsulating Properties</h2> <p> Let&rsquo;s quickly review how to define a JavaScript class and instantiate an instance. Here is a simple JavaScript / ECMAScript 6 class that encapsulates two properties: <code>id</code> and <code>parts</code>. The constructor merely lists the names of the parameters, which happen to be the same as the names of the class properties. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide042cab9e212'><button class='copyBtn' data-clipboard-target='#ide042cab9e212' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span> js <span class='unselectable'>Welcome to Node.js v12.18.2. Type ".help" for more information. > </span>class Ingredient { <span class='unselectable'>... </span> constructor(id, parts) { <span class='unselectable'>..... </span> this.id = id; <span class='unselectable'>..... </span> this.parts = parts; <span class='unselectable'>..... </span> } <span class='unselectable'>... </span>} <span class='unselectable'>undefined </span></pre> <p> New <code>Ingredient</code> instances can be created using this familiar syntax: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb0d3e5402426'><button class='copyBtn' data-clipboard-target='#idb0d3e5402426' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredient = new Ingredient("123", 10); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredient <span class='unselectable'>Ingredient { id: '123', parts: 10 } </span></pre> <h2 id="lits">Object Literals</h2> <p> Object literals look like JSON objects, but without quotes around property names. For example, the following defines an object literal called <code>lit</code> with 2 properties, called <code>id</code> and <code>parts</code>, with values <code>"123"</code> and <code>10</code>, respectively. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc9dfb4551274'><button class='copyBtn' data-clipboard-target='#idc9dfb4551274' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span> var lit = {id: "123", parts: 10}; <span class='unselectable'>undefined </span> <span class='unselectable'>$ </span>lit <span class='unselectable'>{ id: '123', parts: 10 } </span> <span class='unselectable'>> </span>lit.id <span class='unselectable'>'123' </span> <span class='unselectable'>> </span>lit.parts <span class='unselectable'>10 </span></pre> <h2 id="jsonArgs">Use Object Literals to Define Arguments</h2> <p> We can define a class similar to <code>Ingredient</code>, but with the arguments replaced by a something that looks like an object literal without values. For want of a better term I call this an <i>object name literal</i>. The following class definition encapsulates the same two properties as before as an object name literal. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id033a621f7e96'><button class='copyBtn' data-clipboard-target='#id033a621f7e96' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>class IngredientX { <span class='unselectable'>... </span> constructor(<span class="bg_yellow">{id, parts}</span>) { <span class='unselectable'>..... </span> this.id = id; <span class='unselectable'>..... </span> this.parts = parts; <span class='unselectable'>..... </span> } <span class='unselectable'>... </span>} <span class='unselectable'>undefined </span></pre> <p> New <code>IngredientX</code> instances can be created from an object literal: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5c8538f34fb2'><button class='copyBtn' data-clipboard-target='#id5c8538f34fb2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredientX1 = new IngredientX(<span class="bg_yellow">{id: "123", parts: 10 }</span>); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientX1 <span class='unselectable'>IngredientX { id: '123', parts: 10 } </span></pre> <p> Because the <code>IngredientX</code> class definition requires an object name literal (or a JSON object, more on that later) to provide constructor arguments, constructor invocations must specify the names of each parameter being passed to the constructor arguments. This has the benefit of making your software more robust in the face of changing method signatures. </p> <p> Caution: new <code>IngredientX</code> instances cannot be created from scalar arguments. JavaScript gives no error or warning if you do not: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2db4662d2c2c'><button class='copyBtn' data-clipboard-target='#id2db4662d2c2c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredientX2 = new IngredientX("123", 10); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientX2 <span class='unselectable'>IngredientX { id: <span class="bg_yellow">undefined</span>, parts: <span class="bg_yellow">undefined</span> } </span></pre> <h2 id="jsonArgs">JSON Object Can Be Supplied Instead of Object Literals</h2> <p> JSON objects can be provided as arguments instead of object literals. This is extremely handy. Replacing several arguments with a JSON object would possibly be the most significant improvement in robustness that could be made to a JavaScript project. The number of runtime errors encountered as a code base evolves would be greatly reduced. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3513daf02ebf'><button class='copyBtn' data-clipboard-target='#id3513daf02ebf' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredientX3 = new IngredientX({ <span class='unselectable'>... </span> <span class="bg_yellow">"id"</span>: "123", <span class='unselectable'>... </span> <span class="bg_yellow">"parts"</span>: 10 <span class='unselectable'>... </span> }); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientX3 <span class='unselectable'>IngredientX { id: '123', parts: 10 } </span></pre> <h2 id="jsonArgs">Arguments and Parameters Can Be Provided In Any Order</h2> <p> This definition of <code>ingredientX4</code> is identical to the definition of <code>ingredientX3</code>, even though the order of the arguments has been reversed: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddeea4e704cd0'><button class='copyBtn' data-clipboard-target='#iddeea4e704cd0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredientX4 = new IngredientX({ <span class='unselectable'>... </span> <span class="bg_yellow">"parts"</span>: 10, <span class='unselectable'>... </span> <span class="bg_yellow">"id"</span>: "123" <span class='unselectable'>... </span> }); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientX4 <span class='unselectable'>IngredientX { id: '123', parts: 10 } </span></pre> <p> The parameters in the function or method declaration are also insensitive to ordering: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida41a62e2f3b5'><button class='copyBtn' data-clipboard-target='#ida41a62e2f3b5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>class IngredientXReordered { <span class='unselectable'>... </span> constructor(<span class="bg_yellow">{parts, id}</span>) { <span class='unselectable'>..... </span> this.parts = parts; <span class='unselectable'>..... </span> this.id = id; <span class='unselectable'>..... </span> } <span class='unselectable'>... </span>} <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>var ingredientX5 = new IngredientXReordered({ <span class='unselectable'>... </span> <span class="bg_yellow">"parts"</span>: 10, <span class='unselectable'>... </span> <span class="bg_yellow">"id"</span>: "123" <span class='unselectable'>... </span> }); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientX5 <span class='unselectable'>IngredientXReordered { id: '123', parts: 10 } </span></pre> <h2 id="litArgs">Object Literals Can Be Used With Any Method</h2> <p> Object literals / named arguments can be used to define the signature of any function or method, not just class constructors. For example: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id734c3591c507'><button class='copyBtn' data-clipboard-target='#id734c3591c507' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>class IngredientY { <span class='unselectable'>... </span> constructor({id, parts}) { <span class='unselectable'>..... </span> this.id = id; <span class='unselectable'>..... </span> this.parts = parts; <span class='unselectable'>..... </span> } <span class='unselectable'>... </span> <span class='unselectable'>... </span> mix(<span class="bg_yellow">{duration, intensity}</span>) { <span class='unselectable'>... </span> console.log(`Shake for ${duration} hours at intensity ${intensity}.`); <span class='unselectable'>... </span> } <span class='unselectable'>... </span> } <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>var ingredientY = new IngredientY({id: "123", parts: 10 }); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientY.mix(<span class="bg_yellow">{duration: 2.5, intensity: 2}</span>); <span class='unselectable'>Shake for 2.5 hours at intensity 2. </span> <span class='unselectable'>undefined </span></pre> <h2 id="jsonArgs">Default Values for Named Arguments</h2> <p> To make this example more interesting, the default value for <code>id</code> will be generated as a GUID. <a href='https://stackoverflow.com/questions/105034/how-to-create-a-guid-uuid' target='_blank' rel='nofollow'>Here are some other GUID implementations</a>, but the best implementations have dependencies and that would just make the article more complex than necessary. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idece1d7ea7da2'><button class='copyBtn' data-clipboard-target='#idece1d7ea7da2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>function uuidv4() { <span class='unselectable'>... </span> return 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, <span class='unselectable'>... </span> function(c) { <span class='unselectable'>..... </span> var r = Math.random() * 16 | 0, v = c == 'x' ? r : (r & 0x3 | 0x8); <span class='unselectable'>..... </span> return v.toString(16); <span class='unselectable'>..... </span> }); <span class='unselectable'>... </span>} <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>uuidv4() <span class='unselectable'>'b13137c1-1598-42ca-9498-c1502e5405ed' </span></pre> <p> A JavaScript object literal or JSON object must be passed to a method whose parameters were defined by object literal names. If a name/value pair is not provided in the argument then the default parameter value is used. Some examples should help demonstrate how this works: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcadf7f87e517'><button class='copyBtn' data-clipboard-target='#idcadf7f87e517' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>class IngredientZ { <span class='unselectable'>... </span> constructor({id<span class="bg_yellow">=uuidv4()</span>, parts<span class="bg_yellow">=10</span>}) { <span class='unselectable'>..... </span> this.id = id; <span class='unselectable'>..... </span> this.parts = parts; <span class='unselectable'>..... </span> } <span class='unselectable'>... </span> <span class='unselectable'>... </span> mix({duration<span class="bg_yellow">=1.2</span>, intensity<span class="bg_yellow">=6</span>}) { <span class='unselectable'>... </span> console.log(`Shake for ${duration} hours at intensity ${intensity}.`); <span class='unselectable'>... </span> } <span class='unselectable'>... </span> } <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>var ingredientZ1 = new IngredientZ(<span class="bg_yellow">{parts: 4}</span>); <span class='unselectable'>undefined </span> <span class='unselectable'>> </span>ingredientZ1 <span class='unselectable'>IngredientZ { id: <span class="bg_yellow">'4290dc1a-4f4c-4579-9e27-39b68085ad97'</span>, parts: <span class="bg_yellow">4</span> } </span> <span class='unselectable'>undefined </span></pre> <p> Empty objects are allowed as arguments. All this means is that default values are used for all parameters of the object name literal. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id92f161472bad'><button class='copyBtn' data-clipboard-target='#id92f161472bad' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>var ingredientZ2 = new IngredientZ(<span class="bg_yellow">{}</span>); <span class='unselectable'>undefined </span> > ingredientZ2 <span class='unselectable'>IngredientZ { id: '9e70dc12-1f4c-3579-6a17-49a68385bf73', parts: 10 } </span> <span class='unselectable'>> </span>ingredientZ2.mix(<span class="bg_yellow">{}</span>); <span class='unselectable'>Shake for 2.5 hours at intensity 2. </span></pre> <p> Missing objects result in a syntax error. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id050af5928da1'><button class='copyBtn' data-clipboard-target='#id050af5928da1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>> </span>ingredientZ2.mix<span class="bg_yellow">()</span>; <span class='unselectable'>Uncaught TypeError: Cannot read property 'id' of undefined at new IngredientZ2 (repl:3:17) {% noselect undefined </span></pre> <h2 id="info">For More Information</h2> <p> For more information, please see <a href='https://exploringjs.com/impatient-js/ch_callables.html#named-parameters' target='_blank' rel='nofollow'>JavaScript for impatient programmers (ES2021 edition)</a>. </p> JavaScript Linter Configuration 2021-02-08T00:00:00-05:00 https://mslinn.github.io/blog/2021/02/08/js-linter-config <p> Using a lint tool can really help improve your code in a hurry. I am using <a href='https://jshint.com' target='_blank' rel='nofollow'>JSHint</a> for a project that has a big JavaScript file that needs some love. </p> <h2 id=".jshintrc"><span class="code">.jshintrc</span></h2> <p> All modern web browsers support at least the version of JavaScript that conforms to <a href='https://en.wikipedia.org/wiki/ECMAScript#6th_Edition_%E2%80%93_ECMAScript_2015' target='_blank' rel='nofollow'>ECMAScript 6th Edition</a>, also known as ECMAScript 2015. Neither the documentation for the Atom <a href='https://github.com/AtomLinter/linter-jshint' target='_blank' rel='nofollow'>linter-jshint</a> plugin nor <a href='https://jshint.com/docs/' target='_blank' rel='nofollow'>JSHint</a> itself explicitly state that in order to work with the version of JavaScript supported by all modern web browsers, you need to provide a JSON formatted configuration file that sets the <code>esversion</code> property to 6, like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>.jshintrc</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd747803b523c'><button class='copyBtn' data-clipboard-target='#idd747803b523c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{ "esversion": 6 }</pre> <p> If you do not do this, then JSHint will indicate errors if it encounters class definitions, for example. </p> <p> I put <code>.jshintrc</code> in the top-level directory of my project. </p> <p> I created a <a href='https://github.com/mslinn/linter-jshint/pull/1' target='_blank' rel='nofollow'>pull request</a> for the `linter-jshint` GitHub project so this documentation would be included. </p> <h2 id="stdin">Reading from <span class="code">stdin</span></h2> <p> The <a href='https://jshint.com/docs/cli/' target='_blank' rel='nofollow'>JSHint CLI docs</a> say: </p> <div class="quote"> If a file path is a dash (-) then JSHint will read from standard input. </div> <p> I needed to preprocess my JavaScript source files before invoking JSHint. Because JSHint can read from standard input, there is no need to write the preprocessed file contents to a temporary file. </p> <h2 id="stdin">Removing Jekyll Front Matter for JSHint</h2> <p> Jekyll can process any text file, including JavaScript files, if they contain front matter markers. This is useful for invoking Jekyll plugins and/or using Liquid expressions. My big JavaScript file has some information injected into it when Jekyll generates the site. </p> <p> Front matter is marked (delimited by) by two lines at the top of a file, consisting of three dashes, like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id10da0dfcd68d'><button class='copyBtn' data-clipboard-target='#id10da0dfcd68d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>--- ---</pre> <p> Here is how the empty front matter can be stripped from <code>myfile.js</code> so JSHint can inspect the remaining lines: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idea497ed42c87'><button class='copyBtn' data-clipboard-target='#idea497ed42c87' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sed '/---/d' < myfile.js | jshint -</pre> Functional and Non-Functional E-Commerce Requirements 2021-02-02T00:00:00-05:00 https://mslinn.github.io/blog/2021/02/02/ecommerce-requirements <p> In a <a href='/blog/2021/01/30/opencart-postgres.html'>previous blog post</a> I described how I searched for <a href='https://www.google.com/search?q=open+source+shopping+cart' target='_blank' rel='nofollow'>open source shopping cart</a> and the disappointing software options that I found. These search results showed software projects that began 20 years ago and were generally of low quality. The businesses that manage these projects use a failed business model for open source, namely software-as-a-service (SaaS) without much added value. The hosted products have not changed much since they were first established, and they have not kept up with the relentless advances in computer technology. Those businesses derive some additional revenue from customizing the open-source projects. </p> <p> I decided to be more rigorous in my needs analysis for a shopping cart with good coupon support, so that I could find more suitable options. I now know that the cart must have: </p> <ul> <li>Strong discount/coupon support.</li> <li>Support for SKUs defined on-the-fly by customers as they interact with the web site.</li> <li>Integration with a variety of payment processors.</li> </ul> <p> Before I share my reviews of candidate shopping carts with you, let&rsquo;s first discuss the state of the e-commerce market, functional and non-functional requirements, the master/detail pattern, and some technology trends for business software. </p> <h2 id="hot">E-Commerce Is Hot, Hot, Hot</h2> <div style="text-align: center;"> <a href="https://www.digitalcommerce360.com/article/us-ecommerce-sales/" target="_blank" ><picture> <source srcset="/blog/images/django/usEcommerce.webp" type="image/webp"> <source srcset="/blog/images/django/usEcommerce.png" type="image/png"> <img src="/blog/images/django/usEcommerce.png" title="US ecommerce grows 44% in 2020<br />Online spending was $861 billion: 21% of total retail sales" class="center halfsize liImg2 rounded shadow" alt="US ecommerce grows 44% in 2020<br />Online spending was $861 billion: 21% of total retail sales" /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://www.digitalcommerce360.com/article/us-ecommerce-sales/" target="_blank" > US ecommerce grows 44% in 2020<br />Online spending was $861 billion: 21% of total retail sales </a> </figcaption> </figure> </div> <div style=""> <a href="https://www.amazon.com/gp/product/0062292986/ref=as_li_qf_sp_asin_il_tl" target="_blank" ><picture> <source srcset="/blog/images/ecommerce/ecommerceGrowth.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/ecommerceGrowth.png" type="image/png"> <img src="/blog/images/ecommerce/ecommerceGrowth.png" title=" US e-commerce vs. retail sales, 2010-2020<br /> Source: Digital Commerce 360, U.S. Department of Commerce; January 2021 " class=" liImg2 rounded shadow" alt=" US e-commerce vs. retail sales, 2010-2020<br /> Source: Digital Commerce 360, U.S. Department of Commerce; January 2021 " /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://www.amazon.com/gp/product/0062292986/ref=as_li_qf_sp_asin_il_tl" target="_blank" > US e-commerce vs. retail sales, 2010-2020<br /> Source: Digital Commerce 360, U.S. Department of Commerce; January 2021 </a> </figcaption> </figure> </div> <p> There are a huge number of e-commerce sites, and that market is experiencing strong growth in part due to the COVID-19 epidemic. E-commerce has <a href='https://www.amazon.com/gp/product/0062292986/ref=as_li_qf_sp_asin_il_tl' target='_blank' rel='nofollow'>entered Main Street</a>, as per Geoffrey Moore&rsquo;s technology adoption lifecycle. </p> <div style="text-align: center;"> <a href="https://www.amazon.com/gp/product/0062292986/ref=as_li_qf_sp_asin_il_tl" target="_blank" ><picture> <source srcset="/blog/images/ecommerce/crossingTheChasm.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/crossingTheChasm.png" type="image/png"> <img src="/blog/images/ecommerce/crossingTheChasm.png" title="&ldquo;Crossing the Chasm&rdquo;, which introduced the technology adoption lifecycle" class="center quartersize liImg2 rounded shadow" alt="&ldquo;Crossing the Chasm&rdquo;, which introduced the technology adoption lifecycle" /> </picture></a> <figcaption class="quartersize" style="width: 100%; text-align: center;"> <a href="https://www.amazon.com/gp/product/0062292986/ref=as_li_qf_sp_asin_il_tl" target="_blank" > &ldquo;Crossing the Chasm&rdquo;, which introduced the technology adoption lifecycle </a> </figcaption> </figure> </div> <h2 id="nfrs">Functional requirements</h2> <p> QRA Corp has a good definition of functional requirements. Paraphrasing information from <a href='https://qracorp.com/functional-vs-non-functional-requirements/' target='_blank' rel='nofollow'>this page</a>: </p> <div class="quote"> <p> Functional requirements describe what the system does or must not do, and can be thought of in terms of how the system responds to inputs. Functional requirements usually define if/then behaviors and include calculations, data input, and business processes. </p> <p> Functional requirements are features that allow the system to function as it was intended. Put another way, if the functional requirements are not met, the system will not work. Functional requirements are product features and focus on user requirements. </p> </div> <p> Functionally, a shopping cart is well understood, yet in order to be certain that a shopping cart actually works properly, test data and test procedures would need to be designed to verify that all the corner cases were exercised. That would be a significant amount of work to do properly, but at least the work would be rather straightforward. </p> <h2 id="nfrs">Non-functional requirements (NFRs)</h2> <p> Non-functional requirements relate to user expectations, and include security, reliability, performance, maintainability, scalability, and usability. Here are some paraphrased excerpts from the <a href='https://en.wikipedia.org/wiki/Non-functional_requirement' target='_blank' rel='nofollow'>Wikipedia article on NFRs</a>: </p> <div class="quote"> <p> An NFR is a requirement that specifies criteria that can be used to judge the operation of a system, rather than specific behaviors&mldr;<br /><br /> The plan for implementing NFRs is detailed in the system architecture, because they are usually architecturally significant requirements&mldr;<br /><br /> Broadly, functional requirements define what a system is supposed to <i>do</i> and NFRs define how a system is supposed to <i>be</i>&mldr;<br /><br /> NFRs can be divided into two main categories: </p> <ul> <li>Execution qualities, such as safety, security and usability, which are observable during operation (at run time).</li> <li>Evolution qualities, such as testability, maintainability, extensibility and scalability, which are embodied in the static structure of the system.</li> </ul> </div> <p> I need a shopping cart that has been properly tested, is flexible to configure, and is easy to use. These non-functional requirements are more important to me than the mechanics of how the cart was built or the technology that was used, especially for a proof of concept. In my reviews of shopping cart candidates I will focus on how well they fulfill these NFRs. </p> <p> I also want to use technology that is current, properly maintained, full-featured and has a future. </p> <h2 id="master-detail">Master-Detail Structures, Interfaces and Reporting</h2> <p> Shopping carts are a classic example of a master-detail structure. The user interface of a shopping cart must exploit the master-detail paradigm effectively. </p> <div style=""> <a href="https://www.oracle.com/webfolder/ux/middleware/alta/patterns/MasterDetail.html" target="_blank" ><picture> <source srcset="/blog/images/ecommerce/MasterDetail.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/MasterDetail.png" type="image/png"> <img src="/blog/images/ecommerce/MasterDetail.png" title="Image from Oracle Alta UI Patterns: Master-Detail" class=" liImg2 rounded shadow" alt="Image from Oracle Alta UI Patterns: Master-Detail" /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://www.oracle.com/webfolder/ux/middleware/alta/patterns/MasterDetail.html" target="_blank" > Image from Oracle Alta UI Patterns: Master-Detail </a> </figcaption> </figure> </div> <p> <a href='https://en.wikipedia.org/wiki/Master%E2%80%93detail_interface' target='_blank' rel='nofollow'>Wikipedia defines master/detail data models</a> as: </p> <p class="quote"> A master–detail relationship is a one-to-many type relationship. Examples of a master-detail relationship are: a set of purchase orders and a set of line items belonging to each purchase order, an expense report with a set of expense line items or a department with a list of employees belonging to it. An application can use this master-detail relationship to enable users to navigate through the purchase order data and see the detail data for line items only related to the master purchase order selected. </p> <p> In the book <a href='https://www.oreilly.com/library/view/sap-businessobjects-bi/9780071773126/lev1sec120.html' target='_blank'>SAP BusinessObjects BI 4.0 The Complete Reference 3/E</a>, authors Cindi Howson and Elizabeth Newbould provide this rather abstract definition of <i>master/detail reports</i>: </p> <p class="quote"> A master/detail report is a particular kind of report in which a dimension value (master) is used to group data (detail) into separate sections. Master/detail reports allow you to analyze and format data for each unique master data value. </p> <p> Page 13 of <a href='https://books.google.ca/books?id=wgBMWCvTEHQC&pg=PA13&dq=First+normal+form+removes+repetition+by+creating+one-to-many+relationships&hl=en&sa=X&ved=2ahUKEwjB-5_NrMvuAhWCElkFHV0aA-MQ6AEwAHoECAEQAg#v=onepage&q&f=false' target='_blank' rel='nofollow'>Oracle&reg; Performance Tuning for 10gR2, Second Edition</a> by Gavin JT Powell has a more concrete definition: </p> <p class="quote"> <b>First normal form removes repetition by creating one-to-many relationships.</b> Data repeated many times in one entity is removed to a subset entity, which becomes the container for the removed repeating data. Each row in the subset entity will contain a single reference to each row in the original entity. The original entity will then contain only nonduplicated data. This one-to-many relationship is commonly known as a master-detail relationship, where repeating columns are removed to a new entity. The new entity gets a primary key consisting of a composite of the primary key in the master entity and a unique identifier (within each master primary key) on the detail entity. </p> <p> Master-detail is such a common architectural pattern that most business software vendors provide support for it. Other vendors include <a href='https://www.ibm.com/support/knowledgecenter/SSEP7J_11.1.0/com.ibm.swg.ba.cognos.ug_cr_rptstd.doc/t_cr_rptstd_modrep_create_master_detail_relationship.html#cr_rptstd_modrep_create_master_detail_relationship' target='_blank'>IBM</a>, <a href='https://docs.oracle.com/cd/E15586_01/web.1111/b31974/web_masterdetail.htm' target='_blank'>Oracle</a>, <a href='https://help.salesforce.com/articleView?id=sf.overview_of_custom_object_relationships.htm&type=5' target='_blank'>Salesforce</a>, and <a href='https://channel9.msdn.com/Blogs/OneCode/How-to-create-a-master-detail-ListBox-in-universal-Windows-apps' target='_blank'>Microsoft</a>. Apple&rsquo;s iPad is used as a client for master-detail applications so often that the iOS SDK even provides a <a href='https://www.oreilly.com/library/view/beginning-ios-5/9781118144251/ch004-sec007.html' target='_blank'>Master-Detail Application template</a>. The master/detail pattern for Google Android applications <a href='https://github.com/lurbas/MaterialMasterDetail' target='_blank' rel='nofollow'>can be implemented using the Material Design</a> visual language. </p> <h2 id="master-detail-frameworks">Master-Detail Frameworks</h2> <p> Shopping carts must have engaging and effective implementations for master/detail patterns throughout: in the user interface, in the design of reports, in the data structures, in the types of software modules and their interfaces to each other, and in the internal data flow. The best-known web framework explicitly designed to support the master/detail pattern is <a href='https://rubyonrails.org/' target='_blank'>Ruby on Rails</a>. Many other web frameworks can support master/detail, of course. </p> <div style="text-align: center;"> <a href="https://rubyonrails.org/" target="_blank" ><picture> <source srcset="/blog/images/ecommerce/Ruby-on-Rails.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/Ruby-on-Rails.png" type="image/png"> <img src="/blog/images/ecommerce/Ruby-on-Rails.png" title="Ruby on Rails" class="center halfsize liImg2 rounded shadow" alt="Ruby on Rails" /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://rubyonrails.org/" target="_blank" > Ruby on Rails </a> </figcaption> </figure> </div> <h2 id="momentum">Momentum</h2> <h3 id="webserver">Web Servers</h3> <p> Web servers are usually placed in front of e-commerce servers for scalability and security reasons. <a href='https://www.netcraft.com/about/' target='_blank' rel='nofollow'>Netcraft</a> has been reporting on web server deployments since 1995. The <a href='https://news.netcraft.com/archives/2021/01/28/january-2021-web-server-survey.html' target='_blank' rel='nofollow'>Netcraft January 2021 Web Server Survey</a> has some interesting trends. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/ecommerce/wss-active-share.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/wss-active-share.png" type="image/png"> <img src="/blog/images/ecommerce/wss-active-share.png" title="Web server developer market share by server type.<br />Apache <code>httpd</code> is losing ground, as developers move to Microsoft and nginx." class="center liImg2 rounded shadow" alt="Web server developer market share by server type.<br />Apache <code>httpd</code> is losing ground, as developers move to Microsoft and nginx." /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> Web server developer market share by server type.<br />Apache <code>httpd</code> is losing ground, as developers move to Microsoft and nginx. </figcaption> </figure> </div> <div style="text-align: center;"> <picture> <source srcset="/blog/images/ecommerce/wss-top-1m-share.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/wss-top-1m-share.png" type="image/png"> <img src="/blog/images/ecommerce/wss-top-1m-share.png" title="For the busiest 1 million websites, Microsoft has taken the lead." class="center liImg2 rounded shadow" alt="For the busiest 1 million websites, Microsoft has taken the lead." /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> For the busiest 1 million websites, Microsoft has taken the lead. </figcaption> </figure> </div> <p> Microsoft&rsquo;s <a href='https://dotnet.microsoft.com/apps/aspnet' target='_blank' rel='nofollow'>ASP.NET</a> web framework is free, and web hosting is free (to a point), but those enticements fulfill their purpose by locking in customers to Microsoft's software stack and tools. The nopCommerce e-commerce server is free, very capable, and has enjoyed terrific adoption. This technology might be a good business decision for e-commerce sites that have typical requirements, but not if significant customization or integration with non-Microsoft services becomes important. This article was written by an experienced developer: <a href='https://www.freecodecamp.org/news/i-built-a-web-api-with-express-flask-aspnet/' target='_blank' rel='nofollow'>I rebuilt the same web API using Express, Flask, and ASP.NET. Here's what I found</a>. </p> <h3 id="language">Computer Languages</h3> <div style="text-align: center;"> <a href="https://github.com/nasa/fprime" target="_blank" ><picture> <source srcset="/blog/images/ecommerce/mars_drone.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/mars_drone.png" type="image/png"> <img src="/blog/images/ecommerce/mars_drone.png" title="Python runs <a rel='nofollow' href='https://www.nasa.gov/feature/jpl/6-things-to-know-about-nasas-ingenuity-mars-helicopter/' target='_blank'>NASA&rsquo;s Mars drone <i>Ingenuity</i></a>." class="center liImg2 rounded shadow" alt="Python runs <a rel='nofollow' href='https://www.nasa.gov/feature/jpl/6-things-to-know-about-nasas-ingenuity-mars-helicopter/' target='_blank'>NASA&rsquo;s Mars drone <i>Ingenuity</i></a>." /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://github.com/nasa/fprime" target="_blank" > Python runs <a rel='nofollow' href='https://www.nasa.gov/feature/jpl/6-things-to-know-about-nasas-ingenuity-mars-helicopter/' target='_blank'>NASA&rsquo;s Mars drone <i>Ingenuity</i></a>. </a> </figcaption> </figure> </div> <p> Of all computer languages, Python has arguably the most momentum at present. Ruby on Rails is terrific, but its market share has dropped dramatically since 2011, and without that framework the Ruby language would probably be much less important that it is. <a href='https://python.org' target='_blank' rel='nofollow'>Python</a> has been growing in all directions for a very long time. Microsoft&rsquo;s C# language, which requires a .NET compatible runtime, is also popular, but never had the momentum that Python has. </p> <p> The following graph compares the popularity of the Python Django library for making web applications against the popularity of the Ruby on Rails framework, for the time period 2009 to the present day. As you can see, in 2011 Ruby on Rails was at its peak popularity; at that time it was 300% more popular than Django. Now the situation is reversed: today Django is 400% more popular than Ruby on Rails. </p> <div style=""> <a href="https://insights.stackoverflow.com/trends?tags=django%2Cruby-on-rails" target="_blank" ><picture> <source srcset="/blog/images/django/django_vs_ror.webp" type="image/webp"> <source srcset="/blog/images/django/django_vs_ror.png" type="image/png"> <img src="/blog/images/django/django_vs_ror.png" title="Stack Overflow Trends: Django vs Ruby on Rails" class=" liImg2 rounded shadow" alt="Stack Overflow Trends: Django vs Ruby on Rails" /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://insights.stackoverflow.com/trends?tags=django%2Cruby-on-rails" target="_blank" > Stack Overflow Trends: Django vs Ruby on Rails </a> </figcaption> </figure> </div> <p> The one thing about Python that I like better than all other languages is the <a href='https://www.python.org/dev/peps/pep-0206/#id3' target='_blank' rel='nofollow'>&ldquo;batteries included&rdquo;</a> feature-driven approach. This means that Python projects can aspire to more ambitious goals for a given amount of programmer effort, as compared to implementing with other languages. That was a significant factor in my decision to focus on Python-powered semi-custom shopping carts, instead of investigating shopping carts programmed with other computer languages. Ruby on Rails would doubtless provide an excellent foundation for shopping carts, but I think Python is a better strategic choice for me at the present time in the world of open source. </p> <p> This is not a decision I would have made in years past. Python's runtime expends a lot of electrical power to run Python programs. As available computing power continues to grow year-on-year within devices everywhere, the &ldquo;Python runtime tax&rdquo; has become much easier to bear. </p> <div style="text-align: center;"> <a href="https://python.org/" target="_blank" ><picture> <source srcset="/blog/images/python.webp" type="image/webp"> <source srcset="/blog/images/python.png" type="image/png"> <img src="/blog/images/python.png" title="The Python Language" class="center halfsize liImg2 rounded shadow" alt="The Python Language" /> </picture></a> <figcaption class="halfsize" style="width: 100%; text-align: center;"> <a href="https://python.org/" target="_blank" > The Python Language </a> </figcaption> </figure> </div> <h2 id="open">Beyond Open Source</h2> <p> A lot of open source software suffers from limited or no funding. This seriously impacts long-term viabilty. I am happy to consider all appropriate technology, and I am not fixated on open source options. Two major technical components need to be considered: </p> <ul> <li>E-commerce framework &ndash; this dictates the computer language and runtime library.</li> <li>Database &ndash; the e-commerce framework often dictates the choice of database.</li> </ul> <h2 id="arch">Monolithic vs Serverless Architectures</h2> <h3 id="serverless">Modular Yet Monolithic Architectures</h3> <div style="text-align: center;"> <picture> <source srcset="/blog/images/ecommerce/easter_island.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/easter_island.png" type="image/png"> <img src="/blog/images/ecommerce/easter_island.png" title="Monoliths on Easter Island" class="center liImg2 rounded shadow" alt="Monoliths on Easter Island" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> Monoliths on Easter Island </figcaption> </figure> </div> <p> Traditionally, software has been built by first constructing functional modules, then combining the modules into a complete program (the monolith). For e-commerce, entire programs are deployed to production by adding configuration information and integrating with external services, such as payment processors. This works well, however, the result is a centralized computing resource that has a fixed capability and resource requirements. The expense of operating the software does not change much, regardless of the traffic volume. In order to handle periods of heavy traffic, complex mechanisms are required to scale up transactional capacity. Conversely, during periods of light traffic, the minimum cost to maintain the system in an operational state can be significant, especially for a startup company. </p> <h3 id="serverless">Serverless Architecture</h3> <div style="text-align: center;"> <a href="https://martinfowler.com/articles/serverless.html" target="_blank" ><picture> <source srcset="/blog/images/ecommerce/serverless_fowler.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/serverless_fowler.png" type="image/png"> <img src="/blog/images/ecommerce/serverless_fowler.png" title="Martin Fowler on Serverless Architectures" class="center liImg2 rounded shadow" alt="Martin Fowler on Serverless Architectures" /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://martinfowler.com/articles/serverless.html" target="_blank" > Martin Fowler on Serverless Architectures </a> </figcaption> </figure> </div> <p> One of the big architectural advances in recent years has been the introduction of <i>serverless computing</i>. It provides near-infinite scalability without paying for unused resources. <a href='https://www.cloudflare.com/learning/serverless/what-is-serverless/' target='_blank' rel='nofollow'>CloudFlare has a good definition</a>: </p> <p class="quote"> Serverless computing is a method of providing backend services on an as-used basis. A serverless provider allows users to write and deploy code without the hassle of worrying about the underlying infrastructure. A company that gets backend services from a serverless vendor is charged based on their computation and do not have to reserve and pay for a fixed amount of bandwidth or number of servers, as the service is auto-scaling. Note that despite the name serverless, physical servers are still used but developers do not need to be aware of them. </p> <p> Vendors that provide serverless computing platforms include <a href='https://aws.amazon.com/lambda/' target='_blank' rel='nofollow'>AWS Lambda</a>, <a href='https://azure.microsoft.com/services/functions/' target='_blank' rel='nofollow'>Azure Functions</a>, <a href='https://workers.cloudflare.com/' target='_blank' rel='nofollow'>CloudFlare Workers</a>, <a href='https://cloud.google.com/functions' target='_blank' rel='nofollow'>Google Cloud Functions</a> </p> <p> The <a href='https://www.serverless.com' target='_blank' rel='nofollow'>Serverless Framework</a> is a language- and platform- agnostic framework. Languages supported include Node.js, Python, Java, Go, C#, Ruby, Swift, Kotlin, PHP, Scala, & F#. Platforms supported include <a href='https://www.serverless.com/framework/docs/providers/' target='_blank' rel='nofollow'>Alibaba Cloud, AWS, Microsoft Azure, Fn Project, Google Cloud Platform, Apache OpenWhisk, CloudFlare Workers, Knative, Kubeless, Spotinst and Tencent Cloud</a>. The <a href='https://github.com/serverless/serverless' target='_blank' rel='nofollow'>GitHub project</a> has the code. </p> <h2 id="responsive">Responsive Web Pages</h2> <div style="text-align: center;"> <a href="https://medium.com/level-up-web/best-practices-of-responsive-web-design-6da8578f65c4" target="_blank" ><picture> <source srcset="/blog/images/ecommerce/responsive_web.webp" type="image/webp"> <source srcset="/blog/images/ecommerce/responsive_web.png" type="image/png"> <img src="/blog/images/ecommerce/responsive_web.png" title="Image from &lsquo;Best Practices of Responsive Web Design&rsquo; by Bradley Nice" class="center liImg2 rounded shadow" alt="Image from &lsquo;Best Practices of Responsive Web Design&rsquo; by Bradley Nice" /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://medium.com/level-up-web/best-practices-of-responsive-web-design-6da8578f65c4" target="_blank" > Image from &lsquo;Best Practices of Responsive Web Design&rsquo; by Bradley Nice </a> </figcaption> </figure> </div> <p> <a href='https://developer.mozilla.org/en-US/docs/Learn/CSS/CSS_layout/Responsive_Design' target='_blank' rel='nofollow'>Responsive web pages</a> alter their layout and appearance to suit different screen widths and resolutions. Many of the shopping carts I looked at did not support responsive web pages. </p> <h2 id="next">Next: Check Out Django Shopping Carts</h2> <p> OK, so Python, and therefore Django is winning. I admit that I like the Django slogan: &ldquo;Django &ndash; The web framework for perfectionists with deadlines&rdquo;. When an option dominates the competition, you would need a really special reason to consider other options. My functional and non-functional requirements are mainstream, so I'm going to check out the leading option. Today the leading option is Django. Stay tuned&mldr; </p> OpenCart - Postgres - ngnix - Ubuntu 2021-01-30T00:00:00-05:00 https://mslinn.github.io/blog/2021/01/30/opencart-postgres <p> I need a shopping cart that has good coupon/discount support with flexible pricing. My requirements are unique in that each item in the cart might be a custom product, with the price computed according to a formula on our server. There are very few standard SKUs. </p> <p> I started to look into 3 options for obtaining a shopping cart with good coupon support: building my own, or using a commercial product, or customizing an open-source project. This blog post is the story of the &lsquo;customize an open source&rsquo; track. </p> <h2 id="opencart">OpenCart</h2> <p> I wanted to evaluate the leading open-source shopping cart contender by installing it on a development machine and giving it real data. OpenCart is renowned as one of the better open-source shopping carts available today. As with many open-source projects, the company that provides the source code have a conflict of interest: if they make installing and configuring the software effortless then their revenue would be much less than if they had a cadre of interested but frustrated developers. I looked at the hosting options and did not like the price/performance and customization options. </p> <p> OpenCart shows its age by using MySQL and its descendants, like Maria. Long ago I moved on from MySQL to Postgres, and I have been pleased with Postgres. I decided to try to make OpenCart run on Postgres and Ubuntu. I knew before I started that OpenCart&rsquo;s support for Postgres was weak. </p> <div class="pullQuote"> Spoiler alert: OpenCart would not pass a proper source code quality review and any claims of PostgreSQL support are bogus </div> <p> If you find watching videos of gory slow-motion accidents entertaining, please read on. </p> <h2 id="tests">Unit Tests</h2> <p> First thing I look for when evaluating software for possible incorporation into a project is unit tests. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id697d6246a5fd'><button class='copyBtn' data-clipboard-target='#id697d6246a5fd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git grep -Ii test \ ":(exclude)*.css" \ ":(exclude)*.yml" \ ":(exclude)*.js" <span class='unselectable'>$ </span>git grep -IiE 'phptest|Codeception' \ ":(exclude)*.css" \ ":(exclude)*.yml" \ ":(exclude)*.js"</pre> <p> There were no tests and no references to PHPUnit or Codeception. This was a big black mark against OpenCart. </p> <h2 id="deps">Install Dependencies</h2> <p> If you need information about PostgreSQL, <a href='https://www.postgresql.org/' target='_blank' rel='nofollow'>here is the mother ship</a>. This is a good description of <a href='https://www.digitalocean.com/community/tutorials/how-to-install-postgresql-on-ubuntu-20-04-quickstart' target='_blank' rel='nofollow'>how to install Postgres</a>. </p> <p> I use PGAdmin when I want a graphical interface to PostgreSQL. Installation instructions are <a href='https://www.pgadmin.org/download/pgadmin-4-apt/' target='_blank' rel='nofollow'>here</a>. In a nutshell: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4d1d041eb1f0'><button class='copyBtn' data-clipboard-target='#id4d1d041eb1f0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl https://www.pgadmin.org/static/packages_pgadmin_org.pub | \ sudo apt-key add <span class='unselectable'>$ </span>sudo sh -c \ 'echo "deb https://ftp.postgresql.org/pub/pgadmin/pgadmin4/apt/$(lsb_release -cs) pgadmin4 main" > /etc/apt/sources.list.d/pgadmin4.list && apt update' <span class='unselectable'>$ </span>sudo apt install pgadmin4</pre> <p> I installed the rest of the dependencies like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc171aa462153'><button class='copyBtn' data-clipboard-target='#idc171aa462153' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install \ postgresql postgresql-contrib \ software-properties-common lynx \ php7.4 php-fpm php-gd php-curl php-postgre php-zip \ php5-pgsql</pre> <p> <code>phpenmod</code> is a Debian / Ubuntu command for enabling PHP extensions. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id21f370fa61d2'><button class='copyBtn' data-clipboard-target='#id21f370fa61d2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo phpenmod pgsql</pre> <p> I verified that the desired PHP extensions were installed typing <code>php -m</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id792aca067d46'><button class='copyBtn' data-clipboard-target='#id792aca067d46' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>php -m | grep pg <span class='unselectable'>pdo_pgsql pgsql</span></pre> <h2 id="debug">Figuring Out Problems</h2> <p> I also installed the PHP debugger, when I realized needed to do some debugging. IntelliJ (which is the big brother of PHP Storm) did a terrific job of providing debug capability for command-line PHP. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7eaf400a97c0'><button class='copyBtn' data-clipboard-target='#id7eaf400a97c0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install php-xdebug</pre> <p> I opened a new console and continuously viewed the <code>nginx</code>, PHP and PostgreSQL error logs. This was a big help whenever I needed to figure out problems. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id756e795bdeb8'><button class='copyBtn' data-clipboard-target='#id756e795bdeb8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>tail -f \ /var/log/nginx/*.log \ /var/log/php*.log \ /var/log/postgresql/postgresql-12*.log</pre> <h2 id="nginx_config">Configure <span class='code'>nginx</span></h2> <p> The only change I made to <code>/etc/nginx/nginx.conf</code> was to change the default MIME type from <code>application/octet-stream</code> to <code>text/html</code>. The change is highlighted in yellow. </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/nginx/nginx.conf</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbf73cf1030da'><button class='copyBtn' data-clipboard-target='#idbf73cf1030da' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>user www-data; worker_processes auto; pid /run/nginx.pid; include /etc/nginx/modules-enabled/*.conf; events { worker_connections 768; # multi_accept on; } http { ## # Basic Settings ## sendfile on; tcp_nopush on; types_hash_max_size 2048; # server_tokens off; # server_names_hash_bucket_size 64; # server_name_in_redirect off; include /etc/nginx/mime.types; <span style="background-color: yellow">#default_type application/octet-stream;</span> <span style="background-color: yellow">default_type text/html;</span> ## # SSL Settings ## ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE ssl_prefer_server_ciphers on; ## # Logging Settings ## access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log notice; ## # Gzip Settings ## gzip on; # gzip_vary on; # gzip_proxied any; # gzip_comp_level 6; # gzip_buffers 16 8k; # gzip_http_version 1.1; # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript; ## # Virtual Host Configs ## include /etc/nginx/conf.d/*.conf; include /etc/nginx/sites-enabled/*; } #mail { # # See sample authentication script at: # # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript # # # auth_http localhost/auth.php; # # pop3_capabilities "TOP" "USER"; # # imap_capabilities "IMAP4rev1" "UIDPLUS"; # # server { # listen localhost:110; # protocol pop3; # proxy on; # } # # server { # listen localhost:143; # protocol imap; # proxy on; # } #}</pre> <p> I deleted <code>/etc/nginx/sites-enabled/default</code> and replaced it with <code>/etc/nginx/sites-enabled/php</code> so PHP files would be parsed properly, no matter what directory they resided in. </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/nginx/sites_enabled/php</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idaa17a48533ca'><button class='copyBtn' data-clipboard-target='#idaa17a48533ca' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># See https://tecadmin.net/setup-nginx-php-fpm-on-ubuntu-20-04/ server { listen 80; root /var/www/html; index index.php index.html index.htm; server_name example.com; location / { try_files $uri $uri/ =404; } location ~ \.php$ { include snippets/fastcgi-php.conf; fastcgi_pass unix:/var/run/php/php7.4-fpm.sock; } }</pre> <p> Now it was time to restart <code>nginx</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id059a0de2fb88'><button class='copyBtn' data-clipboard-target='#id059a0de2fb88' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl restart nginx.service</pre> <h2 id="restart">Verify Services Are Running</h2> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4768bb89689f'><button class='copyBtn' data-clipboard-target='#id4768bb89689f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl status nginx <span class='unselectable'>● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-01-29 16:58:04 EST; 15h ago Docs: man:nginx(8) Main PID: 1584845 (nginx) Tasks: 9 (limit: 38389) Memory: 10.0M CGroup: /system.slice/nginx.service ├─1584845 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; ├─1584846 nginx: worker process ├─1584847 nginx: worker process ├─1584848 nginx: worker process ├─1584849 nginx: worker process ├─1584850 nginx: worker process ├─1584851 nginx: worker process ├─1584852 nginx: worker process └─1584853 nginx: worker process Jan 29 16:58:04 localhost systemd[1]: Starting A high performance web server and a reverse proxy server... Jan 29 16:58:04 localhost systemd[1]: Started A high performance web server and a reverse proxy server.</span></pre> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd3939540cb34'><button class='copyBtn' data-clipboard-target='#idd3939540cb34' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl status php7.4-fpm <span class='unselectable'>● php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2021-01-30 08:32:36 EST; 10min ago Docs: man:php-fpm7.4(8) Process: 3430785 ExecStartPost=/usr/lib/php/php-fpm-socket-helper install /run/php/php-fpm.sock /etc/php/7.4/fp> Main PID: 3430782 (php-fpm7.4) Status: "Processes active: 0, idle: 2, Requests: 0, slow: 0, Traffic: 0req/sec" Tasks: 3 (limit: 38389) Memory: 8.1M CGroup: /system.slice/php7.4-fpm.service ├─3430782 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf) ├─3430783 php-fpm: pool www └─3430784 php-fpm: pool www Jan 30 08:32:36 localhost systemd[1]: Starting The PHP 7.4 FastCGI Process Manager... Jan 30 08:32:36 localhost systemd[1]: Started The PHP 7.4 FastCGI Process Manager. </span></pre> <h2 id="restart">Verify PHP Works</h2> <p> I made this file, which is very common when working with PHP. Just ensure that it does not appear in your production site, or hackers will know more about your website than they should. </p> <div class='codeLabel unselectable' data-lt-active='false'>/var/www/html/info.php</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2f6db0b00c94'><button class='copyBtn' data-clipboard-target='#id2f6db0b00c94' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>&lt;?php phpinfo(); ?></pre> <p> Now I verified that PHP worked by viewed information about the setup. The <code>-I</code> option causes <code>curl</code> to just return HTML headers, not the HTML body. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id79f158cce698'><button class='copyBtn' data-clipboard-target='#id79f158cce698' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -I http://localhost/info.php <span class='unselectable'>HTTP/1.1 200 OK Server: nginx/1.18.0 (Ubuntu) Date: Sat, 30 Jan 2021 18:50:30 GMT Content-Type: text/html; charset=UTF-8 Connection: keep-alive </span></pre> <p> View <code>info.php</code> in a web browser to see the details. Lynx is good for that from a command line: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7f26dea58367'><button class='copyBtn' data-clipboard-target='#id7f26dea58367' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>lynx http://localhost/info.php</pre> <h2 id="postgres_setup">Set Up ‎PostgreSQL</h2> <p> I only changed the value of <code>listen_addresses</code> in <code>postgresql.conf</code>. Again, this change is highlighted in yellow. </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/postgresql/12/main/postgresql.conf</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc45c84d0d4fa'><button class='copyBtn' data-clipboard-target='#idc45c84d0d4fa' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># ----------------------------- # PostgreSQL configuration file # ----------------------------- # # This file consists of lines of the form: # # name = value # # (The "=" is optional.) Whitespace may be used. Comments are introduced with # "#" anywhere on a line. The complete list of parameter names and allowed # values can be found in the PostgreSQL documentation. # # The commented-out settings shown in this file represent the default values. # Re-commenting a setting is NOT sufficient to revert it to the default value; # you need to reload the server. # # This file is read on server startup and when the server receives a SIGHUP # signal. If you edit the file on a running system, you have to SIGHUP the # server for the changes to take effect, run "pg_ctl reload", or execute # "SELECT pg_reload_conf()". Some parameters, which are marked below, # require a server shutdown and restart to take effect. # # Any parameter can also be given as a command-line option to the server, e.g., # "postgres -c log_connections=on". Some parameters can be changed at run time # with the "SET" SQL command. # # Memory units: kB = kilobytes Time units: ms = milliseconds # MB = megabytes s = seconds # GB = gigabytes min = minutes # TB = terabytes h = hours # d = days #------------------------------------------------------------------------------ # FILE LOCATIONS #------------------------------------------------------------------------------ # The default values of these variables are driven from the -D command-line # option or PGDATA environment variable, represented here as ConfigDir. data_directory = '/var/lib/postgresql/12/main' # use data in another directory # (change requires restart) hba_file = '/etc/postgresql/12/main/pg_hba.conf' # host-based authentication file # (change requires restart) ident_file = '/etc/postgresql/12/main/pg_ident.conf' # ident configuration file # (change requires restart) # If external_pid_file is not explicitly set, no extra PID file is written. external_pid_file = '/var/run/postgresql/12-main.pid' # write an extra PID file # (change requires restart) #------------------------------------------------------------------------------ # CONNECTIONS AND AUTHENTICATION #------------------------------------------------------------------------------ # - Connection Settings - <span style="background-color: yellow">listen_addresses = '*'</span> <span style="background-color: yellow">#listen_addresses = 'localhost' # what IP address(es) to listen on;</span> # comma-separated list of addresses; # defaults to 'localhost'; use '*' for all # (change requires restart) port = 5432 # (change requires restart) max_connections = 100 # (change requires restart) #superuser_reserved_connections = 3 # (change requires restart) unix_socket_directories = '/var/run/postgresql' # comma-separated list of directories # (change requires restart) #unix_socket_group = '' # (change requires restart) #unix_socket_permissions = 0777 # begin with 0 to use octal notation # (change requires restart) #bonjour = off # advertise server via Bonjour # (change requires restart) #bonjour_name = '' # defaults to the computer name # (change requires restart) # - TCP settings - # see "man 7 tcp" for details #tcp_keepalives_idle = 0 # TCP_KEEPIDLE, in seconds; # 0 selects the system default #tcp_keepalives_interval = 0 # TCP_KEEPINTVL, in seconds; # 0 selects the system default #tcp_keepalives_count = 0 # TCP_KEEPCNT; # 0 selects the system default #tcp_user_timeout = 0 # TCP_USER_TIMEOUT, in milliseconds; # 0 selects the system default # - Authentication - #authentication_timeout = 1min # 1s-600s #password_encryption = md5 # md5 or scram-sha-256 #db_user_namespace = off # GSSAPI using Kerberos #krb_server_keyfile = '' #krb_caseins_users = off # - SSL - #ssl = on #ssl_ca_file = '' ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem' #ssl_crl_file = '' ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key' #ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers #ssl_prefer_server_ciphers = on #ssl_ecdh_curve = 'prime256v1' #ssl_min_protocol_version = 'TLSv1' #ssl_max_protocol_version = '' #ssl_dh_params_file = '' #ssl_passphrase_command = '' #ssl_passphrase_command_supports_reload = off #------------------------------------------------------------------------------ # RESOURCE USAGE (except WAL) #------------------------------------------------------------------------------ # - Memory - shared_buffers = 128MB # min 128kB # (change requires restart) #huge_pages = try # on, off, or try # (change requires restart) #temp_buffers = 8MB # min 800kB #max_prepared_transactions = 0 # zero disables the feature # (change requires restart) # Caution: it is not advisable to set max_prepared_transactions nonzero unless # you actively intend to use prepared transactions. #work_mem = 4MB # min 64kB #maintenance_work_mem = 64MB # min 1MB #autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem #max_stack_depth = 2MB # min 100kB #shared_memory_type = mmap # the default is the first option # supported by the operating system: # mmap # sysv # windows # (change requires restart) dynamic_shared_memory_type = posix # the default is the first option # supported by the operating system: # posix # sysv # windows # mmap # (change requires restart) # - Disk - #temp_file_limit = -1 # limits per-process temp file space # in kB, or -1 for no limit # - Kernel Resources - #max_files_per_process = 1000 # min 25 # (change requires restart) # - Cost-Based Vacuum Delay - #vacuum_cost_delay = 0 # 0-100 milliseconds (0 disables) #vacuum_cost_page_hit = 1 # 0-10000 credits #vacuum_cost_page_miss = 10 # 0-10000 credits #vacuum_cost_page_dirty = 20 # 0-10000 credits #vacuum_cost_limit = 200 # 1-10000 credits # - Background Writer - #bgwriter_delay = 200ms # 10-10000ms between rounds #bgwriter_lru_maxpages = 100 # max buffers written/round, 0 disables #bgwriter_lru_multiplier = 2.0 # 0-10.0 multiplier on buffers scanned/round #bgwriter_flush_after = 512kB # measured in pages, 0 disables # - Asynchronous Behavior - #effective_io_concurrency = 1 # 1-1000; 0 disables prefetching #max_worker_processes = 8 # (change requires restart) #max_parallel_maintenance_workers = 2 # taken from max_parallel_workers #max_parallel_workers_per_gather = 2 # taken from max_parallel_workers #parallel_leader_participation = on #max_parallel_workers = 8 # maximum number of max_worker_processes that # can be used in parallel operations #old_snapshot_threshold = -1 # 1min-60d; -1 disables; 0 is immediate # (change requires restart) #backend_flush_after = 0 # measured in pages, 0 disables #------------------------------------------------------------------------------ # WRITE-AHEAD LOG #------------------------------------------------------------------------------ # - Settings - #wal_level = replica # minimal, replica, or logical # (change requires restart) #fsync = on # flush data to disk for crash safety # (turning this off can cause # unrecoverable data corruption) #synchronous_commit = on # synchronization level; # off, local, remote_write, remote_apply, or on #wal_sync_method = fsync # the default is the first option # supported by the operating system: # open_datasync # fdatasync (default on Linux) # fsync # fsync_writethrough # open_sync #full_page_writes = on # recover from partial page writes #wal_compression = off # enable compression of full-page writes #wal_log_hints = off # also do full page writes of non-critical updates # (change requires restart) #wal_init_zero = on # zero-fill new WAL files #wal_recycle = on # recycle WAL files #wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers # (change requires restart) #wal_writer_delay = 200ms # 1-10000 milliseconds #wal_writer_flush_after = 1MB # measured in pages, 0 disables #commit_delay = 0 # range 0-100000, in microseconds #commit_siblings = 5 # range 1-1000 # - Checkpoints - #checkpoint_timeout = 5min # range 30s-1d max_wal_size = 1GB min_wal_size = 80MB #checkpoint_completion_target = 0.5 # checkpoint target duration, 0.0 - 1.0 #checkpoint_flush_after = 256kB # measured in pages, 0 disables #checkpoint_warning = 30s # 0 disables # - Archiving - #archive_mode = off # enables archiving; off, on, or always # (change requires restart) #archive_command = '' # command to use to archive a logfile segment # placeholders: %p = path of file to archive # %f = file name only # e.g. 'test ! -f /mnt/server/archivedir/%f && cp %p /mnt/server/archivedir/%f' #archive_timeout = 0 # force a logfile segment switch after this # number of seconds; 0 disables # - Archive Recovery - # These are only used in recovery mode. #restore_command = '' # command to use to restore an archived logfile segment # placeholders: %p = path of file to restore # %f = file name only # e.g. 'cp /mnt/server/archivedir/%f %p' # (change requires restart) #archive_cleanup_command = '' # command to execute at every restartpoint #recovery_end_command = '' # command to execute at completion of recovery # - Recovery Target - # Set these only when performing a targeted recovery. #recovery_target = '' # 'immediate' to end recovery as soon as a # consistent state is reached # (change requires restart) #recovery_target_name = '' # the named restore point to which recovery will proceed # (change requires restart) #recovery_target_time = '' # the time stamp up to which recovery will proceed # (change requires restart) #recovery_target_xid = '' # the transaction ID up to which recovery will proceed # (change requires restart) #recovery_target_lsn = '' # the WAL LSN up to which recovery will proceed # (change requires restart) #recovery_target_inclusive = on # Specifies whether to stop: # just after the specified recovery target (on) # just before the recovery target (off) # (change requires restart) #recovery_target_timeline = 'latest' # 'current', 'latest', or timeline ID # (change requires restart) #recovery_target_action = 'pause' # 'pause', 'promote', 'shutdown' # (change requires restart) #------------------------------------------------------------------------------ # REPLICATION #------------------------------------------------------------------------------ # - Sending Servers - # Set these on the master and on any standby that will send replication data. #max_wal_senders = 10 # max number of walsender processes # (change requires restart) #wal_keep_segments = 0 # in logfile segments; 0 disables #wal_sender_timeout = 60s # in milliseconds; 0 disables #max_replication_slots = 10 # max number of replication slots # (change requires restart) #track_commit_timestamp = off # collect timestamp of transaction commit # (change requires restart) # - Master Server - # These settings are ignored on a standby server. #synchronous_standby_names = '' # standby servers that provide sync rep # method to choose sync standbys, number of sync standbys, # and comma-separated list of application_name # from standby(s); '*' = all #vacuum_defer_cleanup_age = 0 # number of xacts by which cleanup is delayed # - Standby Servers - # These settings are ignored on a master server. #primary_conninfo = '' # connection string to sending server # (change requires restart) #primary_slot_name = '' # replication slot on sending server # (change requires restart) #promote_trigger_file = '' # file name whose presence ends recovery #hot_standby = on # "off" disallows queries during recovery # (change requires restart) #max_standby_archive_delay = 30s # max delay before canceling queries # when reading WAL from archive; # -1 allows indefinite delay #max_standby_streaming_delay = 30s # max delay before canceling queries # when reading streaming WAL; # -1 allows indefinite delay #wal_receiver_status_interval = 10s # send replies at least this often # 0 disables #hot_standby_feedback = off # send info from standby to prevent # query conflicts #wal_receiver_timeout = 60s # time that receiver waits for # communication from master # in milliseconds; 0 disables #wal_retrieve_retry_interval = 5s # time to wait before retrying to # retrieve WAL after a failed attempt #recovery_min_apply_delay = 0 # minimum delay for applying changes during recovery # - Subscribers - # These settings are ignored on a publisher. #max_logical_replication_workers = 4 # taken from max_worker_processes # (change requires restart) #max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers #------------------------------------------------------------------------------ # QUERY TUNING #------------------------------------------------------------------------------ # - Planner Method Configuration - #enable_bitmapscan = on #enable_hashagg = on #enable_hashjoin = on #enable_indexscan = on #enable_indexonlyscan = on #enable_material = on #enable_mergejoin = on #enable_nestloop = on #enable_parallel_append = on #enable_seqscan = on #enable_sort = on #enable_tidscan = on #enable_partitionwise_join = off #enable_partitionwise_aggregate = off #enable_parallel_hash = on #enable_partition_pruning = on # - Planner Cost Constants - #seq_page_cost = 1.0 # measured on an arbitrary scale #random_page_cost = 4.0 # same scale as above #cpu_tuple_cost = 0.01 # same scale as above #cpu_index_tuple_cost = 0.005 # same scale as above #cpu_operator_cost = 0.0025 # same scale as above #parallel_tuple_cost = 0.1 # same scale as above #parallel_setup_cost = 1000.0 # same scale as above #jit_above_cost = 100000 # perform JIT compilation if available # and query more expensive than this; # -1 disables #jit_inline_above_cost = 500000 # inline small functions if query is # more expensive than this; -1 disables #jit_optimize_above_cost = 500000 # use expensive JIT optimizations if # query is more expensive than this; # -1 disables #min_parallel_table_scan_size = 8MB #min_parallel_index_scan_size = 512kB #effective_cache_size = 4GB # - Genetic Query Optimizer - #geqo = on #geqo_threshold = 12 #geqo_effort = 5 # range 1-10 #geqo_pool_size = 0 # selects default based on effort #geqo_generations = 0 # selects default based on effort #geqo_selection_bias = 2.0 # range 1.5-2.0 #geqo_seed = 0.0 # range 0.0-1.0 # - Other Planner Options - #default_statistics_target = 100 # range 1-10000 #constraint_exclusion = partition # on, off, or partition #cursor_tuple_fraction = 0.1 # range 0.0-1.0 #from_collapse_limit = 8 #join_collapse_limit = 8 # 1 disables collapsing of explicit # JOIN clauses #force_parallel_mode = off #jit = on # allow JIT compilation #plan_cache_mode = auto # auto, force_generic_plan or # force_custom_plan #------------------------------------------------------------------------------ # REPORTING AND LOGGING #------------------------------------------------------------------------------ # - Where to Log - #log_destination = 'stderr' # Valid values are combinations of # stderr, csvlog, syslog, and eventlog, # depending on platform. csvlog # requires logging_collector to be on. # This is used when logging to stderr: #logging_collector = off # Enable capturing of stderr and csvlog # into log files. Required to be on for # csvlogs. # (change requires restart) # These are only used if logging_collector is on: #log_directory = 'log' # directory where log files are written, # can be absolute or relative to PGDATA #log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # log file name pattern, # can include strftime() escapes #log_file_mode = 0600 # creation mode for log files, # begin with 0 to use octal notation #log_truncate_on_rotation = off # If on, an existing log file with the # same name as the new log file will be # truncated rather than appended to. # But such truncation only occurs on # time-driven rotation, not on restarts # or size-driven rotation. Default is # off, meaning append to existing files # in all cases. #log_rotation_age = 1d # Automatic rotation of logfiles will # happen after that time. 0 disables. #log_rotation_size = 10MB # Automatic rotation of logfiles will # happen after that much log output. # 0 disables. # These are relevant when logging to syslog: #syslog_facility = 'LOCAL0' #syslog_ident = 'postgres' #syslog_sequence_numbers = on #syslog_split_messages = on # This is only relevant when logging to eventlog (win32): # (change requires restart) #event_source = 'PostgreSQL' # - When to Log - #log_min_messages = warning # values in order of decreasing detail: # debug5 # debug4 # debug3 # debug2 # debug1 # info # notice # warning # error # log # fatal # panic #log_min_error_statement = error # values in order of decreasing detail: # debug5 # debug4 # debug3 # debug2 # debug1 # info # notice # warning # error # log # fatal # panic (effectively off) #log_min_duration_statement = -1 # -1 is disabled, 0 logs all statements # and their durations, > 0 logs only # statements running at least this number # of milliseconds #log_transaction_sample_rate = 0.0 # Fraction of transactions whose statements # are logged regardless of their duration. 1.0 logs all # statements from all transactions, 0.0 never logs. # - What to Log - #debug_print_parse = off #debug_print_rewritten = off #debug_print_plan = off #debug_pretty_print = on #log_checkpoints = off #log_connections = off #log_disconnections = off #log_duration = off #log_error_verbosity = default # terse, default, or verbose messages #log_hostname = off log_line_prefix = '%m [%p] %q%u@%d ' # special values: # %a = application name # %u = username # %d = database name # %r = remote host and port # %h = remote host # %p = process ID # %t = timestamp without milliseconds # %m = timestamp with milliseconds # %n = timestamp with milliseconds (as a Unix epoch) # %i = command tag # %e = SQL state # %c = session ID # %l = session line number # %s = session start timestamp # %v = virtual transaction ID # %x = transaction ID (0 if none) # %q = stop here in non-session # processes # %% = '%' # e.g. '<%u%%%d> ' #log_lock_waits = off # log lock waits >= deadlock_timeout #log_statement = 'none' # none, ddl, mod, all #log_replication_commands = off #log_temp_files = -1 # log temporary files equal or larger # than the specified size in kilobytes; # -1 disables, 0 logs all temp files log_timezone = 'America/New_York' #------------------------------------------------------------------------------ # PROCESS TITLE #------------------------------------------------------------------------------ cluster_name = '12/main' # added to process titles if nonempty # (change requires restart) #update_process_title = on #------------------------------------------------------------------------------ # STATISTICS #------------------------------------------------------------------------------ # - Query and Index Statistics Collector - #track_activities = on #track_counts = on #track_io_timing = off #track_functions = none # none, pl, all #track_activity_query_size = 1024 # (change requires restart) stats_temp_directory = '/var/run/postgresql/12-main.pg_stat_tmp' # - Monitoring - #log_parser_stats = off #log_planner_stats = off #log_executor_stats = off #log_statement_stats = off #------------------------------------------------------------------------------ # AUTOVACUUM #------------------------------------------------------------------------------ #autovacuum = on # Enable autovacuum subprocess? 'on' # requires track_counts to also be on. #log_autovacuum_min_duration = -1 # -1 disables, 0 logs all actions and # their durations, > 0 logs only # actions running at least this number # of milliseconds. #autovacuum_max_workers = 3 # max number of autovacuum subprocesses # (change requires restart) #autovacuum_naptime = 1min # time between autovacuum runs #autovacuum_vacuum_threshold = 50 # min number of row updates before # vacuum #autovacuum_analyze_threshold = 50 # min number of row updates before # analyze #autovacuum_vacuum_scale_factor = 0.2 # fraction of table size before vacuum #autovacuum_analyze_scale_factor = 0.1 # fraction of table size before analyze #autovacuum_freeze_max_age = 200000000 # maximum XID age before forced vacuum # (change requires restart) #autovacuum_multixact_freeze_max_age = 400000000 # maximum multixact age # before forced vacuum # (change requires restart) #autovacuum_vacuum_cost_delay = 2ms # default vacuum cost delay for # autovacuum, in milliseconds; # -1 means use vacuum_cost_delay #autovacuum_vacuum_cost_limit = -1 # default vacuum cost limit for # autovacuum, -1 means use # vacuum_cost_limit #------------------------------------------------------------------------------ # CLIENT CONNECTION DEFAULTS #------------------------------------------------------------------------------ # - Statement Behavior - #client_min_messages = notice # values in order of decreasing detail: # debug5 # debug4 # debug3 # debug2 # debug1 # log # notice # warning # error #search_path = '"$user", public' # schema names #row_security = on #default_tablespace = '' # a tablespace name, '' uses the default #temp_tablespaces = '' # a list of tablespace names, '' uses # only default tablespace #default_table_access_method = 'heap' #check_function_bodies = on #default_transaction_isolation = 'read committed' #default_transaction_read_only = off #default_transaction_deferrable = off #session_replication_role = 'origin' #statement_timeout = 0 # in milliseconds, 0 is disabled #lock_timeout = 0 # in milliseconds, 0 is disabled #idle_in_transaction_session_timeout = 0 # in milliseconds, 0 is disabled #vacuum_freeze_min_age = 50000000 #vacuum_freeze_table_age = 150000000 #vacuum_multixact_freeze_min_age = 5000000 #vacuum_multixact_freeze_table_age = 150000000 #vacuum_cleanup_index_scale_factor = 0.1 # fraction of total number of tuples # before index cleanup, 0 always performs # index cleanup #bytea_output = 'hex' # hex, escape #xmlbinary = 'base64' #xmloption = 'content' #gin_fuzzy_search_limit = 0 #gin_pending_list_limit = 4MB # - Locale and Formatting - datestyle = 'iso, mdy' #intervalstyle = 'postgres' timezone = 'America/New_York' #timezone_abbreviations = 'Default' # Select the set of available time zone # abbreviations. Currently, there are # Default # Australia (historical usage) # India # You can create your own file in # share/timezonesets/. #extra_float_digits = 1 # min -15, max 3; any value >0 actually # selects precise output mode #client_encoding = sql_ascii # actually, defaults to database # encoding # These settings are initialized by initdb, but they can be changed. lc_messages = 'en_US.UTF-8' # locale for system error message # strings lc_monetary = 'en_US.UTF-8' # locale for monetary formatting lc_numeric = 'en_US.UTF-8' # locale for number formatting lc_time = 'en_US.UTF-8' # locale for time formatting # default configuration for text search default_text_search_config = 'pg_catalog.english' # - Shared Library Preloading - #shared_preload_libraries = '' # (change requires restart) #local_preload_libraries = '' #session_preload_libraries = '' #jit_provider = 'llvmjit' # JIT library to use # - Other Defaults - #dynamic_library_path = '$libdir' #------------------------------------------------------------------------------ # LOCK MANAGEMENT #------------------------------------------------------------------------------ #deadlock_timeout = 1s #max_locks_per_transaction = 64 # min 10 # (change requires restart) #max_pred_locks_per_transaction = 64 # min 10 # (change requires restart) #max_pred_locks_per_relation = -2 # negative values mean # (max_pred_locks_per_transaction # / -max_pred_locks_per_relation) - 1 #max_pred_locks_per_page = 2 # min 0 #------------------------------------------------------------------------------ # VERSION AND PLATFORM COMPATIBILITY #------------------------------------------------------------------------------ # - Previous PostgreSQL Versions - #array_nulls = on #backslash_quote = safe_encoding # on, off, or safe_encoding #escape_string_warning = on #lo_compat_privileges = off #operator_precedence_warning = off #quote_all_identifiers = off #standard_conforming_strings = on #synchronize_seqscans = on # - Other Platforms and Clients - #transform_null_equals = off #------------------------------------------------------------------------------ # ERROR HANDLING #------------------------------------------------------------------------------ #exit_on_error = off # terminate session on any error? #restart_after_crash = on # reinitialize after backend crash? #data_sync_retry = off # retry or panic on failure to fsync # data? # (change requires restart) #------------------------------------------------------------------------------ # CONFIG FILE INCLUDES #------------------------------------------------------------------------------ # These options allow settings to be loaded from files other than the # default postgresql.conf. Note that these are directives, not variable # assignments, so they can usefully be given more than once. include_dir = 'conf.d' # include files ending in '.conf' from # a directory, e.g., 'conf.d' #include_if_exists = '...' # include file only if it exists #include = '...' # include file #------------------------------------------------------------------------------ # CUSTOMIZED OPTIONS #------------------------------------------------------------------------------</pre> <p> All of my changes, highlighted in yellow, are at the bottom of <code>pg_hba.conf</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>/etc/postgresql/12/main/pg_hba.conf</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc977c7d65eed'><button class='copyBtn' data-clipboard-target='#idc977c7d65eed' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># PostgreSQL Client Authentication Configuration File # =================================================== # # Refer to the "Client Authentication" section in the PostgreSQL # documentation for a complete description of this file. A short # synopsis follows. # # This file controls: which hosts are allowed to connect, how clients # are authenticated, which PostgreSQL usernames they can use, which # databases they can access. Records take one of these forms: # # local DATABASE USER METHOD [OPTIONS] # host DATABASE USER ADDRESS METHOD [OPTIONS] # hostssl DATABASE USER ADDRESS METHOD [OPTIONS] # hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] # hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS] # hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS] # # (The uppercase items must be replaced by actual values.) # # The first field is the connection type: "local" is a Unix-domain # socket, "host" is either a plain or SSL-encrypted TCP/IP socket, # "hostssl" is an SSL-encrypted TCP/IP socket, and "hostnossl" is a # non-SSL TCP/IP socket. Similarly, "hostgssenc" uses a # GSSAPI-encrypted TCP/IP socket, while "hostnogssenc" uses a # non-GSSAPI socket. # # DATABASE can be "all", "sameuser", "samerole", "replication", a # database name, or a comma-separated list thereof. The "all" # keyword does not match "replication". Access to replication # must be enabled in a separate record (see example below). # # USER can be "all", a username, a group name prefixed with "+", or a # comma-separated list thereof. In both the DATABASE and USER fields # you can also write a file name prefixed with "@" to include names # from a separate file. # # ADDRESS specifies the set of hosts the record matches. It can be a # host name, or it is made up of an IP address and a CIDR mask that is # an integer (between 0 and 32 (IPv4) or 128 (IPv6) inclusive) that # specifies the number of significant bits in the mask. A host name # that starts with a dot (.) matches a suffix of the actual host name. # Alternatively, you can write an IP address and netmask in separate # columns to specify the set of hosts. Instead of a CIDR-address, you # can write "samehost" to match any of the server's own IP addresses, # or "samenet" to match any address in any subnet that the server is # directly connected to. # # METHOD can be "trust", "reject", "md5", "password", "scram-sha-256", # "gss", "sspi", "ident", "peer", "pam", "ldap", "radius" or "cert". # Note that "password" sends passwords in clear text; "md5" or # "scram-sha-256" are preferred since they send encrypted passwords. # # OPTIONS are a set of options for the authentication in the format # NAME=VALUE. The available options depend on the different # authentication methods -- refer to the "Client Authentication" # section in the documentation for a list of which options are # available for which authentication methods. # # Database and usernames containing spaces, commas, quotes and other # special characters must be quoted. Quoting one of the keywords # "all", "sameuser", "samerole" or "replication" makes the name lose # its special character, and just match a database or username with # that name. # # This file is read on server startup and when the server receives a # SIGHUP signal. If you edit the file on a running system, you have to # SIGHUP the server for the changes to take effect, run "pg_ctl reload", # or execute "SELECT pg_reload_conf()". # # Put your actual configuration here # ---------------------------------- # # If you want to allow non-local connections, you need to add more # "host" records. In that case you will also need to make PostgreSQL # listen on a non-local interface via the listen_addresses # configuration parameter, or via the -i or -h command line switches. # DO NOT DISABLE! # If you change this first entry you will need to make sure that the # database superuser can access the database using some other method. # Noninteractive access to all databases is required during automatic # maintenance (custom daily cronjobs, replication, and similar tasks). # # Database administrative login by Unix domain socket <span style="background-color: yellow">local all postgres md5</span> # TYPE DATABASE USER ADDRESS METHOD # "local" is for Unix domain socket connections only <span style="background-color: yellow">local all all peer</span> # IPv4 local connections: <span style="background-color: yellow">host all all 0.0.0.0/0 md5</span> <span style="background-color: yellow">host all all 0.0.0.0/32 md5</span> # IPv6 local connections: host all all ::1/128 md5 # Allow replication connections from localhost, by a user with the # replication privilege. local replication all peer host replication all 127.0.0.1/32 md5 host replication all ::1/128 md5</pre> <p> Now that PostgreSQL was configured, I restarted it: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id89949d7b48cb'><button class='copyBtn' data-clipboard-target='#id89949d7b48cb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl restart postgresql</pre> <h2 id="together">Connecting PostgreSQL to PHP</h2> <p> I created a new database called <code>opencart</code> with this command: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7af95225e9ca'><button class='copyBtn' data-clipboard-target='#id7af95225e9ca' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>psql -U postgres -c "create database opencart;" <span class='unselectable'>CREATE DATABASE Time: 1057.887 ms (00:01.058)</span></pre> <p> I entered PHP interactive mode to verify that PHP could connect properly to the new PostgreSQL database. This command sequence just creates a simple table and deletes it. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0ec257c985c3'><button class='copyBtn' data-clipboard-target='#id0ec257c985c3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>php -a <span class='unselectable'>Interactive mode enabled php > </span>pg_connect("host=localhost dbname=opencart user=postgres password=hithere"); <span class='unselectable'>php > </span>pg_query("create table test(id integer)"); <span class='unselectable'>php > </span>pg_query("drop table test"); <span class='unselectable'>php > </span>exit</pre> <h2 id="opencart_configuration1">Configuring OpenCart</h2> <p> The two configuration files that OpenCart provides are empty. They need to be renamed before OpenCart can be installed. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb2aaed01dc9a'><button class='copyBtn' data-clipboard-target='#idb2aaed01dc9a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo mv /work/ecommerce/opencart/upload/config{-dist,}.php <span class='unselectable'>$ </span>sudo mv /work/ecommerce/opencart/upload/admin/config{-dist,}.php</pre> <p> These files will contain configuration information after the OpenCart <code>admin</code> user configures the system. The files also need to have their owner or group set to the same user that the web server runs as. For <code>nginx</code>, this username and group are both called <code>www-data</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2af93bd47a38'><button class='copyBtn' data-clipboard-target='#id2af93bd47a38' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>find . -name config.php -exec sudo chown www-data:dev {} \;</pre> <p> I dislike the idea of having a web application modify its configuration data while running. This is inherently insecure. However, many PHP programs from the era that OpenCart was originally written operated that way. I have always been acutely uncomfortable with this practice. </p> <p> Equally distasteful to me was the hack that PHP programmers often do in order to support multi-tenant web applications where users who self-administer their sites have limited storage options (20 years ago this was an issue, the rest of the world has moved on): storing logs within the program file structure. This is insecure. OpenCart logs belong in <code>/var/log/opencart</code>. I did not modify the code, instead I rolled my eyes and made the log files in <code>opencart/upload/system/storage/logs/</code> group writable. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4b48746d644d'><button class='copyBtn' data-clipboard-target='#id4b48746d644d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo chmod g+w opencart/upload/system/storage/logs/*.log</pre> <h2 id="install">Installing OpenCart</h2> <h3 id="web-install">Web-Based Installer</h3> <p> The web-based OpenCart installer is fragile and <a href='https://github.com/opencart/opencart/commits/master/upload/install/index.php' target='_blank' rel='nofollow'>not well maintained</a>. It dies near the end of its work when attempting to install using a Postgres database. </p> <p> Clicking on <code><a href='https://http://localhost/upload/install/' target='_blank' rel='nofollow'><code>http://localhost/upload/install/</code></a></code> starts the web-based OpenCart installation process by displaying the GNU license agreement from 2007. The installation fails on page 3. This problem was first reported on <a href='https://github.com/opencart/opencart/issues/7521' target='_blank' rel='nofollow'>July 15, 2019</a> but it was not addressed. </p> <p> Completing the installation only requires that the database be set up. <code>system/helper/db_schema.php</code> contains PHP code for defining the database schema using MySQL, and the SQL to populate the database is found in <code>upload/install/opencart.sql</code>. </p> <p> At this point I gave up and tried the command-line installer. </p> <h3 id="cli-install">Command-Line Installer</h3> <p> OpenCart has a command-line installer which is not mentioned in the online installation documentation. I always prefer to use a command line installer, if possible because any problems encountered are easier to diagnose and fix than with web-based installers. </p> <p> In contrast to the publicly promoted web-based installer, the command-line installer appears to be <a href='https://github.com/opencart/opencart/commits/master/upload/install/cli_install.php' target='_blank' rel='nofollow'>well maintained</a> for and by the current authors, who obviously also operate OpenCart Cloud (more on that in a minute). Once again, we see the inherent conflict of interest in traditional open-source software. </p> <p> Here is a sample command line for installing OpenCart. The script should be run from the <code>opencart/upload/install</code> directory. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id02dbd3fe805f'><button class='copyBtn' data-clipboard-target='#id02dbd3fe805f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>php cli_install.php install \ --db_database opencart \ --db_driver postgre \ --db_hostname localhost \ --db_password postgres_password \ --db_port 5432 \ --db_username postgres \ --email email@example.com \ --http_server http://localhost/opencart/ \ --password admin_password \ --username admin \ | lynx -stdin</pre> <p> I needed to provide proper values for the following options: </p> <dl> <dt><code>--db_database</code></dt> <dd> There is no default value for this option. It makes sense to name the database <code>opencart</code>, but one might have reasons to give it another name. </dd> <dt><code>--db_hostname</code></dt> <dd>The database might not run on the same network node as OpenCart's web server.</dd> <dt><code>--db_password</code></dt> <dd>Friends do not let friends use empty passwords, even on personal machines.</dd> <dt><code>--db_port</code></dt> <dd>I usually use the default PostgreSQL port, 5432.</dd> <dt><code>--db_username</code></dt> <dd>It is more secure to not use the <code>postgres</code> default username.</dd> <dt><code>--email</code></dt> <dd>Email address of the OpenCart administrator.</dd> <dt><code>--http_server</code></dt> <dd>More than just the domain name, this option also specifies the protocol, HTTP port and the path to the OpenCart directory on the web server.</dd> <dt><code>--password</code></dt> <dd>OpenCart admin user password.</dd> </dl> <h4 id="db_prefix">The <span class="code">--db-prefix</span> Option</h4> <p> The above omits the <code>--db_prefix</code> option, whose default value is <code>oc_</code>. This is because the installer uses <code>upload/install/opencart.sql</code>, which is hard-coded to use the default value. </p> <h4 id="cloud">The <span class="code">--cloud</span> Option</h4> <p> The above also omits the <code>--cloud</code> option. This option has no documentation. After looking at the source code, I think this parameter is exclusively for <a href='https://www.opencart.com/index.php?route=cloud/landing' target='_blank' rel='nofollow'>OpenCart Cloud</a> installations. This means that most people could omit the option because its value defaults to 0, which means the installation is not intended for Open Cloud. </p> <p> Why do I think that? <a href='https://github.com/opencart/opencart/blob/master/upload/install/cli_install.php' target='_blank' rel='nofollow'>Looking at the code</a> I see this parameter suppresses database configuration and saving of configuration information. Also, cloud installations require that the admin user password be pre-hashed, which suggests to me that this script can be initiated from another installation script used by Open Cloud. </p> <h3 id="cloud">Using Default Values</h3> <p> If you are installing on a development machine, it is likely to run both the Postgres database and the PHP website, and software is likely to be set up using default values. Assuming that the PostgreSQL username is the default, <code>postgres</code>, and the database is called <code>opencart</code>, you just need to specify the following parameters: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9f8c052881da'><button class='copyBtn' data-clipboard-target='#id9f8c052881da' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>php cli_install.php install \ --db_driver postgre \ --db_database opencart \ --db_password postgres_password \ --db_port 5432 \ --db_username postgres \ --email email@example.com \ --http_server http://localhost/opencart/ \ --password admin_password \ | lynx -stdin</pre> <h3 id="patch">Patching Source Code</h3> <p> I ran <code>cli_install.php</code>, found problems, fixed them, reran <code>cli_install.php</code>, found more problems, fixed them, etc. etc. </p> <p> Clearly no-one has ever run <code>cli_install.php</code> to completion when <code>--cloud</code> option was set to 0. <p> <h4 id="constants">Defining Constants</h4> <p> The people who use <code>cli_install.php</code> provide values for two constants before the program runs. I found some code in <code>upload/index.php</code> that defined them. Using those statements as a guide, I added some lines after line 57 of <code>upload/system/startup.php</code> so these constants were defined: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbfa13da05739'><button class='copyBtn' data-clipboard-target='#idbfa13da05739' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>// mslinn added: define('DIR_EXTENSION', DIR_OPENCART . 'extension/'); define('HTTP_SERVER', 'file:' . $_SERVER['HTTP_HOST'] . rtrim(dirname($_SERVER['SCRIPT_NAME']), '/.\\') . '/'); // end mslinn</pre> <h4 id="db_driver">Checking Postgres Driver</h4> <p> Clearly no-one has ever tried to install using the PostgreSQL driver before. I had to modify line 198 of <code>upload/install/cli_install.php</code> to add a check, highlighted in yellow, for the PostgreSQL driver extension: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idaa002d6baef8'><button class='copyBtn' data-clipboard-target='#idaa002d6baef8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>if (!extension_loaded('mysqli') <span style="background-color: yellow">&& !extension_loaded('pgsql')</span>) { $error .= 'ERROR: MySQLi extension needs to be loaded for OpenCart to work!' . "\n"; }</pre> <h2 id="results">Running the Command-Line Installer</h2> <p> The command-line installer spewed out miles and miles of HTML (mostly the GNU license) and died with an error message. So, I fixed the problem and reran it. It would die somewhere else with a different error. So, I fixed that problem too and reran it again. It would die yet somewhere else with yet another error. </p> <p> The last problem I found before quitting was an error message resulting from the command-line installer attempting to rewrite an HTML header. Really! A command-line installer does not need to present HTML to the user. This command-line installer program is clearly a cheap hack. </p> <h2 id="stop">Evaluation Results</h2> <p> At this point I felt that I now had a good idea of the quality this open-source project: OpenCart is very poorly constructed. </p> <p> The business model for the company that stewards OpenCart is also clear: keep a few programmers of modest ability busy, and charge for their time by the hour. OpenCart is not something I would want to base an e-commerce business on. </p> <p> Since this is supposedly the best open-source shopping cart today, <a href='/django/index.html'>I will next look into</a> building something just for me that provides me with competitive advantage. </p> Propagating Git Template Changes Downstream 2020-11-30T00:00:00-05:00 https://mslinn.github.io/blog/2020/11/30/propagating-git-template-changes <p> I have developed a <a href='https://jekyllrb.com/' target='_blank' rel='nofollow'>Jekyll</a> template that I use as the starting point for most of my websites. Whenever I improve the template I can easily incorporate the changes into all the websites that are based on it. This article describes how I set that up, and the information applies to all templates in general &ndash; Jekyll is not required. Templates do not need to have any special characteristics, beyond being generally useful in some sense. </p> <p> <a href='https://docs.github.com/en/free-pro-team@latest/github/creating-cloning-and-archiving-repositories/creating-a-template-repository' target='_blank' rel='nofollow'><code>GitHub</code> has a template feature</a> but this article does not require <code>GitHub</code> and works with all <code>git</code> hosts. </p> <p> This diagram shows the local and hosted versions of a template repository and a project repository based on the template. </p> <div style=""> <picture> <source srcset="/blog/images/gitTemplates/gitTemplates.webp" type="image/webp"> <source srcset="/blog/images/gitTemplates/gitTemplates.png" type="image/png"> <img src="/blog/images/gitTemplates/gitTemplates.png" title="Local and Hosted versions of a project and its template" class=" liImg2 rounded shadow" alt="Local and Hosted versions of a project and its template" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> Local and Hosted versions of a project and its template </figcaption> </figure> </div> <h2 id="template_clone">Copy from the Template Repository</h2> <p> To make a new local repository called <code>new_project</code> based on the repository called <code>template</code> from GitHub user with ID <code>mslinn</code>, type the following incantation. Please modify this command to suit your project, which might be hosted on AWS CodeCommit, Bitbucket, GitLab, etc. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbabd88282ea0'><button class='copyBtn' data-clipboard-target='#idbabd88282ea0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git clone git@github.com:mslinn/template.git new_project</pre> <p><i>I have no <code>git</code> repository called <code>template</code>, so the above is just for explanation purposes.</i></p> <p> <code>git</code> automatically sets up a remote origin from the local repository pointing to <code>template</code> for <code>git fetch</code> and <code>git push</code> commands, as we can see from the following: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4301cdb625b7'><button class='copyBtn' data-clipboard-target='#id4301cdb625b7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git remote -v <span class='unselectable'>origin git@github.com:mslinn/template.git (fetch) origin git@github.com:mslinn/template.git (push)</span></pre> <h2 id="upstream_downstream">Define Upstream and Downstream Repositories</h2> <p> To separately obtain updates from the <code>new_project</code> repository tracked at <code>origin</code> and updates from the <code>upstream</code> template, we need to define remote URLs for both the <code>origin</code> and <code>template</code> repositories. </p> <h3 id="upstream">Define Upstream Repository</h3> <p> The template will be an upstream remote repository. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide64f6fa3fed5'><button class='copyBtn' data-clipboard-target='#ide64f6fa3fed5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git remote rename origin upstream</pre> <p> To ensure read-only status we should disable pushing from <code>new_project</code> to the <code>upstream</code> repository, like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddd28363e3295'><button class='copyBtn' data-clipboard-target='#iddd28363e3295' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git remote set-url --push upstream no_push</pre> <h3 id="create_downstream">Create New Downstream Repository</h3> <p> We need to create a new hosted repository for <code>new_project</code>. All the git hosting sites provide a way to do this using a web browser. However, I much prefer to use command line interfaces (CLIs). </p> <h4 id="gitlab_create">AWS CodeCommit</h4> <p> <a href='https://docs.aws.amazon.com/codecommit/latest/userguide/how-to-create-repository.html#how-to-create-repository-cli' target='_blank' rel='nofollow'>The AWS CLI</a> incantation looks something like this: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1a9e35f062e6'><button class='copyBtn' data-clipboard-target='#id1a9e35f062e6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws codecommit create-repository --repository-name new_project \ --repository-description "My downstream project"</pre> </p> <h4 id="bitbucket_create">Bitbucket</h4> <p> <a href='https://marketplace.atlassian.com/apps/1211193/bitbucket-command-line-interface-cli?hosting=server&tab=overview' target='_blank' rel='nofollow'>The Bitbucket CLI</a> incantation looks something like this: </p> <p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id112c25aa1fb7'><button class='copyBtn' data-clipboard-target='#id112c25aa1fb7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>acli bobswift9 --action createRepository \ --project new_project --repository new_project --name new_project</pre> </p> <h4 id="github_create">GitHub</h4> <p> The shiny new official <a href='https://cli.github.com' target='_blank' rel='nofollow'>GitHub CLI</a> unfortunately cannot do something that the tried-and-true <a href='https://hub.github.com/' target='_blank' rel='nofollow'>Hub <code>gh</code></a> does: decouple the creation of a local git repo from the creation of the remote repo on GitHub. That means we must use Hub. like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf61c9a854ca3'><button class='copyBtn' data-clipboard-target='#idf61c9a854ca3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cd new_project <span class='unselectable'>$ </span>hub create new_project <span class='unselectable'>Existing repository detected Updating origin Warning: No xauth data; using fake authentication data for X11 forwarding. X11 forwarding request failed on channel 0 https://github.com/mslinn/new_project</span></pre> <h4 id="gitlab_create">GitLab</h4> <p> All the GitLab CLIs I found had been abandoned. </p> <h3 id="git_all">All Git Host Sites</h3> <p> We can verify that the remotes for the downstream git project on your computer are now set up appropriately. The output shown shows that I used the GitHub CLI: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc4ed012068c4'><button class='copyBtn' data-clipboard-target='#idc4ed012068c4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git remote -v <span class='unselectable'>origin git@github.com:mslinn/new_project.git (fetch) origin git@github.com:mslinn/new_project.git (push) upstream git@github.com:mslinn/template.git (fetch) upstream no_push (push)</span></pre> <h2 id="template_update">Updating From the Downstream Repository</h2> <p> We can pull changes from the downstream <code>new_project</code> <code>origin</code> repository into the local copy of the downstream project like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7836ec79eefe'><button class='copyBtn' data-clipboard-target='#id7836ec79eefe' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git pull <span class='unselectable'>From github.com:mslinn/template * branch master -> FETCH_HEAD Already up-to-date.</span></pre> <p> We could have typed this more verbose version, which accomplishes the same thing: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb37602424d44'><button class='copyBtn' data-clipboard-target='#idb37602424d44' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git pull origin <span class='unselectable'>Everything up-to-date </span></pre> <h2 id="template_update">Updating From the Upstream Template</h2> <p> We can pull changes from the upstream <code>template</code> repository into the local copy of the downstream <code>new_project</code> repository like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddc5c70983de6'><button class='copyBtn' data-clipboard-target='#iddc5c70983de6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git pull upstream <span class='unselectable'>From github.com:mslinn/template * branch master -> FETCH_HEAD Already up-to-date.</span></pre> <h2 id="pushing">Pushing Changes</h2> <p> We can push changes from the local copy of the <code>new_project</code> repository to the hosted copy. From the top-level <code>new_project</code> directory, type: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd313f7bec439'><button class='copyBtn' data-clipboard-target='#idd313f7bec439' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git add -A <span class='unselectable'>$ </span>git commit -m "Commit message goes here" <span class='unselectable'>$ </span>git push origin <span class='unselectable'>Everything up-to-date </span></pre> <p> However, if we try to push changes from <code>new_project</code> to <code>upstream</code>, we get the following error message. This is good because it means we cannot accidentally modify the upstream template when working on a project derived from the template. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3835d15ba6f3'><button class='copyBtn' data-clipboard-target='#id3835d15ba6f3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git push upstream <span class='unselectable'>fatal: 'no_push' does not appear to be a git repository fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.</span></pre> <h2 id="update_upstream">Updating the Upstream Template From a Downstream Repo</h2> <p> It is convenient to use two-way merge utilities to propagate selected changes in a downstream repository with the upstream repository. My favorite such utilities are: </p> <ul> <li><a href='https://meldmerge.org' target='_blank' rel='nofollow'>Meld</a> &ndash; Available for Linux and Windows. Also, available for Mac without support. </li> <li><a href='https://www.jetbrains.com/help/idea/command-line-merge-tool.html' target='_blank' rel='nofollow'>IntelliJ</a> &ndash; Available as a command-line utility for Windows, Mac, and Linux. <a href='https://www.jetbrains.com/help/idea/settings-tools-diff-and-merge.html' target='_blank' rel='nofollow'>Also integrated with the IDEA GUI.</a> </li> </ul> <p> Once you have propagated selected changes from the downstream project to the upstream template repo, commit the changes to the upstream repo. From the top-level template project directory, type: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idede3bdf14f95'><button class='copyBtn' data-clipboard-target='#idede3bdf14f95' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git add -A <span class='unselectable'>$ </span>git commit -m "Commit message goes here" <span class='unselectable'>$ </span>git push origin <span class='unselectable'># the word 'origin' is optional here</span> <span class='unselectable'>Everything up-to-date </span></pre> <!-- <p> The following makes it possible to push changes to <code>new_project</code> without affecting <code>template</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4406079f08e2'><button class='copyBtn' data-clipboard-target='#id4406079f08e2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git remote add origin git@github.com:mslinn/new_project.git <span class='unselectable'>$ </span>git push -u origin master</pre> --> <h2 id="2nd">Setting Up Another Computer</h2> <p> You might need to work on your project on another computer, and update from <code>upstream</code> the same way you set up the first computer. The process to do this is much the same as what was just described, but with fewer steps. It looks something like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1e50d18bcc9b'><button class='copyBtn' data-clipboard-target='#id1e50d18bcc9b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>git clone git@github.com:mslinn/new_project.git <span class='unselectable'>Cloning into 'new_project'... Warning: No xauth data; using fake authentication data for X11 forwarding. X11 forwarding request failed on channel 0 remote: Enumerating objects: xxxx, done. remote: Total xxxx (delta 0), reused 0 (delta 0), pack-reused 1139 Receiving objects: 100% (xxxx/xxxx), xx.xx MiB | x.x MiB/s, done. Resolving deltas: 100% (xxx/xxx), done.</span> <span class='unselectable'>$ </span>cd new_project <span class='unselectable'>$ </span>git remote add upstream https://github.com/mslinn/template <span class='unselectable'>$ </span>git remote set-url --push upstream no_push</pre> <h2 id="script">GitHub Script</h2> <p> The following bash script is an example of how to automate the process of creating a project based on a template, using GitHub as the repository service. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,cloneTemplate' download='cloneTemplate' title='Click on the file name to download the file'>cloneTemplate</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idf2f08197d701">#!/bin/bash # See https://www.mslinn.com/blog/2020/11/30/propagating-git-template-changes.html function help &#123; if [ "$1" ]; then printf "\nError: $1\n\n"; fi echo "Usage: $0 templateUrl newProjectName" exit 1 &#125; if [ -z "$(which git)" ]; then echo "Please install git and rerun this script" exit 2 fi if [ -z "$(which hub)" ]; then echo "Please install hub and rerun this script" exit 3 fi if [ -z "$1" ]; then help "No git project was specified as a template."; fi if [ -z "$2" ]; then help "Please provide the name of the new project based on the template"; fi git clone "$1" "$2" cd "$2" git remote rename origin upstream git remote set-url --push upstream no_push # Add the -p option to create a private repository hub create "$2" git branch -M master git push -u origin master </pre> <h2 id="acks">Acknowledgements</h2> <p> This posting was inspired by <a href='https://medium.com/@smrgrace/having-a-git-repo-that-is-a-template-for-new-projects-148079b7f178' target='_blank' rel='nofollow'>this article</a>. </p> Installing a New SSH Key on AWS EC2 with User Data 2020-10-27T00:00:00-04:00 https://mslinn.github.io/blog/2020/10/27/installing-a-new-ssh-key-on-awc-ec2-with-user-data <editor-fold Intro> <p> For some reason the <code>ssh</code> certificates that AWS generated for me 3 years ago are no longer recognized by Ubuntu 20.10. This article shows how to create new certificates and push them to an AWS server that was just upgraded to Ubuntu 20.01, and now cannot be logged into. I decided to use OpenSSH to generate the new keypairs instead of AWS to generate the keypairs because the current problem stems from <a href='https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html' target='_blank' rel='nofollow'>AWS-generated keys</a> gradually becoming incompatible with OpenSSH servers. </p> <p> This article describes the following: </p> <ol> <li>Tracking down the problem</li> <li>Create a new <code>ssh</code> certificate keypair.</li> <li>Even though the system cannot accept logins, the new <code>ssh</code> public key must be copied to the <code>ubuntu</code> user&rsquo;s <code>~/.ssh</code> directory on the problem server. This is done by defining a user data script on the server instance prior to booting it.</li> <li>Log into the problem server using the new certificates.</li> <li>Complete the upgrade to XUbuntu 20.10.</li> <li>Remove the user data script from the problem server instance.</li> </ol> </editor-fold Intro> <editor-fold Discovery> <h2 class="numbered" id="discovery">Discovery</h2> <p> I was able to log into another of my machines (<code>gojira</code>), so first I wanted to know if the problem machines (<code>va</code> and <code>france</code>) had OpenSSH configured differently. I used the <a href='https://linux.die.net/man/1/comm' target='_blank' rel='nofollow'>comm</a> Linux utility to perform the comparison. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idec2cc5de29ff'><button class='copyBtn' data-clipboard-target='#idec2cc5de29ff' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>for x in cipher mac key kex; do comm -3 <(ssh -Q $x france|sort) <(ssh -Q $x gojira|sort) done <span class='unselectable'>$ </span>for x in cipher mac key kex; do comm -3 <(ssh -Q $x france|sort) <(ssh -Q $x gojira|sort) done</pre> <p> So the problem was not OpenSSH configuration per se. Next I wondered if the ssh connections to the problem machines were different somehow from the ssh connection to the working machine. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9ab06d77ae3f'><button class='copyBtn' data-clipboard-target='#id9ab06d77ae3f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>comm -3 <(ssh -Gva|sort) <(ssh -G gojira|sort) <span class='unselectable'>hostname gojira hostname va identityfile ~/.ssh/id_rsa identityfile ~/.ssh/sslTest user mslinn user ubuntu </span></pre> <p>So the only differences were the hostnames and the keys offered.</p> <p> One of the problem machines, <code>france</code>, resided on <a href='https://scaleway.com' target='_blank' rel='nofollow'><code>scaleway</code></a>. I used the most recently available <a href='https://medium.com/@moul/scaleway-bootscript-simple-kernel-management-for-your-c1-server-de0c301de721' target='_blank' rel='nofollow'>bootscript</a> to launch the server and examined <code>/var/log/auth.log</code>. I found this: <code>sshd[27025]: Unable to negotiate with 205.185.123.173 port 40555: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1 [preauth]</code> </p> <p> This error message is produced by OpenSSH 7.0+. <a href='https://www.openssh.com/txt/release-7.0' target='_blank' rel='nofollow'>The release notes say</a> &ldquo;Support for the 1024-bit diffie-hellman-group1-sha1 key exchange is disabled by default at run-time. It may be re-enabled using the instructions at <code><a href='https://http://www.openssh.com/legacy.html' target='_blank' rel='nofollow'><code>http://www.openssh.com/legacy.html</code></a></code>&rdquo; </p> <p> So it seems that the version of OpenSSH installed with Ubuntu 20.10 rejects my old keys. The release notes for the new version of OpenSSH also indicate that OpenSSH 7.1 will be even stricter: </p> <ul> <li> &ldquo;This focus of this release is primarily to deprecate weak, legacy and/or unsafe cryptography&rdquo; </li> <li> &ldquo;Refusing all RSA keys smaller than 1024 bits (the current minimum is 768 bits)&rdquo; </li> <li> &ldquo;Several ciphers will be disabled by default: <code>blowfish-cbc</code>, <code>cast128-cbc</code>, all <code>arcfour</code> variants and the <code>rijndael-cbc</code> aliases for AES.&rdquo; </li> <li> &ldquo;MD5-based <code>HMAC</code> algorithms will be disabled by default.&rdquo; </li> </ul> <p> Clearly I need to generate better <code>ssh</code> keys. </p> <p> The version of OpenSSH installed by Ubuntu 20.10 is 8.3: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id37404eeb8dc5'><button class='copyBtn' data-clipboard-target='#id37404eeb8dc5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>$ sshd -V <span class='unselectable'>unknown option -- V OpenSSH_<span class="bg_yellow">8.3p1</span> Ubuntu-1, OpenSSL 1.1.1f 31 Mar 2020 usage: sshd [-46DdeiqTt] [-C connection_spec] [-c host_cert_file] [-E log_file] [-f config_file] [-g login_grace_time] [-h host_key_file] [-o option] [-p port] [-u len]</span> $ ssh -V <span class='unselectable'>OpenSSH_<span class="bg_yellow">8.3p1</span> Ubuntu-1, OpenSSL 1.1.1f 31 Mar 2020 </span></pre> </editor-fold Discovery> <editor-fold Setup> <h2 class="numbered" id="setup">Setup</h2> <h3 class="numbered" id="setupAwsCli">AWS CLI</h3> <p> I prefer to use the AWS CLI instead of the <a href='https://console.aws.amazon.com' target='_blank' rel='nofollow'>web console</a>. Installation instructions are <a href='https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html' target='_blank' rel='nofollow'>here</a>. This article uses the AWS CLI exclusively in favor of the AWS web console. </p> <p> The code in the remainder of this blog post references an AWS EC2 instance, whose id I stored in <code>AWS_PROBLEM_INSTANCE_ID</code>. The <a href='/blog/2020/10/25/rescuing-catastrophic-upgrades-to-ubuntu-20_10.html#info'>previous blog post</a> shows how I did that. </p> <h3 class="numbered" id="setupJq"><span class="code">jq</span></h3> <p> I also use <a href='https://stedolan.github.io/jq/' target='_blank' rel='nofollow'>jq</a> for parsing JSON in the bash shell. Install it on Debian-style Linux distros such as Ubuntu like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc3ad38a24e4f'><button class='copyBtn' data-clipboard-target='#idc3ad38a24e4f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install jq</pre> </editor-fold Setup> <editor-fold nameKeys> <h2 class="numbered" id="nameKeys">Name the New Keypair</h2> <p> I wanted to make new <code>ecdsa</code> keys because <a href='https://goteleport.com/blog/comparing-ssh-keys/' target='_blank' rel='nofollow'>this algorithm is the currently accepted best practice</a> for commercial security concerns. <code>ecdsa</code> stands for <a href='https://en.bitcoin.it/wiki/Elliptic_Curve_Digital_Signature_Algorithm' target='_blank' rel='nofollow'>Elliptic Curve Digital Signature Algorithm</a>. </p> <p> Unfortunately, AWS EC2 only accepts <code>RSA</code> keys. The name of the new key pair will be of the form <code>~/.ssh/rsa-YYYY-MM-DD</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbb3f37de5b39'><button class='copyBtn' data-clipboard-target='#idbb3f37de5b39' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_KEY_PAIR_FILE="$HOME/.ssh/rsa-$( date '+%Y-%m-%d-%M-%S' )" <span class='unselectable'>$ </span>echo $AWS_KEY_PAIR_FILE <span class='unselectable'>/home/mslinn/.ssh/rsa-2020-11-04-08-24-22 </span></pre> <p> The new public key will be called <code>~/.ssh/rsa-2020-11-04-08-24-22.pub</code> and the new private key will be called <code>~/.ssh/rsa-2020-11-04-08-24-22</code>. </p> </editor-fold nameKeys> <editor-fold createKeys> <h2 class="numbered" id="createKeys">Create a New Keypair</h2> <ol> <li type='a'>This is how I would have created a new <code>ECDSA</code> keypair, if AWS supported that type of encryption. <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb14f06948b29'><button class='copyBtn' data-clipboard-target='#idb14f06948b29' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ssh-keygen -b 512 -C "mslinn@mslinn.com" -f "$AWS_KEY_PAIR_FILE" -P "" -t ecdsa <span class='unselectable'>Generating public/private ecdsa key pair. Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-08-24-22 Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-08-24-22.pub The key fingerprint is: SHA256:HEKjAA1GZxHbpwqjm85DXQpQEeIWrcjZ6fl84RHQaHE mslinn@mslinn.com The key's randomart image is: +---[RSA 512]----+ |=O*Bo.*E | |+.=oo*.o | |+o+.+.o.. | |o= o .o+ . | | o+ +. S | |..o=. o | |o .o . o | |.+ o o | |+o. . | +----[SHA256]-----+ $ </span>chmod 400 $AWS_KEY_PAIR_FILE</pre> </li> <li type='a'> Instead, I created a new <code>RSA</code> keypair like this: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id74c6a483430e'><button class='copyBtn' data-clipboard-target='#id74c6a483430e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ssh-keygen -b 2048 -C "mslinn@mslinn.com" -f "$AWS_KEY_PAIR_FILE" -m PEM -P "" -t rsa <span class='unselectable'>Generating public/private rsa key pair. Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-08-24-22 Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-08-24-22.pub The key fingerprint is: SHA256:bQScX0UMn0xGDorxSvElMZzwMyyk7hgs2FNbshBNenA mslinn@mslinn.com The key's randomart image is: +---[RSA 2048]----+ | ooE .*++o+** | | =. ooXo=.B.. | | o + o +.X. = | | o = * . =.o | |. + = . S o | | o + . | | . . | | | | | +----[SHA256]-----+ $ </span>chmod 400 $AWS_KEY_PAIR_FILE</pre> </li> <li type='a'> I would have liked to copy the keypair to the problem system using <code>ssh-copy-id</code>, but that only works when login is possible. <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3bf47ed8f072'><button class='copyBtn' data-clipboard-target='#id3bf47ed8f072' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ssh-copy-id -i $AWS_KEY_PAIR_FILE ubuntu@$AWS_PROBLEM_IP</pre> </li> <li type='a'> Instead, I decided to paste the public key into an AWS user data script and execute that script on the problem server the next time it booted. The purpose of the script is to copy the new public key that was just made to <code>~/.ssh/</code> on the problem server. This is the user data script I wrote to install the new public key, called <code>rescue_ubuntu2010.sh</code>: <br /> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,rescue_ubuntu2010.sh' download='rescue_ubuntu2010.sh' title='Click on the file name to download the file'>rescue_ubuntu2010.sh</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id68184d5b2f88">#!/bin/bash KEY_FILE_NAME=/home/ubuntu/.ssh/rsa-2020-11-03.pub cat > "$KEY_FILE_NAME" &lt;&lt;EOF ssh-rsa ABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFA== mslinn@mslinn.com EOF chown ubuntu: /home/ubuntu/.ssh/* chmod 400 "$KEY_FILE_NAME" cat "$KEY_FILE_NAME" >> /home/ubuntu/.ssh/authorized_keys </pre> The script runs on the problem server as <code>root</code> next time the system boots, and it reboots the server on the last line. </li> <li type='a'> The script need to be converted into base 64, in a file called <code>rescue_ubuntu2010.b64</code>. <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id00b5fc715c8b'><button class='copyBtn' data-clipboard-target='#id00b5fc715c8b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>base64 rescue_ubuntu2010.sh > rescue_ubuntu2010.b64</pre> </li> <li type='a'> The problem EC2 instance can be shut down like this: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id529d1f755822'><button class='copyBtn' data-clipboard-target='#id529d1f755822' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 stop-instances --instance-id $AWS_PROBLEM_INSTANCE_ID <span class='unselectable'>$ </span>aws ec2 wait instance-stopped --instance-ids $AWS_PROBLEM_INSTANCE_ID</pre> </li> <li type='a'> With the problem EC2 instance stopped, its user data was set to the base64-encoded version of the rescue script. <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcaae548f4338'><button class='copyBtn' data-clipboard-target='#idcaae548f4338' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 modify-instance-attribute \ --instance-id $AWS_PROBLEM_INSTANCE_ID \ --attribute userData \ --value file://rescue_ubuntu2010.b64</pre> </li> <li type='a'> Now the problem EC2 instance can be restarted. The script will add the new key to <code>/home/ubuntu/.ssh/authorized_keys</code> and login should be possible. <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9d368b1ea527'><button class='copyBtn' data-clipboard-target='#id9d368b1ea527' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 start-instances --instance-id $AWS_PROBLEM_INSTANCE_ID <span class='unselectable'>{ "StartingInstances": [ { "CurrentState": { "Code": 0, "Name": "pending" }, "InstanceId": "i-d3b03954", "PreviousState": { "Code": 80, "Name": "stopped" } } ] }</span> <span class='unselectable'>$ </span>aws ec2 wait instance-running --instance-ids $AWS_PROBLEM_INSTANCE_ID</pre> </li> </ol> </editor-fold createKeys> <editor-fold resetData> <h2 class="numbered" id="resetData">Reset User Data for Next Time</h2> <p> Next time the problem server is stopped, clear the user data so it is not provided the next time the server restarts. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3380b8a3b89c'><button class='copyBtn' data-clipboard-target='#id3380b8a3b89c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 modify-instance-attribute \ --instance-id $AWS_PROBLEM_INSTANCE_ID \ --user-data Value=</pre> </editor-fold resetData> Rescuing a Catastrophic Upgrade to Ubuntu 20.10 2020-10-25T00:00:00-04:00 https://mslinn.github.io/blog/2020/10/25/rescuing-catastrophic-upgrades-to-ubuntu-20_10 <editor-fold Intro> <p> The upgrade from Ubuntu 20.04 to 20.10 has been especially problematic for each of the half-dozen XUbuntu systems that I manage. One important server that I run on <a href='https://www.scaleway.com' target='_blank' rel='nofollow'>Scaleway</a> became unresponsive and would not boot shortly after starting the installation, and another important server on <a href='https://aws.amazon.com' target='_blank' rel='nofollow'>AWS</a> ran fine, but did not allow logins. </p> <p> This blog post details what I did to recover the AWS server using a standard <a href='https://en.wikipedia.org/wiki/Unix-like' target='_blank' rel='nofollow'>*nix</a> procedure that any competent system administrator would be comfortable with: <a href='https://wiki.archlinux.org/title/chroot' target='_blank' rel='nofollow'><code>chroot</code></a>. </p> <p> Before Linux had <a href='https://wiki.archlinux.org/title/Cgroups' target='_blank' rel='nofollow'><code>cgroups</code></a>, we used <code>chroot</code> and its close cousin, <code>jail</code>. I used <code>chroot</code> for the technical basis of <a href="/blog/index.html#Zamples">Zamples</a> back in 2001. </p> <p> Because the <code>chroot</code> environment will be set up in a way that it shares the rescue system&rsquo;s <code>/var/run</code> directory, the rescue system should have all upgrades in place and should be rebooted if <code>/var/run/reboot-required</code> exists. </p> <p> AWS also provides a tool called <a href='https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-ec2rescue.html' target='_blank' rel='nofollow'><code>EC2Rescue</code></a>, which does a complicated series of actions to accomplish something similar. <a href='https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html' target='_blank' rel='nofollow'>Here is additional documentation.</a> I find the AWS documentation is frequently obtuse, and the approach taken by most AWS products and tools is extremely general. Consequently I often find myself wasting a lot of time trying to get things to work. I don&rsquo;t subscribe to AWS support; if I had subscribed to expensive enterprise-level support, complete with an AWS expert to hold my hand while I attempted to resurrect the server, I might have tried using <code>EC2Rescue</code>. On the other hand, when pressed with an emergency, I prefer to lean on tried-and-true methods like <code>chroot</code>. </p> <p> This blog post concludes with two Bash scripts to automate the details. I wrote the second script in late January 2022, approximately 2 years after this post was first published. The second script is less user-friendly, but more fine-grained. That is a reasonable tradeoff, and both approaches have merit. </p> </editor-fold Intro> <editor-fold Setup> <h2 class="numbered" id="setup">Setup</h2> <h3 class="numbered" id="setupAwsCli">AWS CLI</h3> <p> I prefer to use the AWS CLI instead of the <a href='https://console.aws.amazon.com' target='_blank' rel='nofollow'>web console</a>. Installation instructions are <a href='https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html' target='_blank' rel='nofollow'>here</a>. This article uses the AWS CLI exclusively in favor of the AWS web console. </p> <h3 class="numbered" id="setupJq"><span class="code">jq</span></h3> <p> I also use <a href='https://stedolan.github.io/jq/' target='_blank' rel='nofollow'>jq</a> for parsing JSON in the bash shell. Install it on Debian-style Linux distros such as Ubuntu like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6ad74e4d93eb'><button class='copyBtn' data-clipboard-target='#id6ad74e4d93eb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install jq</pre> <editor-fold info> <h2 class="numbered" id="info">Discover information about the Problem EC2 instance</h2> <h3 class="numbered" id="ec2Info">Getting the AWS EC2 Instance Information</h3> <p> Because my problem EC2 instance has a tag called <code>Name</code> with <code>Value</code> <code>production</code>, I was able to easily obtain a JSON representation of all the information about it. I stored the JSON in an environment variable called <code>AWS_EC2_PRODUCTION</code>. </p> <p> The results are shown in unselectable text. This is so you can easily use this sample code yourself. You can copy the code to run into your clipboard. Just click on the little copy icon at the top right hand corner of the scrolling code display area. Because the prompt and the results and are unselectable, your clipboard will only pick up the code you need to paste in order to run the code example yourself. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc250e3562cc4'><button class='copyBtn' data-clipboard-target='#idc250e3562cc4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_EC2_PRODUCTION="$( aws ec2 describe-instances | \ jq '.Reservations[].Instances[] | select((.Tags[]?.Key=="Name") and (.Tags[]?.Value=="production"))' )" <span class='unselectable'>$ </span>echo "$AWS_EC2_PRODUCTION" <span class='unselectable'>{ "AmiLaunchIndex": 0, "ImageId": "ami-e29b9388", "InstanceId": "i-825eb905", "InstanceType": "t2.small", "KeyName": "sslTest", "LaunchTime": "2017-10-12T16:24:14.000Z", "Monitoring": { "State": "disabled" }, "Placement": { "AvailabilityZone": "us-east-1c", "GroupName": "", "Tenancy": "default" }, "PrivateDnsName": "ip-10-0-0-201.ec2.internal", "PrivateIpAddress": "10.0.0.201", "ProductCodes": [], "PublicDnsName": "", "PublicIpAddress": "52.207.225.143", "State": { "Code": 16, "Name": "running" }, "StateTransitionReason": "", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "Architecture": "x86_64", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "AttachTime": "2016-04-05T19:07:17.000Z", "DeleteOnTermination": true, "Status": "attached", "VolumeId": "vol-1c8903b4" } } ], "ClientToken": "GykZz1459883236367", "EbsOptimized": false, "Hypervisor": "xen", "NetworkInterfaces": [ { "Association": { "IpOwnerId": "amazon", "PublicDnsName": "", "PublicIp": "52.207.225.143" }, "Attachment": { "AttachTime": "2016-04-05T19:07:16.000Z", "AttachmentId": "eni-attach-a58bd15f", "DeleteOnTermination": true, "DeviceIndex": 0, "Status": "attached" }, "Description": "Primary network interface", "Groups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "Ipv6Addresses": [], "MacAddress": "0a:a4:be:1b:8e:eb", "NetworkInterfaceId": "eni-fa4f65bb", "OwnerId": "031372724784", "PrivateIpAddress": "10.0.0.201", "PrivateIpAddresses": [ { "Association": { "IpOwnerId": "amazon", "PublicDnsName": "", "PublicIp": "52.207.225.143" }, "Primary": true, "PrivateIpAddress": "10.0.0.201" } ], "SourceDestCheck": true, "Status": "in-use", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "InterfaceType": "interface" } ], "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SecurityGroups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "SourceDestCheck": true, "Tags": [ { "Key": "Name", "Value": "production" } ], "VirtualizationType": "hvm", "CpuOptions": { "CoreCount": 1, "ThreadsPerCore": 1 }, "CapacityReservationSpecification": { "CapacityReservationPreference": "open" }, "HibernationOptions": { "Configured": false }, "MetadataOptions": { "State": "applied", "HttpTokens": "optional", "HttpPutResponseHopLimit": 1, "HttpEndpoint": "enabled" } }</span></pre> <h3 class="numbered" id="problemIdGet">Getting the AWS EC2 Problem Instance Id</h3> <p> The instance ID for the problem EC2 instance can be extracted from the JSON returned by the preceding results easily: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id28822814a20c'><button class='copyBtn' data-clipboard-target='#id28822814a20c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_PROBLEM_INSTANCE_ID="$( jq -r .InstanceId <<< "$AWS_EC2_PRODUCTION" )" <span class='unselectable'>$ </span>echo "$AWS_PROBLEM_INSTANCE_ID" <span class='unselectable'>i-825eb905 </span></pre> <h3 class="numbered" id="problemIdIP">Getting the AWS EC2 Problem Instance IP Address</h3> <p> The IP address for the problem EC2 instance can be extracted from the JSON returned by the preceding results easily: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd9908e1211cc'><button class='copyBtn' data-clipboard-target='#idd9908e1211cc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_PROBLEM_IP="$( jq -r .PublicIpAddress <<< "$AWS_EC2_PRODUCTION" )" <span class='unselectable'>$ </span>echo "$AWS_PROBLEM_IP" <span class='unselectable'>52.207.225.143</span></pre> <h3 class="numbered" id="problemAZ">Getting the AWS EC2 Problem Availability Zone</h3> <p> The AWS availability zone for the problem EC2 instance can be extracted from the JSON returned by the preceding results easily: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id44eb2fa8f69d'><button class='copyBtn' data-clipboard-target='#id44eb2fa8f69d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_AVAILABILITY_ZONE="$( jq -r .Placement.AvailabilityZone <<< "$AWS_EC2_PRODUCTION" )" <span class='unselectable'>$ </span>echo "$AWS_AVAILABILITY_ZONE" <span class='unselectable'>us-east-1c </span></pre> <h3 class="numbered" id="volumeId">Getting the AWS EC2 Problem Volume ID</h3> <p> The following command line extracts the volume id of the problem server&rsquo;s system drive into an environment variable called <code>$AWS_PROBLEM_VOLUME_ID</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1a25aad2b9fa'><button class='copyBtn' data-clipboard-target='#id1a25aad2b9fa' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_PROBLEM_VOLUME_ID="$( jq -r '.BlockDeviceMappings[].Ebs.VolumeId' <<< "$AWS_EC2_PRODUCTION" )" <span class='unselectable'>$ </span>echo "$AWS_PROBLEM_VOLUME_ID" <span class='unselectable'>vol-1c8903b4 </span></pre> </editor-fold info> <editor-fold snapshotProblem> <h2 class="numbered" id="snapshotProblem">Make a Snapshot of the Problem Server</h2> <p> One approach, which would be living dangerously, would be to mount the system volume of the problem server on another server, set up <code>chroot</code>, attempt to repair the drive image, remount the repaired drive on the problem server, and reboot the server. I am never that optimistic. Things invariably go wrong. Instead, we will take a snapshot of the problem drive, turn the snapshot into a volume, repair the volume, swap in the repaired volume on the problem system, and reboot that system. </p> <p> It is better to shut down the EC2 instance before making a snapshot, however a snapshot can be taken whenever the server is idling. We will need to shut down the server anyway, so that could be done now, or at the last minute. </p> <p> I made a snapshot with a tag called <code>Name</code> with the value like <code>production 2020-10-25</code> and saved the snapshot id in an environment variable called <code>AWS_PROBLEM_SNAPSHOT_ID</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id27f4c2d4dc87'><button class='copyBtn' data-clipboard-target='#id27f4c2d4dc87' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_PROBLEM_SNAPSHOT_ID="$( aws ec2 create-snapshot --volume-id "$AWS_PROBLEM_VOLUME_ID" \ --description "production `date '+%Y-%m-%d'`" \ --tag-specifications "ResourceType=snapshot,Tags=[{Key=Created, Value=`date '+%Y-%m-%d'`},{Key=Name, Value=\"Broken do-release-upgrade 20.{04,10\"}]" | \ jq -r .SnapshotId )" <span class='unselectable'>$ </span>echo "$AWS_PROBLEM_SNAPSHOT_ID" <span class='unselectable'>snap-0a856be1f58b8a856 </span> <span class='unselectable'>$ </span>aws ec2 wait snapshot-completed --snapshot-ids "$AWS_PROBLEM_SNAPSHOT_ID"</pre> <p> Snapshots only take a few minutes to complete. The <code>aws ec2 wait</code> command blocks until the specified operation finishes. </p> </editor-fold snapshotProblem> <editor-fold rescueVolume> <h2 class="numbered" id="rescueVolue">Create Rescue Volume From Snapshot</h2> <p> Once the snapshot process has completed, create a new volume from the snapshot. The default volume type is <a href='https://aws.amazon.com/ebs/features/#Amazon_EBS_volume_types' target='_blank' rel='nofollow'><code>gp2</code></a>. We&rsquo;ll refer to this volume as <code>$AWS_RESCUE_VOLUME_ID</code>. It is important to create the volume in the same availability zone as the problem EC2 instance so that it can easily be attached. This command applies a tag called <code>Name</code>, with the value <code>rescue</code>, for easy identification. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id056a34ca665e'><button class='copyBtn' data-clipboard-target='#id056a34ca665e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_RESCUE_VOLUME_ID="$( aws ec2 create-volume \ --availability-zone $AWS_AVAILABILITY_ZONE \ --snapshot-id $AWS_PROBLEM_SNAPSHOT_ID \ --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=rescue}]' | \ jq -r .VolumeId )" <span class='unselectable'>$ </span>echo "$AWS_RESCUE_VOLUME_ID" <span class='unselectable'>vol-0e20fd22d2dc5a933 </span> <span class='unselectable'>$ </span>aws ec2 wait volume-available --volume-id "$AWS_RESCUE_VOLUME_ID"</pre> </editor-fold rescueVolume> <editor-fold> <h2 class="numbered" id="snapshotRescue">Use an EC2 Spot Instance For the Rescue Server</h2> <p> Now that the rescue volume is <code>available</code>, we need to mount it on a server, which I&rsquo;ll call the rescue server. We&rsquo;ll refer to the server where the rescue volume is prepared via its instance id, saved as <code>AWS_EC2_RESCUE_ID</code>. You can either create a new EC2 instance for this purpose, or use an existing EC2 instance. </p> <p> The rescue server does not need to be anything special; a tiny virtual machine of any description will do fine. However, some rescue operations will be much easier if the type of operating system is the same as that on the problem drive. <a href='https://www.mslinn.com/blog/2020/10/24/ec2-spot-instances-cli.html' target='_blank' rel='nofollow'>Yesterday</a> I blogged about how to find a suitable AMI, and determine its <code>image-id</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8b8e02b0c7be'><button class='copyBtn' data-clipboard-target='#id8b8e02b0c7be' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_AMI="$( aws ec2 describe-images \ --owners 099720109477 \ --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-????????-????????-amd64-server-????????" \ "Name=state,Values=available" \ --query "reverse(sort_by(Images, &CreationDate))[:1]" | \ jq -r '.[0]' )" <span class='unselectable'>$ </span>echo "$AWS_AMI" <span class='unselectable'>{ "Architecture": "x86_64", "CreationDate": "2020-10-30T14:07:42.000Z", "ImageId": "ami-0c71ec98278087e60", "ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030", "ImageType": "machine", "Public": true, "OwnerId": "099720109477", "PlatformDetails": "Linux/UNIX", "UsageOperation": "RunInstances", "State": "available", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "DeleteOnTermination": true, "SnapshotId": "snap-00bf581086dd686e5", "VolumeSize": 8, "VolumeType": "gp2", "Encrypted": false } }, { "DeviceName": "/dev/sdb", "VirtualName": "ephemeral0" }, { "DeviceName": "/dev/sdc", "VirtualName": "ephemeral1" } ], "Description": "Canonical, Ubuntu, 20.10, amd64 groovy image build on 2020-10-30", "EnaSupport": true, "Hypervisor": "xen", "Name": "ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030", "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SriovNetSupport": "simple", "VirtualizationType": "hvm" } </span></pre> <p> Now let's extract the ID of the AMI image and save it as <code>AWS_AMI_ID</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9305137a4213'><button class='copyBtn' data-clipboard-target='#id9305137a4213' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_AMI_ID="$( jq -r '.[0].ImageId' <<< "$AWS_AMI" )" <span class='unselectable'>$ </span>echo "$AWS_AMI_ID" <span class='unselectable'>ami-0c71ec98278087e60 </span></pre> <p> Volumes can be attached to running and stopped server instances. The load on the rescue server will likely be light and short-lived. An EC2 spot instance is ideal, and only costs two cents per hour! The spot instance will likely only be needed for 15 minutes. I specified my VPC id as <code>SubnetId</code>, the security group <code>sg-4cbc6f35</code> and the <code>AvailabilityZone</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idad22fc1b7ed5'><button class='copyBtn' data-clipboard-target='#idad22fc1b7ed5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_EC2_RESCUE="$( aws ec2 run-instances \ --image-id "$AWS_AMI_ID" \ --instance-market-options '{ "MarketType": "spot" }' \ --instance-type t2.medium \ --key-name rsa-2020-11-03.pub \ --network-interfaces '[ { "DeviceIndex": 0, "Groups": ["sg-4cbc6f35"], "SubnetId": "subnet-49de033f", "DeleteOnTermination": true, "AssociatePublicIpAddress": true } ]' \ --placement '{ "AvailabilityZone": "us-east-1c" }' )" <span class='unselectable'>$ </span>echo "$AWS_EC2_RESCUE" <span class='unselectable'>{ "Groups": [], "Instances": [ { "AmiLaunchIndex": 0, "ImageId": "ami-0dba2cb6798deb6d8", "InstanceId": "i-012a54aefcd333de9", "InstanceType": "t2.small", "KeyName": "rsa-2020-11-03.pub", "LaunchTime": "2020-11-03T23:19:50.000Z", "Monitoring": { "State": "disabled" }, "Placement": { "AvailabilityZone": "us-east-1c", "GroupName": "", "Tenancy": "default" }, "PrivateDnsName": "ip-10-0-0-210.ec2.internal", "PrivateIpAddress": "10.0.0.210", "ProductCodes": [], "PublicDnsName": "", "State": { "Code": 0, "Name": "pending" }, "StateTransitionReason": "", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "Architecture": "x86_64", "BlockDeviceMappings": [], "ClientToken": "026583fb-c94e-4bca-bdd2-8dcdcaa3aae9", "EbsOptimized": false, "EnaSupport": true, "Hypervisor": "xen", "InstanceLifecycle": "spot", "NetworkInterfaces": [ { "Attachment": { "AttachTime": "2020-11-03T23:19:50.000Z", "AttachmentId": "eni-attach-04feb4d36cf5c6792", "DeleteOnTermination": true, "DeviceIndex": 0, "Status": "attaching" }, "Description": "", "Groups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "Ipv6Addresses": [], "MacAddress": "0a:6d:ba:c5:65:4b", "NetworkInterfaceId": "eni-09ef90920cfb29dd9", "OwnerId": "031372724784", "PrivateIpAddress": "10.0.0.210", "PrivateIpAddresses": [ { "Primary": true, "PrivateIpAddress": "10.0.0.210" } ], "SourceDestCheck": true, "Status": "in-use", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "InterfaceType": "interface" } ], "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SecurityGroups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "SourceDestCheck": true, "SpotInstanceRequestId": "sir-rrs9gm3j", "StateReason": { "Code": "pending", "Message": "pending" }, "VirtualizationType": "hvm", "CpuOptions": { "CoreCount": 1, "ThreadsPerCore": 1 }, "CapacityReservationSpecification": { "CapacityReservationPreference": "open" }, "MetadataOptions": { "State": "pending", "HttpTokens": "optional", "HttpPutResponseHopLimit": 1, "HttpEndpoint": "enabled" } } ], "OwnerId": "031372724784", "ReservationId": "r-0d45e1919e7bad5c9" } </span></pre> <p> We can use <code>jq</code> to extract the EC2 <code>InstanceId</code> of the spot instance: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9af647673513'><button class='copyBtn' data-clipboard-target='#id9af647673513' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_SPOT_INSTANCE_ID="$( jq -r '.Instances[].InstanceId' <<< "$AWS_EC2_RESCUE" )" <span class='unselectable'>$ </span>echo "$AWS_SPOT_INSTANCE_ID" <span class='unselectable'>ami-0dba2cb6798deb6d8 </span></pre> <p> We need to retrieve the IP address of the newly created EC2 spot instance. This instance will disappear (terminate) once it shuts down, so do not reboot it. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7c4d2e2d3d06'><button class='copyBtn' data-clipboard-target='#id7c4d2e2d3d06' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 describe-instances --instance-ids "$AWS_SPOT_INSTANCE_ID" <span class='unselectable'>{ "Reservations": [ { "Groups": [], "Instances": [ { "AmiLaunchIndex": 0, "ImageId": "ami-0dba2cb6798deb6d8", "InstanceId": "i-012a54aefcd333de9", "InstanceType": "t2.small", "KeyName": "rsa-2020-11-03.pub", "LaunchTime": "2020-11-03T23:19:50.000Z", "Monitoring": { "State": "disabled" }, "Placement": { "AvailabilityZone": "us-east-1c", "GroupName": "", "Tenancy": "default" }, "PrivateDnsName": "ip-10-0-0-210.ec2.internal", "PrivateIpAddress": "10.0.0.210", "ProductCodes": [], "PublicDnsName": "", "PublicIpAddress": "54.242.88.254", "State": { "Code": 16, "Name": "running" }, "StateTransitionReason": "", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "Architecture": "x86_64", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "AttachTime": "2020-11-03T23:19:51.000Z", "DeleteOnTermination": true, "Status": "attached", "VolumeId": "vol-0c44c8c009d1fafda" } } ], "ClientToken": "026583fb-c94e-4bca-bdd2-8dcdcaa3aae9", "EbsOptimized": false, "EnaSupport": true, "Hypervisor": "xen", "InstanceLifecycle": "spot", "NetworkInterfaces": [ { "Association": { "IpOwnerId": "amazon", "PublicDnsName": "", "PublicIp": "54.242.88.254" }, "Attachment": { "AttachTime": "2020-11-03T23:19:50.000Z", "AttachmentId": "eni-attach-04feb4d36cf5c6792", "DeleteOnTermination": true, "DeviceIndex": 0, "Status": "attached" }, "Description": "", "Groups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "Ipv6Addresses": [], "MacAddress": "0a:6d:ba:c5:65:4b", "NetworkInterfaceId": "eni-09ef90920cfb29dd9", "OwnerId": "031372724784", "PrivateIpAddress": "10.0.0.210", "PrivateIpAddresses": [ { "Association": { "IpOwnerId": "amazon", "PublicDnsName": "", "PublicIp": "54.242.88.254" }, "Primary": true, "PrivateIpAddress": "10.0.0.210" } ], "SourceDestCheck": true, "Status": "in-use", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "InterfaceType": "interface" } ], "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SecurityGroups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "SourceDestCheck": true, "SpotInstanceRequestId": "sir-rrs9gm3j", "VirtualizationType": "hvm", "CpuOptions": { "CoreCount": 1, "ThreadsPerCore": 1 }, "CapacityReservationSpecification": { "CapacityReservationPreference": "open" }, "HibernationOptions": { "Configured": false }, "MetadataOptions": { "State": "applied", "HttpTokens": "optional", "HttpPutResponseHopLimit": 1, "HttpEndpoint": "enabled" } } ], "OwnerId": "031372724784", "ReservationId": "r-0d45e1919e7bad5c9" } ] }</span></pre> <!-- <p> It is possible that the work necessary to rescue the problem disk image might make changes to the rescue system. The rescue system should therefore have a snapshot taken before going any further. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6558235f0c8e'><button class='copyBtn' data-clipboard-target='#id6558235f0c8e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_RESCUE_SNAPSHOT_ID="$( aws ec2 create-snapshot --volume-id $AWS_RESCUE_VOLUME_ID %} \ --description "production `date '+%Y-%m-%d'`" \ --tag-specifications "ResourceType=snapshot,Tags=[{Key=Created, Value=`date '+%Y-%m-%d'`},{Key=Name, Value=\"Broken do-release-upgrade 20.{04,10\"}]" | \ jq -r .SnapshotId )" $ echo "$AWS_RESCUE_SNAPSHOT_ID" <span class='unselectable'>snap-0a856be1f58b8359a </span> <span class='unselectable'>$ </span>aws ec2 wait snapshot-completed --snapshot-ids $AWS_RESCUE_SNAPSHOT_ID</pre> --> </editor-fold> <editor-fold mount> <h2 class="numbered" id="mount">Mount the Rescue Volume On the Rescue Server</h2> <p> We need to select a device name to be assigned to the rescue disk once it is attached to an EC2 instance. The available names depend on what names are already in use on the rescue server. After logging into the rescue server, I ran the <code>lsblk</code> Linux command to see the available disk devices and their mount points. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6bcd590ca6d2'><span class='unselectable'>$ </span>lsblk <span class="unselectable">NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop1 7:1 0 53.1M 1 loop /snap/lxd/10984 loop2 7:2 0 88.4M 1 loop /snap/core/7169 loop3 7:3 0 97.8M 1 loop /snap/core/10185 loop4 7:4 0 53.1M 1 loop /snap/lxd/11348 xvda 202:0 0 8G 0 disk └─xvda1 202:1 0 8G 0 part /</span></pre> <p> The lsblk output does not show full device paths, instead, the <code>/dev/</code> prefix is omitted. With that in mind we can see that the only <code>disk</code> device on the rescue server is <code>/dev/xvda</code>, and its only partition called <code>/dev/xvda1</code> is mounted on the root directory. Because Linux drives are normally named sequentially, we should name the rescue disk <code>/dev/xvdb</code>. Let&rsquo;s define an environment variable called <code>AWS_RESCUE_DRIVE</code> to memorialize that decision. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb7ed35dbfcf5'><span class='unselectable'>$ </span>AWS_RESCUE_DRIVE=/dev/xvdb</pre> <p> The <code>aws ec2 attach-volume</code> command will attach the rescue volume to the rescue server. It automatically selects an appropriate device name for the rescue volume, which in the following example is <code>/dev/xvdb</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc69382f5d2d0'><button class='copyBtn' data-clipboard-target='#idc69382f5d2d0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_ATTACH_VOLUME="$( aws ec2 attach-volume \ --device $AWS_RESCUE_DRIVE \ --instance-id $AWS_EC2_RESCUE_ID \ --volume-id $AWS_RESCUE_VOLUME_ID )" <span class='unselectable'>$ </span>echo "$AWS_ATTACH_VOLUME" <span class='unselectable'>{ "AttachTime": "2020-10-26T14:34:55.222Z", "InstanceId": "i-d3b03954", "VolumeId": "vol-0e20fd22d2dc5a933", "State": "attaching", "Device": "/dev/xvdb" }</span> <span class='unselectable'>$ </span>aws ec2 wait volume-in-use --volume-id "$AWS_RESCUE_VOLUME_ID"</pre> <p> The details of the mounted rescue drive are provided by <code>fdisk -l</code>: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd04383c15ef4'><button class='copyBtn' data-clipboard-target='#idd04383c15ef4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo fdisk -l | sed -n -e '/xvdb/,$p' <span class='unselectable'>Disk /dev/xvdb: 12 GiB, 12884901888 bytes, 25165824 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x00000000 Device Boot Start End Sectors Size Id Type /dev/xvdb1 * 16065 25165790 25149726 12G 83 Linux</span></pre> <p> Now it is time to mount the rescue drive on the rescue server. Ubuntu has a directory called <code>/mnt</code> whose purpose is to act as a mount point: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id119e61e037aa'><button class='copyBtn' data-clipboard-target='#id119e61e037aa' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo mount /dev/xvdb1 /mnt</pre> <p> Let&rsquo;s confirm that the drive is mounted: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide657387a55e6'><button class='copyBtn' data-clipboard-target='#ide657387a55e6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>df -h | grep '^/dev/' | grep -v '^/dev/loop' <span class='unselectable'>/dev/xvda1 7.8G 6.3G 1.1G 86% / /dev/xvdb1 12G 9.0G 2.2G 82% /mnt</span></pre> <p> The last line shows that this drive is mounted on <code>/mnt</code> and it is 82% full. </p> </editor-fold mount> <editor-fold chroot> <h2 class="numbered" id="chroot">Set Up a <span class='code'>chroot</span> to Establish an Environment for Making Repairs</h2> <p> We need to mount some more file systems before we perform the <code>chroot</code>. The following mounts the rescue server&rsquo;s <code>/dev</code>, <code>/dev/shm</code>, <code>/sys</code>, and <code>/run</code> to the same paths within the rescue volume. Because programs like <code>do-release-upgrade</code> need a <code>tty</code>, I also mount <code>devtps</code> and <code>proc</code>. These mounts only last until the next server reboot. After all the mounts the <code>chroot</code> is issued. </p> <p class="warning"> <b>Warning</b> - mounting <code>/run</code> and then updating the system on the rescue disk from within a chroot may change the host system&rsquo;s <code>/run</code> contents; if the package managers (<code>apt</code> and <code>dpkg</code>) get out of sync with the actual state on the host system you won&rsquo;t be able to update the host system until you restore the host system&rsquo;s image from the snapshot that we made <a href='#snapshotRescue'>earlier</a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf38c9d120648'><button class='copyBtn' data-clipboard-target='#idf38c9d120648' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo mount -o bind /dev /mnt/dev <span class='unselectable'>$ </span>sudo mount -o bind /dev/shm /mnt/dev/shm <span class='unselectable'>$ </span>sudo mount -o bind /sys /mnt/sys <span class='unselectable'>$ </span>sudo mount -o bind /run /mnt/run <span class='unselectable'>$ </span>sudo mount -t proc proc /mnt/proc <span class='unselectable'>$ </span>sudo mount -t devpts devpts /mnt/dev/pts <span class='unselectable'>$ </span>sudo chroot /mnt <span class='unselectable'>root@ip-10-0-0-189:/#</span></pre> <p> Notice how the prompt changed after the <code>chroot</code>. That is your clue that it is active. </p> <!-- <p> I edited <code>/etc/hosts</code> in the <code>chroot</code> to add the name of the system to the entry for <code>localhost</code> (<code>127.0.0.1</code>): </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc979f622a618'><button class='copyBtn' data-clipboard-target='#idc979f622a618' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>127.0.0.1 localhost ip-10-0-0-189</pre> --> </editor-fold chroot> <editor-fold fix> <h2 class="numbered" id="fix">Correct the Problem</h2> <p> This step depends on whatever is wrong. I won&rsquo;t bore you with the problem I had. </p> </editor-fold fix> <editor-fold unmountRescue> <h2 class="numbered" id="unmount">Unmount the New Volume</h2> <p> Exit the <code>chroot</code> and unmount the rescue volume from the rescue server. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddeadbc77ccf2'><button class='copyBtn' data-clipboard-target='#iddeadbc77ccf2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'># </span>exit <span class='unselectable'>$ </span>sudo umount /mnt/dev <span class='unselectable'>$ </span>sudo umount /mnt/dev/shm <span class='unselectable'>$ </span>sudo umount /mnt/sys <span class='unselectable'>$ </span>sudo umount /mnt/run <span class='unselectable'>$ </span>sudo umount /mnt/proc <span class='unselectable'>$ </span>sudo umount /mnt/dev/pts <span class='unselectable'>$ </span>sudo umount /mnt</pre> <p> Detach the rescue volume from the rescue server. This can be done from any machine that is configured with <code>aws cli</code> for use with your account credentials. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6f856876db58'><button class='copyBtn' data-clipboard-target='#id6f856876db58' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 detach-volume --volume-id $AWS_RESCUE_VOLUME_ID <span class='unselectable'>$ </span>aws ec2 wait volume-available --volume-id $AWS_RESCUE_VOLUME_ID</pre> </editor-fold unmountRescue> <editor-fold unmountProblem> <h2 class="numbered" id="unmount">Unmount the Problem Volume</h2> <p> The problem server must be shut down for this to work. Detach the problem volume from the problem server. This can be done from any machine that is configured with <code>aws cli</code> for use with your account credentials. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id24a3811d218f'><button class='copyBtn' data-clipboard-target='#id24a3811d218f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 stop-instances --instance-id $AWS_PROBLEM_INSTANCE_ID <span class='unselectable'>$ </span>aws ec2 wait instance-stopped --instance-ids $AWS_PROBLEM_INSTANCE_ID <span class='unselectable'>$ </span>aws ec2 detach-volume --volume-id $AWS_PROBLEM_VOLUME_ID <span class='unselectable'>$ </span>aws ec2 wait volume-available --volume-id $AWS_PROBLEM_VOLUME_ID</pre> </editor-fold unmountProblem> <editor-fold replace> <h2 class="numbered" id="replace">Replace the Problem Volume</h2> <p> Now it is time to replace the problem volume containing the problem boot drive on the problem system with the newly created volume. BTW, AWS EC2 always refers to boot drives as <code>/dev/sda1</code>, even when the device has a different name, such as <code>/dev/xvdb1</code>. </p> <h3 id="replaceSystemVolume"><span>replaceSystemVolume</span> Bash function</h3> <p> This Bash function detaches the volume containing the current boot drive of an EC2 instance and replaces it with another volume. If the EC2 instance is running then it is first stopped. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,replaceSystemVolume' download='replaceSystemVolume' title='Click on the file name to download the file'>replaceSystemVolume</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id372fdba31497">#!/bin/bash function replaceSystemVolume &#123; # $1 - EC2 instance id # $2 - new volume to mount as system boot drive export EC2_INSTANCE="$( aws ec2 describe-instances --instance-ids "$1" | \ jq -r ".Reservations[].Instances[0]" )" export EC2_NAME="$( jq -r ".Tags[] | select(.Key==\"Name\") | .Value" &lt;&lt;&lt; "$EC2_INSTANCE" )" export ATTACHED_VOLUME_ID="$( jq -r ".BlockDeviceMappings[].Ebs.VolumeId" &lt;&lt;&lt; "$EC2_INSTANCE" )" if [[ "$ATTACHED_VOLUME_ID" == "$2" ]]; then >&amp;2 echo "VolumeId $2 is already attached to EC2 instance $1" exit 1 fi export EC2_STATE="$( jq -r ".State.Name" &lt;&lt;&lt; "$EC2_INSTANCE" )" if [ "$EC2_STATE" == running ]; then echo "Stopping EC2 instance $1" aws ec2 stop-instances --instance-ids "$1" aws ec2 wait instance-stopped --instance-ids "$1" fi aws ec2 detach-volume --volume-id "$ATTACHED_VOLUME_ID" aws ec2 wait volume-available --volume-id "$ATTACHED_VOLUME_ID" aws ec2 attach-volume \ --device /dev/sda1 \ --instance-id "$1" \ --volume-id "$2" aws ec2 wait volume-in-use --volume-id "$2" aws ec2 start-instances --instance-ids "$1" aws ec2 wait instance-started --instance-ids "$1" &#125; set -e replaceSystemVolume "$@" </pre> <p> Here is how to use it: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8fa612a09011'><button class='copyBtn' data-clipboard-target='#id8fa612a09011' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>replaceSystemVolume "$AWS_PROBLEM_INSTANCE_ID" "$AWS_RESCUE_VOLUME_ID"</pre> <p> Preview 2 instance id is <code>AWS_EC2_RESCUE_ID</code>. Replace rescue volume on preview with preview's original volume: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc3a445a4c757'><button class='copyBtn' data-clipboard-target='#idc3a445a4c757' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>replaceSystemVolume "$AWS_EC2_RESCUE_ID" "$AWS_PREVIEW_VOLUME_ID"</pre> </editor-fold replace> <editor-fold boot> <h2 class="numbered" id="boot">Boot the problem system</h2> <p> Boot the problem system and verify the problem is solved. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6cf0a148526d'><button class='copyBtn' data-clipboard-target='#id6cf0a148526d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 start-instances --instance-ids $AWS_PROBLEM_INSTANCE_ID <span class='unselectable'>$ </span>aws ec2 wait instance-started --instance-ids $AWS_PROBLEM_INSTANCE_ID</pre> </editor-fold boot> <editor-fold acknowledgements> <h2 id="acknowledgements">Acknowledgements</h2> <p> This article was inspired by <a href='https://www.rootusers.com/how-to-repair-an-aws-ec2-instance-without-console' target='_blank'>this excellent article</a>, which uses the AWS web console to achieve similar results. </p> </editor-fold acknowledgements> Working With EC2 Spot Instances From AWS CLI 2020-10-24T00:00:00-04:00 https://mslinn.github.io/blog/2020/10/24/ec2-spot-instances-cli <editor-fold Intro> <p> AWS EC2 <code>T2.medium</code> spot instances <a href='https://aws.amazon.com/ec2/spot/pricing/' target='_blank' rel='nofollow'>cost less than 2 cents per hour</a> for Linux and can be created very easily from the command line. They self-destruct once shut down. These powerful virtual machines can do an incredible amount of work in an hour for less than 2 cents! </p> <p> This article shows how all this can be done via the command line. I also provide an <a href='#createEc2Spot'>interactive bash script</a> for automating the process of obtaining and releasing an EC2 spot instance. </p> <p> Once again this article uses <a href='https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html' target='_blank' rel='nofollow'><code>AWS CLI</code></a> and <a href='https://stedolan.github.io/jq/' target='_blank' rel='nofollow'><code>jq</code></a>. </p> </editor-fold Intro> <editor-fold nameKeys> <h2 class="numbered" id="nameKeys">Create and Import a New Keypair</h2> <p> I want to create a new temporary ssh keypair that will just be used for this spot instance. The name of the new key pair will be of the form <code>~/.ssh/rsa-YYYY-MM-DD-mm-ss</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id69800f2c50ed'><button class='copyBtn' data-clipboard-target='#id69800f2c50ed' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_KEY_PAIR_NAME="rsa-$( date '+%Y-%m-%d-%H-%M-%S' )" <span class='unselectable'>$ </span>echo "$AWS_KEY_PAIR_NAME" <span class='unselectable'>rsa-2020-11-04-10-43-54 </span> <span class='unselectable'>$ </span>AWS_KEY_PAIR_FILE="~/.ssh/$AWS_KEY_PAIR_NAME" <span class='unselectable'>$ </span>echo "$AWS_KEY_PAIR_FILE" <span class='unselectable'>~/.ssh/rsa-2020-11-04-10-43-54 </span></pre> <p> Now we can make the keypair. AWS EC2 does not accept keys longer than 2048 bits. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4fec746ec3a3'><button class='copyBtn' data-clipboard-target='#id4fec746ec3a3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ssh-keygen -b 2048 -f "$AWS_KEY_PAIR_FILE" -P "" -N "" -t rsa <span class='unselectable'>Generating public/private rsa key pair. Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-10-43-54 Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-10-43-54.pub The key fingerprint is: SHA256:bQScX0UMn0xGDorxSvElMZzwMyyk7hgs2FNbshBNenA mslinn@mslinn.com The key's randomart image is: +---[RSA 2048]----+ | ooE .*++o+** | | =. ooXo=.B.. | | o + o +.X. = | | o = * . =.o | |. + = . S o | | o + . | | . . | | | | | +----[SHA256]-----+</span></pre> <p> The new public key will be stored in <code>~/.ssh/2020-11-04-10-43-54.pub</code> and the new private key will be stored in <code>~/.ssh/2020-11-04-10-43-54</code>. </p> <p> Now set the permissions for the key. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id23bbf651b9ff'><button class='copyBtn' data-clipboard-target='#id23bbf651b9ff' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>chmod 400 "$AWS_KEY_PAIR_FILE"</pre> <p> Now we can import the key pair into AWS: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id46755bb4f6a4'><button class='copyBtn' data-clipboard-target='#id46755bb4f6a4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 import-key-pair \ --key-name "$AWS_KEY_PAIR_NAME" \ --public-key-material "fileb://${AWS_KEY_PAIR_FILE}.pub" <span class='unselectable'>{ "KeyFingerprint": "c7:76:90:53:17:d0:fc:ba:45:dd:93:d2:93:03:c2:19", "KeyName": "2020-11-04-10-43-54", "KeyPairId": "key-092a2306ec3f4aff6" }</span></pre> </editor-fold nameKeys> <editor-fold AMI> <h2 class="numbered" id="ami">Select an AMI</h2> <p> New <a href='https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html' target='_blank' rel='nofollow'>AMIs</a> become available every day. You probably want your EC2 spot instance to be created from the most recent AMI that matches your needs. For most of my work I want an Ubuntu 64-bit Intel/AMD server distribution. <a href='https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/finding-an-ami.html' target='_blank' rel='nofollow'>AWS documentation</a> is helpful and gives us a head start in automating the AMI selection. </p> <p> The following incantation sets an environment variable called <code>AWS_AMI</code> to the details in JSON syntax of the AMI for the most recent 64-bit Ubuntu server release for Intel/AMD architecture. The <code>OwnerId</code> of Canonical, the publisher of Ubuntu, is <a href='https://ubuntu.com/server/docs/cloud-images/amazon-ec2' target='_blank' rel='nofollow'><code>099720109477</code></a>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5c77778a73d6'><button class='copyBtn' data-clipboard-target='#id5c77778a73d6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_AMI="$( aws ec2 describe-images \ --owners <span class="bg_yellow">099720109477</span> \ --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-????????-????????-amd64-server-????????" \ "Name=state,Values=available" \ --query "reverse(sort_by(Images, &CreationDate))[:1]" | \ jq -r '.[0]' )" <span class='unselectable'>$ </span>echo "$AWS_AMI" <span class='unselectable'>{ "Architecture": "x86_64", "CreationDate": "2020-10-30T14:07:42.000Z", "ImageId": "ami-0c71ec98278087e60", "ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030", "ImageType": "machine", "Public": true, "OwnerId": "099720109477", "PlatformDetails": "Linux/UNIX", "UsageOperation": "RunInstances", "State": "available", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "DeleteOnTermination": true, "SnapshotId": "snap-00bf581086dd686e5", "VolumeSize": 8, "VolumeType": "gp2", "Encrypted": false } }, { "DeviceName": "/dev/sdb", "VirtualName": "ephemeral0" }, { "DeviceName": "/dev/sdc", "VirtualName": "ephemeral1" } ], "Description": "Canonical, Ubuntu, 20.10, amd64 groovy image build on 2020-10-30", "EnaSupport": true, "Hypervisor": "xen", "Name": "ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030", "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SriovNetSupport": "simple", "VirtualizationType": "hvm" } </span></pre> <p> Now let's extract the ID of the AMI image and save it as <code>AWS_AMI_ID</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4f41e6bb38c6'><button class='copyBtn' data-clipboard-target='#id4f41e6bb38c6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_AMI_ID="$( jq -r '.[0].ImageId' <<< "$AWS_AMI" )" <span class='unselectable'>$ </span>echo "$AWS_AMI_ID" <span class='unselectable'>ami-0c71ec98278087e60 </span></pre> </editor-fold AMI> <editor-fold Spot> <h2 class="numbered" id="create">Create an EC2 Spot Instance</h2> <p> For my work I often want my spot instance to be created in the same VPC subnet as my other resources, with the same security group. That is why the following environment variables are defined for the <code>Groups</code> and <code>SubnetId</code> values within the <code>network-interfaces</code> option, as well as the AWS region. The script at the end of this article offers an easier way of obtaining all these values. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf0ab11f5e021'><button class='copyBtn' data-clipboard-target='#idf0ab11f5e021' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_GROUP=sg-4cbc6f35 <span class='unselectable'>$ </span>AWS_SUBNET=subnet-49de033f <span class='unselectable'>$ </span>AWS_ZONE=us-east-1c <span class='unselectable'>$ </span>AWS_EC2_TYPE=t2.medium</pre> <p> The following creates an AWS EC2 spot instance with a public IP address and runs it. Details about the newly created spot instance are stored as JSON in <code>AWS_SPOT_INSTANCE</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idec7bde0227e6'><button class='copyBtn' data-clipboard-target='#idec7bde0227e6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_SPOT_INSTANCE="$( aws ec2 run-instances \ --image-id "$AWS_AMI_ID" \ --instance-market-options '{ "MarketType": "spot" }' \ --instance-type "$AWS_EC2_TYPE" \ --key-name "$AWS_KEY_PAIR_NAME" \ --network-interfaces "[ { \"DeviceIndex\": 0, \"Groups\": [\"$AWS_GROUP\"], \"SubnetId\": \"$AWS_SUBNET\", \"DeleteOnTermination\": true, \"AssociatePublicIpAddress\": true } ]" \ --placement "{ \"AvailabilityZone\": \"$AWS_ZONE\" }" | \ jq -r .Instances[0] )" <span class='unselectable'>$ </span>echo "$AWS_SPOT_INSTANCE" <span class='unselectable'>{ "AmiLaunchIndex": 0, "ImageId": "ami-0dba2cb6798deb6d8", "InstanceId": "i-012a54aefcd333de9", "InstanceType": "t2.small", "KeyName": "rsa-2020-11-03.pub", "LaunchTime": "2020-11-03T23:19:50.000Z", "Monitoring": { "State": "disabled" }, "Placement": { "AvailabilityZone": "us-east-1c", "GroupName": "", "Tenancy": "default" }, "PrivateDnsName": "ip-10-0-0-210.ec2.internal", "PrivateIpAddress": "10.0.0.210", "ProductCodes": [], "PublicDnsName": "", "State": { "Code": 0, "Name": "pending" }, "StateTransitionReason": "", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "Architecture": "x86_64", "BlockDeviceMappings": [], "ClientToken": "026583fb-c94e-4bca-bdd2-8dcdcaa3aae9", "EbsOptimized": false, "EnaSupport": true, "Hypervisor": "xen", "InstanceLifecycle": "spot", "NetworkInterfaces": [ { "Attachment": { "AttachTime": "2020-11-03T23:19:50.000Z", "AttachmentId": "eni-attach-04feb4d36cf5c6792", "DeleteOnTermination": true, "DeviceIndex": 0, "Status": "attaching" }, "Description": "", "Groups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "Ipv6Addresses": [], "MacAddress": "0a:6d:ba:c5:65:4b", "NetworkInterfaceId": "eni-09ef90920cfb29dd9", "OwnerId": "031372724784", "PrivateIpAddress": "10.0.0.210", "PrivateIpAddresses": [ { "Primary": true, "PrivateIpAddress": "10.0.0.210" } ], "SourceDestCheck": true, "Status": "in-use", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "InterfaceType": "interface" } ], "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SecurityGroups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "SourceDestCheck": true, "SpotInstanceRequestId": "sir-rrs9gm3j", "StateReason": { "Code": "pending", "Message": "pending" }, "VirtualizationType": "hvm", "CpuOptions": { "CoreCount": 1, "ThreadsPerCore": 1 }, "CapacityReservationSpecification": { "CapacityReservationPreference": "open" }, "MetadataOptions": { "State": "pending", "HttpTokens": "optional", "HttpPutResponseHopLimit": 1, "HttpEndpoint": "enabled" } } </span></pre> <p> Now extract the EC2 spot instance id and save it in <code>AWS_SPOT_ID</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ideec7a841df87'><button class='copyBtn' data-clipboard-target='#ideec7a841df87' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_SPOT_ID="$( jq -r .InstanceId <<< "$AWS_SPOT_INSTANCE" )" <span class='unselectable'>$ </span>echo "$AWS_SPOT_ID" <span class='unselectable'>i-012a54aefcd333de9 </span></pre> <p> Wait for the instance to start. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2defa818a685'><button class='copyBtn' data-clipboard-target='#id2defa818a685' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 wait instance-running --instance-ids "$AWS_SPOT_ID"</pre> </editor-fold Spot> <editor-fold Connect> <h2 class="numbered" id="connect">Connect to the Spot Instance</h2> <p> In order to <code>ssh</code> into the spot instance we first need to discover its IP address, which is saved in <code>AWS_SPOT_IP</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7caa22e1bf11'><button class='copyBtn' data-clipboard-target='#id7caa22e1bf11' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>AWS_SPOT_IP="$( aws ec2 describe-instances \ --instance-ids $AWS_SPOT_ID | \ jq -r .Reservations[0].Instances[0].PublicIpAddress )" <span class='unselectable'>$ </span>echo "$AWS_SPOT_IP" <span class='unselectable'>54.242.88.254 </span></pre> <p> Now we can connect to the spot instance via <code>ssh</code>. The default userid for Ubuntu is <code>ubuntu</code>. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id197ca7d00f20'><button class='copyBtn' data-clipboard-target='#id197ca7d00f20' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ssh -i "$AWS_KEY_PAIR_FILE" "ubuntu@$AWS_SPOT_IP" <span class='unselectable'>Warning: No xauth data; using fake authentication data for X11 forwarding. Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.11.0-1027-aws x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage System information as of Thu Jan 27 20:28:22 UTC 2022 System load: 0.06 Processes: 113 Usage of /: 18.3% of 7.69GB Users logged in: 0 Memory usage: 5% IPv4 address for eth0: 10.0.0.29 Swap usage: 0% 1 update can be applied immediately. To see these additional updates run: apt list --upgradable The list of available updates is more than a week old. To check for new updates run: sudo apt update The programs included with the Ubuntu system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. /usr/bin/xauth: file /home/ubuntu/.Xauthority does not exist To run a command as administrator (user "root"), use "sudo <command>". See "man sudo_root" for details. ubuntu@ip-10-0-0-29:~$ </span></pre> <p> Do your work on the spot instance now. We'll disconnect and clean up next. </p> </editor-fold Connect> <editor-fold Disconnect> <h2 class="numbered" id="connect">Disconnect from the Spot Instance and Clean Up</h2> <h3 class="numbered" id="consoleCheck">Using the Command Line</h3> <p> Once the spot instance stops it is automatically terminated. The instance will survive a <code>reboot</code>, but not a <code>halt</code>. From a prompt on the spot instance, type: </p> <div class='codeLabel unselectable' data-lt-active='false'>shell&nbsp;(Spot&nbsp;Instance)</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8b656df5743d'><button class='copyBtn' data-clipboard-target='#id8b656df5743d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo halt</pre> <p> Back in the shell that launched the spot instance, wait for the spot instance to stop before cleaning up. </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2ce3a11af8fa'><button class='copyBtn' data-clipboard-target='#id2ce3a11af8fa' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 wait instance-stopped --instance-ids $AWS_SPOT_ID</pre> <p> Delete the temporary <code>ssh</code> keypair we created. Copies exist on AWS and the local machine; we need to remove all of them, like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfaa6f751e920'><button class='copyBtn' data-clipboard-target='#idfaa6f751e920' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 delete-key-pair --key-name $AWS_KEY_PAIR_NAME <span class='unselectable'>$ </span>rm $AWS_KEY_PAIR_FILE $AWS_KEY_PAIR_FILE.pub</pre> <h3 class="numbered" id="consoleCheck">Checking With Web Console</h3> <p> You can use the web console to verify that all the <a href='https://console.aws.amazon.com/ec2sp/v2/#/spot' target='_blank' rel='nofollow'>spot instances</a> were shut down, and the <a href='https://console.aws.amazon.com/ec2/v2/home?#KeyPairs:' target='_blank' rel='nofollow'>key pairs</a> were deleted. </p> </editor-fold Disconnect> <editor-fold Script> <h2 class="numbered" id="createEc2Spot">Bash Script <span class="code">createEc2Spot</span></h2> <h3 class="numbered" id="createEc2Spot_code">Source Code</h3> <p> This script does everything discussed above, plus it prompts the user with default values for parameters unique to each invocation. Click on the name of the script and save it this script to a directory on your <code>PATH</code>. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,createEc2Spot' download='createEc2Spot' title='Click on the file name to download the file'>createEc2Spot</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id2b2c455838ee">#!/bin/bash # Author: Mike Slinn mslinn@mslinn.com # Initial version 2020-1-25 # Last modified 2022-01-27 set -e function readWithDefault &#123; >&amp;2 printf "\n$1: " read -e -i "$2" VALUE echo "$VALUE" &#125; echo "The AWS EC2 spot instance needs to share settings with the existing EC2 instance you want to affect." echo "The easiest way to do this is to reference an EC2 instance with a Name tag, so it can be identified." echo "If there is no such EC2 instance, delete the default value in the next prompt, and you will be able to specify the details manually." AWS_EC2_NAME="$( readWithDefault "AWS EC2 Name tag value" production )" if [ "$AWS_EC2_NAME" ]; then AWS_EC2_PRODUCTION="$( aws ec2 describe-instances | \ jq ".Reservations[].Instances[] | select((.Tags[]?.Key==\"Name\") and (.Tags[]?.Value==\"$AWS_EC2_NAME\"))" )" AWS_GROUP="$( jq -r ".NetworkInterfaces[].Groups[].GroupId" &lt;&lt;&lt; "$AWS_EC2_PRODUCTION" )" AWS_SUBNET="$( jq -r ".SubnetId" &lt;&lt;&lt; "$AWS_EC2_PRODUCTION" )" AWS_ZONE="$( jq -r ".Placement.AvailabilityZone" &lt;&lt;&lt; "$AWS_EC2_PRODUCTION" )" else echo "Please answer a few questions so the AWS EC2 spot instance can be created." AWS_GROUP="$( readWithDefault "AWS security group" sg-4cbc6f35 )" AWS_SUBNET="$( readWithDefault "EC2 subnet" subnet-49de033f )" AWS_ZONE="$( readWithDefault "AWS availability zone" us-east-1c )" fi echo "EC2 spot instances are really inexpensive, so be generous with the size of the machine type for this spot instance." AWS_EC2_TYPE="$( readWithDefault "EC2 machine type" t2.medium )" AWS_KEY_PAIR_NAME="rsa-$( date '+%Y-%m-%d-%H-%M-%S' )" AWS_KEY_PAIR_FILE="$HOME/.ssh/$AWS_KEY_PAIR_NAME" ssh-keygen -b 2048 -f "$AWS_KEY_PAIR_FILE" -P "" -N "" -t rsa # -q chmod 400 "$AWS_KEY_PAIR_FILE" aws ec2 import-key-pair \ --key-name "$AWS_KEY_PAIR_NAME" \ --public-key-material "fileb://$AWS_KEY_PAIR_FILE.pub" echo "Searching for the latest 64-bit Intel/AMD Ubuntu AMI by Canonical." AWS_AMI="$( aws ec2 describe-images \ --owners 099720109477 \ --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-????????-????????-amd64-server-????????" \ "Name=state,Values=available" \ --query "reverse(sort_by(Images, &amp;CreationDate))[:1]" | \ jq -r '.[0]' )" echo "Obtaining the AMI image ID." AWS_AMI_ID="$( jq -r '.ImageId' &lt;&lt;&lt; "$AWS_AMI" )" echo "Creating the EC2 spot instance." AWS_SPOT_INSTANCE="$( aws ec2 run-instances \ --image-id $AWS_AMI_ID \ --instance-market-options '&#123; "MarketType": "spot" &#125;' \ --instance-type $AWS_EC2_TYPE \ --key-name $AWS_KEY_PAIR_NAME \ --network-interfaces "[ &#123; \"DeviceIndex\": 0, \"Groups\": [\"$AWS_GROUP\"], \"SubnetId\": \"$AWS_SUBNET\", \"DeleteOnTermination\": true, \"AssociatePublicIpAddress\": true &#125; ]" \ --placement "&#123; \"AvailabilityZone\": \"$AWS_ZONE\" &#125;" | \ jq -r .Instances[0] )" echo "Obtaining the EC2 spot instance ID." AWS_SPOT_ID="$( jq -r .InstanceId &lt;&lt;&lt; "$AWS_SPOT_INSTANCE" )" echo "Awaiting for the EC2 spot instance $AWS_SPOT_ID to enter the running state." aws ec2 wait instance-running --instance-ids $AWS_SPOT_ID echo "Obtaining the IP address of the new EC2 spot instance $AWS_SPOT_ID." AWS_SPOT_IP="$( aws ec2 describe-instances \ --instance-ids $AWS_SPOT_ID | \ jq -r .Reservations[0].Instances[0].PublicIpAddress )" echo "About to ssh to the EC2 spot instance as ubuntu@$AWS_SPOT_IP using $AWS_KEY_PAIR_FILE." echo "When you are done, type: sudo halt." echo "The spot instance will then terminate and be gone forever." echo "Any predefined resources, such as volumes that you attach will be freed." ssh -i "$AWS_KEY_PAIR_FILE" "ubuntu@$AWS_SPOT_IP" echo "Waiting for the EC2 spot instance $AWS_SPOT_ID to enter the stopped state." aws ec2 wait instance-stopped --instance-ids "$AWS_SPOT_ID" echo "The spot instance is no longer available. Deleting its keypair." aws ec2 delete-key-pair --key-name AWS_KEY_PAIR_NAME rm $AWS_KEY_PAIR_FILE $AWS_KEY_PAIR_FILE.pub </pre> <p>Make the script executable.</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2197af995a2c'><button class='copyBtn' data-clipboard-target='#id2197af995a2c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>chmod a+x createEc2Spot</pre> <h3 class="numbered" id="createEc2Spot_usage">Sample Usage</h3> <p> The script is easy to use: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcb4ce7ff7513'><button class='copyBtn' data-clipboard-target='#idcb4ce7ff7513' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>createEc2Spot <span class='unselectable'>The AWS EC2 spot instance needs to share settings with the existing EC2 instance you want to affect. The easiest way to do this is to reference an EC2 instance with a Name tag, so it can be identified. If there is no such EC2 instance, delete the default value in the next prompt, and you will be able to specify the details manually. AWS EC2 Name tag value: </span>production <span class='unselectable'>EC2 spot instances are really inexpensive, so be generous with the size of the machine type for this spot instance. EC2 machine type: </span>t2.medium <span class='unselectable'>Generating public/private rsa key pair. Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-10-43-54 Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-10-43-54.pub The key fingerprint is: SHA256:bQScX0UMn0xGDorxSvElMZzwMyyk7hgs2FNbshBNenA mslinn@mslinn.com The key's randomart image is: +---[RSA 2048]----+ | ooE .*++o+** | | =. ooXo=.B.. | | o + o +.X. = | | o = * . =.o | |. + = . S o | | o + . | | . . | | | | | +----[SHA256]-----+ "KeyFingerprint": "be:19:50:59:a1:83:ea:c1:91:1e:2f:d6:31:64:9a:c0", "KeyName": "rsa-2022-01-27-50-02", "KeyPairId": "key-072a3f0545864526a" } Searching for the latest 64-bit Intel/AMD Ubuntu AMI by Canonical. Obtaining the AMI image ID. Creating the EC2 spot instance. Obtaining the EC2 spot instance ID. Awaiting for the EC2 spot instance i-03d09e364ed15a448 to enter the running state. Obtaining the IP address of the new EC2 spot instance i-03d09e364ed15a448. 54.242.88.254 When you are done, type: sudo halt. The spot instance will then terminate and be gone forever. Any predefined resources, such as volumes that you attach will be freed. Waiting for the EC2 spot instance $AWS_SPOT_ID to enter the stopped state. </span></pre> <p> Now do your work on the spot instance, and then run <code>sudo halt</code>. Once the spot instance shuts down, it is destroyed and the script cleans up. Do not try to reboot the spot instance, because that shuts it down and it goes away instead of coming back up. Now do your work on the spot instance. From a prompt on the spot instance, type: </p> <div class='codeLabel unselectable' data-lt-active='false'>shell&nbsp;(Spot&nbsp;Instance)</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id641e9f29b662'><button class='copyBtn' data-clipboard-target='#id641e9f29b662' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo halt</pre> <p> Back in the shell on your computer, you should see: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idedd85c8f1a0d'><button class='copyBtn' data-clipboard-target='#idedd85c8f1a0d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo halt <span class='unselectable'>The spot instance is no longer available. Deleting its keypair. </span></pre> </editor-fold Script> <editor-fold Script2> <h2 class="numbered" id="aws_ec2_functions">Bash Script <span class="code">aws_ec2_functions</span></h2> <h3 class="numbered" id="aws_ec2_functions_code">Source Code</h3> <p> This is another script does the same thing as the previous script, but in steps. Click on the name of the script and save it this script to a directory on your <code>PATH</code>. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,aws_ec2_functions' download='aws_ec2_functions' title='Click on the file name to download the file'>aws_ec2_functions</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idad20a3fc19ef">#!/bin/bash # Author: Mike Slinn mslinn@mslinn.com # Initial version 2022-01-28 # Last modified 2022-01-28 function readWithDefault &#123; # prompt user for a value, with a default >&amp;2 printf "\n$1: " read -e -i "$2" VALUE echo "$VALUE" &#125; function requires &#123; # Halts if any of the supplied arguments is not the name of a defined environment variable for ENV_VAR in "$@"; do if [ -z "$&#123;!ENV_VAR&#125;" ]; then echo "Error: $&#123;ENV_VAR&#125; is undefined." return 1 2> /dev/null || exit 1 #else # echo "$&#123;ENV_VAR&#125; has value $&#123;!ENV_VAR&#125;" fi done &#125; function attachVolumeToSpot &#123; requires AWS_SPOT_ID AWS_NEW_VOLUME_ID || return export AWS_ATTACH_VOLUME="$( aws ec2 attach-volume \ --device /dev/xvdh \ --instance-id $AWS_SPOT_ID \ --volume-id $AWS_NEW_VOLUME_ID )" aws ec2 wait volume-in-use --volume-id "$AWS_NEW_VOLUME_ID" export AWS_ATTACH_VOLUME_DEVICE="$( aws ec2 describe-volumes \ --volume-id "$AWS_NEW_VOLUME_ID" | \ jq -r .Volumes[0].Attachments[0].Device )" &#125; function chroot &#123; requires AWS_NEW_VOLUME_ID || return # Use the /tmp/mounter script built by attachVolumeToSpot aws ec2 detach-volume --volume-id $AWS_NEW_VOLUME_ID aws ec2 wait volume-available --volume-id $AWS_NEW_VOLUME_ID &#125; function copyScriptToSpot &#123; requires AWS_ATTACH_VOLUME_DEVICE cat >/tmp/mounter &lt;&lt;EOF sudo mount "$&#123;AWS_ATTACH_VOLUME_DEVICE&#125;1" /mnt sudo mount -o bind /dev /mnt/dev sudo mount -o bind /dev/shm /mnt/dev/shm sudo mount -o bind /sys /mnt/sys sudo mount -o bind /run /mnt/run sudo mount -t proc proc /mnt/proc sudo mount -t devpts devpts /mnt/dev/pts sudo chroot /mnt # If the user types 'exit' then this script continues. # Otherwise, if the user types 'sudo halt' this script is not needed, AWS does the cleanup. sudo umount /mnt/dev sudo umount /mnt/dev/shm sudo umount /mnt/sys sudo umount /mnt/run sudo umount /mnt/proc sudo umount /mnt/dev/pts sudo umount /mnt EOF set -xv chmod a+x /tmp/mounter scpToSpot /tmp/mounter mounter echo "About to ssh into the spot instance. Run ./mounter to enter chroot using the new volume" sshToSpot # Proves the drive is mounted: #df -h | grep '^/dev/' | grep -v '^/dev/loop' rm /tmp/mounter &#125; function deleteEc2SpotInstance &#123; requires AWS_KEY_PAIR_NAME AWS_SPOT_ID || return # Hope that this does not bomb out if the user types 'sudo halt' aws ec2 cancel-spot-instance-requests --spot-instance-request-ids "$AWS_SPOT_ID" echo "Waiting for the EC2 spot instance $AWS_SPOT_ID to enter the stopped state." aws ec2 wait instance-stopped --instance-ids "$AWS_SPOT_ID" echo "The spot instance is no longer available. Deleting its keypair." aws ec2 delete-key-pair --key-name AWS_KEY_PAIR_NAME rm $AWS_KEY_PAIR_FILE $AWS_KEY_PAIR_FILE.pub &#125; function findEc2 &#123; echo "The existing AWS EC2 instance needs a snapshot to be made, which will then be turned into a volume and then mounted on a new AWS EC2 spot instance." echo "This script looks for an EC2 instance with a Name tag, so it can be identified." export AWS_EC2_ORIGINAL_NAME="$( readWithDefault "AWS EC2 Name tag value" production )" export AWS_EC2_ORIGINAL="$( aws ec2 describe-instances | \ jq ".Reservations[].Instances[] | select((.Tags[]?.Key==\"Name\") and (.Tags[]?.Value==\"$AWS_EC2_ORIGINAL_NAME\"))" )" export AWS_ORIGINAL_EC2_INSTANCE_ID="$( jq -r .InstanceId &lt;&lt;&lt; "$AWS_EC2_ORIGINAL" )" export AWS_ORIGINAL_EC2_IP="$( jq -r .PublicIpAddress &lt;&lt;&lt; "$AWS_EC2_ORIGINAL" )" AWS_ORIGINAL_VOLUME_ID="$( jq -r '.BlockDeviceMappings[].Ebs.VolumeId' &lt;&lt;&lt; "$AWS_EC2_ORIGINAL" )" export AWS_GROUP="$( jq -r ".NetworkInterfaces[].Groups[].GroupId" &lt;&lt;&lt; "$AWS_EC2_ORIGINAL" )" export AWS_SUBNET="$( jq -r ".SubnetId" &lt;&lt;&lt; "$AWS_EC2_ORIGINAL" )" export AWS_ZONE="$( jq -r .Placement.AvailabilityZone &lt;&lt;&lt; "$AWS_EC2_ORIGINAL" )" echo "Original EC2 instance $AWS_ORIGINAL_EC2_INSTANCE_ID is at IP address $AWS_ORIGINAL_EC2_IP, has EBS volume $AWS_ORIGINAL_VOLUME_ID, with security group $AWS_GROUP, in $AWS_SUBNET, in the $AWS_ZONE zone." &#125; function findSnapshot &#123; # Look for a snapshot previously created by this script export AWS_SNAPSHOT_ID="$( aws ec2 describe-snapshots \ --owner-ids self \ --filters Name=tag:Name,Values=TestScript | \ jq -r .Snapshots[].SnapshotId )" &#125; function findVolume &#123; # Look for a snapshot previously created by this script requires AWS_ZONE AWS_SNAPSHOT_ID || return export AWS_ATTACH_VOLUME_DEVICE="$( aws ec2 describe-volumes \ --filters Name=tag:Name,Values=TestScript | \ jq -r .Volumes[0].Attachments[0].Device )" &#125; function fn_help &#123; echo "Bash functions to make working with AWS EC2 easier via the command line. Source this file then call the functions: attachVolumeToSpot chroot copyScriptToSpot deleteEc2SpotInstance findEc2 findSnapshot findVolume latestUbuntuAmi makeEc2SpotInstance makeSnapshot makeVolumeFromSnapshot mountVolumeOnSpot scpToSpot sshToSpot Typical usage requires ' || true' to keep the terminal open if a problem occurs. This is unnecessary if you invoke from another bash script. source aws_ec2_functions findEc2 || true makeSnapshot || true makeVolumeFromSnapshot || true latestUbuntuAmi || true makeEc2SpotInstance || true attachVolumeToSpot || true copyScriptToSpot || true ... or, to perform all of the above: source aws_ec2_functions prepare_spot || true ... another way to perform all of the above: aws_ec2_functions run ... to pick up from a failed attempt, which created a snapshot and a volume, but did not make a spot instance, or the spot instance has been cancelled: findSnapshot || true findVolume || true latestUbuntuAmi || true makeEc2SpotInstance || true attachVolumeToSpot || true copyScriptToSpot || true " &#125; function latestUbuntuAmi &#123; export AWS_AMI="$( aws ec2 describe-images \ --owners 099720109477 \ --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-????????-????????-amd64-server-????????" \ "Name=state,Values=available" \ --query "reverse(sort_by(Images, &amp;CreationDate))[:1]" | \ jq -r '.[0]' )" export AWS_AMI_ID="$( jq -r '.ImageId' &lt;&lt;&lt; "$AWS_AMI" )" echo "The most recent Ubuntu AMI ID is $AWS_AMI_ID" &#125; function makeEc2SpotInstance &#123; requires AWS_AMI_ID AWS_GROUP AWS_SUBNET AWS_ZONE || return export AWS_KEY_PAIR_NAME="rsa-$( date '+%Y-%m-%d-%H-%M-%S' )" export AWS_KEY_PAIR_FILE="$HOME/.ssh/$AWS_KEY_PAIR_NAME" ssh-keygen -b 2048 -f "$AWS_KEY_PAIR_FILE" -P "" -N "" -t rsa # -q chmod 400 "$AWS_KEY_PAIR_FILE" aws ec2 import-key-pair \ --key-name "$AWS_KEY_PAIR_NAME" \ --public-key-material "fileb://$AWS_KEY_PAIR_FILE.pub" echo "EC2 spot instances are really inexpensive, so be generous with the size of the machine type for this spot instance." export AWS_EC2_TYPE="$( readWithDefault "EC2 machine type" t2.medium )" export AWS_SPOT_INSTANCE="$( aws ec2 run-instances \ --image-id $AWS_AMI_ID \ --instance-market-options '&#123; "MarketType": "spot" &#125;' \ --instance-type $AWS_EC2_TYPE \ --key-name $AWS_KEY_PAIR_NAME \ --network-interfaces "[ &#123; \"DeviceIndex\": 0, \"Groups\": [\"$AWS_GROUP\"], \"SubnetId\": \"$AWS_SUBNET\", \"DeleteOnTermination\": true, \"AssociatePublicIpAddress\": true &#125; ]" \ --placement "&#123; \"AvailabilityZone\": \"$AWS_ZONE\" &#125;" | \ jq -r .Instances[0] )" export AWS_SPOT_ID="$( jq -r .InstanceId &lt;&lt;&lt; "$AWS_SPOT_INSTANCE" )" echo "Waiting for the EC2 spot instance $AWS_SPOT_ID to enter the running state. This usually takes about 1 minute." aws ec2 wait instance-running --instance-ids "$AWS_SPOT_ID" export AWS_SPOT_IP="$( aws ec2 describe-instances \ --instance-ids $AWS_SPOT_ID | \ jq -r .Reservations[0].Instances[0].PublicIpAddress )" echo "The IP address of the new EC2 spot instance $AWS_SPOT_ID is $AWS_SPOT_IP." &#125; function makeSnapshot &#123; # Makes an AWS EC2 snapshot with name TestScript from AWS_ORIGINAL_VOLUME_ID requires AWS_ORIGINAL_VOLUME_ID || return COMPLETION_TIME="$(date --date="@$(($(date +%s)+120))" +"%H":"%M":"%S")" echo "Snapshots take about 2 minutes. This one should complete by $COMPLETION_TIME." export AWS_SNAPSHOT_ID="$( aws ec2 create-snapshot --volume-id "$AWS_ORIGINAL_VOLUME_ID" \ --description "production $( date '+%Y-%m-%d' )" \ --tag-specifications "ResourceType=snapshot,Tags=[&#123;Key=Created, Value=`date '+%Y-%m-%d'`&#125;,&#123;Key=Name, Value=\"TestScript\"&#125;]" | \ jq -r .SnapshotId )" aws ec2 wait snapshot-completed --snapshot-ids "$AWS_SNAPSHOT_ID" echo "Snapshot $AWS_SNAPSHOT_ID is complete." &#125; function makeVolumeFromSnapshot &#123; # Makes an AWS EC2 volume with name TestScript requires AWS_ZONE AWS_SNAPSHOT_ID || return export AWS_NEW_VOLUME_ID="$( aws ec2 create-volume \ --availability-zone $AWS_ZONE \ --snapshot-id $AWS_SNAPSHOT_ID \ --tag-specifications 'ResourceType=volume,Tags=[&#123;Key=Name,Value=TestScript&#125;]' | \ jq -r .VolumeId )" echo "Waiting for AWS volume $AWS_NEW_VOLUME_ID to become available. This usually takes about 20 seconds." aws ec2 wait volume-available --volume-id "$AWS_NEW_VOLUME_ID" echo "$AWS_NEW_VOLUME_ID" &#125; function prepare_spot &#123; # Run everything findEc2 makeSnapshot makeVolumeFromSnapshot latestUbuntuAmi makeEc2SpotInstance attachVolumeToSpot copyScriptToSpot &#125; function scpToSpot &#123; requires AWS_KEY_PAIR_FILE AWS_SPOT_IP || return scp -pi "$AWS_KEY_PAIR_FILE" "$1" "ubuntu@$AWS_SPOT_IP:$2" &#125; function sshToSpot &#123; requires AWS_KEY_PAIR_FILE AWS_SPOT_IP || return echo "When you are done, type: sudo halt." echo "The AWS EC2 spot instance will then terminate and be gone forever." echo "Any predefined resources, such as volumes that you attach will be freed." ssh -i "$AWS_KEY_PAIR_FILE" "ubuntu@$AWS_SPOT_IP" "$*" &#125; if [ "$1" == prepare_spot ]; then echo "Starting..." prepare_spot || true elif [ "$1" ]; then fn_help || true else echo "Type fn_help to obtain help information" fi </pre> <p>Make the script executable.</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1772f60c11af'><button class='copyBtn' data-clipboard-target='#id1772f60c11af' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>chmod a+x createEc2Spot</pre> <h3 class="numbered" id="aws_ec2_functions_usage">Sample Usage</h3> <p> View the help like this: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id98cced3d7117'><button class='copyBtn' data-clipboard-target='#id98cced3d7117' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws_ec2_functions -h <span class='unselectable'>Bash functions to make working with AWS EC2 easier via the command line. Source this file then call the functions, listed alphabetically: attachVolumeToSpot chroot copyScriptToSpot deleteEc2SpotInstance findEc2 findSnapshot findVolume latestUbuntuAmi makeEc2SpotInstance makeSnapshot makeVolumeFromSnapshot mountVolumeOnSpot scpToSpot sshToSpot Typical usage requires ' || true' to keep the terminal open if a problem occurs. This is unnecessary if you invoke from another bash script. source aws_ec2_functions findEc2 || true makeSnapshot || true makeVolumeFromSnapshot || true latestUbuntuAmi || true makeEc2SpotInstance || true attachVolumeToSpot || true copyScriptToSpot || true ... or, to perform all of the above: source aws_ec2_functions prepare_spot || true ... another way to perform all of the above: aws_ec2_functions run ... to pick up from a failed attempt, which created a snapshot and a volume, but did not make a spot instance, or the spot instance has been cancelled: findSnapshot || true findVolume || true latestUbuntuAmi || true makeEc2SpotInstance || true attachVolumeToSpot || true copyScriptToSpot || true </span></pre> <h2 class="numbered" id="chroot_usage">Using the chroot</h2> <p> Using either of the preceding two ways to set up the <code>chroot</code>, enter it using the <code>mounter</code> script: </p> <div class='codeLabel unselectable' data-lt-active='false'>shell&nbsp;on&nbsp;Spot&nbsp;Instance</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9c8d03ec1b20'><button class='copyBtn' data-clipboard-target='#id9c8d03ec1b20' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>ubuntu@ip-10-0-0-193:~$ </span>ls <span class='unselectable'>mounter ubuntu@ip-10-0-0-193:~$ </span>./mounter <span class='unselectable'>ubuntu@ip-10-0-0-193:~$ </span>echo "127.0.1.1 $(hostname)" >> /etc/hosts <span class='unselectable'>root@ip-10-0-0-193:/# </span>su ubuntu <span class='unselectable'>ubuntu@ip-10-0-0-193:/$ </span># Do whatever you need to do <span class='unselectable'>ubuntu@ip-10-0-0-193:/$ </span>sudo halt # Shut everything down</pre> </editor-fold Script> Scala-Style Lambda Function Placeholder Syntax in Python 3 2020-10-22T00:00:00-04:00 https://mslinn.github.io/blog/2020/10/22/scala-style-functional-programming-in-python-3 <p> Pipes are the ultimate in functional programming. It is clear when reading code that uses pipes that data is not mutated. Piping data into and out of lambda functions (and regular functions) is a succinct way of elegantly expressing a computation. This article discusses how to do this using similar syntax in Python 3, Scala 2 and Scala 3. </p> <p> After programming in Scala for more than ten years I have grown to appreciate Scala's implementation of lambda functions, including the ability to use the underscore character as a placeholder for variables. Scala 2.13 introduced pipelining between functions, which is rather like <a href='https://en.wikipedia.org/wiki/Unix-like' target='_blank' rel='nofollow'>*nix</a> pipes between processes. </p> <p> Python 3 can also do something similar. This article demonstrates how to use <a href='https://github.com/sspipe/sspipe' target='_blank' rel='nofollow'><code>sspipe</code></a> and <a href='https://github.com/JulienPalard/Pipe' target='_blank' rel='nofollow'>JulienPalard&rsquo;s <code>pipe</code></a> with Scala's underscore placeholder for <a href='https://stackoverflow.com/questions/29767310/pythons-lambda-with-underscore-for-an-argument' target='_blank' rel='nofollow'>Python 3 lambda functions</a>. </p> <h2 id="pythonSetup">Python 3 Setup</h2> The key concept is to use this specific Python import: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7cb0c475a5b6'><button class='copyBtn' data-clipboard-target='#id7cb0c475a5b6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>from sspipe import p, px, px as _</pre> <p> All the Python code examples that follow require this import. The Python code examples are modified versions of the <a href='https://github.com/sspipe/sspipe#examples' target='_blank' rel='nofollow'><code>sspipe</code> examples</a> to illustrate how to use underscores as placeholders. </p> <p> This import is unusual because it imports <code>px</code> twice: once as a normal import, and once aliased to <code>_</code>. I use the <code>_</code> alias to support Scala-like syntax, and I use <code>px</code> when I need to reference a parameter twice. Python is unlike Scala in that the Python compiler does not treat variables called <code>_</code> specially; those variables are merely called <code>_</code>. I could use <code>_</code> in Python code many times to refer to the same value, but a Scala programmer reading that code would expect that each reference to <code>_</code> would be another input parameter, not a regular variable reference. The examples that follow should make this clear. </p> <h3 id="installation">Python 3 Installation</h3> Install <code>sspipe</code> using <code>pip</code>: <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id233e093c01a4'><button class='copyBtn' data-clipboard-target='#id233e093c01a4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pip install --upgrade sspipe</pre> <h2 id="scalaSetup">Scala 2.13+ Setup</h2> <p> Scala 2.13 introduced <a href='https://www.scala-lang.org/api/current/scala/util/ChainingOps.html' target='_blank' rel='nofollow'><code>ChainingOps</code></a>, which adds chaining methods <code>tap</code> and <code>pipe</code> to every type. The key concept is to import the following prior to attempting the code examples below: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0272e59c14bf'><button class='copyBtn' data-clipboard-target='#id0272e59c14bf' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>implicit class Smokin[A](val a: A) { import scala.util.chaining._ import scala.language.implicitConversions implicit def |>[B](f: (A) => B): B = a.pipe(f) }</pre> <h2 id="examples">Python and Scala Usage Examples</h2> <h3 id="2-3">One Lambda Function and 1 Pipe</h3> <p> This Python example employs one lambda function and 1 pipe to add 2 to the number 5: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0ff65e682168'><button class='copyBtn' data-clipboard-target='#id0ff65e682168' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>5 | _ + 2 <span class='unselectable'>7 </span></pre> <p> The Scala equivalent of the above is: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcd00cd986a37'><button class='copyBtn' data-clipboard-target='#idcd00cd986a37' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>scala> </span>5 |> ((_: Int) + 2) <span class='unselectable'>val res0: Int = 7 </span></pre> <h3 id="2-3">Two Lambda Functions and 2 Pipes</h3> <p> This Python example employs two lambda functions and 2 pipes to multiply the previous result by 5 and then add the previous result. Recall that I said that in Python, an underscore when used this way is the name of a normal variable and that the compiler does not treat underscores as placeholders for lambda parameters. A Scala programmer would complain about the following code, because they would expect that the second lambda function would require 2 inputs: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3dc79b7f7ce8'><button class='copyBtn' data-clipboard-target='#id3dc79b7f7ce8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>5 | _ + 2 | _ * 5 + _ <span class='unselectable'>42 </span></pre> <p> A better way to write the above would be to use the special variable <code>px</code>, which was imported above. Now everyone either knows that <code>px</code> holds the piped value, or they complain about <code>px</code> being a magic variable. A possible solution to this complaint would be to alias <code>px</code> to a more descriptive name, such as <code>pipedValue</code> &mldr; which is still magical, but at least it is more descriptive. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide5742be68f12'><button class='copyBtn' data-clipboard-target='#ide5742be68f12' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>5 | _ + 2 | px * 5 + px <span class='unselectable'>42 </span></pre> The Scala equivalent of the above is: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida1cec3b5e716'><button class='copyBtn' data-clipboard-target='#ida1cec3b5e716' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>scala> </span>5 |> ((_: Int) + 2) |> ((x: Int) => x * 5 + x) <span class='unselectable'>val res1: Int = 42 </span></pre> <h3 id="2-3">Two Lambda Functions and 3 Pipes</h3> <p> This Python example employs 2 lambda functions and 3 pipes to add 10 to the even numbers from 0 to 5, exclusive. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6d63803141e9'><button class='copyBtn' data-clipboard-target='#id6d63803141e9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>( range(5) | p(filter, _ % 2 == 0) | p(map, _ + 10) | p(list) ) <span class='unselectable'>[10, 12, 14] </span></pre> <p> Scala has a better way of performing this type of computation that does not require pipes or computation. It is better because it is simpler to understand. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id98f82ad78162'><button class='copyBtn' data-clipboard-target='#id98f82ad78162' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>scala> </span>for { | x <- (0 until 5).toList if x % 2 == 0 | y = x + 10 | } yield y <span class='unselectable'>val res12: List[Int] = List(10, 12, 14) </span></pre> <h2 id="other">Other examples of placeholder syntax</h2> <p> <code>NumPy</code> expressions (NumPy is Python-specific): </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide6685cd2a128'><button class='copyBtn' data-clipboard-target='#ide6685cd2a128' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>range(10) | np.sin(_)+1 | p(plt.plot)</pre> <p> Pandas expressions (Pandas is Python-specific): </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id741c364a43d9'><button class='copyBtn' data-clipboard-target='#id741c364a43d9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>people_df | _.loc[_.age > 10, 'name']</pre> <p> Solution for the <a href='https://projecteuler.net/problem=2' target='_blank' rel='nofollow'>2nd Project Euler exercise</a>: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id44a627c63fc1'><button class='copyBtn' data-clipboard-target='#id44a627c63fc1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>def fib(): a, b = 0, 1 while True: yield a a, b = b, a + b <span class='unselectable'>>>> </span>euler2 = ( fib() | p.where(_ % 2 == 0) | p.take_while(_ < 4000000) | p.add() ) <span class='unselectable'>>>> </span>euler2 4613732</pre> <h2 id="dotty">Looking Ahead to Scala 3 (Dotty)</h2> <p> The next major version of Scala, due out in a few months, will probably allow a Scala 3 extension method to define the vertical bar as a method for more readabile code: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9cf35ae1f8ba'><button class='copyBtn' data-clipboard-target='#id9cf35ae1f8ba' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>def [A,B](a: A) |(f: (A) => B): B = a.pipe(f) # Sample usage: val x = 5 | doSomething | doSomethingElse | doSomethingMore</pre> <h2 id="scalacourses">To Learn More</h2> <p> My <a href='https://www.scalacourses.com/showCourse/40' target='_blank'>Introduction to Scala</a> course on ScalaCourses.com teaches Scala lambda functions. </p> Converting All Images in a Website to webp Format 2020-08-15T00:00:00-04:00 https://mslinn.github.io/blog/2020/08/15/converting-all-images-to-webp-format <p> I first launched this website in 1996. Since then, it has been re-incarnated using many different technologies. Presently I use <a href='https://jekyllrb.com/' target='_blank' rel='nofollow'>Jekyll</a> to assemble the site, then push the image to a web-enabled AWS S3 bucket that is edge-cached by an AWS CloudFront distribution. </p> <p> Until yesterday, the site contained images with a mixture of image formats. I decided to convert them all to the new <a href='https://developers.google.com/speed/webp' target='_blank' rel='nofollow'><code>webp</code></a> format. Because there are hundreds of images in over 120 web pages, I wrote a bash script called <code>toWebP</code> to do the work. This posting provides the <code>toWebP</code> script plus instructions on how you could use it for your website. </p> <p> The script converts image types <code>gif</code>, <code>jpg</code>, <code>jpeg</code>, <code>png</code>, <code>tif</code>, and <code>tiff</code>. It also modifies the HTML pages, CSS and SCSS that reference those images. </p> <p> The conversions are set for maximum fidelity (lossless where possible), and maximum compression. This means the images look great and load quickly. </p> <h3 id="caveat">Caveat</h3> <p> The script assumes that all images are local to your website, which makes sense because the converted images need to be stored, and local storage is the only sensible option. It renames all references to images in HTML, CSS and SCSS files to <code>webp</code> format. If the images are remote (for example, on a CDN), they are not converted, but the image file types in the HTML, CSS and SCSS are adjusted anyway. I suppose I could fix the script, but I don't need to do that for myself. If someone needs that feature, go ahead and enhance the script... and please provide me the enhanced script, so I can update this blog posting. </p> <h2 id="prerequisites">Prerequisites</h2> <p> You need to install the WebP package.<br> </p> <h3 id="mac">Mac</h3> <p> Use <a href='https://formulae.brew.sh/formula/webp' target='_blank' rel='nofollow'>Homebrew</a> or <a href='https://ports.macports.org/?search=webp&search_by=name' target='_blank' rel='nofollow'>Macports</a>. </p> <h3 id="ubuntu">Ubuntu (this is the default Linux distribution for Windows Subsystem for Linux)</h3> <p>At a shell prompt type:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0adf7e6c759b'><button class='copyBtn' data-clipboard-target='#id0adf7e6c759b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install webp</pre> <h2 id="running">Running <span class="code">toWebp</span></h2> <p> The program may emit warnings when it runs. Those warnings can be safely ignored. </p> <p> Hopefully, your website is managed by git. I suggest that you commit your work before running the script. That way if something goes wrong you just have to type <code>git stash</code> to return your website to its previous state. </p> <h3 id="usage">Usage</h3> <p>The general form of the command to convert all images and modify the HTML pages that they are referenced from is:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3a490bc43ff3'><button class='copyBtn' data-clipboard-target='#id3a490bc43ff3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>toWebp &lt;directoryName></pre> <h3 id="examples">Examples</h3> <p>To convert the website (images, html, scss & css) rooted at the current directory, type:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd0cea7d4d247'><button class='copyBtn' data-clipboard-target='#idd0cea7d4d247' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>toWebp .</pre> <p>To convert the website called <code>mySite</code> rooted under your home directory, type:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf1b4f0416a70'><button class='copyBtn' data-clipboard-target='#idf1b4f0416a70' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>toWebp ~/mySite</pre> <p>To just convert 1 specific image to <code>webp</code>, type:</p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id19a19201e121'><button class='copyBtn' data-clipboard-target='#id19a19201e121' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>toWebp images/blah.jpg</pre> <h2 id="toWebP">The <span class="code">toWebP</span> Bash Script.</h2> <p>Put this file in one of the directories on your <code>PATH</code>, for example <code>/usr/local/bin</code>: <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,toWebp' download='toWebp' title='Click on the file name to download the file'>toWebp</a> </div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id9d9eb801c4ab">#!/bin/bash # Convert all images to webp files in place and update HTML, CSS and SCSS to suit # Usage: toWebp directory|imageFileName # Example: toWebp . # convert images, html, scss &amp; css in current directory tree # Example: toWebp images/blah.jpg # just convert 1 specific image # # SPDX-License-Identifier: Apache-2.0 shopt -s extglob export CMD="cwebp alpha_q 10 -exact -lossless -m 6 -short -q 100 -z 9" function checkDependencies &#123; if [ -z "$( which cwebp )" ]; then echo "Installing webp" yes | sudo apt install webp fi &#125; function convertGifs &#123; for F in $( find "$1" -iname '*.gif' ); do TOF="$&#123;F%.*&#125;.webp" gif2webp -loop_compatibility -m 6 -mixed "$F" -q 100 -o "$TOF" done &#125; function convertMost &#123; # See "Extended pattern" extglob for Bash # https://wiki.bash-hackers.org/syntax/pattern#extended_pattern_language cd "$1" for F in $( listImages ); do TOF="$&#123;F%.*&#125;.webp" echo "Converting '$F' to '$TOF'" # Warning messages might be emitted, but don't worry $CMD "$F" -o "$TOF" if [ "$DELETE_OLD" ]; then rm "$F"; fi done cd - &#125; function listImages &#123; # Does not return gifs because cwebp cannot handle them find . -iregex '.*\.\(jpg\|png\|jpeg\|tif\|tiff\)' -printf '%f\n' &#125; function renameImage &#123; sed -i "s,\b$1\b,.webp,g" "$2" &#125; function swapImages &#123; for F in $( find "$1" -iname '*.html' -o -iname '*.css' -o -iname '*.scss' ); do for X in .gif .jpg .jpeg .tif .tiff .png; do echo "Swapping $X images for .webp in '$F'" renameImage "$X" "$F" done done &#125; # Set cwd to project root GIT_ROOT="$( git rev-parse --show-toplevel )" cd "$&#123;GIT_ROOT&#125;" || exit source _bin/loadConfigEnvVars checkDependencies unset DELETE_OLD # TODO make this a command line option if [ -f "$1" ]; then # just convert a single image to webp F="$1" $CMD "$F" -o "$&#123;F%.*&#125;.webp" if [ "$DELETE_OLD" ]; then rm "$F"; fi elif [ -d "$1" ]; then # process all images, css and scss in directory shopt -s globstar convertMost "$1" convertGifs "$1" swapImages "$1" else >&amp;2 echo "Error: you must either specify a valid file or a directory" fi </pre> <h3 id="chmod">Make it Executable</h3> <p> Remember to make the <code>toWebp</code> script executable before trying to use it: </p> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div><pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcf9162576e3e'><button class='copyBtn' data-clipboard-target='#idcf9162576e3e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>chmod a+x /usr/local/bin/toWebp</pre> Dotty (Scala 3 Preview) Presentation at Hopper, Montreal 2019-11-28T00:00:00-05:00 https://mslinn.github.io/blog/2019/11/28/dotty-scala-3-preview <div style=""> <picture> <source srcset="/blog/images/dotty/dottyLambda_690x388.webp" type="image/webp"> <source srcset="/blog/images/dotty/dottyLambda_690x388.png" type="image/png"> <img src="/blog/images/dotty/dottyLambda_690x388.png" title="Mike Slinn presents" class=" liImg " alt="Mike Slinn presents" /> </picture> </div> <div style="text-align: center"> <p> Yesterday I presented <a href='https://www.meetup.com/lambda-montreal/events/266306046/' target='_blank'>Dotty (Scala 3 Preview)</a> to Lambda Montreal. </p> <p> The slides are <a href='https://www.slideshare.net/mslinn/dotty-scala-3-preview' target='_blank'>here</a>. </p> <p> The code is <a href='https://github.com/mslinn/dotty-example-project/' target='_blank'>here</a>. </p> <p> The video recording is <a href='https://www.youtube.com/watch?v=7S68TY0S2e0' target='_blank'>here</a>. </p> </div> <div style="text-align: center;"> <picture> <source srcset="/blog/images/dotty/lambdaMontreal.webp" type="image/webp"> <source srcset="/blog/images/dotty/lambdaMontreal.png" type="image/png"> <img src="/blog/images/dotty/lambdaMontreal.png" title="Mike Slinn presents" class="center quartersize liImg2 rounded shadow" alt="Mike Slinn presents" /> </picture> </div> A Hybrid Machine Learning / Personality Simulation Platform 2019-10-24T00:00:00-04:00 https://mslinn.github.io/blog/2019/10/24/hybrid-ml-simulation <div style="text-align: center;"> <a href="https://www.meetup.com/MTL-Machine-Learning/events/265039754/" target="_blank" rel="nofollow"><picture> <source srcset="/blog/images/empathyWorks/mtlMLconference.webp" type="image/webp"> <source srcset="/blog/images/empathyWorks/mtlMLconference.png" type="image/png"> <img src="/blog/images/empathyWorks/mtlMLconference.png" class="center liImg2 rounded shadow" /> </picture></a> </div> <p> Yesterday I presented <a href='https://www.meetup.com/MTL-Machine-Learning/events/265039754/' target='_blank' rel='nofollow'>&ldquo;EmpathyWorks: A Hybrid Machine Learning / Personality Simulation Platform&rdquo;</a> to the <a href='https://www.meetup.com/MTL-Machine-Learning/events/265039754/' target='_blank' rel='nofollow'>Fall 2019 Montreal Machine Learning Mini-Conference</a>. </p> <p> The slides are <a href='https://www.slideshare.net/mslinn/empathyworks-towards-an-eventbased-simulationml-hybrid-platform' target='_blank' rel='nofollow'>here</a>. The video recording is <a href='https://youtu.be/PiDsiyJIMmo' target='_blank' rel='nofollow'>here</a>. </p> <div style="text-align: center"> <div style="display: inline-block; margin: 0.5em; vertical-align: top;"> <picture> <source srcset="/assets/images/robotCircle207x207.webp" type="image/webp"> <source srcset="/assets/images/robotCircle207x207.png" type="image/png"> <img src="/assets/images/robotCircle207x207.png" title="Mike Slinn presents" class=" liImg " alt="Mike Slinn presents" /> </picture> </div> <div style="display: inline-block; margin: 0.5em; vertical-align: top;"> <picture> <source srcset="/blog/images/empathyWorks/montrealMachineLearningMeetup.webp" type="image/webp"> <source srcset="/blog/images/empathyWorks/montrealMachineLearningMeetup.png" type="image/png"> <img src="/blog/images/empathyWorks/montrealMachineLearningMeetup.png" title="Montreal Machine Learning Meetup" class=" quartersize liImg2 rounded shadow" style="margin-left: 3em; margin-top: 3em;" alt="Montreal Machine Learning Meetup" /> </picture> </div> </div> Decentralized Ponytails 2018-09-13T00:00:00-04:00 https://mslinn.github.io/blog/2018/09/13/decentralized-ponytails <p> I&rsquo;d like to point out the similarity of the early days of the open-source movement with today&rsquo;s decentralized blockchain movement. </p> <p> Open-source software was brought to mainstream attention during the last technology bubble at the end of the last millennium. The open-source software movement had a loyal cadre of zealots who believed that their cause would overcome any need for a business case. Sun Microsystems was the hardware company whose servers powered the Internet, and their software included the Java programming language, plus many other important networking-related products. Sun's slogan was &ldquo;The network is the computer&rdquo;. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/ponytails/Sun-Logo_225x99.webp" type="image/webp"> <source srcset="/blog/images/ponytails/Sun-Logo_225x99.png" type="image/png"> <img src="/blog/images/ponytails/Sun-Logo_225x99.png" title="Sun Microsystems logo" class="center quartersize liImg2 rounded shadow" style="padding: 1em" alt="Sun Microsystems logo" /> </picture> </div> <p> Jonathan Schwartz, the CEO of Sun Microsystems was one of the open-source zealots. He was famous for his ponytail. Unfortunately, zealotry and dogma is bad for business, and as a result Sun Microsystems is no longer with us. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/ponytails/jonathanSchwartz.webp" type="image/webp"> <source srcset="/blog/images/ponytails/jonathanSchwartz.png" type="image/png"> <img src="/blog/images/ponytails/jonathanSchwartz.png" title="Jonathan Schwartz and his ponytail" class="center liImg rounded shadow" alt="Jonathan Schwartz and his ponytail" /> </picture> </div> <p> Eventually companies like <a href='https://redhat.com' target='_blank' rel='nofollow'>Red Hat</a> developed solid business models for open-source software, but that took years to develop. Today we see many companies attempting using decentralized blockchain technology to create cryptocurrencies, other token-based economies, and evangelizing decentralized dogma without a solid business case. Most of these ventures will die a horrible death, and the investors will get nothing. It will take years for solid business models based on decentralization to be proven. </p> <p> Mr. Schwartz's ponytail was the fashion statement that fueled the YouTube parody below. For background, <a href='https://en.wikipedia.org/wiki/Scott_McNealy' target='_blank' rel='nofollow'>Scott McNealy</a> was the previous CEO at Sun Microsystems. </p> <iframe width="690" height="388" src="https://www.youtube.com/embed/5r3JSciJf5M" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen class="rounded shadow liImg"></iframe> <p> Full disclosure: I also had a ponytail in 2008, and for a few years I had my Sun Spark 2 workstation at home. </p> <div style="text-align: center;"> <picture> <source srcset="/images/mike/mikeclose3.webp" type="image/webp"> <source srcset="/images/mike/mikeclose3.png" type="image/png"> <img src="/images/mike/mikeclose3.png" title="Mike Slinn and his ponytail back in 2008" class="center quartersize liImg2 rounded shadow" alt="Mike Slinn and his ponytail back in 2008" /> </picture> </div> <p> Here is a transcription of the video, which I paraphrased for clarity:</p> </p> <div class="quote"> <p><b>Steve Gilmore:</b> Hi, this is Steve Gilmore and this is a video special edition of the Gilmore gang. I'm here with Jonathan Schwartz. It's a great pleasure &ndash; it's been a long, long time coming &ndash; I haven't seen Jonathan for quite a while. Jonathan Schwartz, who is the president and CEO of Sun Microsystems, agreed to sit down for the first time in three or four years and talk about what's going on with Sun. I wanna start, Jonathan, by thanking you for joining us. </p> <p><b>Jonathan Schwartz:</b> Thank you for having me, Steve. It has been a long time, nice to see you. </p> <p><b>Steve Gilmore:</b> So, you know there's been a lot of turmoil on Wall Street as I know you know. </p> <p><b>Jonathan Schwartz:</b> Yes. </p> <p><b>Steve Gilmore:</b> What's your take on that? </p> <p><b>Jo