Mike Slinn's Blog 2024-04-22T12:13:49-04:00 https://mslinn.github.io/blog Mike Slinn mslinn@gmail.com Product Development Collaboration 2024-02-18T00:00:00-05:00 https://mslinn.github.io/blog/2024/02/18/collaboration <!-- #region intro --> <p> For product development, the goal of effective collaboration is to fully engage all parties. Good product-oriented collaborators maintain a flow of information, ideas and commensurate action. Hesitation is minimized. Feedback is rapid. Reactions are prompt, yet measured through constant practice. </p> <!-- endregion --> <!-- #region Product Collaborator Checklist --> <h2 id="check">Product Collaborator Checklist</h2> <p> The following criteria are a guide to effective collaboration candidates for product development. </p> <!-- #region Desirable Traits --> <h3 id="destr">Desirable Traits</h3> <ul> <li> Strong interest in optimizing the architecture and engineering processes. </li> <li> A focus on the contributing to the desired results for the team, which are: reliably providing an easily understood and valuable benefit to a well-defined customer base. </li> <li> Manage the difference between desired outcomes and actual outcomes. </li> <li> Strive for process and product consistency. </li> <li> Willing to expend the overhead of time and energy required for effective communication: <ul> <li>Driving their activities from a Kanban-style status board</li> <li>Planning interactions with others, minimizing impulsive chatter</li> <li>Preparing and distributing notes before meetings</li> <li>Describing alternatives for decisions</li> <li>Documenting the rationale for design decisions</li> <li>Summarizing discussions in project documentation</li> <li>Contributing to an up-to-date product roadmap</li> </ul> </li> </ul> <!-- endregion --> <!-- #region Undesirable Traits --> <h3 id="undestr">Undesirable Traits</h3> <p> People with the following traits disqualify themselves from being a member of the core team because they are unable to consistently contribute effectively to the ever-evolving vision for the project. </p> <ul> <li> "Guns for hire" just want to get paid for time spent; they have no vested interest in the long-term outcome. They are tools to be exploited, not valuable resources to be cultivated. The best use of a mercenary is to give them orders, supervise them, learn from them, and then dispose of them as soon as possible. </li> <li> "Putting in time" demonstrates a lack of personal engagement. Those people require constant supervision, and the cost of the supervision must be subtracted from any value they might provide when computing their net contribution. </li> </ul> <!-- endregion --> <!-- #region Software Architecture and Engineering Processes --> <h2 id="defs">Software Architecture and Engineering Processes</h2> <p> ChatGPT gave me the following response when I asked, &ldquo;Compare and contrast the processes of architecture and engineering for software&rdquo;. I like the quality of this response, and provide it verbatim because it exactly matches my understanding of those terms. </p> <p> When it comes to software development, there are parallels between the processes of architecture and engineering, but there are also significant differences due to the intangible and rapidly evolving nature of software. Let's compare and contrast the processes of software architecture and engineering: </p> <h3 id="pf">Purpose and Focus</h3> <p> Software Architecture: Similar to architectural design in physical structures, software architecture focuses on designing the overall structure and organization of a software system. It involves making high-level decisions about the system's components, their interactions, and the principles that guide their design. </p> <p> Software Engineering: Software engineering encompasses the systematic approach to developing software solutions. It involves various disciplines such as requirements analysis, design, coding, testing, deployment, and maintenance. The focus is on creating reliable, efficient, and maintainable software systems. </p> <h3 id="dp">Design Process</h3> <p> Software Architecture: The architectural design process involves defining the system's key components, interfaces, and their interactions. Architectural patterns, such as layered architecture, microservices, or event-driven architecture, are often used to guide the design. Considerations include scalability, performance, security, and maintainability. </p> <p> Software Engineering: Software engineering encompasses the entire software development lifecycle (SDLC). It begins with requirements gathering and analysis, followed by design, implementation (coding), testing, deployment, and maintenance. Various methodologies like Waterfall, Agile, and DevOps guide this process. </p> <h3 id="ct"> Creativity vs. Technicality</h3> <p> Software Architecture: Software architects must balance creative design decisions with technical constraints. They need to envision the overall structure of the system and choose appropriate technologies and patterns to achieve the desired functionality and quality attributes. </p> <p> Software Engineering: Software engineers focus on implementing the design created by architects. While creativity is still important, engineers primarily deal with the technical aspects of coding, testing, and debugging. They translate the architectural design into working software components. </p> <h3 id="coll">Collaboration</h3> <p> Software Architecture: Collaboration between architects, stakeholders, and development teams is crucial to ensure that architectural decisions align with business goals and technical requirements. Architects often communicate their designs through diagrams, documentation, and discussions. </p> <p> Software Engineering: Software engineers collaborate closely with other team members, including architects, designers, testers, and project managers. They work together to implement features, resolve technical challenges, and deliver software increments according to project timelines. </p> <h3>Regulatory and Environmental Considerations</h3> <p> Software Architecture: Architectural decisions may need to comply with regulatory standards, industry best practices, and security guidelines. Architects must consider factors like data privacy, compliance requirements (such as GDPR or HIPAA), and interoperability with other systems. </p> <p> Software Engineering: Engineers implement features and functionalities while adhering to coding standards, security protocols, and testing practices. They ensure that the software meets quality standards, performance targets, and regulatory requirements specified by the architecture and project stakeholders. </p> <p> In summary, while both software architecture and engineering involve designing and building software systems, they have distinct focuses and processes. Architecture concentrates on high-level design decisions and system structure, while engineering encompasses the entire software development lifecycle, from requirements to deployment. Collaboration and communication are essential in both disciplines to deliver successful software solutions.</p> </p> <!-- endregion --> <!-- #region Collaboration --> <h2 id="collabl">Collaboration</h2> <p> ChatGPT defined collaboration as the following. Again, I very much like the response, so I provide it verbatim. </p> <p class="quote"> Collaboration refers to the process of individuals or groups working together to achieve a common goal or objective. It involves the sharing of ideas, resources, responsibilities, and tasks in a coordinated manner to accomplish tasks more effectively and efficiently than if done individually. <br><br> Collaboration often emphasizes communication, cooperation, and mutual respect among participants. It can occur within various contexts, including businesses, organizations, teams, communities, and even across different disciplines or sectors. <br><br> Successful collaboration typically results in synergy, where the combined efforts of the participants produce outcomes that exceed what could be achieved by any individual or subgroup working alone. </p> <p> Following is the response when I asked "what is required for effective collaboration in a new software project?" I <span class="bg_yellow">highlighted</span> a passage that is important, and <s>struck out</s> portions that I disagree with. </p> <p> Effective collaboration in a new software project requires several key elements: </p> <ol> <li> <b>Clear Goals and Objectives</b>: Establishing clear, well-defined goals and objectives for the project ensures that all collaborators understand what they are working towards and why. </li> <li> <b>Communication Channels</b>: Implementing reliable communication channels is essential for keeping all team members informed and facilitating the exchange of ideas, updates, and feedback. This can include tools like instant messaging, email, project management software, or regular team meetings. </li> <li> <b>Defined Roles and Responsibilities</b>: Clearly defining roles and responsibilities for each team member helps avoid confusion and ensures that tasks are completed efficiently. This includes designating a project manager, developers, testers, designers, and other relevant roles. </li> <li> <b>Collaboration Tools</b>: Utilizing collaboration tools such as version control systems (e.g., Git), issue tracking systems <s>(e.g., Jira)</s>, document sharing platforms (e.g., Google Drive), and collaborative coding environments (e.g., GitHub) can streamline the development process and enhance teamwork. </li> <li class="bg_yellow"> <b>Regular Feedback and Reviews</b>: Encouraging open and constructive feedback throughout the project lifecycle allows team members to address issues promptly, make necessary adjustments, and continuously improve the software. </li> <li class="bg_yellow"> <b>Flexibility and Adaptability</b>: Remaining flexible and adaptable to changes in requirements, timelines, and priorities is crucial for navigating unforeseen challenges and ensuring the project stays on track. </li> <li> <b>Respect and Trust</b>: Cultivating an environment of respect and trust among team members fosters collaboration and encourages open communication, creativity, and innovation. </li> <li class="bg_yellow"> <b>Documentation</b>: Maintaining thorough documentation of project requirements, design decisions, codebase, and processes helps ensure continuity and facilitates knowledge transfer among team members. </li> <li> <b>Regular Progress Tracking</b>: Implementing mechanisms to track progress, <s>such as sprint planning, daily stand-up meetings, and</s> regular status reports, enables the team to monitor project milestones, identify potential bottlenecks, and make necessary adjustments to stay on schedule. </li> <li> <b>Quality Assurance and Testing</b>: Integrating quality assurance processes and testing procedures into the development workflow ensures that the software meets the specified requirements, functions as intended, and delivers value to end-users. </li> </ol> <p> By incorporating these elements into the collaborative process, teams can increase the likelihood of success in their new software projects. </p> <!-- endregion --> <!-- #region Design Decisions --> <h2 id="dd">Design Decisions</h2> <p> I asked ChatGPT "how would someone document design decisions in a software project?" The highlighted portion is covered in more detail later in this article. </p> <p> Documenting design decisions in a software project is essential for ensuring clarity, facilitating collaboration, and providing a reference for future development. Here's a structured approach to documenting design decisions: </p> <ol> <li> <b>Establish a Design Document</b>: Create a designated document or section within project documentation specifically for recording design decisions. This could be a separate document, a section in the project wiki, or part of the version control repository. </li> <li> <b>Provide Context</b>: Start by providing context for the decision. Explain the problem or requirement that the design decision aims to address. Include any relevant background information, constraints, or considerations that influenced the decision-making process. </li> <li class="bg_yellow"> <b>Describe Alternatives</b>: Outline the various alternatives considered during the decision-making process. This could include different approaches, technologies, architectures, or design patterns. Briefly discuss the pros and cons of each alternative. </li> <li> <b>Justify the Chosen Solution</b>: Explain why the chosen solution was selected over the alternatives. Highlight the reasons, benefits, and trade-offs associated with the chosen approach. Consider factors such as performance, scalability, maintainability, ease of implementation, and alignment with project goals. </li> <li> <b>Detail the Design</b>: Provide a detailed description of the chosen design solution. Break down the design into its constituent components, modules, or layers. Use diagrams, flowcharts, UML diagrams, or other visual aids to illustrate the architecture, structure, and relationships between components. </li> <li> <b>Document Dependencies and Interactions</b>: Document dependencies between different components or modules within the design. Describe how these components interact with each other and with external systems or services. Identify any interfaces, APIs, or protocols used for communication. </li> <li> <b>Address Risks and Mitigations</b>: Identify potential risks or challenges associated with the chosen design. Discuss strategies for mitigating these risks and ensuring the robustness and resilience of the design solution. </li> <li> <b>Include Implementation Details</b>: If applicable, include implementation details or guidelines for developers to follow when implementing the design. This could include coding standards, best practices, design patterns, or frameworks to be used. </li> <li> <b>Update and Maintain</b>: Keep the design document up to date as the project evolves. Update it to reflect any changes, refinements, or new decisions made during the development process. Ensure that the document remains a reliable reference for developers and stakeholders. </li> <li> <b>Review and Iterate</b>: Encourage team members to review the design document periodically and provide feedback. Iterate on the design as necessary based on feedback, changes in requirements, or lessons learned from implementation. </li> </ol> <p> By following these steps, you can effectively document design decisions in a software project, providing a comprehensive and structured reference for developers, stakeholders, and future contributors. </p> <!-- endregion --> <!-- #region Comparative Product Matrix --> <h2 id="mat">Comparative Product Matrix</h2> <p> Comparative product matrices are useful for guiding and documenting design decisions. Search for them when you need to identify pre-existing solutions to a problem, and generate them when considering the implementation of various approaches. </p> <div class='imgWrapper imgFlex inline' style=' '> <figure> <a href='https://github.com/snowfort-ai/awesome-llm-webapps?tab=readme-ov-file#project-table' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/comprative_product_matrix.svg" type="image/svg"> <!---<source srcset="/blog/images/comprative_product_matrix.avif" type="image/avif">--> <source srcset="/blog/images/comprative_product_matrix.webp" type="image/webp"> <source srcset="/blog/images/comprative_product_matrix.apng" type="image/apng"> <source srcset="/blog/images/comprative_product_matrix.png" type="image/png"> <source srcset="/blog/images/comprative_product_matrix.jpg" type="image/jpeg"> <source srcset="/blog/images/comprative_product_matrix.jpeg" type="image/jpeg"> <source srcset="/blog/images/comprative_product_matrix.jfif" type="image/jpeg"> <source srcset="/blog/images/comprative_product_matrix.pjpeg" type="image/jpeg"> <source srcset="/blog/images/comprative_product_matrix.pjp" type="image/jpeg"> <source srcset="/blog/images/comprative_product_matrix.gif" type="image/gif"> <source srcset="/blog/images/comprative_product_matrix.tif" type="image/tiff"> <source srcset="/blog/images/comprative_product_matrix.tiff" type="image/tiff"> <source srcset="/blog/images/comprative_product_matrix.bmp" type="image/bmp"> <source srcset="/blog/images/comprative_product_matrix.ico" type="image/x-icon"> <source srcset="/blog/images/comprative_product_matrix.cur" type="image/x-icon"> <img alt='Comparative product matrix' class="imgImg " src="/blog/images/comprative_product_matrix.png" style='width: 100%; ' title='Comparative product matrix' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://github.com/snowfort-ai/awesome-llm-webapps?tab=readme-ov-file#project-table" target='_blank' > Comparative product matrix </a> </figcaption> </figure> </div> <p> Comparative product matrices take time to read and to prepare, but they help guide the team to the best technical decisions. Without them, you are just hoping to get lucky. </p><!-- endregion --> Expanding the WSL virtual hard drive 2024-02-12T00:00:00-05:00 https://mslinn.github.io/blog/2024/02/12/wsl-expand <!-- #region intro --> <p> I ran out of room on my WSL2 virtual hard drive. <a href='https://learn.microsoft.com/en-us/windows/wsl/disk-space#how-to-expand-the-size-of-your-wsl-2-virtual-hard-disk' target='_blank' rel="nofollow">Microsoft&rsquo;s instructions</a> worked perfectly to increase the virtual drive size. </p> <!-- endregion --> <!-- #region Shut Down WSL --> <h2 id="stop">Shut Down WSL</h2> <p> The following command work from WSL bash, and cmd, and PowerShell: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id48c2b4d17c70'><button class='copyBtn' data-clipboard-target='#id48c2b4d17c70' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>wsl.exe --shutdown</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Expand VHD With Diskpart --> <h2 id="Diskpart">Expand VHD With Diskpart</h2> <p style="text-align: left;"> The virtual disk was in <code>%LocalAppData%\<wbr>Packages\<wbr>CanonicalGroupLimited.<wbr>UbuntuonWindows_<wbr>79rhkp1fndgsc\<wbr>LocalState\<wbr>ext4.vhdx</code> </p> <p> Open a <code>cmd.exe</code> window with admininstrator privileges and run <code>diskpart</code>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>diskpart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc3ddc23d419f'><button class='copyBtn' data-clipboard-target='#idc3ddc23d419f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>Select vdisk file="%LocalAppData%\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState\ext4.vhdx"<br> <span class='unselectable'>DISKPART> </span>detail vdisk <span class='unselectable'>Device type ID: 0 (Unknown) Vendor ID: {00000000-0000-0000-0000-000000000000} (Unknown) State: Added Virtual size: 256 GB Physical size: 247 GB Filename: C:\Users\Mike Slinn\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState\ext4.vhdx Is Child: No Parent Filename: Associated disk#: Not found. </span></pre> </div> <!-- endregion --> <p> The virtual disk was 256 GB (256,000 MB). I want it to be twice as big: 512,000 MB. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>diskpart (continued)</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4475bcdf96ec'><button class='copyBtn' data-clipboard-target='#id4475bcdf96ec' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>expand vdisk maximum=512000<br> <span class='unselectable'>100 percent completed<br> DiskPart successfully expanded the virtual disk file. </span> <span class='unselectable'>DISKPART> </span>exit<br> <span class='unselectable'>Leaving DiskPart... </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Restart WSL and resize2fs --> <h2 id="restart">Restart WSL and Run <span class="code">resize2fs</span></h2> <p> Now I restarted WSL from a <code>cmd</code> window. The following command work from WSL bash, and cmd, and PowerShell: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9521fd6a3e6c'><button class='copyBtn' data-clipboard-target='#id9521fd6a3e6c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>wsl.exe</pre> </div> <!-- endregion --> <p> Now tell Linux to use all available space: </p> <!-- #region --> <div class="jekyll_pre" style='margin-bottom: 1em;'> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida4e066fb691e'><button class='copyBtn' data-clipboard-target='#ida4e066fb691e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo resize2fs /dev/sdc 512000M <span class='unselectable'>resize2fs 1.47.0 (5-Feb-2023) Filesystem at /dev/sdc is mounted on /; on-line resizing required old_desc_blocks = 32, new_desc_blocks = 63 The filesystem on /dev/sdc is now 131072000 (4k) blocks long. </span></pre> </div> <!-- endregion --> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <p> Done! </p> <!-- endregion --> Installing JDK 17 on Ubuntu 2023-09-28T00:00:00-04:00 https://mslinn.github.io/blog/2023/09/28/jdk <!-- #region install --> <p> The Java Development Kit (JDK) version 17 is compatible with v11 and v8. Installing the JDK also installs the JRE, as you can see from the highlighted dependency shown below: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id47b00e02b277'><button class='copyBtn' data-clipboard-target='#id47b00e02b277' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>apt show -a openjdk-17-jdk <span class='unselectable'>Package: openjdk-17-jdk Version: 17.0.8.1+1~us1-0ubuntu1~23.04 Priority: optional Section: java Source: openjdk-17 Origin: Ubuntu Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com> Original-Maintainer: OpenJDK Team <openjdk-17@packages.debian.org> Bugs: https://bugs.launchpad.net/ubuntu/+filebug Installed-Size: 1544 kB Provides: java-compiler, java-sdk (= 17), java10-sdk, java11-sdk, java12-sdk, java13-sdk, java14-sdk, java15-sdk, java16-sdk, java17-sdk, java2-sdk, java5-sdk, java6-sdk, java7-sdk, java8-sdk, java9-sdk Depends: <span class="bg_yellow">openjdk-17-jre (= 17.0.8.1+1~us1-0ubuntu1~23.04)</span>, openjdk-17-jdk-headless (= 17.0.8.1+1~us1-0ubuntu1~23.04), libc6 (>= 2.34), zlib1g (>= 1:1.1.4) Recommends: libxt-dev Suggests: openjdk-17-demo, openjdk-17-source, visualvm Homepage: https://openjdk.java.net/ Download-Size: 1486 kB APT-Sources: http://archive.ubuntu.com/ubuntu lunar-updates/main amd64 Packages Description: OpenJDK Development Kit (JDK) OpenJDK is a development environment for building applications, applets, and components using the Java programming language.<br> Package: openjdk-17-jdk Version: 17.0.6+10-1ubuntu2 Priority: optional Section: java Source: openjdk-17 Origin: Ubuntu Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com> Original-Maintainer: OpenJDK Team <openjdk-17@packages.debian.org> Bugs: https://bugs.launchpad.net/ubuntu/+filebug Installed-Size: 4732 kB Provides: java-compiler, java-sdk, java10-sdk, java11-sdk, java12-sdk, java13-sdk, java14-sdk, java15-sdk, java16-sdk, java17-sdk, java2-sdk, java5-sdk, java6-sdk, java7-sdk, java8-sdk, java9-sdk Depends: openjdk-17-jre (= 17.0.6+10-1ubuntu2), openjdk-17-jdk-headless (= 17.0.6+10-1ubuntu2), libc6 (>= 2.34) Recommends: libxt-dev Suggests: openjdk-17-demo, openjdk-17-source, visualvm Homepage: https://openjdk.java.net/ Download-Size: 4585 kB APT-Sources: http://archive.ubuntu.com/ubuntu lunar/main amd64 Packages Description: OpenJDK Development Kit (JDK) OpenJDK is a development environment for building applications, applets, and components using the Java programming language. </span></pre> </div> <!-- endregion --> <p> Install OpenJDK 17 and its dependencies: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5ab2fbe1b433'><button class='copyBtn' data-clipboard-target='#id5ab2fbe1b433' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install openjdk-17-jdk</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region set default java --> <h2 id="def">Setting the Default Java Version</h2> <p> The man page for <code>update-java-alternatives</code> is: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idef5258f65c6f'><button class='copyBtn' data-clipboard-target='#idef5258f65c6f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>man update-java-alternatives <span class='unselectable'>UPDATE-JAVA-ALTERNATIVESystem Manager&#39;s MUPDATE-JAVA-ALTERNATIVES(8)<br/> NAME update-java-alternatives - update alternatives for jre/sdk installations<br/> SYNOPSIS update-java-alternatives [--jre] [--plugin] [-v|--verbose] -l|--list [&lt;jname&gt;] -s|--set &lt;jname&gt; -a|--auto -h|-?|--help<br/> DESCRIPTION update-java-alternatives updates all alternatives belonging to one runtime or development kit for the Java language. A package does provide these information of it&#39;s alternatives in /usr/lib/jvm/.&lt;jname&gt;.jinfo.<br/> OPTIONS -l|--list [&lt;jname&gt;] List all installed packages (or just &lt;jname&gt;) provid&#8208; ing information to set a bunch of java alternatives. Verbose output shows each alternative provided by the packages.<br/> -a|--auto Switch all alternatives of registered jre/sdk instal&#8208; lations to automatic mode.<br/> -s|--set &lt;jname&gt; Set all alternatives of the registered jre/sdk instal&#8208; lation to the program path provided by the &lt;jname&gt; in&#8208; stallation.<br/> --jre Limit the actions to alternatives belong to a runtime environment, not a development kit.<br/> --jre-headless Limit the actions to alternatives belong to the head&#8208; less part of a runtime environment.<br/> --plugin Limit the actions to alternatives providing browser plugins.<br/> -h|--help Display a help message.<br/> -v|--verbose Verbose output.<br/> FILES /usr/lib/jvm/.*.jinfo A text file describing a jre/sdk installation. Con&#8208; sists of some variables of the form &lt;var&gt;=&lt;value&gt; and a list of alternatives of the form jre|jdk &lt;name&gt; &lt;path&gt;.<br/> AUTHOR update-java-alternatives and this manual page was written by Matthias Klose &lt;doko@ubuntu.com&gt;.<br/> May 2006 UPDATE-JAVA-ALTERNATIVES(8) </span></pre> </div> <!-- endregion --> <p> The above man page neglects to say that a package version&rsquo;s priority is set from its version number. Thus a newer version would normally have a higher priority. To get a list of the installed Java versions, type: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id16df505f2cb2'><button class='copyBtn' data-clipboard-target='#id16df505f2cb2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo update-java-alternatives --list <span class='unselectable'>java-1.11.0-openjdk-amd64 1111 /usr/lib/jvm/java-1.11.0-openjdk-amd64 java-1.17.0-openjdk-amd64 1711 /usr/lib/jvm/java-1.17.0-openjdk-amd64 </span></pre> </div> <!-- endregion --> <p> As you can see, my machine had Java 11 and 17 installed. To set the newest version to be the default, type: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7b7e6a91a4c2'><button class='copyBtn' data-clipboard-target='#id7b7e6a91a4c2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo update-java-alternatives -a <span class='unselectable'>$ </span>java -version <span class='unselectable'>openjdk version "17.0.8.1" 2023-08-24 OpenJDK Runtime Environment (build 17.0.8.1+1-Ubuntu-0ubuntu123.04) OpenJDK 64-Bit Server VM (build 17.0.8.1+1-Ubuntu-0ubuntu123.04, mixed mode, sharing) </span></pre> </div> <!-- endregion --> <!-- endregion --> C++ Boost library 2023-09-14T00:00:00-04:00 https://mslinn.github.io/blog/2023/09/14/boost <!-- #region intro --> <h2 id="about">About Boost</h2> <p> <a href='www.boost.org'>Boost</a> is a general-purpose open source library of utility functions for C++. The <a href='https://www.boost.org/doc/libs/?view=condensed' target='_blank' rel="nofollow">list of categories of fuctionality</a> is formidable. </p> <div class='imgWrapper imgFlex inline' style=' '> <a href='https://www.boost.org/doc/libs/?view=condensed' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/c/boostCategories.svg" type="image/svg"> <!---<source srcset="/blog/c/boostCategories.avif" type="image/avif">--> <source srcset="/blog/c/boostCategories.webp" type="image/webp"> <source srcset="/blog/c/boostCategories.apng" type="image/apng"> <source srcset="/blog/c/boostCategories.png" type="image/png"> <source srcset="/blog/c/boostCategories.jpg" type="image/jpeg"> <source srcset="/blog/c/boostCategories.jpeg" type="image/jpeg"> <source srcset="/blog/c/boostCategories.jfif" type="image/jpeg"> <source srcset="/blog/c/boostCategories.pjpeg" type="image/jpeg"> <source srcset="/blog/c/boostCategories.pjp" type="image/jpeg"> <source srcset="/blog/c/boostCategories.gif" type="image/gif"> <source srcset="/blog/c/boostCategories.tif" type="image/tiff"> <source srcset="/blog/c/boostCategories.tiff" type="image/tiff"> <source srcset="/blog/c/boostCategories.bmp" type="image/bmp"> <source srcset="/blog/c/boostCategories.ico" type="image/x-icon"> <source srcset="/blog/c/boostCategories.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/c/boostCategories.png" style='width: 100%; ' /> </picture> </a> </div> <p> Boost is used in many <a href='https://www.boost.org/users/uses_shrink.html' target='_blank' rel="nofollow">commercial products</a>, <a href='https://www.boost.org/users/uses_inhouse.html' target='_blank' rel="nofollow">in-house projects</a>, and <a href='https://www.boost.org/users/uses_open.html' target='_blank' rel="nofollow">open-source projects</a>. </p> <p> Ostensibly a C++ library for wide usage, it has become an techical underpinning for Python; that is why Boost documentation also contains information about Python. </p> <h2 id="start">Getting Started</h2> <p> I started by reading the online <a href='https://www.boost.org/doc/libs/1_83_0/more/getting_started/index.html' target='_blank' rel="nofollow">Getting Started</a> guide. </p> <div class='imgWrapper imgFlex inline' style=' '> <a href='https://www.boost.org/doc/libs/1_83_0/more/getting_started/index.html' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/c/boost_getting_started.svg" type="image/svg"> <!---<source srcset="/blog/c/boost_getting_started.avif" type="image/avif">--> <source srcset="/blog/c/boost_getting_started.webp" type="image/webp"> <source srcset="/blog/c/boost_getting_started.apng" type="image/apng"> <source srcset="/blog/c/boost_getting_started.png" type="image/png"> <source srcset="/blog/c/boost_getting_started.jpg" type="image/jpeg"> <source srcset="/blog/c/boost_getting_started.jpeg" type="image/jpeg"> <source srcset="/blog/c/boost_getting_started.jfif" type="image/jpeg"> <source srcset="/blog/c/boost_getting_started.pjpeg" type="image/jpeg"> <source srcset="/blog/c/boost_getting_started.pjp" type="image/jpeg"> <source srcset="/blog/c/boost_getting_started.gif" type="image/gif"> <source srcset="/blog/c/boost_getting_started.tif" type="image/tiff"> <source srcset="/blog/c/boost_getting_started.tiff" type="image/tiff"> <source srcset="/blog/c/boost_getting_started.bmp" type="image/bmp"> <source srcset="/blog/c/boost_getting_started.ico" type="image/x-icon"> <source srcset="/blog/c/boost_getting_started.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/c/boost_getting_started.png" style='width: 100%; ' /> </picture> </a> </div> <p> For most Boost functionality, only headers are required, and libraries need not be built. The exceptions are described <a href='https://www.boost.org/doc/libs/1_83_0/more/getting_started/unix-variants.html#header-only-libraries' target='_blank' rel="nofollow">here</a>. </p> <div class='imgWrapper imgFlex inline' style=' '> <a href='https://www.boost.org/doc/libs/1_83_0/more/getting_started/unix-variants.html#header-only-libraries' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/c/boost_header_only.svg" type="image/svg"> <!---<source srcset="/blog/c/boost_header_only.avif" type="image/avif">--> <source srcset="/blog/c/boost_header_only.webp" type="image/webp"> <source srcset="/blog/c/boost_header_only.apng" type="image/apng"> <source srcset="/blog/c/boost_header_only.png" type="image/png"> <source srcset="/blog/c/boost_header_only.jpg" type="image/jpeg"> <source srcset="/blog/c/boost_header_only.jpeg" type="image/jpeg"> <source srcset="/blog/c/boost_header_only.jfif" type="image/jpeg"> <source srcset="/blog/c/boost_header_only.pjpeg" type="image/jpeg"> <source srcset="/blog/c/boost_header_only.pjp" type="image/jpeg"> <source srcset="/blog/c/boost_header_only.gif" type="image/gif"> <source srcset="/blog/c/boost_header_only.tif" type="image/tiff"> <source srcset="/blog/c/boost_header_only.tiff" type="image/tiff"> <source srcset="/blog/c/boost_header_only.bmp" type="image/bmp"> <source srcset="/blog/c/boost_header_only.ico" type="image/x-icon"> <source srcset="/blog/c/boost_header_only.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/c/boost_header_only.png" style='width: 100%; ' /> </picture> </a> </div> <p> <a href='https://theboostcpplibraries.com/' target='_blank' rel="nofollow">The Boost C++ Libraries</a> is a good free online book, and you can order non-free printed copies. </p> <div class='imgWrapper imgFlex inline' style=' '> <a href='https://theBoostCppLibraries.com' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/c/the_boost_cpp_libraries.svg" type="image/svg"> <!---<source srcset="/blog/c/the_boost_cpp_libraries.avif" type="image/avif">--> <source srcset="/blog/c/the_boost_cpp_libraries.webp" type="image/webp"> <source srcset="/blog/c/the_boost_cpp_libraries.apng" type="image/apng"> <source srcset="/blog/c/the_boost_cpp_libraries.png" type="image/png"> <source srcset="/blog/c/the_boost_cpp_libraries.jpg" type="image/jpeg"> <source srcset="/blog/c/the_boost_cpp_libraries.jpeg" type="image/jpeg"> <source srcset="/blog/c/the_boost_cpp_libraries.jfif" type="image/jpeg"> <source srcset="/blog/c/the_boost_cpp_libraries.pjpeg" type="image/jpeg"> <source srcset="/blog/c/the_boost_cpp_libraries.pjp" type="image/jpeg"> <source srcset="/blog/c/the_boost_cpp_libraries.gif" type="image/gif"> <source srcset="/blog/c/the_boost_cpp_libraries.tif" type="image/tiff"> <source srcset="/blog/c/the_boost_cpp_libraries.tiff" type="image/tiff"> <source srcset="/blog/c/the_boost_cpp_libraries.bmp" type="image/bmp"> <source srcset="/blog/c/the_boost_cpp_libraries.ico" type="image/x-icon"> <source srcset="/blog/c/the_boost_cpp_libraries.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/c/the_boost_cpp_libraries.png" style='width: 100%; ' /> </picture> </a> </div> <!-- endregion --> <!-- #region --> <h2 id="install">Ubuntu Installation</h2> <h3 id="boost_ver">Discover the Latest Boost Version Number</h3> <p> The latest version of Boost is shown on <a href='https://https://www.boost.org/users/history/?ref=learnubuntu.com' target='_blank' rel="nofollow"><code>https://www.boost.org/users/history/?ref=learnubuntu.com</code></a>: </p> <p> Let&rsquo;s scrape the lastest version number from the above web page: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idecd0babb5abc'><button class='copyBtn' data-clipboard-target='#idecd0babb5abc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install libxml2-utils <span class='unselectable'>$ </span>VER=$( wget -q -O - https://www.boost.org/users/history/?ref=learnubuntu.com | \ xmllint --html --xpath '//*[@id="intro"]/div/h2[1]/a[2]' - 2>/dev/null | \ grep -oEi 'Version ([0-9].)*' | \ cut -d' ' -f 2 ) <span class='unselectable'>$ </span>echo $VER <span class='unselectable'>1.83 </span></pre> </div> <!-- endregion --> <h3 id="dl">Download Source &amp; Dependencies</h3> <p> Download and unpack the library source to a new directory within your home directory as follows: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idab41ed4e9ca3'><button class='copyBtn' data-clipboard-target='#idab41ed4e9ca3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cd <span class='unselectable'>$ </span>V_R=`echo $VER | tr . _` <span class='unselectable'>$ </span>echo $V_R <span class='unselectable'>1_83 </span> <span class='unselectable'>$ </span>NAME=boost_${V_R}_0 <span class='unselectable'>$ </span>echo $NAME <span class='unselectable'>boost_1_83_0 </span> <span class='unselectable'>$ </span>wget -O $NAME.tar.gz \ https://sourceforge.net/projects/boost/files/boost/$VER.0/$NAME.tar.gz/download <span class='unselectable'>$ </span>tar xzvf $NAME.tar.gz <span class='unselectable'>$ </span>cd $NAME/</pre> </div> <!-- endregion --> <p> You should now have the following files and directories: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7668dc51d4b2'><button class='copyBtn' data-clipboard-target='#id7668dc51d4b2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls -w72 <span class='unselectable'>INSTALL boost/ boostcpp.jam index.htm rst.css Jamroot boost-build.jam bootstrap.bat index.html status/ LICENSE_1_0.txt boost.css bootstrap.sh* libs/ tools/ README.md boost.png doc/ more/ </span></pre> </div> <!-- endregion --> <p> Install the required dependencies for building Boost. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id784d77a11ce3'><button class='copyBtn' data-clipboard-target='#id784d77a11ce3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt-get update <span class='unselectable'>$ </span>yes | sudo apt install autotools-dev build-essential g++ \ libbz2-dev libicu-dev python3-dev</pre> </div> <!-- endregion --> <h3 id="bootstrap">Build Installer Using <span class="code">Bootstrap</span></h3> <p> Two scripts for building the installation program Boost are provided: <code>bootstrap.sh</code> (for Bash) and <code>bootstrap.bat</code> (for Windows/DOS). By default the scripts build all possible static/shared debug/release Boost libraries. Another default is that the script uses all the CPU cores your computer has for the build process. Even using all the cores on a fast laptop, the build might take 10 minutes. </p> <p> Here is the <code>bootstrap.sh</code> help message: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idec62e571933d'><button class='copyBtn' data-clipboard-target='#idec62e571933d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>./bootstrap.sh -h <span class='unselectable'>`./bootstrap.sh\&#39; builds the Boost build system B2 and prepares Boost for building. This includes setting defaults in the project-config.jam which you can adjust prior to invoking B2.<br/> Usage: ./bootstrap.sh [OPTION]...<br/> Defaults for the options are specified in brackets.<br/> Configuration: -h, --help display this help and exit --with-bjam=BJAM use existing Boost.Jam executable (bjam) [automatically built] --with-toolset=TOOLSET use specific TOOLSET to build B2 and as default for building Boost [automatically detected] --show-libraries show the set of libraries that require build and installation steps (i.e., those libraries that can be used with --with-libraries or --without-libraries), then exit --with-libraries=list build only a particular set of libraries, describing using either a comma-separated list of library names or &quot;all&quot; [all] --without-libraries=list build all libraries except the ones listed [] --with-icu enable Unicode/ICU support in Regex [automatically detected] --without-icu disable Unicode/ICU support in Regex --with-icu=DIR specify the root of the ICU library installation and enable Unicode/ICU support in Regex [automatically detected] --with-python=PYTHON specify the Python executable [python] --with-python-root=DIR specify the root of the Python installation [automatically detected] --with-python-version=X.Y specify the Python version as X.Y [automatically detected]<br/> Installation directories: --prefix=PREFIX install Boost into the given PREFIX [/usr/local] --exec-prefix=EPREFIX install Boost binaries into the given EPREFIX [PREFIX]<br/> More precise control over installation directories: --libdir=DIR install libraries here [EPREFIX/lib] --includedir=DIR install headers here [PREFIX/include] </span></pre> </div> <p> <code>Bootstrap.sh</code> compiles and links a build program called <code>b2</code>, which by default installs all of Boost into <code>/usr/<wbr>local/</code>. To be more specific, the default location for Boost include files to be installed into is <code>/usr/<wbr>local/<wbr>include/<wbr>boost/</code>, and Boost libraries are installed by default into <code>/usr/<wbr>local/<wbr>lib/</code>. </p> <!-- endregion --> <p> Create (and recreate) the <code>b2</code> installation program for Boost for <i>each</i> <a href='/blog/2021/04/09/python-venvs.html'>Python virtual environment</a>. By default, <code>bootstrap.sh</code> uses the currently active Python virtual environment, like this: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id26f2472aff69'><button class='copyBtn' data-clipboard-target='#id26f2472aff69' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>./bootstrap.sh</pre> </div> <p> Following is how to build <code>b2</code> for the virtual environment at <code>~/venv/blah/</code>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id729fe4277eec'><button class='copyBtn' data-clipboard-target='#id729fe4277eec' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>./bootstrap.sh --with-python-root=~/venv/blah <span class='unselectable'>Building B2 engine..<br/> ### ### ### Using &#39;gcc&#39; toolset. ### ###<br/> g++ (Ubuntu 12.3.0-1ubuntu1~23.04) 12.3.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.<br/><br/> ### ###<br/> &gt; g++ -x c++ -std=c++11 -O2 -s -DNDEBUG builtins.cpp class.cpp command.cpp compile.cpp constants.cpp cwd.cpp debug.cpp debugger.cpp execcmd.cpp execnt.cpp execunix.cpp filesys.cpp filent.cpp fileunix.cpp frames.cpp function.cpp glob.cpp hash.cpp hcache.cpp hdrmacro.cpp headers.cpp jam_strings.cpp jam.cpp jamgram.cpp lists.cpp make.cpp make1.cpp md5.cpp mem.cpp modules.cpp native.cpp object.cpp option.cpp output.cpp parse.cpp pathnt.cpp pathsys.cpp pathunix.cpp regexp.cpp rules.cpp scan.cpp search.cpp startup.cpp subst.cpp sysinfo.cpp timestamp.cpp variable.cpp w32_getreg.cpp modules/order.cpp modules/path.cpp modules/property-set.cpp modules/regex.cpp modules/sequence.cpp modules/set.cpp -o b2 tools/build/src/engine/b2 Detecting Python version... 3.11 Unicode/ICU support for Boost.Regex?... /usr Backing up existing B2 configuration in project-config.jam.2 Generating B2 configuration in project-config.jam for gcc...<br/> Bootstrapping is done. To build, run:<br/> ./b2<br/> To generate header files, run:<br/> ./b2 headers<br/> The configuration generated uses gcc to build by default. If that is unintended either use the --with-toolset option or adjust configuration, by editing &#39;project-config.jam&#39;.<br/> Further information:<br/> - Command line help: ./b2 --help<br/> - Getting started guide: http://www.boost.org/more/getting_started/unix-variants.html<br/> - B2 documentation: http://www.boost.org/build/ </span></pre> </div> <!-- endregion --> <h3 id="b2">Running <span class="code">B2</span>, the Boost Installer</h3> <p> Following is the help message for the newly created <code>b2</code> Boost installation program. As you can see, the defaults specified when creating the <code>b2</code> script can be overidden. Note that not all of the available options are described in this help message; for example, the <code>-j</code> option is only described in the <a href='https://www.boost.org/doc/libs/1_83_0/tools/build/doc/html/index.html' target='_blank' rel="nofollow">online help for <code>b2</code></a>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd64fc57b26a1'><button class='copyBtn' data-clipboard-target='#idd64fc57b26a1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>./b2 --help <span class='unselectable'>B2 4.10-git<br/> Project-specific help:<br/> Project has jamfile at Jamroot<br/> Usage:<br/> b2 [options] [properties] [install|stage]<br/> Builds and installs Boost.<br/> Targets and Related Options:<br/> install Install headers and compiled library files to the ======= configured locations (below).<br/> --prefix=&lt;PREFIX&gt; Install architecture independent files here. Default: C:\Boost on Windows Default: /usr/local on Unix, Linux, etc.<br/> --exec-prefix=&lt;EPREFIX&gt; Install architecture dependent files here. Default: &lt;PREFIX&gt;<br/> --libdir=&lt;LIBDIR&gt; Install library files here. Default: &lt;EPREFIX&gt;/lib<br/> --includedir=&lt;HDRDIR&gt; Install header files here. Default: &lt;PREFIX&gt;/include<br/> --cmakedir=&lt;CMAKEDIR&gt; Install CMake configuration files here. Default: &lt;LIBDIR&gt;/cmake<br/> --no-cmake-config Do not install CMake configuration files.<br/> stage Build and install only compiled library files to the ===== stage directory.<br/> --stagedir=&lt;STAGEDIR&gt; Install library files here Default: ./stage<br/> Other Options:<br/> --build-type=&lt;type&gt; Build the specified pre-defined set of variations of the libraries. Note, that which variants get built depends on what each library supports.<br/> -- minimal -- (default) Builds a minimal set of variants. On Windows, these are static multithreaded libraries in debug and release modes, using shared runtime. On Linux, these are static and shared multithreaded libraries in release mode.<br/> -- complete -- Build all possible variations.<br/> --build-dir=DIR Build in this location instead of building within the distribution tree. Recommended!<br/> --show-libraries Display the list of Boost libraries that require build and installation steps, and then exit.<br/> --layout=&lt;layout&gt; Determine whether to choose library names and header locations such that multiple versions of Boost or multiple compilers can be used on the same system.<br/> -- versioned -- Names of boost binaries include the Boost version number, name and version of the compiler and encoded build properties. Boost headers are installed in a subdirectory of &lt;HDRDIR&gt; whose name contains the Boost version number.<br/> -- tagged -- Names of boost binaries include the encoded build properties such as variant and threading, but do not including compiler name and version, or Boost version. This option is useful if you build several variants of Boost, using the same compiler.<br/> -- system -- Binaries names do not include the Boost version number or the name and version number of the compiler. Boost headers are installed directly into &lt;HDRDIR&gt;. This option is intended for system integrators building distribution packages.<br/> The default value is &#39;versioned&#39; on Windows, and &#39;system&#39; on Unix.<br/> --buildid=ID Add the specified ID to the name of built libraries. The default is to not add anything.<br/> --python-buildid=ID Add the specified ID to the name of built libraries that depend on Python. The default is to not add anything. This ID is added in addition to --buildid.<br/> --help This message.<br/> --with-&lt;library&gt; Build and install the specified &lt;library&gt;. If this option is used, only libraries specified using this option will be built.<br/> --without-&lt;library&gt; Do not build, stage, or install the specified &lt;library&gt;. By default, all libraries are built.<br/> Properties:<br/> toolset=toolset Indicate the toolset to build with.<br/> variant=debug|release Select the build variant<br/> link=static|shared Whether to build static or shared libraries<br/> threading=single|multi Whether to build single or multithreaded binaries<br/> runtime-link=static|shared Whether to link to static or shared C and C++ runtime.<br/><br/> General command line usage:<br/> b2 [options] [properties] [targets]<br/> Options, properties and targets can be specified in any order.<br/> Important Options:<br/> * --clean Remove targets instead of building * -a Rebuild everything * -n Don&#39;t execute the commands, only print them * -d+2 Show commands as they are executed * -d0 Suppress all informational messages * -q Stop at first error * --reconfigure Rerun all configuration checks * --durations[=N] Report top N targets by execution time * --debug-configuration Diagnose configuration * --debug-building Report which targets are built with what properties * --debug-generator Diagnose generator search/execution<br/> Further Help:<br/> The following options can be used to obtain additional documentation.<br/> * --help-options Print more obscure command line options. * --help-internal B2 implementation details. * --help-doc-options Implementation details doc formatting.<br/> ...found 1 target... </span></pre> </div> <!-- endregion --> <p> The following runs the newly created <code>b2</code> program, which builds and installs Boost using all CPU cores. </p> <ol> <li> The Python version is stored into an environment variable called <code>PVER</code>. </li> <li> The <code>CPLUS_INCLUDE_PATH</code> environment variable is pointed at the location of the Python 3 include files. <code>PVER</code> was used to set the proper value for <code>CPLUS_INCLUDE_PATH</code>. </li> <li> <code>B2</code> generates screen upon screen of compiler warnings. The <code>-d 1</code> option suppresses them. </li> </ol> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5d54947246ce'><button class='copyBtn' data-clipboard-target='#id5d54947246ce' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>PVER="$(python --version | cut -d' ' -f 2 | grep -oEi '(3.[0-9]*)')" <span class='unselectable'>$ </span>echo $PVER <span class='unselectable'>3.11 </span> <span class='unselectable'>$ </span>sudo CPLUS_INCLUDE_PATH=/usr/include/python$PVER ./b2 -d 1 install <span class='unselectable'>Performing configuration checks<br/> - default address-model : 64-bit (cached) [1] - default architecture : x86 (cached) [1] - compiler supports SSE2 : yes (cached) [2] - compiler supports SSE4.1 : yes (cached) [2] - has std::atomic_ref : no (cached) [2] - has -Wl,--no-undefined : yes (cached) [2] - has statx : yes (cached) [2] - has init_priority attribute : yes (cached) [2] - has stat::st_blksize : yes (cached) [2] - has stat::st_mtim : yes (cached) [2] - has stat::st_mtimensec : no (cached) [2] - has stat::st_mtimespec : no (cached) [2] - has stat::st_birthtim : no (cached) [2] - has stat::st_birthtimensec : no (cached) [2] - has stat::st_birthtimespec : no (cached) [2] - has fdopendir(O_NOFOLLOW) : yes (cached) [2] - has dirent::d_type : yes (cached) [2] - has POSIX *at APIs : yes (cached) [2] - cxx11_auto_declarations : yes (cached) [2] - cxx11_constexpr : yes (cached) [2] - cxx11_defaulted_functions : yes (cached) [2] - cxx11_final : yes (cached) [2] - cxx11_hdr_mutex : yes (cached) [2] - cxx11_hdr_tuple : yes (cached) [2] - cxx11_lambdas : yes (cached) [2] - cxx11_noexcept : yes (cached) [2] - cxx11_nullptr : yes (cached) [2] - cxx11_rvalue_references : yes (cached) [2] - cxx11_template_aliases : yes (cached) [2] - cxx11_thread_local : yes (cached) [2] - cxx11_variadic_templates : yes (cached) [2] - has_icu builds : yes (cached) [2] warning: Graph library does not contain MPI-based parallel components. note: to enable them, add &quot;using mpi ;&quot; to your user-config.jam. note: to suppress this message, pass &quot;--without-graph_parallel&quot; to bjam. - zlib : yes (cached) - bzip2 : yes (cached) - lzma : yes (cached) - zstd : yes (cached) - has_lzma_cputhreads builds : yes (cached) [2] - cxx11_decltype : yes (cached) [2] - cxx11_basic_alignas : yes (cached) [2] - iconv (libc) : yes (cached) [2] - icu : yes (cached) [2] - cxx11_defaulted_moves : yes (cached) [2] - cxx11_hdr_functional : yes (cached) [2] - cxx11_hdr_type_traits : yes (cached) [2] - cxx11_override : yes (cached) [2] - cxx11_range_based_for : yes (cached) [2] - cxx11_scoped_enums : yes (cached) [2] - cxx11_smart_ptr : yes (cached) [2] - cxx11_static_assert : yes (cached) [2] - lockfree boost::atomic_flag : yes (cached) [2] - native atomic int32 supported : yes (cached) [2] - native syslog supported : yes (cached) [2] - pthread supports robust mutexes : yes (cached) [2] - compiler supports SSSE3 : yes (cached) [2] - compiler supports AVX2 : yes (cached) [2] - gcc visibility : yes (cached) [2] - sfinae_expr : yes (cached) [2] - cxx11_unified_initialization_syntax : yes (cached) [2] - cxx11_hdr_initializer_list : yes (cached) [2] - cxx11_hdr_chrono : yes (cached) [2] - cxx11_numeric_limits : yes (cached) [2] - cxx11_hdr_array : yes (cached) [2] - cxx11_hdr_atomic : yes (cached) [2] - cxx11_allocator : yes (cached) [2] - cxx11_explicit_conversion_operators : yes (cached) [2] - long double support : yes (cached) [2] warning: skipping optional Message Passing Interface (MPI) library. note: to enable MPI support, add &quot;using mpi ;&quot; to user-config.jam. note: to suppress this message, pass &quot;--without-mpi&quot; to bjam. note: otherwise, you can safely ignore this message. - cxx11_char16_t : yes (cached) [2] - cxx11_char32_t : yes (cached) [2] - Has Large File Support : yes (cached) [2] - Has attribute init_priority : yes (cached) [2] - libbacktrace builds : yes (cached) [2] - addr2line builds : yes (cached) [2] - WinDbg builds : no (cached) [2] - WinDbg builds : no (cached) [3] - WinDbgCached builds : no (cached) [2] - WinDbgCached builds : no (cached) [3] - BOOST_COMP_GNUC &gt;= 4.3.0 : yes (cached) [2] - BOOST_COMP_GNUC &gt;= 4.3.0 : yes (cached) [4] - cxx11_hdr_thread : yes (cached) [2] - cxx11_hdr_regex : yes (cached) [2] - compiler supports SSE2 : yes (cached) [4] - compiler supports SSE4.1 : yes (cached) [4] - has std::atomic_ref : no (cached) [4] - has statx : yes (cached) [4] - has init_priority attribute : yes (cached) [4] - has stat::st_blksize : yes (cached) [4] - has stat::st_mtim : yes (cached) [4] - has stat::st_mtimensec : no (cached) [4] - has stat::st_mtimespec : no (cached) [4] - has stat::st_birthtim : no (cached) [4] - has stat::st_birthtimensec : no (cached) [4] - has stat::st_birthtimespec : no (cached) [4] - has fdopendir(O_NOFOLLOW) : yes (cached) [4] - has dirent::d_type : yes (cached) [4] - has POSIX *at APIs : yes (cached) [4] - cxx11_auto_declarations : yes (cached) [4] - cxx11_constexpr : yes (cached) [4] - cxx11_defaulted_functions : yes (cached) [4] - cxx11_final : yes (cached) [4] - cxx11_hdr_mutex : yes (cached) [4] - cxx11_hdr_tuple : yes (cached) [4] - cxx11_lambdas : yes (cached) [4] - cxx11_noexcept : yes (cached) [4] - cxx11_nullptr : yes (cached) [4] - cxx11_rvalue_references : yes (cached) [4] - cxx11_template_aliases : yes (cached) [4] - cxx11_thread_local : yes (cached) [4] - cxx11_variadic_templates : yes (cached) [4] - has_icu builds : yes (cached) [4] - zlib : yes (cached) [5] - bzip2 : yes (cached) [5] - lzma : yes (cached) [5] - zstd : yes (cached) [5] - has_lzma_cputhreads builds : yes (cached) [4] - cxx11_decltype : yes (cached) [4] - cxx11_basic_alignas : yes (cached) [4] - iconv (libc) : yes (cached) [4] - icu : yes (cached) [4] - cxx11_defaulted_moves : yes (cached) [4] - cxx11_hdr_functional : yes (cached) [4] - cxx11_hdr_type_traits : yes (cached) [4] - cxx11_override : yes (cached) [4] - cxx11_range_based_for : yes (cached) [4] - cxx11_scoped_enums : yes (cached) [4] - cxx11_smart_ptr : yes (cached) [4] - cxx11_static_assert : yes (cached) [4] - lockfree boost::atomic_flag : yes (cached) [4] - native atomic int32 supported : yes (cached) [4] - native syslog supported : yes (cached) [4] - pthread supports robust mutexes : yes (cached) [4] - compiler supports SSSE3 : yes (cached) [4] - compiler supports AVX2 : yes (cached) [4] - gcc visibility : yes (cached) [4] - sfinae_expr : yes (cached) [4] - cxx11_unified_initialization_syntax : yes (cached) [4] - cxx11_hdr_initializer_list : yes (cached) [4] - cxx11_hdr_chrono : yes (cached) [4] - cxx11_numeric_limits : yes (cached) [4] - cxx11_hdr_array : yes (cached) [4] - cxx11_hdr_atomic : yes (cached) [4] - cxx11_allocator : yes (cached) [4] - cxx11_explicit_conversion_operators : yes (cached) [4] - long double support : yes (cached) [4] - cxx11_char16_t : yes (cached) [4] - cxx11_char32_t : yes (cached) [4] - Has Large File Support : yes (cached) [4] - Has attribute init_priority : yes (cached) [4] - libbacktrace builds : yes (cached) [4] - addr2line builds : yes (cached) [4] - WinDbg builds : no (cached) [4] - WinDbg builds : no (cached) [6] - WinDbgCached builds : no (cached) [4] - WinDbgCached builds : no (cached) [6] - cxx11_hdr_thread : yes (cached) [4] - cxx11_hdr_regex : yes (cached) [4]<br/> [1] gcc-12 [2] gcc-12/release/python-3.11/threading-multi/visibility-hidden [3] gcc-12/release/build-no/python-3.11/threading-multi/visibility-hidden [4] gcc-12/release/link-static/python-3.11/threading-multi/visibility-hidden [5] link-static [6] gcc-12/release/build-no/link-static/python-3.11/threading-multi/visibility-hidden<br/> Component configuration:<br/> - atomic : building - chrono : building - container : building - context : building - contract : building - coroutine : building - date_time : building - exception : building - fiber : building - filesystem : building - graph : building - graph_parallel : building - headers : building - iostreams : building - json : building - locale : building - log : building - math : building - mpi : building - nowide : building - program_options : building - python : building - random : building - regex : building - serialization : building - stacktrace : building - system : building - test : building - thread : building - timer : building - type_erasure : building - url : building - wave : building<br/> ...patience... ...patience... ...patience... ...patience... ...patience... ...patience... ...patience... ...found 52031 targets... </span></pre> </div> <!-- endregion --> <p> The <code>b2</code> installation program stored the following include files in <code>/usr/<wbr>local/<wbr>include/<wbr>boost/</code> </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idde7196018ef0'><button class='copyBtn' data-clipboard-target='#idde7196018ef0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls -w72 /usr/local/include/boost/ <span class='unselectable'>accumulators/ make_unique.hpp algorithm/ math/ align/ math_fwd.hpp align.hpp mem_fn.hpp aligned_storage.hpp memory_order.hpp any/ metaparse/ any.hpp metaparse.hpp archive/ move/ array.hpp mp11/ asio/ mp11.hpp asio.hpp mpi/ assert/ mpi.hpp assert.hpp mpl/ assign/ msm/ assign.hpp multi_array/ atomic/ multi_array.hpp atomic.hpp multi_index/ beast/ multi_index_container.hpp beast.hpp multi_index_container_fwd.hpp bimap/ multiprecision/ bimap.hpp mysql/ bind/ mysql.hpp bind.hpp next_prior.hpp blank.hpp non_type.hpp blank_fwd.hpp noncopyable.hpp call_traits.hpp nondet_random.hpp callable_traits/ none.hpp callable_traits.hpp none_t.hpp cast.hpp nowide/ cerrno.hpp numeric/ checked_delete.hpp operators.hpp chrono/ operators_v1.hpp chrono.hpp optional/ circular_buffer/ optional.hpp circular_buffer.hpp outcome/ circular_buffer_fwd.hpp outcome.hpp compat/ parameter/ compatibility/ parameter.hpp compressed_pair.hpp pending/ compute/ pfr/ compute.hpp pfr.hpp concept/ phoenix/ concept_archetype.hpp phoenix.hpp concept_check/ pointee.hpp concept_check.hpp pointer_cast.hpp config/ pointer_to_other.hpp config.hpp poly_collection/ container/ polygon/ container_hash/ polymorphic_cast.hpp context/ polymorphic_pointer_cast.hpp contract/ pool/ contract.hpp predef/ contract_macro.hpp predef.h convert/ preprocessor/ convert.hpp preprocessor.hpp core/ process/ coroutine/ process.hpp coroutine2/ program_options/ crc.hpp program_options.hpp cregex.hpp progress.hpp cstdfloat.hpp property_map/ cstdint.hpp property_tree/ cstdlib.hpp proto/ current_function.hpp ptr_container/ cxx11_char_types.hpp python/ date_time/ python.hpp date_time.hpp qvm/ describe/ qvm.hpp describe.hpp qvm_lite.hpp detail/ random/ dll/ random.hpp dll.hpp range/ dynamic_bitset/ range.hpp dynamic_bitset.hpp ratio/ dynamic_bitset_fwd.hpp ratio.hpp enable_shared_from_this.hpp rational.hpp endian/ ref.hpp endian.hpp regex/ exception/ regex.h exception_ptr.hpp regex.hpp fiber/ regex_fwd.hpp filesystem/ safe_numerics/ filesystem.hpp scope_exit.hpp flyweight/ scoped_array.hpp flyweight.hpp scoped_ptr.hpp foreach.hpp serialization/ foreach_fwd.hpp shared_array.hpp format/ shared_container_iterator.hpp format.hpp shared_ptr.hpp function/ signals2/ function.hpp signals2.hpp function_equal.hpp smart_ptr/ function_output_iterator.hpp smart_ptr.hpp function_types/ sort/ functional/ spirit/ functional.hpp spirit.hpp fusion/ stacktrace/ generator_iterator.hpp stacktrace.hpp geometry/ statechart/ geometry.hpp static_assert.hpp get_pointer.hpp static_string/ gil/ static_string.hpp gil.hpp stl_interfaces/ graph/ swap.hpp hana/ system/ hana.hpp system.hpp heap/ test/ histogram/ thread/ histogram.hpp thread.hpp hof/ throw_exception.hpp hof.hpp timer/ icl/ timer.hpp implicit_cast.hpp token_functions.hpp indirect_reference.hpp token_iterator.hpp integer/ tokenizer.hpp integer.hpp tti/ integer_fwd.hpp tuple/ integer_traits.hpp type.hpp interprocess/ type_erasure/ intrusive/ type_index/ intrusive_ptr.hpp type_index.hpp io/ type_traits/ io_fwd.hpp type_traits.hpp iostreams/ typeof/ is_placeholder.hpp units/ iterator/ unordered/ iterator.hpp unordered_map.hpp iterator_adaptors.hpp unordered_set.hpp json/ url/ json.hpp url.hpp lambda/ utility/ lambda2/ utility.hpp lambda2.hpp uuid/ leaf/ variant/ leaf.hpp variant.hpp lexical_cast/ variant2/ lexical_cast.hpp variant2.hpp limits.hpp version.hpp local_function/ visit_each.hpp local_function.hpp vmd/ locale/ wave/ locale.hpp wave.hpp lockfree/ weak_ptr.hpp log/ winapi/ logic/ xpressive/ make_default.hpp yap/ make_shared.hpp </span></pre> </div> <!-- endregion --> <p> The <code>b2</code> installation program stored the following libraries in <code>/usr/<wbr>local/<wbr>lib/</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id91957827c679'><button class='copyBtn' data-clipboard-target='#id91957827c679' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls -w72 /usr/local/lib <span class='unselectable'>cmake/ libboost_atomic.a libboost_atomic.so@ libboost_atomic.so.1.83.0* libboost_chrono.a libboost_chrono.so@ libboost_chrono.so.1.83.0* libboost_container.a libboost_container.so@ libboost_container.so.1.83.0* libboost_context.a libboost_context.so@ libboost_context.so.1.83.0* libboost_contract.a libboost_contract.so@ libboost_contract.so.1.83.0* libboost_coroutine.a libboost_coroutine.so@ libboost_coroutine.so.1.83.0* libboost_date_time.a libboost_date_time.so@ libboost_date_time.so.1.83.0* libboost_exception.a libboost_fiber.a libboost_fiber.so@ libboost_fiber.so.1.83.0* libboost_filesystem.a libboost_filesystem.so@ libboost_filesystem.so.1.83.0* libboost_graph.a libboost_graph.so@ libboost_graph.so.1.83.0* libboost_iostreams.a libboost_iostreams.so@ libboost_iostreams.so.1.83.0* libboost_json.a libboost_json.so@ libboost_json.so.1.83.0* libboost_locale.a libboost_locale.so@ libboost_locale.so.1.83.0* libboost_log.a libboost_log.so@ libboost_log.so.1.83.0* libboost_log_setup.a libboost_log_setup.so@ libboost_log_setup.so.1.83.0* libboost_math_c99.a libboost_math_c99.so@ libboost_math_c99.so.1.83.0* libboost_math_c99f.a libboost_math_c99f.so@ libboost_math_c99f.so.1.83.0* libboost_math_c99l.a libboost_math_c99l.so@ libboost_math_c99l.so.1.83.0* libboost_math_tr1.a libboost_math_tr1.so@ libboost_math_tr1.so.1.83.0* libboost_math_tr1f.a libboost_math_tr1f.so@ libboost_math_tr1f.so.1.83.0* libboost_math_tr1l.a libboost_math_tr1l.so@ libboost_math_tr1l.so.1.83.0* libboost_nowide.a libboost_nowide.so@ libboost_nowide.so.1.83.0* libboost_prg_exec_monitor.a libboost_prg_exec_monitor.so@ libboost_prg_exec_monitor.so.1.83.0* libboost_program_options.a libboost_program_options.so@ libboost_program_options.so.1.83.0* libboost_random.a libboost_random.so@ libboost_random.so.1.83.0* libboost_regex.a libboost_regex.so@ libboost_regex.so.1.83.0* libboost_serialization.a libboost_serialization.so@ libboost_serialization.so.1.83.0* libboost_stacktrace_addr2line.a libboost_stacktrace_addr2line.so@ libboost_stacktrace_addr2line.so.1.83.0* libboost_stacktrace_backtrace.a libboost_stacktrace_backtrace.so@ libboost_stacktrace_backtrace.so.1.83.0* libboost_stacktrace_basic.a libboost_stacktrace_basic.so@ libboost_stacktrace_basic.so.1.83.0* libboost_stacktrace_noop.a libboost_stacktrace_noop.so@ libboost_stacktrace_noop.so.1.83.0* libboost_system.a libboost_system.so@ libboost_system.so.1.83.0* libboost_test_exec_monitor.a libboost_thread.a libboost_thread.so@ libboost_thread.so.1.83.0* libboost_timer.a libboost_timer.so@ libboost_timer.so.1.83.0* libboost_type_erasure.a libboost_type_erasure.so@ libboost_type_erasure.so.1.83.0* libboost_unit_test_framework.a libboost_unit_test_framework.so@ libboost_unit_test_framework.so.1.83.0* libboost_url.a libboost_url.so@ libboost_url.so.1.83.0* libboost_wave.a libboost_wave.so@ libboost_wave.so.1.83.0* libboost_wserialization.a libboost_wserialization.so@ libboost_wserialization.so.1.83.0* node_modules/ python3.11/ </span></pre> </div> <!-- endregion --> <h3 id="load_path">Verify Boost System Libraries</h3> <p> The <code>ldconfig</code> command manages Linux system libraries. Here is the help message: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id996d779b114d'><button class='copyBtn' data-clipboard-target='#id996d779b114d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>man ldconfig <span class='unselectable'>ldconfig(8) System Manager&#39;s Manual ldconfig(8)<br/> NAME ldconfig - configure dynamic linker run-time bindings<br/> SYNOPSIS /sbin/ldconfig [-nNvVX] [-C cache] [-f conf] [-r root] directory ...<br/> /sbin/ldconfig -l [-v] library ...<br/> /sbin/ldconfig -p<br/> DESCRIPTION ldconfig creates the necessary links and cache to the most re&#8208; cent shared libraries found in the directories specified on the command line, in the file /etc/ld.so.conf, and in the trusted directories, /lib and /usr/lib. On some 64-bit architectures such as x86-64, /lib and /usr/lib are the trusted directories for 32-bit libraries, while /lib64 and /usr/lib64 are used for 64-bit libraries.<br/> The cache is used by the run-time linker, ld.so or ld-linux.so. ldconfig checks the header and filenames of the libraries it encounters when determining which versions should have their links updated. ldconfig should normally be run by the supe&#8208; ruser as it may require write permission on some root owned di&#8208; rectories and files.<br/> ldconfig will look only at files that are named lib*.so* (for regular shared objects) or ld-*.so* (for the dynamic loader it&#8208; self). Other files will be ignored. Also, ldconfig expects a certain pattern to how the symbolic links are set up, like this example, where the middle file (libfoo.so.1 here) is the SONAME for the library:<br/> libfoo.so -&gt; libfoo.so.1 -&gt; libfoo.so.1.12<br/> Failure to follow this pattern may result in compatibility is&#8208; sues after an upgrade.<br/> OPTIONS -c fmt --format=fmt (Since glibc 2.2) Use cache format fmt, which is one of old, new, or compat. Since glibc 2.32, the default is new. Before that, it was compat.<br/> -C cache Use cache instead of /etc/ld.so.cache.<br/> -f conf Use conf instead of /etc/ld.so.conf.<br/> -i --ignore-aux-cache (Since glibc 2.7) Ignore auxiliary cache file.<br/> -l (Since glibc 2.2) Interpret each operand as a libary name and configure its links. Intended for use only by experts.<br/> -n Process only the directories specified on the command line; don&#39;t process the trusted directories, nor those specified in /etc/ld.so.conf. Implies -N.<br/> -N Don&#39;t rebuild the cache. Unless -X is also specified, links are still updated.<br/> -p --print-cache Print the lists of directories and candidate libraries stored in the current cache.<br/> -r root Change to and use root as the root directory.<br/> -v --verbose Verbose mode. Print current version number, the name of each directory as it is scanned, and any links that are created. Overrides quiet mode.<br/> -V --version Print program version.<br/> -X Don&#39;t update links. Unless -N is also specified, the cache is still rebuilt.<br/> FILES /lib/ld.so is the run-time linker/loader. /etc/ld.so.conf contains a list of directories, one per line, in which to search for libraries. /etc/ld.so.cache contains an ordered list of libraries found in the di&#8208; rectories specified in /etc/ld.so.conf, as well as those found in the trusted directories.<br/> SEE ALSO ldd(1), ld.so(8)<br/> Linux man-pages 6.03 2023-01-07 ldconfig(8) </span></pre> </div> <!-- endregion --> <p> The Boost installation procedure automatically causes the newly built Boost libraries to be added to the configured system libraries when <code>bootstrap.sh</code> is not invoked with a target path, for example by specifying the <code>--with-python-root</code> option. </p> <p> To manually add the newly built Boost libraries to the system load path, use <code>ldconfig</code>. The following adds the newly compiled and installed Boost libraries in <code>/usr/local/lib/</code> (and any other libraries that might happen to be in that directory) to the <code>ld.so</code> cache. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb93dd565ab7b'><button class='copyBtn' data-clipboard-target='#idb93dd565ab7b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo ldconfig -n /usr/local/lib</pre> </div> <!-- endregion --> <p> We can verify that the library cache now contains the newly built Boost libraries by using <code>ldconfig</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddbf09df5a6c9'><button class='copyBtn' data-clipboard-target='#iddbf09df5a6c9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ldconfig -p | grep boost <span class='unselectable'>libboost_wserialization.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_wserialization.so.1.83.0 libboost_wserialization.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_wserialization.so libboost_wave.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_wave.so.1.83.0 libboost_wave.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_wave.so libboost_url.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_url.so.1.83.0 libboost_url.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_url.so libboost_unit_test_framework.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_unit_test_framework.so.1.83.0 libboost_unit_test_framework.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_unit_test_framework.so libboost_type_erasure.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_type_erasure.so.1.83.0 libboost_type_erasure.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_type_erasure.so libboost_timer.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_timer.so.1.83.0 libboost_timer.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_timer.so libboost_thread.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_thread.so.1.83.0 libboost_thread.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_thread.so libboost_system.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_system.so.1.83.0 libboost_system.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_system.so libboost_stacktrace_noop.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_stacktrace_noop.so.1.83.0 libboost_stacktrace_noop.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_stacktrace_noop.so libboost_stacktrace_basic.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_stacktrace_basic.so.1.83.0 libboost_stacktrace_basic.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_stacktrace_basic.so libboost_stacktrace_backtrace.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_stacktrace_backtrace.so.1.83.0 libboost_stacktrace_backtrace.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_stacktrace_backtrace.so libboost_stacktrace_addr2line.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_stacktrace_addr2line.so.1.83.0 libboost_stacktrace_addr2line.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_stacktrace_addr2line.so libboost_serialization.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_serialization.so.1.83.0 libboost_serialization.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_serialization.so libboost_regex.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_regex.so.1.83.0 libboost_regex.so.1.74.0 (libc6,x86-64) =&gt; /lib/x86_64-linux-gnu/libboost_regex.so.1.74.0 libboost_regex.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_regex.so libboost_random.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_random.so.1.83.0 libboost_random.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_random.so libboost_python311.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_python311.so.1.83.0 libboost_python311.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_python311.so libboost_program_options.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_program_options.so.1.83.0 libboost_program_options.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_program_options.so libboost_prg_exec_monitor.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_prg_exec_monitor.so.1.83.0 libboost_prg_exec_monitor.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_prg_exec_monitor.so libboost_nowide.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_nowide.so.1.83.0 libboost_nowide.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_nowide.so libboost_math_tr1l.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_tr1l.so.1.83.0 libboost_math_tr1l.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_tr1l.so libboost_math_tr1f.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_tr1f.so.1.83.0 libboost_math_tr1f.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_tr1f.so libboost_math_tr1.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_tr1.so.1.83.0 libboost_math_tr1.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_tr1.so libboost_math_c99l.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_c99l.so.1.83.0 libboost_math_c99l.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_c99l.so libboost_math_c99f.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_c99f.so.1.83.0 libboost_math_c99f.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_c99f.so libboost_math_c99.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_c99.so.1.83.0 libboost_math_c99.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_c99.so libboost_log_setup.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_log_setup.so.1.83.0 libboost_log_setup.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_log_setup.so libboost_log.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_log.so.1.83.0 libboost_log.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_log.so libboost_locale.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_locale.so.1.83.0 libboost_locale.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_locale.so libboost_json.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_json.so.1.83.0 libboost_json.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_json.so libboost_iostreams.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_iostreams.so.1.83.0 libboost_iostreams.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_iostreams.so libboost_graph.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_graph.so.1.83.0 libboost_graph.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_graph.so libboost_filesystem.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_filesystem.so.1.83.0 libboost_filesystem.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_filesystem.so libboost_fiber.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_fiber.so.1.83.0 libboost_fiber.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_fiber.so libboost_date_time.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_date_time.so.1.83.0 libboost_date_time.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_date_time.so libboost_coroutine.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_coroutine.so.1.83.0 libboost_coroutine.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_coroutine.so libboost_contract.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_contract.so.1.83.0 libboost_contract.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_contract.so libboost_context.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_context.so.1.83.0 libboost_context.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_context.so libboost_container.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_container.so.1.83.0 libboost_container.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_container.so libboost_chrono.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_chrono.so.1.83.0 libboost_chrono.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_chrono.so libboost_atomic.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_atomic.so.1.83.0 libboost_atomic.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_atomic.so </span></pre> </div> <!-- endregion --> <p> As a further check, examine the full path of an arbitrary include file (<code>src.hpp</code>) and list the Boost libraries: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddf9e426c43ea'><button class='copyBtn' data-clipboard-target='#iddf9e426c43ea' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>locate src.hpp <span class='unselectable'>/usr/include/boost/asio/impl/src.hpp /usr/include/boost/asio/ssl/impl/src.hpp /usr/include/boost/beast/src.hpp </span> <span class='unselectable'>$ </span>find /usr/local/lib/ -iname *boost* <span class='unselectable'>/usr/local/lib/libboost_math_tr1f.a /usr/local/lib/libboost_log_setup.so /usr/local/lib/libboost_system.so.1.83.0 /usr/local/lib/libboost_system.so /usr/local/lib/libboost_serialization.a /usr/local/lib/libboost_context.a /usr/local/lib/libboost_nowide.so /usr/local/lib/libboost_contract.so.1.83.0 /usr/local/lib/libboost_prg_exec_monitor.a /usr/local/lib/libboost_stacktrace_backtrace.so /usr/local/lib/libboost_iostreams.so /usr/local/lib/libboost_math_c99.so.1.83.0 /usr/local/lib/libboost_stacktrace_addr2line.so /usr/local/lib/libboost_locale.so.1.83.0 /usr/local/lib/libboost_math_c99l.so.1.83.0 /usr/local/lib/libboost_timer.a /usr/local/lib/libboost_atomic.so.1.83.0 /usr/local/lib/libboost_url.so.1.83.0 /usr/local/lib/libboost_math_c99.a /usr/local/lib/libboost_date_time.so /usr/local/lib/libboost_fiber.a /usr/local/lib/libboost_filesystem.so.1.83.0 /usr/local/lib/libboost_date_time.a /usr/local/lib/libboost_log_setup.a /usr/local/lib/libboost_type_erasure.so /usr/local/lib/libboost_json.so.1.83.0 /usr/local/lib/libboost_wave.so.1.83.0 /usr/local/lib/libboost_unit_test_framework.so /usr/local/lib/libboost_type_erasure.so.1.83.0 /usr/local/lib/libboost_random.so.1.83.0 /usr/local/lib/libboost_filesystem.so /usr/local/lib/libboost_stacktrace_basic.a /usr/local/lib/libboost_math_c99.so /usr/local/lib/libboost_regex.a /usr/local/lib/libboost_coroutine.so.1.83.0 /usr/local/lib/libboost_filesystem.a /usr/local/lib/libboost_date_time.so.1.83.0 /usr/local/lib/libboost_wave.a /usr/local/lib/libboost_regex.so /usr/local/lib/libboost_container.a /usr/local/lib/libboost_locale.a /usr/local/lib/libboost_graph.so /usr/local/lib/libboost_graph.so.1.83.0 /usr/local/lib/libboost_math_tr1f.so.1.83.0 /usr/local/lib/libboost_math_tr1.so.1.83.0 /usr/local/lib/libboost_program_options.so.1.83.0 /usr/local/lib/libboost_iostreams.so.1.83.0 /usr/local/lib/libboost_math_c99f.so.1.83.0 /usr/local/lib/libboost_nowide.so.1.83.0 /usr/local/lib/libboost_thread.so.1.83.0 /usr/local/lib/libboost_unit_test_framework.a /usr/local/lib/cmake/boost_math_c99f-1.83.0 /usr/local/lib/cmake/boost_math_c99f-1.83.0/libboost_math_c99f-variant-shared.cmake /usr/local/lib/cmake/boost_math_c99f-1.83.0/boost_math_c99f-config.cmake /usr/local/lib/cmake/boost_math_c99f-1.83.0/libboost_math_c99f-variant-static.cmake /usr/local/lib/cmake/boost_math_c99f-1.83.0/boost_math_c99f-config-version.cmake /usr/local/lib/cmake/boost_test_exec_monitor-1.83.0 /usr/local/lib/cmake/boost_test_exec_monitor-1.83.0/libboost_test_exec_monitor-variant-static.cmake /usr/local/lib/cmake/boost_test_exec_monitor-1.83.0/libboost_test_exec_monitor-variant-shared.cmake /usr/local/lib/cmake/boost_test_exec_monitor-1.83.0/boost_test_exec_monitor-config.cmake /usr/local/lib/cmake/boost_test_exec_monitor-1.83.0/boost_test_exec_monitor-config-version.cmake /usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0 /usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0/libboost_stacktrace_backtrace-variant-shared.cmake /usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0/boost_stacktrace_backtrace-config-version.cmake /usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0/libboost_stacktrace_backtrace-variant-static.cmake /usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0/boost_stacktrace_backtrace-config.cmake /usr/local/lib/cmake/boost_wave-1.83.0 /usr/local/lib/cmake/boost_wave-1.83.0/libboost_wave-variant-shared.cmake /usr/local/lib/cmake/boost_wave-1.83.0/boost_wave-config-version.cmake /usr/local/lib/cmake/boost_wave-1.83.0/libboost_wave-variant-static.cmake /usr/local/lib/cmake/boost_wave-1.83.0/boost_wave-config.cmake /usr/local/lib/cmake/boost_log-1.83.0 /usr/local/lib/cmake/boost_log-1.83.0/libboost_log-variant-shared.cmake /usr/local/lib/cmake/boost_log-1.83.0/boost_log-config.cmake /usr/local/lib/cmake/boost_log-1.83.0/libboost_log-variant-static.cmake /usr/local/lib/cmake/boost_log-1.83.0/boost_log-config-version.cmake /usr/local/lib/cmake/boost_stacktrace_basic-1.83.0 /usr/local/lib/cmake/boost_stacktrace_basic-1.83.0/boost_stacktrace_basic-config.cmake /usr/local/lib/cmake/boost_stacktrace_basic-1.83.0/boost_stacktrace_basic-config-version.cmake /usr/local/lib/cmake/boost_stacktrace_basic-1.83.0/libboost_stacktrace_basic-variant-static.cmake /usr/local/lib/cmake/boost_stacktrace_basic-1.83.0/libboost_stacktrace_basic-variant-shared.cmake /usr/local/lib/cmake/boost_coroutine-1.83.0 /usr/local/lib/cmake/boost_coroutine-1.83.0/boost_coroutine-config-version.cmake /usr/local/lib/cmake/boost_coroutine-1.83.0/boost_coroutine-config.cmake /usr/local/lib/cmake/boost_coroutine-1.83.0/libboost_coroutine-variant-shared.cmake /usr/local/lib/cmake/boost_coroutine-1.83.0/libboost_coroutine-variant-static.cmake /usr/local/lib/cmake/boost_timer-1.83.0 /usr/local/lib/cmake/boost_timer-1.83.0/boost_timer-config-version.cmake /usr/local/lib/cmake/boost_timer-1.83.0/boost_timer-config.cmake /usr/local/lib/cmake/boost_timer-1.83.0/libboost_timer-variant-shared.cmake /usr/local/lib/cmake/boost_timer-1.83.0/libboost_timer-variant-static.cmake /usr/local/lib/cmake/boost_program_options-1.83.0 /usr/local/lib/cmake/boost_program_options-1.83.0/boost_program_options-config.cmake /usr/local/lib/cmake/boost_program_options-1.83.0/libboost_program_options-variant-static.cmake /usr/local/lib/cmake/boost_program_options-1.83.0/libboost_program_options-variant-shared.cmake /usr/local/lib/cmake/boost_program_options-1.83.0/boost_program_options-config-version.cmake /usr/local/lib/cmake/boost_math_tr1f-1.83.0 /usr/local/lib/cmake/boost_math_tr1f-1.83.0/libboost_math_tr1f-variant-static.cmake /usr/local/lib/cmake/boost_math_tr1f-1.83.0/boost_math_tr1f-config.cmake /usr/local/lib/cmake/boost_math_tr1f-1.83.0/boost_math_tr1f-config-version.cmake /usr/local/lib/cmake/boost_math_tr1f-1.83.0/libboost_math_tr1f-variant-shared.cmake /usr/local/lib/cmake/boost_date_time-1.83.0 /usr/local/lib/cmake/boost_date_time-1.83.0/libboost_date_time-variant-static.cmake /usr/local/lib/cmake/boost_date_time-1.83.0/libboost_date_time-variant-shared.cmake /usr/local/lib/cmake/boost_date_time-1.83.0/boost_date_time-config.cmake /usr/local/lib/cmake/boost_date_time-1.83.0/boost_date_time-config-version.cmake /usr/local/lib/cmake/boost_math_c99l-1.83.0 /usr/local/lib/cmake/boost_math_c99l-1.83.0/libboost_math_c99l-variant-static.cmake /usr/local/lib/cmake/boost_math_c99l-1.83.0/libboost_math_c99l-variant-shared.cmake /usr/local/lib/cmake/boost_math_c99l-1.83.0/boost_math_c99l-config.cmake /usr/local/lib/cmake/boost_math_c99l-1.83.0/boost_math_c99l-config-version.cmake /usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0 /usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0/libboost_prg_exec_monitor-variant-shared.cmake /usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0/libboost_prg_exec_monitor-variant-static.cmake /usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0/boost_prg_exec_monitor-config.cmake /usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0/boost_prg_exec_monitor-config-version.cmake /usr/local/lib/cmake/boost_wserialization-1.83.0 /usr/local/lib/cmake/boost_wserialization-1.83.0/libboost_wserialization-variant-shared.cmake /usr/local/lib/cmake/boost_wserialization-1.83.0/boost_wserialization-config-version.cmake /usr/local/lib/cmake/boost_wserialization-1.83.0/libboost_wserialization-variant-static.cmake /usr/local/lib/cmake/boost_wserialization-1.83.0/boost_wserialization-config.cmake /usr/local/lib/cmake/boost_serialization-1.83.0 /usr/local/lib/cmake/boost_serialization-1.83.0/boost_serialization-config.cmake /usr/local/lib/cmake/boost_serialization-1.83.0/libboost_serialization-variant-static.cmake /usr/local/lib/cmake/boost_serialization-1.83.0/boost_serialization-config-version.cmake /usr/local/lib/cmake/boost_serialization-1.83.0/libboost_serialization-variant-shared.cmake /usr/local/lib/cmake/boost_random-1.83.0 /usr/local/lib/cmake/boost_random-1.83.0/boost_random-config.cmake /usr/local/lib/cmake/boost_random-1.83.0/libboost_random-variant-static.cmake /usr/local/lib/cmake/boost_random-1.83.0/libboost_random-variant-shared.cmake /usr/local/lib/cmake/boost_random-1.83.0/boost_random-config-version.cmake /usr/local/lib/cmake/boost_log_setup-1.83.0 /usr/local/lib/cmake/boost_log_setup-1.83.0/boost_log_setup-config-version.cmake /usr/local/lib/cmake/boost_log_setup-1.83.0/libboost_log_setup-variant-shared.cmake /usr/local/lib/cmake/boost_log_setup-1.83.0/libboost_log_setup-variant-static.cmake /usr/local/lib/cmake/boost_log_setup-1.83.0/boost_log_setup-config.cmake /usr/local/lib/cmake/boost_exception-1.83.0 /usr/local/lib/cmake/boost_exception-1.83.0/boost_exception-config.cmake /usr/local/lib/cmake/boost_exception-1.83.0/boost_exception-config-version.cmake /usr/local/lib/cmake/boost_system-1.83.0 /usr/local/lib/cmake/boost_system-1.83.0/libboost_system-variant-shared.cmake /usr/local/lib/cmake/boost_system-1.83.0/boost_system-config-version.cmake /usr/local/lib/cmake/boost_system-1.83.0/libboost_system-variant-static.cmake /usr/local/lib/cmake/boost_system-1.83.0/boost_system-config.cmake /usr/local/lib/cmake/boost_python-1.83.0 /usr/local/lib/cmake/boost_python-1.83.0/boost_python-config.cmake /usr/local/lib/cmake/boost_python-1.83.0/boost_python-config-version.cmake /usr/local/lib/cmake/boost_thread-1.83.0 /usr/local/lib/cmake/boost_thread-1.83.0/libboost_thread-variant-shared.cmake /usr/local/lib/cmake/boost_thread-1.83.0/boost_thread-config.cmake /usr/local/lib/cmake/boost_thread-1.83.0/boost_thread-config-version.cmake /usr/local/lib/cmake/boost_thread-1.83.0/libboost_thread-variant-static.cmake /usr/local/lib/cmake/boost_json-1.83.0 /usr/local/lib/cmake/boost_json-1.83.0/libboost_json-variant-shared.cmake /usr/local/lib/cmake/boost_json-1.83.0/libboost_json-variant-static.cmake /usr/local/lib/cmake/boost_json-1.83.0/boost_json-config.cmake /usr/local/lib/cmake/boost_json-1.83.0/boost_json-config-version.cmake /usr/local/lib/cmake/boost_iostreams-1.83.0 /usr/local/lib/cmake/boost_iostreams-1.83.0/libboost_iostreams-variant-shared.cmake /usr/local/lib/cmake/boost_iostreams-1.83.0/boost_iostreams-config.cmake /usr/local/lib/cmake/boost_iostreams-1.83.0/libboost_iostreams-variant-static.cmake /usr/local/lib/cmake/boost_iostreams-1.83.0/boost_iostreams-config-version.cmake /usr/local/lib/cmake/boost_math_c99-1.83.0 /usr/local/lib/cmake/boost_math_c99-1.83.0/libboost_math_c99-variant-static.cmake /usr/local/lib/cmake/boost_math_c99-1.83.0/boost_math_c99-config-version.cmake /usr/local/lib/cmake/boost_math_c99-1.83.0/libboost_math_c99-variant-shared.cmake /usr/local/lib/cmake/boost_math_c99-1.83.0/boost_math_c99-config.cmake /usr/local/lib/cmake/boost_unit_test_framework-1.83.0 /usr/local/lib/cmake/boost_unit_test_framework-1.83.0/libboost_unit_test_framework-variant-static.cmake /usr/local/lib/cmake/boost_unit_test_framework-1.83.0/boost_unit_test_framework-config-version.cmake /usr/local/lib/cmake/boost_unit_test_framework-1.83.0/libboost_unit_test_framework-variant-shared.cmake /usr/local/lib/cmake/boost_unit_test_framework-1.83.0/boost_unit_test_framework-config.cmake /usr/local/lib/cmake/boost_filesystem-1.83.0 /usr/local/lib/cmake/boost_filesystem-1.83.0/libboost_filesystem-variant-static.cmake /usr/local/lib/cmake/boost_filesystem-1.83.0/boost_filesystem-config.cmake /usr/local/lib/cmake/boost_filesystem-1.83.0/boost_filesystem-config-version.cmake /usr/local/lib/cmake/boost_filesystem-1.83.0/libboost_filesystem-variant-shared.cmake /usr/local/lib/cmake/Boost-1.83.0 /usr/local/lib/cmake/Boost-1.83.0/BoostConfigVersion.cmake /usr/local/lib/cmake/Boost-1.83.0/BoostConfig.cmake /usr/local/lib/cmake/boost_headers-1.83.0 /usr/local/lib/cmake/boost_headers-1.83.0/boost_headers-config.cmake /usr/local/lib/cmake/boost_headers-1.83.0/boost_headers-config-version.cmake /usr/local/lib/cmake/boost_nowide-1.83.0 /usr/local/lib/cmake/boost_nowide-1.83.0/libboost_nowide-variant-static.cmake /usr/local/lib/cmake/boost_nowide-1.83.0/libboost_nowide-variant-shared.cmake /usr/local/lib/cmake/boost_nowide-1.83.0/boost_nowide-config.cmake /usr/local/lib/cmake/boost_nowide-1.83.0/boost_nowide-config-version.cmake /usr/local/lib/cmake/boost_atomic-1.83.0 /usr/local/lib/cmake/boost_atomic-1.83.0/boost_atomic-config-version.cmake /usr/local/lib/cmake/boost_atomic-1.83.0/boost_atomic-config.cmake /usr/local/lib/cmake/boost_atomic-1.83.0/libboost_atomic-variant-shared.cmake /usr/local/lib/cmake/boost_atomic-1.83.0/libboost_atomic-variant-static.cmake /usr/local/lib/cmake/boost_graph-1.83.0 /usr/local/lib/cmake/boost_graph-1.83.0/boost_graph-config-version.cmake /usr/local/lib/cmake/boost_graph-1.83.0/libboost_graph-variant-static.cmake /usr/local/lib/cmake/boost_graph-1.83.0/libboost_graph-variant-shared.cmake /usr/local/lib/cmake/boost_graph-1.83.0/boost_graph-config.cmake /usr/local/lib/cmake/boost_math_tr1-1.83.0 /usr/local/lib/cmake/boost_math_tr1-1.83.0/boost_math_tr1-config-version.cmake /usr/local/lib/cmake/boost_math_tr1-1.83.0/libboost_math_tr1-variant-static.cmake /usr/local/lib/cmake/boost_math_tr1-1.83.0/libboost_math_tr1-variant-shared.cmake /usr/local/lib/cmake/boost_math_tr1-1.83.0/boost_math_tr1-config.cmake /usr/local/lib/cmake/boost_graph_parallel-1.83.0 /usr/local/lib/cmake/boost_graph_parallel-1.83.0/boost_graph_parallel-config-version.cmake /usr/local/lib/cmake/boost_graph_parallel-1.83.0/boost_graph_parallel-config.cmake /usr/local/lib/cmake/boost_stacktrace_noop-1.83.0 /usr/local/lib/cmake/boost_stacktrace_noop-1.83.0/libboost_stacktrace_noop-variant-static.cmake /usr/local/lib/cmake/boost_stacktrace_noop-1.83.0/boost_stacktrace_noop-config-version.cmake /usr/local/lib/cmake/boost_stacktrace_noop-1.83.0/boost_stacktrace_noop-config.cmake /usr/local/lib/cmake/boost_stacktrace_noop-1.83.0/libboost_stacktrace_noop-variant-shared.cmake /usr/local/lib/cmake/boost_mpi-1.83.0 /usr/local/lib/cmake/boost_mpi-1.83.0/boost_mpi-config-version.cmake /usr/local/lib/cmake/boost_mpi-1.83.0/boost_mpi-config.cmake /usr/local/lib/cmake/boost_type_erasure-1.83.0 /usr/local/lib/cmake/boost_type_erasure-1.83.0/libboost_type_erasure-variant-shared.cmake /usr/local/lib/cmake/boost_type_erasure-1.83.0/libboost_type_erasure-variant-static.cmake /usr/local/lib/cmake/boost_type_erasure-1.83.0/boost_type_erasure-config-version.cmake /usr/local/lib/cmake/boost_type_erasure-1.83.0/boost_type_erasure-config.cmake /usr/local/lib/cmake/boost_chrono-1.83.0 /usr/local/lib/cmake/boost_chrono-1.83.0/boost_chrono-config-version.cmake /usr/local/lib/cmake/boost_chrono-1.83.0/libboost_chrono-variant-static.cmake /usr/local/lib/cmake/boost_chrono-1.83.0/libboost_chrono-variant-shared.cmake /usr/local/lib/cmake/boost_chrono-1.83.0/boost_chrono-config.cmake /usr/local/lib/cmake/boost_math_tr1l-1.83.0 /usr/local/lib/cmake/boost_math_tr1l-1.83.0/libboost_math_tr1l-variant-shared.cmake /usr/local/lib/cmake/boost_math_tr1l-1.83.0/boost_math_tr1l-config.cmake /usr/local/lib/cmake/boost_math_tr1l-1.83.0/boost_math_tr1l-config-version.cmake /usr/local/lib/cmake/boost_math_tr1l-1.83.0/libboost_math_tr1l-variant-static.cmake /usr/local/lib/cmake/boost_contract-1.83.0 /usr/local/lib/cmake/boost_contract-1.83.0/libboost_contract-variant-shared.cmake /usr/local/lib/cmake/boost_contract-1.83.0/boost_contract-config-version.cmake /usr/local/lib/cmake/boost_contract-1.83.0/boost_contract-config.cmake /usr/local/lib/cmake/boost_contract-1.83.0/libboost_contract-variant-static.cmake /usr/local/lib/cmake/boost_fiber-1.83.0 /usr/local/lib/cmake/boost_fiber-1.83.0/libboost_fiber-variant-static.cmake /usr/local/lib/cmake/boost_fiber-1.83.0/boost_fiber-config-version.cmake /usr/local/lib/cmake/boost_fiber-1.83.0/libboost_fiber-variant-shared.cmake /usr/local/lib/cmake/boost_fiber-1.83.0/boost_fiber-config.cmake /usr/local/lib/cmake/BoostDetectToolset-1.83.0.cmake /usr/local/lib/cmake/boost_context-1.83.0 /usr/local/lib/cmake/boost_context-1.83.0/boost_context-config.cmake /usr/local/lib/cmake/boost_context-1.83.0/libboost_context-variant-shared.cmake /usr/local/lib/cmake/boost_context-1.83.0/libboost_context-variant-static.cmake /usr/local/lib/cmake/boost_context-1.83.0/boost_context-config-version.cmake /usr/local/lib/cmake/boost_container-1.83.0 /usr/local/lib/cmake/boost_container-1.83.0/libboost_container-variant-shared.cmake /usr/local/lib/cmake/boost_container-1.83.0/boost_container-config-version.cmake /usr/local/lib/cmake/boost_container-1.83.0/boost_container-config.cmake /usr/local/lib/cmake/boost_container-1.83.0/libboost_container-variant-static.cmake /usr/local/lib/cmake/boost_regex-1.83.0 /usr/local/lib/cmake/boost_regex-1.83.0/libboost_regex-variant-shared.cmake /usr/local/lib/cmake/boost_regex-1.83.0/boost_regex-config.cmake /usr/local/lib/cmake/boost_regex-1.83.0/libboost_regex-variant-static.cmake /usr/local/lib/cmake/boost_regex-1.83.0/boost_regex-config-version.cmake /usr/local/lib/cmake/boost_url-1.83.0 /usr/local/lib/cmake/boost_url-1.83.0/boost_url-config-version.cmake /usr/local/lib/cmake/boost_url-1.83.0/libboost_url-variant-static.cmake /usr/local/lib/cmake/boost_url-1.83.0/libboost_url-variant-shared.cmake /usr/local/lib/cmake/boost_url-1.83.0/boost_url-config.cmake /usr/local/lib/cmake/boost_locale-1.83.0 /usr/local/lib/cmake/boost_locale-1.83.0/libboost_locale-variant-static.cmake /usr/local/lib/cmake/boost_locale-1.83.0/boost_locale-config.cmake /usr/local/lib/cmake/boost_locale-1.83.0/libboost_locale-variant-shared.cmake /usr/local/lib/cmake/boost_locale-1.83.0/boost_locale-config-version.cmake /usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0 /usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0/libboost_stacktrace_addr2line-variant-static.cmake /usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0/libboost_stacktrace_addr2line-variant-shared.cmake /usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0/boost_stacktrace_addr2line-config-version.cmake /usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0/boost_stacktrace_addr2line-config.cmake /usr/local/lib/libboost_fiber.so.1.83.0 /usr/local/lib/libboost_chrono.a /usr/local/lib/libboost_serialization.so /usr/local/lib/libboost_fiber.so /usr/local/lib/libboost_wserialization.a /usr/local/lib/libboost_unit_test_framework.so.1.83.0 /usr/local/lib/libboost_stacktrace_noop.a /usr/local/lib/libboost_iostreams.a /usr/local/lib/libboost_math_c99f.a /usr/local/lib/libboost_exception.a /usr/local/lib/libboost_type_erasure.a /usr/local/lib/libboost_thread.a /usr/local/lib/libboost_chrono.so.1.83.0 /usr/local/lib/libboost_math_c99l.so /usr/local/lib/libboost_log_setup.so.1.83.0 /usr/local/lib/libboost_stacktrace_addr2line.a /usr/local/lib/libboost_container.so.1.83.0 /usr/local/lib/libboost_context.so /usr/local/lib/libboost_program_options.so /usr/local/lib/libboost_stacktrace_basic.so /usr/local/lib/libboost_graph.a /usr/local/lib/libboost_prg_exec_monitor.so.1.83.0 /usr/local/lib/libboost_atomic.a /usr/local/lib/libboost_math_tr1l.a /usr/local/lib/libboost_log.so.1.83.0 /usr/local/lib/libboost_random.a /usr/local/lib/libboost_math_tr1f.so /usr/local/lib/libboost_contract.a /usr/local/lib/libboost_atomic.so /usr/local/lib/libboost_serialization.so.1.83.0 /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_wave.so /usr/local/lib/libboost_program_options.a /usr/local/lib/libboost_context.so.1.83.0 /usr/local/lib/libboost_coroutine.a /usr/local/lib/libboost_json.a /usr/local/lib/libboost_stacktrace_noop.so.1.83.0 /usr/local/lib/libboost_wserialization.so.1.83.0 /usr/local/lib/libboost_math_tr1.a /usr/local/lib/libboost_chrono.so /usr/local/lib/libboost_stacktrace_backtrace.a /usr/local/lib/libboost_stacktrace_addr2line.so.1.83.0 /usr/local/lib/libboost_stacktrace_basic.so.1.83.0 /usr/local/lib/libboost_math_c99l.a /usr/local/lib/libboost_log.so /usr/local/lib/libboost_log.a /usr/local/lib/libboost_math_tr1l.so /usr/local/lib/libboost_container.so /usr/local/lib/libboost_coroutine.so /usr/local/lib/libboost_prg_exec_monitor.so /usr/local/lib/libboost_nowide.a /usr/local/lib/libboost_url.so /usr/local/lib/libboost_url.a /usr/local/lib/libboost_math_c99f.so /usr/local/lib/libboost_stacktrace_backtrace.so.1.83.0 /usr/local/lib/libboost_test_exec_monitor.a /usr/local/lib/libboost_json.so /usr/local/lib/libboost_wserialization.so /usr/local/lib/libboost_system.a /usr/local/lib/libboost_regex.so.1.83.0 /usr/local/lib/libboost_contract.so /usr/local/lib/libboost_stacktrace_noop.so /usr/local/lib/libboost_locale.so /usr/local/lib/libboost_math_tr1.so /usr/local/lib/libboost_timer.so.1.83.0 /usr/local/lib/libboost_random.so /usr/local/lib/libboost_math_tr1l.so.1.83.0 /usr/local/lib/libboost_timer.so </span></pre> </div> <!-- endregion --> <!-- #region clean up --> <h3 id="clean">Clean Up</h3> <p> Many Ubuntu/Debian packages are left behind by the Boost installation program. You can view them as follows: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idac3bb486cbb0'><button class='copyBtn' data-clipboard-target='#idac3bb486cbb0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>apt --dry-run autoremove | grep -Po '^Remv \K[^ ]+' <span class='unselectable'>libboost-mpi-python-dev libboost-mpi-python1.74-dev libboost-mpi-python1.74.0 mpi-default-bin libboost-mpi-dev libboost-mpi1.74-dev mpi-default-dev libopenmpi-dev openmpi-bin libcoarrays-openmpi-dev libopenmpi3 libucx0 libfabric1 ibverbs-providers libboost-atomic-dev libboost-coroutine-dev libboost-coroutine1.74-dev libboost-fiber-dev libboost-fiber1.74-dev libboost-context1.74-dev libboost-type-erasure-dev libboost-type-erasure1.74-dev libboost-thread1.74-dev libboost-log-dev libboost-log1.74-dev libboost-atomic1.74-dev libboost-atomic1.74.0 libboost-chrono-dev libboost-timer-dev libboost-timer1.74-dev libboost-chrono1.74-dev libboost-timer1.74.0 libboost-chrono1.74.0 libboost-container-dev libboost-container1.74-dev libboost-container1.74.0 libboost-context-dev libboost-fiber1.74.0 libboost-coroutine1.74.0 libboost-context1.74.0 libboost-date-time-dev libboost-date-time1.74-dev libboost-date-time1.74.0 libboost-dev libboost-exception-dev libboost-exception1.74-dev libboost-filesystem-dev libboost-wave-dev libboost-wave1.74-dev libboost-filesystem1.74-dev libboost-graph-dev libboost-graph-parallel-dev libboost-graph-parallel1.74-dev libboost-graph-parallel1.74.0 libboost-graph1.74-dev libboost-graph1.74.0 libboost-iostreams-dev libboost-iostreams1.74-dev libboost-locale-dev libboost-locale1.74-dev libboost-log1.74.0 libboost-math-dev libboost-math1.74-dev libboost-math1.74.0 libboost-mpi1.74.0 libboost-nowide-dev libboost-nowide1.74-dev libboost-nowide1.74.0 libboost-numpy-dev libboost-numpy1.74-dev libboost-numpy1.74.0 libboost-program-options-dev libboost-program-options1.74-dev libboost-program-options1.74.0 libboost-python-dev libboost-python1.74-dev libboost-python1.74.0 libboost-random-dev libboost-random1.74-dev libboost-random1.74.0 libboost-regex-dev libboost-regex1.74-dev libboost-serialization-dev libboost-serialization1.74-dev libboost-serialization1.74.0 libboost-stacktrace-dev libboost-stacktrace1.74-dev libboost-stacktrace1.74.0 libboost-system-dev libboost-system1.74-dev libboost-system1.74.0 libboost-test-dev libboost-test1.74-dev libboost-test1.74.0 libboost-thread-dev libboost-tools-dev libboost-type-erasure1.74.0 libboost-wave1.74.0 libboost1.74-dev libboost1.74-tools-dev libcaf-openmpi-3 libpmix-dev libevent-dev libevent-extra-2.1-7 libevent-openssl-2.1-7 libpmix2 libevent-pthreads-2.1-7 libhwloc-dev libhwloc-plugins libhwloc15 libibverbs-dev librdmacm1 libibverbs1 libjs-jquery-ui libmunge2 libnl-route-3-dev libnl-3-dev libnuma-dev libpsm-infinipath1 libpsm2-2 openmpi-common </span></pre> </div> <!-- endregion --> <p> Remove the packages as follows: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id16d4d3515d80'><button class='copyBtn' data-clipboard-target='#id16d4d3515d80' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt autoremove <span class='unselectable'>Reading package lists... Done Building dependency tree... Done Reading state information... Done The following packages will be REMOVED: ibverbs-providers libboost-atomic-dev libboost-atomic1.74-dev libboost-atomic1.74.0 libboost-chrono-dev libboost-chrono1.74-dev libboost-chrono1.74.0 libboost-container-dev libboost-container1.74-dev libboost-container1.74.0 libboost-context-dev libboost-context1.74-dev libboost-context1.74.0 libboost-coroutine-dev libboost-coroutine1.74-dev libboost-coroutine1.74.0 libboost-date-time-dev libboost-date-time1.74-dev libboost-date-time1.74.0 libboost-dev libboost-exception-dev libboost-exception1.74-dev libboost-fiber-dev libboost-fiber1.74-dev libboost-fiber1.74.0 libboost-filesystem-dev libboost-filesystem1.74-dev libboost-graph-dev libboost-graph-parallel-dev libboost-graph-parallel1.74-dev libboost-graph-parallel1.74.0 libboost-graph1.74-dev libboost-graph1.74.0 libboost-iostreams-dev libboost-iostreams1.74-dev libboost-locale-dev libboost-locale1.74-dev libboost-log-dev libboost-log1.74-dev libboost-log1.74.0 libboost-math-dev libboost-math1.74-dev libboost-math1.74.0 libboost-mpi-dev libboost-mpi-python-dev libboost-mpi-python1.74-dev libboost-mpi-python1.74.0 libboost-mpi1.74-dev libboost-mpi1.74.0 libboost-nowide-dev libboost-nowide1.74-dev libboost-nowide1.74.0 libboost-numpy-dev libboost-numpy1.74-dev libboost-numpy1.74.0 libboost-program-options-dev libboost-program-options1.74-dev libboost-program-options1.74.0 libboost-python-dev libboost-python1.74-dev libboost-python1.74.0 libboost-random-dev libboost-random1.74-dev libboost-random1.74.0 libboost-regex-dev libboost-regex1.74-dev libboost-serialization-dev libboost-serialization1.74-dev libboost-serialization1.74.0 libboost-stacktrace-dev libboost-stacktrace1.74-dev libboost-stacktrace1.74.0 libboost-system-dev libboost-system1.74-dev libboost-system1.74.0 libboost-test-dev libboost-test1.74-dev libboost-test1.74.0 libboost-thread-dev libboost-thread1.74-dev libboost-timer-dev libboost-timer1.74-dev libboost-timer1.74.0 libboost-tools-dev libboost-type-erasure-dev libboost-type-erasure1.74-dev libboost-type-erasure1.74.0 libboost-wave-dev libboost-wave1.74-dev libboost-wave1.74.0 libboost1.74-dev libboost1.74-tools-dev libcaf-openmpi-3 libcoarrays-openmpi-dev libevent-dev libevent-extra-2.1-7 libevent-openssl-2.1-7 libevent-pthreads-2.1-7 libfabric1 libhwloc-dev libhwloc-plugins libhwloc15 libibverbs-dev libibverbs1 libjs-jquery-ui libmunge2 libnl-3-dev libnl-route-3-dev libnuma-dev libopenmpi-dev libopenmpi3 libpmix-dev libpmix2 libpsm-infinipath1 libpsm2-2 librdmacm1 libucx0 mpi-default-bin mpi-default-dev openmpi-bin openmpi-common 0 upgraded, 0 newly installed, 121 to remove and 4 not upgraded. After this operation, 413 MB disk space will be freed. (Reading database ... 399799 files and directories currently installed.) Removing libboost-mpi-python-dev (1.74.0.3ubuntu7) ... Removing libboost-mpi-python1.74-dev (1.74.0-18.1ubuntu3) ... Removing libboost-mpi-python1.74.0 (1.74.0-18.1ubuntu3) ... Removing mpi-default-bin (1.14) ... Removing libboost-mpi-dev (1.74.0.3ubuntu7) ... Removing libboost-mpi1.74-dev (1.74.0-18.1ubuntu3) ... Removing mpi-default-dev (1.14) ... Removing libopenmpi-dev:amd64 (4.1.4-3ubuntu2) ... Removing libcoarrays-openmpi-dev:amd64 (2.10.1-1) ... update-alternatives: warning: alternative /usr/bin/caf.openmpi (part of link group caf) doesn't exist; removing from list of alternatives update-alternatives: warning: /etc/alternatives/caf is dangling; it will be updated with best choice Removing libboost-atomic-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-coroutine-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-coroutine1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-fiber-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-fiber1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-type-erasure-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-type-erasure1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-log-dev (1.74.0.3ubuntu7) ... Removing libboost-log1.74-dev (1.74.0-18.1ubuntu3) ... Removing libboost-chrono-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-timer-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-timer1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-timer1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-container-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-container1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-container1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-context-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-fiber1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-coroutine1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-date-time-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-exception-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-exception1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-filesystem-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-wave-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-wave1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-filesystem1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-graph-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-graph-parallel-dev (1.74.0.3ubuntu7) ... Removing libboost-graph-parallel1.74-dev (1.74.0-18.1ubuntu3) ... Removing libboost-graph-parallel1.74.0 (1.74.0-18.1ubuntu3) ... Removing libboost-graph1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-graph1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-iostreams-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-iostreams1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-locale-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-locale1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-log1.74.0 (1.74.0-18.1ubuntu3) ... Removing libboost-math-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-math1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-math1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-mpi1.74.0 (1.74.0-18.1ubuntu3) ... Removing libboost-nowide-dev (1.74.0.3ubuntu7) ... Removing libboost-nowide1.74-dev (1.74.0-18.1ubuntu3) ... Removing libboost-nowide1.74.0 (1.74.0-18.1ubuntu3) ... Removing libboost-numpy-dev (1.74.0.3ubuntu7) ... Removing libboost-numpy1.74-dev (1.74.0-18.1ubuntu3) ... Removing libboost-numpy1.74.0 (1.74.0-18.1ubuntu3) ... Removing libboost-program-options-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-program-options1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-program-options1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-python-dev (1.74.0.3ubuntu7) ... Removing libboost-python1.74-dev (1.74.0-18.1ubuntu3) ... Removing libboost-python1.74.0 (1.74.0-18.1ubuntu3) ... Removing libboost-random-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-random1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-random1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-regex-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-regex1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-serialization-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-stacktrace-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-stacktrace1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-stacktrace1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-system-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-test-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-test1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-test1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-thread-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-tools-dev (1.74.0.3ubuntu7) ... Removing libboost-type-erasure1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-wave1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost1.74-tools-dev (1.74.0-18.1ubuntu3) ... Removing libcaf-openmpi-3:amd64 (2.10.1-1) ... Removing libpmix-dev:amd64 (4.2.2-1) ... Removing libevent-dev (2.1.12-stable-8ubuntu3) ... Removing libevent-extra-2.1-7:amd64 (2.1.12-stable-8ubuntu3) ... Removing libevent-openssl-2.1-7:amd64 (2.1.12-stable-8ubuntu3) ... Removing libhwloc-dev:amd64 (2.9.0-1) ... Removing libibverbs-dev:amd64 (44.0-2) ... Removing libjs-jquery-ui (1.13.2+dfsg-1) ... Removing libnl-route-3-dev:amd64 (3.7.0-0.2) ... Removing libnl-3-dev:amd64 (3.7.0-0.2) ... Removing libnuma-dev:amd64 (2.0.16-1) ... Removing openmpi-bin (4.1.4-3ubuntu2) ... Removing libopenmpi3:amd64 (4.1.4-3ubuntu2) ... Removing libucx0:amd64 (1.13.1-1) ... Removing libfabric1:amd64 (1.17.0-3) ... Removing ibverbs-providers:amd64 (44.0-2) ... Removing libboost-context1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-thread1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-atomic1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-atomic1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-chrono1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-chrono1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-context1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-date-time1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-date-time1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-serialization1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-serialization1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-system1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-system1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libpmix2:amd64 (4.2.2-1) ... Removing libevent-pthreads-2.1-7:amd64 (2.1.12-stable-8ubuntu3) ... Removing libhwloc-plugins:amd64 (2.9.0-1) ... Removing libhwloc15:amd64 (2.9.0-1) ... Removing librdmacm1:amd64 (44.0-2) ... Removing libibverbs1:amd64 (44.0-2) ... Removing libmunge2 (0.5.15-2) ... Removing libpsm-infinipath1 (3.3+20.604758e7-6.2) ... update-alternatives: warning: alternative /usr/lib/libpsm1/libpsm_infinipath.so.1.16 (part of link group libpsm_infinipath.so.1) doesn't exist; removing from list of alternatives update-alternatives: warning: /etc/alternatives/libpsm_infinipath.so.1 is dangling; it will be updated with best choice Removing libpsm2-2 (11.2.185-2) ... Removing openmpi-common (4.1.4-3ubuntu2) ... Processing triggers for man-db (2.11.2-1) ... Processing triggers for libc-bin (2.37-0ubuntu2) ... /sbin/ldconfig.real: /usr/lib/wsl/lib/libcuda.so.1 is not a symbolic link </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region conclusion --> <h2 id="go">Go Forth and Get Boosted</h2> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <p> Boost usage as a C++ general-pupose library is widespread throughout a diverse range of industries and requirements. </p> <!-- endregion --> PDF Manipulation 2023-09-08T00:00:00-04:00 https://mslinn.github.io/blog/2023/09/08/pdftk <!-- #region intro --> <h2 id="install">Installation</h2> <p> The <a href='https://www.pdflabs.com/tools/pdftk-the-pdf-toolkit/' target='_blank' rel="nofollow"><code>pdftk</code> website</a> discusses <code>pdktk</code> (a F/OSS command-line program), a Windows GUI, and mentions the book <a href='https://github.com/InspectorDidi/Hacking-Books/blob/master/PDF%20Hacks.pdf' target='_blank' rel="nofollow"><b>PDF Hacks</b></a>. </p> <p> An alternative free GUI is <a href='https://portableapps.com/apps/office/pdftk_builder_portable' target='_blank' rel="nofollow">PDFTK Builder</a>. </p> <p> Other command-line utilties for manipulating PDF documents are: </p> <ul> <li><a href='https://en.wikipedia.org/wiki/Poppler_(software)' target='_blank' rel="nofollow">Poppler</a> (GPL)</li> <li><a href='https://qpdf.readthedocs.io/en/stable/index.html' target='_blank' rel="nofollow">QPDF</a></li> </ul> <p> Install the command-line <code>pdftk</code> program on Ubuntu: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3eea94fddc40'><button class='copyBtn' data-clipboard-target='#id3eea94fddc40' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install pdftk</pre> </div> <!-- endregion --> <p> This is the <code>pdftk</code> help text: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id052237e1dcb9'><button class='copyBtn' data-clipboard-target='#id052237e1dcb9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>man pdftk <span class='unselectable'>pdftk.pdftk-java(1) General Commands Manual pdftk.pdftk-java(1)<br/> NAME pdftk - A handy tool for manipulating PDF<br/> SYNOPSIS pdftk &lt;input PDF files | - | PROMPT&gt; [ input_pw &lt;input PDF owner passwords | PROMPT&gt; ] [ &lt;operation&gt; &lt;operation arguments&gt; ] [ output &lt;output filename | - | PROMPT&gt; ] [ encrypt_40bit | encrypt_128bit | encrypt_aes128 ] [ allow &lt;permissions&gt; ] [ owner_pw &lt;owner password | PROMPT&gt; ] [ user_pw &lt;user password | PROMPT&gt; ] [ flatten ] [ need_appearances ] [ compress | uncompress ] [ keep_first_id | keep_final_id ] [ drop_xfa ] [ drop_xmp ] [ replacement_font &lt;font name&gt; ] [ verbose ] [ dont_ask | do_ask ] Where: &lt;operation&gt; may be empty, or: [ cat | shuffle | burst | rotate | generate_fdf | fill_form | background | multibackground | stamp | multistamp | dump_data | dump_data_utf8 | dump_data_fields | dump_data_fields_utf8 | dump_data_annots | update_info | update_info_utf8 | attach_files | unpack_files ]<br/> For Complete Help: pdftk --help<br/> DESCRIPTION If PDF is electronic paper, then pdftk is an electronic staple- remover, hole-punch, binder, secret-decoder-ring, and X-Ray- glasses. Pdftk is a simple tool for doing everyday things with PDF documents. Use it to:<br/> * Merge PDF Documents or Collate PDF Page Scans * Split PDF Pages into a New Document * Rotate PDF Documents or Pages * Decrypt Input as Necessary (Password Required) * Encrypt Output as Desired * Fill PDF Forms with X/FDF Data and/or Flatten Forms * Generate FDF Data Stencils from PDF Forms * Apply a Background Watermark or a Foreground Stamp * Report PDF Metrics, Bookmarks and Metadata * Add/Update PDF Metrics, Bookmarks or Metadata * Attach Files to PDF Pages or the PDF Document * Unpack PDF Attachments * Burst a PDF Document into Single Pages * Uncompress and Re-Compress Page Streams * Repair Corrupted PDF (Where Possible)<br/> OPTIONS A summary of options is included below.<br/> --help, -h Show this summary of options.<br/> &lt;input PDF files | - | PROMPT&gt; A list of the input PDF files. If you plan to combine these PDFs (without using handles) then list files in the order you want them combined. Use - to pass a sin&#8208; gle PDF into pdftk via stdin. Input files can be asso&#8208; ciated with handles, where a handle is one or more up&#8208; per-case letters:<br/> &lt;input PDF handle&gt;=&lt;input PDF filename&gt;<br/> Handles are often omitted. They are useful when speci&#8208; fying PDF passwords or page ranges, later.<br/> For example: A=input1.pdf QT=input2.pdf M=input3.pdf<br/> [input_pw &lt;input PDF owner passwords | PROMPT&gt;] Input PDF owner passwords, if necessary, are associated with files by using their handles:<br/> &lt;input PDF handle&gt;=&lt;input PDF file owner password&gt;<br/> If handles are not given, then passwords are associated with input files by order.<br/> Most pdftk features require that encrypted input PDF are accompanied by the ~owner~ password. If the input PDF has no owner password, then the user password must be given, instead. If the input PDF has no passwords, then no password should be given.<br/> When running in do_ask mode, pdftk will prompt you for a password if the supplied password is incorrect or none was given.<br/> [&lt;operation&gt; &lt;operation arguments&gt;] Available operations are: cat, shuffle, burst, rotate, generate_fdf, fill_form, background, multibackground, stamp, multistamp, dump_data, dump_data_utf8, dump_data_fields, dump_data_fields_utf8, dump_data_an&#8208; nots, update_info, update_info_utf8, attach_files, un&#8208; pack_files. Some operations takes additional arguments, described below.<br/> If this optional argument is omitted, then pdftk runs in &#39;filter&#39; mode. Filter mode takes only one PDF input and creates a new PDF after applying all of the output op&#8208; tions, like encryption and compression.<br/> cat [&lt;page ranges&gt;] Assembles (catenates) pages from input PDFs to create a new PDF. Use cat to merge PDF pages or to split PDF pages from documents. You can also use it to rotate PDF pages. Page order in the new PDF is specified by the order of the given page ranges. Page ranges are described like this:<br/> &lt;input PDF handle&gt;[&lt;begin page number&gt;[-&lt;end page number&gt;[&lt;qualifier&gt;]]][&lt;page rotation&gt;]<br/> Where the handle identifies one of the input PDF files, and the beginning and ending page numbers are one-based references to pages in the PDF file. The qualifier can be even, odd, or ~, and the page rota&#8208; tion can be north, south, east, west, left, right, or down.<br/> If a PDF handle is given but no pages are specified, then the entire PDF is used. If no pages are speci&#8208; fied for any of the input PDFs, then the input PDFs&#39; bookmarks are also merged and included in the output.<br/> If the handle is omitted from the page range, then the pages are taken from the first input PDF.<br/> The even qualifier causes pdftk to use only the even- numbered PDF pages, so 1-6even yields pages 2, 4 and 6 in that order. 6-1even yields pages 6, 4 and 2 in that order.<br/> The odd qualifier works similarly to the even.<br/> Pages can be subtracted from a page range using the ~ qualifier followed by a page range. For instance, 1-20~5-6 and 1-20~5~6 are equivalent to 1-4 7-20, and ~5 yields all pages except page 5. Depending on your shell, you may need to quote this argument because of the ~ at the beginning.<br/> The page rotation setting can cause pdftk to rotate pages and documents. Each option sets the page rota&#8208; tion as follows (in degrees): north: 0, east: 90, south: 180, west: 270, left: -90, right: +90, down: +180. left, right, and down make relative adjustments to a page&#39;s rotation.<br/> If no arguments are passed to cat, then pdftk com&#8208; bines all input PDFs in the order they were given to create the output.<br/> NOTES: * &lt;end page number&gt; may be less than &lt;begin page num&#8208; ber&gt;. * The keyword end may be used to reference the final page of a document instead of a page number. * Reference a single page by omitting the ending page number. * The handle may be used alone to represent the en&#8208; tire PDF document, e.g., B1-end is the same as B. * You can reference page numbers in reverse order by prefixing them with the letter r. For example, page r1 is the last page of the document, r2 is the next- to-last page of the document, and rend is the first page of the document. You can use this prefix in ranges, too, for example r3-r1 is the last three pages of a PDF.<br/> Page Range Examples without Handles: 1\-endeast &ndash; rotate entire document 90 degrees 5 11 20 &ndash; take single pages from input PDF 5-25oddwest &ndash; take odd pages in range, rotate 90 de&#8208; grees 6-1 &ndash; reverse pages in range from input PDF<br/> Page Range Examples Using Handles: Say A=in1.pdf B=in2.pdf, then: A1-21 &ndash; take range from in1.pdf Bend-1odd &ndash; take all odd pages from in2.pdf in re&#8208; verse order A72 &ndash; take a single page from in1.pdf A1-21 Beven A72 &ndash; assemble pages from both in1.pdf and in2.pdf Awest &ndash; rotate entire in1.pdf document 90 degrees B &ndash; use all of in2.pdf A2-30evenleft &ndash; take the even pages from the range, remove 90 degrees from each page&#39;s rotation A A &ndash; catenate in1.pdf with in1.pdf Aevenwest Aoddeast &ndash; apply rotations to even pages, odd pages from in1.pdf Awest Bwest Bdown &ndash; catenate rotated documents<br/> shuffle [&lt;page ranges&gt;] Collates pages from input PDFs to create a new PDF. Works like the cat operation except that it takes one page at a time from each page range to assemble the output PDF. If one range runs out of pages, it con&#8208; tinues with the remaining ranges. Ranges can use all of the features described above for cat, like reverse page ranges, multiple ranges from a single PDF, and page rotation. This feature was designed to help collate PDF pages after scanning paper documents.<br/> burst Splits a single input PDF document into individual pages. Also creates a report named doc_data.txt which is the same as the output from dump_data. The output section can contain a printf-styled format string to name these pages. For example, if you want pages named page_01.pdf, page_02.pdf, etc., pass output page_%02d.pdf to pdftk. If the pattern is omitted, then a default pattern g_%04d.pdf is appended and produces pages named pg_0001.pdf, pg_0002.pdf, etc. Encryption can be applied to the output by appending output options such as owner_pw, e.g.:<br/> pdftk in.pdf burst owner_pw foopass<br/> rotate [&lt;page ranges&gt;] Takes a single input PDF and rotates just the speci&#8208; fied pages. All other pages remain unchanged. The page order remains unchanged. Specify the pages to rotate using the same notation as you would with cat, except you omit the pages that you aren&#39;t rotating:<br/> [&lt;begin page number&gt;[-&lt;end page number&gt;[&lt;quali&#8208; fier&gt;]]][&lt;page rotation&gt;]<br/> The qualifier can be even or odd, and the page rota&#8208; tion can be north, south, east, west, left, right, or down.<br/> Each option sets the page rotation as follows (in de&#8208; grees): north: 0, east: 90, south: 180, west: 270, left: -90, right: +90, down: +180. left, right, and down make relative adjustments to a page&#39;s rotation.<br/> The given order of the pages doesn&#39;t change the page order in the output.<br/> generate_fdf Reads a single input PDF file and generates an FDF file suitable for fill_form out of it to the given output filename or (if no output is given) to stdout. Does not create a new PDF.<br/> fill_form &lt;FDF data filename | XFDF data filename | - | PROMPT&gt; Fills the single input PDF&#39;s form fields with the data from an FDF file, XFDF file or stdin. Enter the data filename after fill_form, or use - to pass the data via stdin, like so:<br/> pdftk form.pdf fill_form data.fdf output form.filled.pdf<br/> If the input FDF file includes Rich Text formatted data in addition to plain text, then the Rich Text data is packed into the form fields as well as the plain text. Pdftk also sets a flag that cues Reader/Acrobat to generate new field appearances based on the Rich Text data. So when the user opens the PDF, the viewer will create the Rich Text appear&#8208; ance on the spot. If the user&#39;s PDF viewer does not support Rich Text, then the user will see the plain text data instead. If you flatten this form before Acrobat has a chance to create (and save) new field appearances, then the plain text field data is what you&#39;ll see.<br/> Also see the flatten, need_appearances, and replace&#8208; ment_font options.<br/> background &lt;background PDF filename | - | PROMPT&gt; Applies a PDF watermark to the background of a single input PDF. Pass the background PDF&#39;s filename after background like so:<br/> pdftk in.pdf background back.pdf output out.pdf<br/> Pdftk uses only the first page from the background PDF and applies it to every page of the input PDF. This page is scaled and rotated as needed to fit the input page. You can use - to pass a background PDF into pdftk via stdin.<br/> If the input PDF does not have a transparent back&#8208; ground (such as a PDF created from page scans) then the resulting background won&#39;t be visible &ndash; use the stamp operation instead.<br/> multibackground &lt;background PDF filename | - | PROMPT&gt; Same as the background operation, but applies each page of the background PDF to the corresponding page of the input PDF. If the input PDF has more pages than the stamp PDF, then the final stamp page is re&#8208; peated across these remaining pages in the input PDF.<br/> stamp &lt;stamp PDF filename | - | PROMPT&gt; This behaves just like the background operation ex&#8208; cept it overlays the stamp PDF page on top of the in&#8208; put PDF document&#39;s pages. This works best if the stamp PDF page has a transparent background.<br/> multistamp &lt;stamp PDF filename | - | PROMPT&gt; Same as the stamp operation, but applies each page of the background PDF to the corresponding page of the input PDF. If the input PDF has more pages than the stamp PDF, then the final stamp page is repeated across these remaining pages in the input PDF.<br/> dump_data Reads a single input PDF file and reports its meta&#8208; data, bookmarks (a/k/a outlines), page metrics (me&#8208; dia, rotation and labels), data embedded by STAMPtk (see STAMPtk&#39;s embed option) and other data to the given output filename or (if no output is given) to stdout. Non-ASCII characters are encoded as XML nu&#8208; merical entities. Does not create a new PDF.<br/> dump_data_utf8 Same as dump_data except that the output is encoded as UTF-8.<br/> dump_data_fields Reads a single input PDF file and reports form field statistics to the given output filename or (if no output is given) to stdout. Non-ASCII characters are encoded as XML numerical entities. Does not create a new PDF.<br/> dump_data_fields_utf8 Same as dump_data_fields except that the output is encoded as UTF-8.<br/> dump_data_annots This operation currently reports only link annota&#8208; tions. Reads a single input PDF file and reports an&#8208; notation information to the given output filename or (if no output is given) to stdout. Non-ASCII charac&#8208; ters are encoded as XML numerical entities. Does not create a new PDF.<br/> update_info &lt;info data filename | - | PROMPT&gt; Changes the bookmarks, page labels, page sizes, page rotations, and metadata in a single PDF&#39;s Info dic&#8208; tionary to match the input data file. The input data file uses the same syntax as the output from dump_data. Non-ASCII characters should be encoded as XML numerical entities.<br/> This operation does not change the metadata stored in the PDF&#39;s XMP stream, if it has one. (For this reason you should include a ModDate entry in your updated info with a current date/timestamp, format: D:YYYYM&#8208; MDDHHmmSS, e.g. D:201307241346 &ndash; omitted data after YYYY revert to default values.)<br/> For example:<br/> pdftk in.pdf update_info in.info output out.pdf<br/> update_info_utf8 &lt;info data filename | - | PROMPT&gt; Same as update_info except that the input is encoded as UTF-8.<br/> attach_files &lt;attachment filenames | PROMPT&gt; [to_page &lt;page number | PROMPT&gt; | relation &lt;relationship&gt;] Packs arbitrary files into a PDF using PDF&#39;s file at&#8208; tachment features. More than one attachment may be listed after attach_files. Attachments are added at the document level unless the optional to_page option is given, in which case the files are attached to the given page number (the first page is 1, the final page is end). Attachments at the document level may be tagged with a relationship among Source, Data, Al&#8208; ternative, Supplement, and Unspecified (default).<br/> For example:<br/> pdftk in.pdf attach_files table1.html table2.html to_page 6 output out.pdf<br/> pdftk in.pdf attach_files in.tex relation Source out&#8208; put out.pdf<br/> unpack_files Copies all of the attachments from the input PDF into the current folder or to an output directory given after output. For example:<br/> pdftk report.pdf unpack_files output ~/atts/<br/> or, interactively:<br/> pdftk report.pdf unpack_files output PROMPT<br/> [output &lt;output filename | - | PROMPT&gt;] The output PDF filename may not be set to the name of an input filename. Use - to output to stdout. When using the dump_data operation, use output to set the name of the output data file. When using the unpack_files opera&#8208; tion, use output to set the name of an output directory. When using the burst operation, you can use output to control the resulting PDF page filenames (described above).<br/> [encrypt_40bit | encrypt_128bit | encrypt_aes128] If an output PDF user or owner password is given, the output PDF encryption algorithm defaults to AES-128. The weaker RC4 40-bit and RC4 128-bit algorithms can be cho&#8208; sen by specifying encrypt_40bit or encrypt_128bit (dis&#8208; couraged).<br/> [allow &lt;permissions&gt;] Permissions are applied to the output PDF only if an en&#8208; cryption strength is specified or an owner or user pass&#8208; word is given. If permissions are not specified, they default to &#39;none,&#39; which means all of the following fea&#8208; tures are disabled.<br/> The permissions section may include one or more of the following features:<br/> Printing Top Quality Printing<br/> DegradedPrinting Lower Quality Printing<br/> ModifyContents Also allows Assembly<br/> Assembly<br/> CopyContents Also allows ScreenReaders<br/> ScreenReaders<br/> ModifyAnnotations Also allows FillIn<br/> FillIn<br/> AllFeatures Allows the user to perform all of the above, and top quality printing.<br/> [owner_pw &lt;owner password | PROMPT&gt;]<br/> [user_pw &lt;user password | PROMPT&gt;] If an encryption strength is given but no passwords are supplied, then the owner and user passwords remain empty, which means that the resulting PDF may be opened and its security parameters altered by anybody.<br/> [compress | uncompress] These are only useful when you want to edit PDF code in a text editor like vim or emacs. Remove PDF page stream compression by applying the uncompress filter. Use the compress filter to restore compression.<br/> [flatten] Use this option to merge an input PDF&#39;s interactive form fields (and their data) with the PDF&#39;s pages. Only one input PDF may be given. Sometimes used with the fill_form operation.<br/> [need_appearances] Sets a flag that cues Reader/Acrobat to generate new field appearances based on the form field values. Use this when filling a form with non-ASCII text to ensure the best presentation in Adobe Reader or Acrobat. It won&#39;t work when combined with the flatten option.<br/> [replacement_font &lt;font name&gt;] Use the specified font to display text in form fields. This option is useful when filling a form with non-ASCII text that is not supported by the fonts included in the input PDF. font name may be either the file name or the family name of a font, but using a file name is more re&#8208; liable. Currently only TrueType fonts with Unicode text are supported.<br/> [keep_first_id | keep_final_id] When combining pages from multiple PDFs, use one of these options to copy the document ID from either the first or final input document into the new output PDF. Otherwise pdftk creates a new document ID for the output PDF. When no operation is given, pdftk always uses the ID from the (single) input PDF.<br/> [drop_xfa] If your input PDF is a form created using Acrobat 7 or Adobe Designer, then it probably has XFA data. Filling such a form using pdftk yields a PDF with data that fails to display in Acrobat 7 (and 6?). The workaround solution is to remove the form&#39;s XFA data, either before you fill the form using pdftk or at the time you fill the form. Using this option causes pdftk to omit the XFA data from the output PDF form.<br/> This option is only useful when running pdftk on a sin&#8208; gle input PDF. When assembling a PDF from multiple in&#8208; puts using pdftk, any XFA data in the input is automati&#8208; cally omitted.<br/> [drop_xmp] Many PDFs store document metadata using both an Info dictionary (old school) and an XMP stream (new school). Pdftk&#39;s update_info operation can update the Info dic&#8208; tionary, but not the XMP stream. The proper remedy for this is to include a ModDate entry in your updated info with a current date/timestamp. The date/timestamp format is: D:YYYYMMDDHHmmSS, e.g. D:201307241346 &ndash; omitted data after YYYY revert to default values. This newer ModDate should cue PDF viewers that the Info metadata is more current than the XMP data.<br/> Alternatively, you might prefer to remove the XMP stream from the PDF altogether &ndash; that&#39;s what this option does. Note that objects inside the PDF might have their own, separate XMP metadata streams, and that drop_xmp does not remove those. It only removes the PDF&#39;s document- level XMP stream.<br/> [verbose] By default, pdftk runs quietly. Append verbose to the end and it will speak up.<br/> [dont_ask | do_ask] Depending on the compile-time settings (see ASK_ABOUT_WARNINGS), pdftk might prompt you for further input when it encounters a problem, such as a bad pass&#8208; word. Override this default behavior by adding dont_ask (so pdftk won&#39;t ask you what to do) or do_ask (so pdftk will ask you what to do).<br/> When running in dont_ask mode, pdftk will over-write files with its output without notice.<br/> EXAMPLES Collate scanned pages pdftk A=even.pdf B=odd.pdf shuffle A B output collated.pdf or if odd.pdf is in reverse order: pdftk A=even.pdf B=odd.pdf shuffle A Bend-1 output col&#8208; lated.pdf<br/> The following examples use actual passwords as command line pa&#8208; rameters, which is discouraged (see the SECURITY CONSIDERATIONS section).<br/> Decrypt a PDF pdftk secured.pdf input_pw foopass output unsecured.pdf<br/> Encrypt a PDF using AES-128 (the default), withhold all permis&#8208; sions (the default) pdftk 1.pdf output 1.128.pdf owner_pw foopass<br/> Same as above, except password &#39;baz&#39; must also be used to open output PDF pdftk 1.pdf output 1.128.pdf owner_pw foo user_pw baz<br/> Same as above, except printing is allowed (once the PDF is open) pdftk 1.pdf output 1.128.pdf owner_pw foo user_pw baz allow printing<br/> Apply RCA 40-bit encryption to output, revoking all permissions (the default). Set the owner PW to &#39;foopass&#39;. pdftk 1.pdf 2.pdf cat output 3.pdf encrypt_40bit owner_pw foopass<br/> Join two files, one of which requires the password &#39;foopass&#39;. The output is not encrypted. pdftk A=secured.pdf 2.pdf input_pw A=foopass cat output 3.pdf<br/> Join in1.pdf and in2.pdf into a new PDF, out1.pdf pdftk in1.pdf in2.pdf cat output out1.pdf or (using handles): pdftk A=in1.pdf B=in2.pdf cat A B output out1.pdf or (using wildcards): pdftk *.pdf cat output combined.pdf<br/> Remove page 13 from in1.pdf to create out1.pdf pdftk in.pdf cat 1-12 14-end output out1.pdf or: pdftk A=in1.pdf cat A1-12 A14-end output out1.pdf<br/> Uncompress PDF page streams for editing the PDF in a text edi&#8208; tor (e.g., vim, emacs) pdftk doc.pdf output doc.unc.pdf uncompress<br/> Repair a PDF&#39;s corrupted XREF table and stream lengths, if pos&#8208; sible pdftk broken.pdf output fixed.pdf<br/> Burst a single PDF document into pages and dump its data to doc_data.txt pdftk in.pdf burst<br/> Burst a single PDF document into encrypted pages. Allow low- quality printing pdftk in.pdf burst owner_pw foopass allow DegradedPrinting<br/> Write a report on PDF document metadata and bookmarks to re&#8208; port.txt pdftk in.pdf dump_data output report.txt<br/> Rotate the first PDF page to 90 degrees clockwise pdftk in.pdf cat 1east 2-end output out.pdf<br/> Rotate an entire PDF document to 180 degrees pdftk in.pdf cat 1-endsouth output out.pdf<br/> NOTES This is a port of pdftk to java. See https://gitlab.com/pdftk-java/pdftk The original program can be found at www.pdftk.com<br/> AUTHOR Original author of pdftk is Sid Steward (sid.steward at pdflabs dot com).<br/> SECURITY CONSIDERATIONS Passing a password as a command line parameter is insecure be&#8208; cause it can get saved into the shell&#39;s history and be accessi&#8208; ble by other users via /proc. Use the keyword PROMPT and input any passwords via standard input instead.<br/> December 7, 2020 pdftk.pdftk-java(1) </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region OCR --> <h2 id="ocr">OCR</h2> <p> <a href='https://https://tools.pdf24.org/en/ocr-pdf' target='_blank' rel="nofollow"><code>https://tools.pdf24.org/en/ocr-pdf</code></a> provides fast, free OCR, without ads. </p> <p> This website works well, and offers a total of 30 PDF operations. The other operations of <a href='https://https://tools.pdf24.org/' target='_blank' rel="nofollow"><code>https://tools.pdf24.org/</code></a> include merging, splitting, compressing, editing, signing, redacting, watermarking, locking, unlocking, etc. </p> <!-- endregion --> <!-- #region ops --> <h2 id="ops">PdfTk Operations</h2> <!-- #region rotate one page --> <h3 id="2south">Rotate page 2 only</h3> <p> Given <code>my_file.pdf</code>, a 2-page document: </p> <ol> <li>Do not transform page 1 (keep it as-is)</li> <li>Rotate page 2 only</li> <li>Save as <code>my_file_fixed.pdf</code></li> </ol> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0f0c4cf21b77'><button class='copyBtn' data-clipboard-target='#id0f0c4cf21b77' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk my_file.pdf cat 1 2south output my_file_fixed.pdf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region merge --> <h3 id="merge">Merge PDFs</h3> <p> Concatenate <code>file_a.pdf</code>, <code>file_b.pdf</code> and <code>file_c.pdf</code>, then save as <code>big_file.pdf</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id31f9f09de3eb'><button class='copyBtn' data-clipboard-target='#id31f9f09de3eb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk file_a.pdf file_b.pdf file_c.pdf cat output big_file.pdf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region assemble --> <h3 id="assemble">Assemble Book Excerpt</h3> <p> If you need to create multiple excerpts of a book, one way to proceed is to: </p> <ol> <li> Create a PDF containing 3 pages: the front cover, the page containing the <a href='https://en.wikipedia.org/wiki/Edition_notice' target='_blank' rel="nofollow">edition notice</a>, and the back cover. Let's call this PDF file <code>book_wrapper.pdf</code>. </li> <li> Scan each of the excerpts into separate PDFs. Let's call these PDF files <code>book_excerpt_raw_1.pdf</code>, <code>book_excerpt_raw_2.pdf</code>, etc. </li> <li> Wrap the each excerpt within the front cover and edition notice, and the back cover. To do that, concatenate the first 2 pages of <code>book_wrapper.pdf</code>, <code>book_excerpt_raw_N.pdf</code> and the last page of <code>book_wrapper.pdf</code>, then save as <code>book_excerpt_N.pdf</code>. Following is an example of how to do that for 3 excerpts: </li> </ol> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id41bad7becf6e'><button class='copyBtn' data-clipboard-target='#id41bad7becf6e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk book_wrapper.pdf 1-2 \ book_excerpt_raw_1.pdf \ book_wrapper.pdf 3 \ cat output book_excerpt_1.pdf <span class='unselectable'>$ </span>pdftk book_wrapper.pdf 1-2 \ book_excerpt_raw_2.pdf \ book_wrapper.pdf 3 \ cat output book_excerpt_2.pdf <span class='unselectable'>$ </span>pdftk book_wrapper.pdf 1-2 \ book_excerpt_raw_3.pdf \ book_wrapper.pdf 3 \ cat output book_excerpt_3.pdf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region cut --> <h3 id="cut">Cut a large PDF into 2 Parts</h3> <p> Cut <code>bigfile.pdf</code> into two parts: </p> <ol> <li><code>bigfile_part1.pdf</code> (containing pages 1-150)</li> <li><code>bigfile_part2.pdf</code> (containing pages 151-350)</li> </ol> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id289069743634'><button class='copyBtn' data-clipboard-target='#id289069743634' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk A=bigfile.pdf cat 1-150 output bigfile_part1.pdf <span class='unselectable'>$ </span>pdftk A=bigfile.pdf cat 151-end output bigfile_part2.pdf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Move Pages --> <h2 id="double">Interleaving Double-Sided Originals</h2> <p> If you scan a double-sided document, you might end up with the odd-numbered pages in one PDF document, and the even-numbered pages in another. </p> <p> A problem that you will encouter is that when you turn over the document to scan the even-numbered pages, they will be in reverse order because the pages will be scanned from the back to the front of the document. The following examples show two ways of combining the odd- and even-numbered pages in the proper order. </p> <h3 id="manual_reverse">Manually Reversing Even-Numbered Pages</h3> <p> This example shows how to interleave the pages in both documents to create <code>complete.pdf</code>. </p> <ol> <li>Scan the paper document from front to back, so only the odd pages are saved as <code>odd.pdf</code>.</li> <li>Manually reverse the order of the pages in the paper document.</li> <li>Turn the paper document over.</li> <li>Scan the even pages into <code>even.pdf</code>.</li> <li>Run the following incantation:</li> </ol> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfb9d356b970d'><button class='copyBtn' data-clipboard-target='#idfb9d356b970d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk A=odd.pdf B=even.pdf shuffle A B output complete.pdf<br> <span class='unselectable'>$ </span>rm even.pdf odd.pdf</pre> </div> <!-- endregion --> <h3 id="auto_reverse">Automatically Reversing Even-Numbered Pages</h3> <p> This example does not require you to manually re-order the pages, because <code>pdftk</code> can perform that task. </p> <ol> <li>Scan the document from front to back, so only the odd pages are saved as <code>odd.pdf</code>.</li> <li>Turn over the paper document.</li> <li>Scan the even pages from back to front into <code>even_reversed.pdf</code>.</li> <li>Run the following incantation:</li> </ol> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida9de9bd4825e'><button class='copyBtn' data-clipboard-target='#ida9de9bd4825e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk even_reversed.pdf cat end-1 output even.pdf<br> <span class='unselectable'>$ </span>pdftk A=odd.pdf B=even.pdf shuffle A B output complete.pdf<br> <span class='unselectable'>$ </span>rm even.pdf even_reversed.pdf odd.pdf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region delete 1 page --> <h3 id="cut">Delete a Page</h3> <p> Delete page 69 from <code>bigfile.pdf</code> and save as <code>smaller.pdf</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbdada97b428b'><button class='copyBtn' data-clipboard-target='#idbdada97b428b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk A=bigfile.pdf cat ~69 output smaller.pdf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region delete several pages --> <h3 id="cut">Delete Several Pages</h3> <p> Delete pages 69-96 from <code>bigfile.pdf</code> and save as <code>smaller.pdf</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id09edcaf351ff'><button class='copyBtn' data-clipboard-target='#id09edcaf351ff' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk A=bigfile.pdf cat ~69-96 output smaller.pdf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region remove password --> <h2 id="pwd">Remove Password</h2> <p> Assuming <code>protected.pdf</code> is encrypted with password <code>myPwd</code>, make an unencrypted copy and save as <code>output.pdf</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id91fb4221c985'><button class='copyBtn' data-clipboard-target='#id91fb4221c985' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk protected.pdf input_pw myPwd output output.pdf</pre> </div> <!-- endregion --> <p> You might need to enclose the password within single quotes. </p> <!-- endregion --> <!-- endregion --> <!-- #region Edit Contents --> <h2 id="edit">Edit Contents</h2> <p> Use LibreOffice Writer to edit the PDF, then export as PDF. </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3eb8643489c2'><button class='copyBtn' data-clipboard-target='#id3eb8643489c2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>libreoffice --writer my_file.pdf</pre> </div> <!-- endregion --> <!-- endregion --> Pytest and Visual Studio Code 2023-08-20T00:00:00-04:00 https://mslinn.github.io/blog/2023/08/20/pytest <!-- #region intro --> <p> Python has several well-known testing frameworks. Popular choices for functional and unit testing include <a href='https://docs.pytest.org' target='_blank' rel="nofollow"><code>pytest</code></a>, the <a href='https://robotframework.org/' target='_blank' rel="nofollow">Robot framework</a>, and <a href='https://docs.python.org/3/library/unittest.html' target='_blank' rel="nofollow"><code>unittest</code></a>. <a href='https://github.com/gabrielfalcao/lettuce' target='_blank' rel="nofollow">Lettuce</a> and <a href='https://behave.readthedocs.io/en/latest/' target='_blank' rel="nofollow">Behave</a> are popular for behavior-driven testing. </p> <p> I decided to use <code>pytest</code> for a recent Python project. </p> <p> I found that <code>pytest</code>&rsquo;s test discovery mechanism was finicky, and demanded that Python packages and modules be set up before tests could be run. This is in sharp contrast to writing unit tests for Ruby, for example, using <a href='https://rspec.info/' target='_blank' rel="nofollow"><code>rspec</code></a>, because Ruby modules are so much more flexible, forgiving and malleable. </p> <p> However, getting <code>pytest</code> to work in Visual Studio Code was nearly impossible. Microsoft's documentation leaves a lot unsaid. I share what I learned, and how I made things work in this article. </p> <!-- endregion --> <!-- #region pytest --> <h2 id="pytest">Pytest</h2> <p> You can run <code>pytest</code> from the command line 3 ways. </p> <!-- #region pytest command --> <h3 id="cmd"><span class="code">Pytest</span> Command</h3> <p> The most common way to run <code>pytest</code> is to type the <code>pytest</code> command, like the following, which runs all tests: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide5d00d1c2bff'><button class='copyBtn' data-clipboard-target='#ide5d00d1c2bff' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pytest</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region module --> <h3 id="module"><span class="code">Pytest</span> Module</h3> <p> The second way to run <code>pytest</code> is to invoke the <code>pytest</code> Python module, as follows: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc7f045aa285e'><button class='copyBtn' data-clipboard-target='#idc7f045aa285e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>python -m pytest</pre> </div> <!-- endregion --> <p> Running <code>pytest</code> as a Python module is nearly the same as running the <code>pytest</code> command, except that running it as a module adds the current directory to <code>sys.path</code>, which is standard python behavior, and can be helpful. </p> <p> I had best results running <code>pytest</code> tests from Visual Studio Code when <code>pytest</code> was invoked as a module. </p> <!-- endregion --> <!-- #region calling from python --> <h3 id="pgm">Calling <span class="code">Pytest</span> from Python</h3> <p> Your code can <a href='https://docs.pytest.org/en/7.1.x/how-to/usage.html#calling-pytest-from-python-code' target='_blank' rel="nofollow">call <code>pytest</code></a>. Lots of interesting and advanced development setups could be crafted using this approach. Baby steps. </p> <!-- endregion --> <!-- #region help --> <h3 id="help">Help Message</h3> <p> The help message for <code>pytest</code> is: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id361f530bbe28'><button class='copyBtn' data-clipboard-target='#id361f530bbe28' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pytest -h <span class='unselectable'>usage: pytest [options] [file_or_dir] [file_or_dir] [...]<br/> positional arguments: file_or_dir<br/> general: -k EXPRESSION Only run tests which match the given substring expression. An expression is a Python evaluatable expression where all names are substring-matched against test names and their parent classes. Example: -k &#39;test_method or test_other&#39; matches all test functions and classes whose name contains &#39;test_method&#39; or &#39;test_other&#39;, while -k &#39;not test_method&#39; matches those that don&#39;t contain &#39;test_method&#39; in their names. -k &#39;not test_method and not test_other&#39; will eliminate the matches. Additionally keywords are matched to classes and functions containing extra names in their &#39;extra_keyword_matches&#39; set, as well as functions which have names assigned directly to them. The matching is case-insensitive. -m MARKEXPR Only run tests matching given mark expression. For example: -m &#39;mark1 and not mark2&#39;. --markers show markers (builtin, plugin and per-project ones). -x, --exitfirst Exit instantly on first error or failed test --fixtures, --funcargs Show available fixtures, sorted by plugin appearance (fixtures with leading &#39;_&#39; are only shown with &#39;-v&#39;) --fixtures-per-test Show fixtures per test --pdb Start the interactive Python debugger on errors or KeyboardInterrupt --pdbcls=modulename:classname Specify a custom interactive Python debugger for use with --pdb.For example: --pdbcls=IPython.terminal.debugger:TerminalPdb --trace Immediately break when running each test --capture=method Per-test capturing method: one of fd|sys|no|tee-sys -s Shortcut for --capture=no --runxfail Report the results of xfail tests as if they were not marked --lf, --last-failed Rerun only the tests that failed at the last run (or all if none failed) --ff, --failed-first Run all tests, but run the last failures first. This may re-order tests and thus lead to repeated fixture setup/teardown. --nf, --new-first Run tests from new files first, then the rest of the tests sorted by file mtime --cache-show=[CACHESHOW] Show cache contents, don&#39;t perform collection or tests. Optional argument: glob (default: &#39;*&#39;). --cache-clear Remove all cache contents at start of test run --lfnf={all,none}, --last-failed-no-failures={all,none} Which tests to run with no previously (known) failures --sw, --stepwise Exit on test failure and continue from last failing test next time --sw-skip, --stepwise-skip Ignore the first failing test but stop on the next failing test. Implicitly enables --stepwise.<br/> Reporting: --durations=N Show N slowest setup/test durations (N=0 for all) --durations-min=N Minimal duration in seconds for inclusion in slowest list. Default: 0.005. -v, --verbose Increase verbosity --no-header Disable header --no-summary Disable summary -q, --quiet Decrease verbosity --verbosity=VERBOSE Set verbosity. Default: 0. -r chars Show extra test summary info as specified by chars: (f)ailed, (E)rror, (s)kipped, (x)failed, (X)passed, (p)assed, (P)assed with output, (a)ll except passed (p/P), or (A)ll. (w)arnings are enabled by default (see --disable-warnings), &#39;N&#39; can be used to reset the list. (default: &#39;fE&#39;). --disable-warnings, --disable-pytest-warnings Disable warnings summary -l, --showlocals Show locals in tracebacks (disabled by default) --no-showlocals Hide locals in tracebacks (negate --showlocals passed through addopts) --tb=style Traceback print mode (auto/long/short/line/native/no) --show-capture={no,stdout,stderr,log,all} Controls how captured stdout/stderr/log is shown on failed tests. Default: all. --full-trace Don&#39;t cut any tracebacks (default is to cut) --color=color Color terminal output (yes/no/auto) --code-highlight={yes,no} Whether code should be highlighted (only if --color is also enabled). Default: yes. --pastebin=mode Send failed|all info to bpaste.net pastebin service --junit-xml=path Create junit-xml style report file at given path --junit-prefix=str Prepend prefix to classnames in junit-xml output<br/> pytest-warnings: -W PYTHONWARNINGS, --pythonwarnings=PYTHONWARNINGS Set which warnings to report, see -W option of Python itself --maxfail=num Exit after first num failures or errors --strict-config Any warnings encountered while parsing the `pytest` section of the configuration file raise errors --strict-markers Markers not registered in the `markers` section of the configuration file raise errors --strict (Deprecated) alias to --strict-markers -c FILE, --config-file=FILE Load configuration from `FILE` instead of trying to locate one of the implicit configuration files. --continue-on-collection-errors Force test execution even if collection errors occur --rootdir=ROOTDIR Define root directory for tests. Can be relative path: &#39;root_dir&#39;, &#39;./root_dir&#39;, &#39;root_dir/another_dir/&#39;; absolute path: &#39;/home/user/root_dir&#39;; path with variables: &#39;$HOME/root_dir&#39;.<br/> collection: --collect-only, --co Only collect tests, don&#39;t execute them --pyargs Try to interpret all arguments as Python packages --ignore=path Ignore path during collection (multi-allowed) --ignore-glob=path Ignore path pattern during collection (multi-allowed) --deselect=nodeid_prefix Deselect item (via node id prefix) during collection (multi-allowed) --confcutdir=dir Only load conftest.py&#39;s relative to specified dir --noconftest Don&#39;t load any conftest.py files --keep-duplicates Keep duplicate tests --collect-in-virtualenv Don&#39;t ignore tests in a local virtualenv directory --import-mode={prepend,append,importlib} Prepend/append to sys.path when importing test modules and conftest files. Default: prepend. --doctest-modules Run doctests in all .py modules --doctest-report={none,cdiff,ndiff,udiff,only_first_failure} Choose another output format for diffs on doctest failure --doctest-glob=pat Doctests file matching pattern, default: test*.txt --doctest-ignore-import-errors Ignore doctest ImportErrors --doctest-continue-on-failure For a given doctest, continue to run after the first failure<br/> test session debugging and configuration: --basetemp=dir Base temporary directory for this test run. (Warning: this directory is removed if it exists.) -V, --version Display pytest version and information about plugins. When given twice, also display information about plugins. -h, --help Show help message and configuration info -p name Early-load given plugin module name or entry point (multi-allowed). To avoid loading of plugins, use the `no:` prefix, e.g. `no:doctest`. --trace-config Trace considerations of conftest.py files --debug=[DEBUG_FILE_NAME] Store internal tracing debug information in this log file. This file is opened with &#39;w&#39; and truncated as a result, care advised. Default: pytestdebug.log. -o OVERRIDE_INI, --override-ini=OVERRIDE_INI Override ini option with &quot;option=value&quot; style, e.g. `-o xfail_strict=True -o cache_dir=cache`. --assert=MODE Control assertion debugging tools. &#39;plain&#39; performs no assertion debugging. &#39;rewrite&#39; (the default) rewrites assert statements in test modules on import to provide assert expression information. --setup-only Only setup fixtures, do not execute tests --setup-show Show setup of fixtures while executing tests --setup-plan Show what fixtures and tests would be executed but don&#39;t execute anything<br/> logging: --log-level=LEVEL Level of messages to catch/display. Not set by default, so it depends on the root/parent log handler&#39;s effective level, where it is &quot;WARNING&quot; by default. --log-format=LOG_FORMAT Log format used by the logging module --log-date-format=LOG_DATE_FORMAT Log date format used by the logging module --log-cli-level=LOG_CLI_LEVEL CLI logging level --log-cli-format=LOG_CLI_FORMAT Log format used by the logging module --log-cli-date-format=LOG_CLI_DATE_FORMAT Log date format used by the logging module --log-file=LOG_FILE Path to a file when logging will be written to --log-file-level=LOG_FILE_LEVEL Log file logging level --log-file-format=LOG_FILE_FORMAT Log format used by the logging module --log-file-date-format=LOG_FILE_DATE_FORMAT Log date format used by the logging module --log-auto-indent=LOG_AUTO_INDENT Auto-indent multiline messages passed to the logging module. Accepts true|on, false|off or an integer. --log-disable=LOGGER_DISABLE Disable a logger by name. Can be passed multiple times.<br/> [pytest] ini-options in the first pytest.ini|tox.ini|setup.cfg|pyproject.toml file found:<br/> markers (linelist): Markers for test functions empty_parameter_set_mark (string): Default marker for empty parametersets norecursedirs (args): Directory patterns to avoid for recursion testpaths (args): Directories to search for tests when no files or directories are given on the command line filterwarnings (linelist): Each line specifies a pattern for warnings.filterwarnings. Processed after -W/--pythonwarnings. usefixtures (args): List of default fixtures to be used with this project python_files (args): Glob-style file patterns for Python test module discovery python_classes (args): Prefixes or glob names for Python test class discovery python_functions (args): Prefixes or glob names for Python test function and method discovery disable_test_id_escaping_and_forfeit_all_rights_to_community_support (bool): Disable string escape non-ASCII characters, might cause unwanted side effects(use at your own risk) console_output_style (string): Console output: &quot;classic&quot;, or with additional progress information (&quot;progress&quot; (percentage) | &quot;count&quot; | &quot;progress-even-when-capture-no&quot; (forces progress even when capture=no) xfail_strict (bool): Default for the strict parameter of xfail markers when not given explicitly (default: False) tmp_path_retention_count (string): How many sessions should we keep the `tmp_path` directories, according to `tmp_path_retention_policy`. tmp_path_retention_policy (string): Controls which directories created by the `tmp_path` fixture are kept around, based on test outcome. (all/failed/none) enable_assertion_pass_hook (bool): Enables the pytest_assertion_pass hook. Make sure to delete any previously generated pyc cache files. junit_suite_name (string): Test suite name for JUnit report junit_logging (string): Write captured log messages to JUnit report: one of no|log|system-out|system-err|out-err|all junit_log_passing_tests (bool): Capture log information for passing tests to JUnit report: junit_duration_report (string): Duration time to report: one of total|call junit_family (string): Emit XML for schema: one of legacy|xunit1|xunit2 doctest_optionflags (args): Option flags for doctests doctest_encoding (string): Encoding used for doctest files cache_dir (string): Cache directory path log_level (string): Default value for --log-level log_format (string): Default value for --log-format log_date_format (string): Default value for --log-date-format log_cli (bool): Enable log display during test run (also known as &quot;live logging&quot;) log_cli_level (string): Default value for --log-cli-level log_cli_format (string): Default value for --log-cli-format log_cli_date_format (string): Default value for --log-cli-date-format log_file (string): Default value for --log-file log_file_level (string): Default value for --log-file-level log_file_format (string): Default value for --log-file-format log_file_date_format (string): Default value for --log-file-date-format log_auto_indent (string): Default value for --log-auto-indent pythonpath (paths): Add paths to sys.path faulthandler_timeout (string): Dump the traceback of all threads if a test takes more than TIMEOUT seconds to finish addopts (args): Extra command line options minversion (string): Minimally required pytest version required_plugins (args): Plugins that must be present for pytest to run<br/> Environment variables: PYTEST_ADDOPTS Extra command line options PYTEST_PLUGINS Comma-separated plugins to load during startup PYTEST_DISABLE_PLUGIN_AUTOLOAD Set to disable plugin auto-loading PYTEST_DEBUG Set to enable debug tracing of pytest&#39;s internals<br/><br/> to see available markers type: pytest --markers to see available fixtures type: pytest --fixtures (shown according to specified file_or_dir or current dir if not specified; fixtures with leading &#39;_&#39; are only shown with the &#39;-v&#39; option </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region helpful options --> <h3 id="options">Helpful Options</h3> <p> It is handy to run new and failing tests, but to skip previously passing tests. The <code>&#8209;&#8209;nf</code> and <code>&#8209;&#8209;lf</code> options, respectively, invoke that mode of behavior: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6d22b372c700'><button class='copyBtn' data-clipboard-target='#id6d22b372c700' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pytest --nf --lf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- endregion --> <!-- #region setup --> <h2 id="setup">Setup</h2> <!-- #region config --> <h3 id="config">Configuration</h3> <p> <code>Pytest</code> needed to know that the source code for my Python project is stored in the <code>src</code> directory. I created <code>pyproject.toml</code> as follows, which configured <code>pytest</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>pyproject.toml</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id348184dbb968'><button class='copyBtn' data-clipboard-target='#id348184dbb968' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[tool.pytest.ini_options] addopts = [ "--import-mode=importlib", ] pythonpath = "src"</pre> </div> <!-- endregion --> <p> The way in which <code>pytest</code> discovers tests is <a href='https://docs.pytest.org/en/7.1.x/explanation/pythonpath.htm' target='_blank' rel="nofollow">complex</a>. <a href='https://www.merriam-webster.com/dictionary/TL%3BDR' target='_blank' rel="nofollow">TL;DR</a>: specify the newest and best mechanism for test discovery by using the <code>&#8209;&#8209;import-mode=importlib</code> option as shown above. </p> <p> To always run <code>pytest</code> with the <code>&#8209;&#8209;nf</code> and <code>&#8209;&#8209;lf</code> options, add those options to <code>pyproject.toml</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>pyproject.toml</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9dcf1640e078'><button class='copyBtn' data-clipboard-target='#id9dcf1640e078' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[tool.pytest.ini_options] addopts = [ "--import-mode=importlib", <span class='bg_yellow'>"--lf",</span> <span class='bg_yellow'>"--nf",</span> ] pythonpath = "src"</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Test Directories --> <h3 id="dirs">Test Directories</h3> <p> Flexibility generally adds complexity. <code>Pytest</code> allows for a lot of flexibility regarding the location of where tests can reside. Unfortunately, Python&rsquo;s package and module mechanism in combination with the <code>pytest</code> discovery mechanism for tests can lead to a frustrating experience when setting up a new project. </p> <p> You can read about <a href='https://pytest.org/en/7.4.x/explanation/goodpractices.html' target='_blank' rel="nofollow">good practices for <code>pytest</code></a>. Those recommendations are intimidating for first-time <code>pytest</code> programmers. I did not follow all of them. </p> <p> I found the simplest approach was to create a subdirectory for the bulk of the Python project code. If a file called <code>__init__.py</code> is created in that directory (use <a href='https://man7.org/linux/man-pages/man1/touch.1.html' target='_blank' rel="nofollow"><code>touch</code></a> for this) then the subdirectory becomes a submodule of your Python program. For this type of <code>pytest</code> setup, each submodule needs a <code>tests</code> directory. </p> <p> Here is how you could use <code>touch</code> to define an imaginary subdirectory/submodule called <code>my_submodule</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfbe08bac6148'><button class='copyBtn' data-clipboard-target='#idfbe08bac6148' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mkdir -p src/my_submodule/<br> <span class='unselectable'>$ </span>touch src/my_submodule/__init__.py</pre> </div> <!-- endregion --> <p> My project structure was as follows. <a href='https://linux.die.net/man/1/tree' target='_blank' rel="nofollow"><code>Tree</code> docs are here.</a> </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>pytest project structure</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer tree' id='idc95c3d7f9294'><button class='copyBtn' data-clipboard-target='#idc95c3d7f9294' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>tree -I .git -I __pycache__ <span class='unselectable'>. ├── README.md ├── dl ├── dl.config ├── pyproject.toml ├── requirements.txt └── src ├── __main__.py └── dl ├── __init__.py ├── argument_parse.py ├── dl_config.py ├── media_file.py ├── remote.py ├── tests │   ├── __init__.py │   ├── test_dl_config.py │   └── test_uti.py └── util.py 3 directories, 15 files </span></pre> </div> <!-- endregion --> <p> <code>src/dl/tests/__init__.py</code> was an empty marker file, but the <code>___init__.py</code> file in the <a href='https://docs.python.org/3/tutorial/modules.html' target='_blank' rel="nofollow">parent module</a> defined <code>import</code>s for the submodule: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>src/dl/__init__.py</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3b71a7c1892a'><button class='copyBtn' data-clipboard-target='#id3b71a7c1892a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>from dl.argument_parse import ArgParse from dl.dl_config import DLConfig from dl.media_file import MediaFile from dl.remote import Remote from dl.util import *</pre> </div> <!-- endregion --> <!-- endregion --> <!-- endregion --> <!-- #region sample test --> <h2 id="test">Sample Test</h2> <p> Here is a sample <code>pytest</code> file, containing 2 unit tests. It is simple and clear. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>src/dl/tests/test_util.py</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide065e62c1c84'><button class='copyBtn' data-clipboard-target='#ide065e62c1c84' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>import pytest from pathlib import Path from dl import samba_mount, samba_parse class TestUtil: def test_samba_parse(self): remote_drive, local_path = samba_parse('e:/media/renders') assert remote_drive=='e' assert local_path=='/media/renders' def test_samba_mount(self): samba_root = samba_mount('bear', 'e', False) assert samba_root==Path('/mnt/bear/e') assert samba_root.is_mount()</pre> </div> <!-- endregion --> <h2 id="rap">Secret Sauce</h2> <p> If you have been paying attention, you should realize that one option for running new and failing tests is to type the following from the command line. We will use this method of invocation in the next section to integrate with Visual Studio Code. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idde3f18945200'><button class='copyBtn' data-clipboard-target='#idde3f18945200' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>python -m pytest --nf --lf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region vscode --> <h2 id="vscode">Visual Studio Code Integration</h2> <p> Visual Studio Code has a Testing explorer and a Run and Debug explorer. I find the existance of two very similar explorers annoying, because you often want to debug a test. Yes, you can do that by pushing the right button in the Testing Explorer ... if it works. This did not work for me. </p> <p> <a href='https://marketplace.visualstudio.com/items?itemName=Shopify.ruby-extensions-pack' target='_blank' rel="nofollow">Shopify Ruby</a>, the latest and greatest Ruby extension for Visual Studio Code, uses the Run and Debug explorer to run and debug tests as well as entire programs. This makes sense to me. </p> <p> <a href='https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance' target='_blank' rel="nofollow">Microsoft Pylance</a> (for Python) follows a different convention, and uses the Run and Debug explorer to run and debug Python programs. You must use the Testing explorer for running or debugging Python unit tests. Annoying. </p> <p> In the Testing explorer, I was able to see my <code>dl</code> module, but no tests were discovered. This was really annoying, and this is where more documentation from Microsoft would have been helpful. After wasting lots of time trying to solve this problem, I discovered a solution. </p> <h3 id="">Invoke <span class="code">Pytest</span> As a Module from VSCode</h3> <p> Run and Debug configurations that invoke the <code>pytest</code> module work perfectly. Here is an example <code>.vscode/launch.json</code> that demonstrates this: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>.vscode/launch.json</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idaddddfd7eb79'><button class='copyBtn' data-clipboard-target='#idaddddfd7eb79' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{ "version": "0.2.0", "configurations": [ { "name": "PyTest All", "type": "python", "request": "launch", "module": "pytest", "justMyCode": true }, { "args": ["--nf", "--lf"], "name": "PyTest New and Failing", "type": "python", "request": "launch", "module": "pytest", "justMyCode": true }, ] }</pre> </div> <!-- endregion --> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <p> You can set breakpoints in your code and debug your tests this way. </p> <!-- endregion --> C and C++ Online and On Ubuntu 2023-05-12T00:00:00-04:00 https://mslinn.github.io/blog/2023/05/12/c <!-- #region intro --> <div class="one_column right quartersize"> <div class='imgWrapper imgFlex inline' style=' '> <figure> <a href='/blog/c/Lattice_C_8086_1982.pdf' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/c/lattice_c.svg" type="image/svg"> <!---<source srcset="/blog/c/lattice_c.avif" type="image/avif">--> <source srcset="/blog/c/lattice_c.webp" type="image/webp"> <source srcset="/blog/c/lattice_c.apng" type="image/apng"> <source srcset="/blog/c/lattice_c.png" type="image/png"> <source srcset="/blog/c/lattice_c.jpg" type="image/jpeg"> <source srcset="/blog/c/lattice_c.jpeg" type="image/jpeg"> <source srcset="/blog/c/lattice_c.jfif" type="image/jpeg"> <source srcset="/blog/c/lattice_c.pjpeg" type="image/jpeg"> <source srcset="/blog/c/lattice_c.pjp" type="image/jpeg"> <source srcset="/blog/c/lattice_c.gif" type="image/gif"> <source srcset="/blog/c/lattice_c.tif" type="image/tiff"> <source srcset="/blog/c/lattice_c.tiff" type="image/tiff"> <source srcset="/blog/c/lattice_c.bmp" type="image/bmp"> <source srcset="/blog/c/lattice_c.ico" type="image/x-icon"> <source srcset="/blog/c/lattice_c.cur" type="image/x-icon"> <img alt='Lattice C' class="imgImg rounded shadow" src="/blog/c/lattice_c.png" style='width: 100%; ' title='Lattice C' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="/blog/c/Lattice_C_8086_1982.pdf" target='_blank' > Lattice C </a> </figcaption> </figure> </div> <div class='imgWrapper imgFlex inline' style=' '> <figure> <picture class='imgPicture'> <source srcset="/blog/c/k&r.svg" type="image/svg"> <!---<source srcset="/blog/c/k&r.avif" type="image/avif">--> <source srcset="/blog/c/k&r.webp" type="image/webp"> <source srcset="/blog/c/k&r.apng" type="image/apng"> <source srcset="/blog/c/k&r.png" type="image/png"> <source srcset="/blog/c/k&r.jpg" type="image/jpeg"> <source srcset="/blog/c/k&r.jpeg" type="image/jpeg"> <source srcset="/blog/c/k&r.jfif" type="image/jpeg"> <source srcset="/blog/c/k&r.pjpeg" type="image/jpeg"> <source srcset="/blog/c/k&r.pjp" type="image/jpeg"> <source srcset="/blog/c/k&r.gif" type="image/gif"> <source srcset="/blog/c/k&r.tif" type="image/tiff"> <source srcset="/blog/c/k&r.tiff" type="image/tiff"> <source srcset="/blog/c/k&r.bmp" type="image/bmp"> <source srcset="/blog/c/k&r.ico" type="image/x-icon"> <source srcset="/blog/c/k&r.cur" type="image/x-icon"> <img alt='K&R Book' class="imgImg rounded shadow" src="/blog/c/k&r.png" style='width: 100%; ' title='K&R Book' /> </picture> <figcaption class='imgFigCaption '> K&R Book </figcaption> </figure> </div> <div class='imgWrapper imgFlex inline' style=' '> <figure> <a href='https://developerinsider.co/download-turbo-c-for-windows-7-8-8-1-and-windows-10-32-64-bit-full-screen/' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/c/turbo_c_2.svg" type="image/svg"> <!---<source srcset="/blog/c/turbo_c_2.avif" type="image/avif">--> <source srcset="/blog/c/turbo_c_2.webp" type="image/webp"> <source srcset="/blog/c/turbo_c_2.apng" type="image/apng"> <source srcset="/blog/c/turbo_c_2.png" type="image/png"> <source srcset="/blog/c/turbo_c_2.jpg" type="image/jpeg"> <source srcset="/blog/c/turbo_c_2.jpeg" type="image/jpeg"> <source srcset="/blog/c/turbo_c_2.jfif" type="image/jpeg"> <source srcset="/blog/c/turbo_c_2.pjpeg" type="image/jpeg"> <source srcset="/blog/c/turbo_c_2.pjp" type="image/jpeg"> <source srcset="/blog/c/turbo_c_2.gif" type="image/gif"> <source srcset="/blog/c/turbo_c_2.tif" type="image/tiff"> <source srcset="/blog/c/turbo_c_2.tiff" type="image/tiff"> <source srcset="/blog/c/turbo_c_2.bmp" type="image/bmp"> <source srcset="/blog/c/turbo_c_2.ico" type="image/x-icon"> <source srcset="/blog/c/turbo_c_2.cur" type="image/x-icon"> <img alt='Borland Turbo C' class="imgImg rounded shadow" src="/blog/c/turbo_c_2.png" style='width: 100%; ' title='Borland Turbo C' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://developerinsider.co/download-turbo-c-for-windows-7-8-8-1-and-windows-10-32-64-bit-full-screen/" target='_blank' > Borland Turbo C </a> </figcaption> </figure> </div> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/zamples.svg" type="image/svg"> <!---<source srcset="/blog/images/zamples.avif" type="image/avif">--> <source srcset="/blog/images/zamples.webp" type="image/webp"> <source srcset="/blog/images/zamples.apng" type="image/apng"> <source srcset="/blog/images/zamples.png" type="image/png"> <source srcset="/blog/images/zamples.jpg" type="image/jpeg"> <source srcset="/blog/images/zamples.jpeg" type="image/jpeg"> <source srcset="/blog/images/zamples.jfif" type="image/jpeg"> <source srcset="/blog/images/zamples.pjpeg" type="image/jpeg"> <source srcset="/blog/images/zamples.pjp" type="image/jpeg"> <source srcset="/blog/images/zamples.gif" type="image/gif"> <source srcset="/blog/images/zamples.tif" type="image/tiff"> <source srcset="/blog/images/zamples.tiff" type="image/tiff"> <source srcset="/blog/images/zamples.bmp" type="image/bmp"> <source srcset="/blog/images/zamples.ico" type="image/x-icon"> <source srcset="/blog/images/zamples.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/zamples.png" style='width: 100%; ' /> </picture> </div> </div> <p> Recently I decided to brush up on my C programming. </p> <p> I first discovered C during a visit in 1978 to the <a href='https://cs.berkeley.edu/' target='_blank' rel="nofollow">Computer Science department of University of California, Berkeley</a>, where I also encounted UNIX for the first time. </p> <p> From about 1983 to 1985 I was the unofficial Canadian distributor of the <a href='/blog/c/Lattice_C_8086_1982.pdf'>Lattice C compiler</a>. I used that compiler for most of the programming work I was doing on PCs at the time. FedEx Express Canada did not come into being until 1987, UPS was even worse with the Canadian border than it is now, and Canada Post took forever to bring packages across the border. I bought in bulk at a discount, and sold at list price. Distribution is all about time and place. </p> <p> I also sold the first edition of the K&R book (&ldquo;The C Programming Language&rdquo;) via mail order, and I provided technical support, which brought in interesting consulting work. </p> <p> The Borland Turbo C compiler, released in 1990, was the original IDE. It was eventually replaced by Borland C++ compiler, which also compiled C. In 2005 I became the product marketing manager for most of Borland&rsquo;s compilers. One of my responsibilities was to give free copies to book authors and members of the trade press. Borland C++ shipped in a massive box that was responsible for a lot of dead trees. </p> <p> The C language has evolved since K&R days. The current and previous iterations are well summarized on <a href='https://en.cppreference.com/w/c/language' target='_blank' rel="nofollow"><code>cppreference.com</code></a>, which also provides an online C/C++ IDE for sample code. </p> <!-- endregion --> <!-- #region Revisions --> <h2 id="revisions">Revision History</h2> <p> 1978: <b>K&R book</b> first published. </p> <p> 1989: <b>C89</b>, also known as ANSI X3. 159-1989, or ANSI C. </p> <p>1995: <b>NA1</b> (aka C94 and C95) added support for wide-characters, wide-strings, and international keyboard layouts. </p> <p>1999: <b>C99</b> added restricted pointers, variable-length arrays, flexible array members, complex numbers support, type-generic math, <code>long long int</code>, extended identifiers, hexadecimal floating-point constants, compound literals, designated initializers, single line comments, extended integer type, the ability to mix declarations and code, variadic macros, new math functions, inline functions, boolean types, <code>_Pragma</code> and standard pragmas, and <code>VA_COPY</code>. C99 removed implicit function declaration. </p> <p>2011: <b>C11</b> added concurrency support, type-generic expressions, alignment-related facilities, static assertion, Unicode support, floating-point characteristic macros, no-return functions, anonymous structures and unions, bound-checking, and reentrancy functions. </p> <p>2018: <b>C18</b> addressed defects in C11 without introducing new language features. </p> <p>2023: The most recent publicly available draft of the next release, <a href='https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3096.pdf' target='_blank' rel="nofollow"><b>C23</b></a>, was released on April 1, 2023. Unfortunately, that document was written for insiders, and is very difficult to read. Happily, JeanHeyd Meneide, the Project Editor for ISO/IEC JTC1 SC22 WG14 - Programming Languages, C, wrote an <a href='https://thephd.dev/c23-is-coming-here-is-what-is-on-the-menu' target='_blank' rel="nofollow">easy-to-read <wbr>summary of C23</a>. </p> <h2 id="linux">Linux Is Written in C</h2> <p> It has been <a href='https://www.zdnet.com/article/linus-torvalds-prepares-to-move-the-linux-kernel-to-modern-c/' target='_blank' rel="nofollow">reported</a> that Linux was originally written in C89, but Linux moved to C11 in 2022. I am curious as to why C18 is not used, since C18 just contains bug fixes for C11. Most of the Linux kernel code is written using the <a href='https://gcc.gnu.org/onlinedocs/gcc-10.2.0/gcc/C-Extensions.html#C-Extensions' target='_blank' rel="nofollow">GNU extensions of GCC</a>. </p> <!-- endregion --> <!-- #region online --> <h2 id="online">Online IDEs</h2> <p> I pioneered online programming with SMX Explorer in 1994, <a href='/resume/history/jspexplorer/developerWorks/'>JSP Explorer</a> in 2000, and supported most available languages in 2002 with <a href='/blog/2004/12/09/live-code-examples-for-j2se-6-mustang.html'>Zamples.com</a>. </p> <p> Now it seems that every day another online programming option becomes available. Many are free: </p> <ul> <li> <a href='https://coliru.stacked-crooked.com/' target='_blank' rel="nofollow">Coliru</a> is used by <code>cppreference.com</code>; <a href='https://github.com/StackedCrooked/coliru' target='_blank' rel="nofollow">GitHub project</a> </li> <li> <a href='https://github.com/features/codespaces' target='_blank' rel="nofollow">GitHub Codespaces</a> is tightly integrated with Microsoft Azure. </li> <li><a href='https://www.jetbrains.com/space/' target='_blank' rel="nofollow">JetBrains Space</a></li> <li> <a href='https://www.onlinegdb.com/online_c_compiler' target='_blank' rel="nofollow">OnlineGDB</a> claims to be to first Online IDE, but the <a href='https://web.archive.org/web/20160815000000*/OnlineGDB.com' target='_blank' rel="nofollow">WayBack Machine</a> shows that this site was created in 2016. </li> <li><a href='https://www.programiz.com/c-programming/online-compiler/' target='_blank' rel="nofollow">Programiz</a></li> <li><a href='https://www.tutorialspoint.com/compile_c_online.php' target='_blank' rel="nofollow">TutorialsPoint CodingGround</a></li> <li><a href='https://replit.com/languages/c' target='_blank' rel="nofollow">Replit</a></li> <li><a href='https://onecompiler.com/c' target='_blank' rel="nofollow">OneCompiler</a></li> <li>... and <a href='https://www.google.com/search?q=online+c+programming' target='_blank' rel="nofollow">many more</a> ... <a href='https://arne-mertz.de/2017/05/online-compilers/' target='_blank' rel="nofollow">even more!</a></li> </ul> <!-- endregion --> <!-- #region install --> <h2 id="install">Installing C/C++ on WSL/Ubuntu</h2> <p> Online programming is fine for learning, but I prefer to be independent and self-reliant whenever practical for doing actual work. </p> <h3 id="cli">Command-Line Toolchain</h3> <p> To program in C or C++ on Ubuntu using the traditional Linux command-line tool&shy;chain, install the <a href='https://packages.debian.org/sid/build-essential' target='_blank' rel="nofollow"><code>build-essential</code></a> Debian meta-package on Ubuntu. It includes the <a href='https://gcc.gnu.org/' target='_blank' rel="nofollow">GNU Compiler Collection (GCC)</a>, which is a collection of compilers and libraries for C, C++, Objective-C, Fortran, Ada, Go, and D programming languages. It also includes <a href='https://linux.die.net/man/1/make' target='_blank' rel="nofollow"><code>make</code></a> and <a href='https://packages.debian.org/sid/libc6.1-dev' target='_blank' rel="nofollow"><code>libc6.1-dev</code></a>, the GNU C development libraries and header files. </p> <p> You probably should also install <a href='https://www.sourceware.org/gdb/' target='_blank' rel="nofollow"><code>gdb</code></a>, the GNU Project Debugger, and the <a href='https://packages.debian.org/unstable/manpages-dev' target='_blank' rel="nofollow">manual pages</a> about using GNU/Linux for development. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9179531b3040'><button class='copyBtn' data-clipboard-target='#id9179531b3040' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install build-essential gdb manpages-dev</pre> </div> <!-- endregion --> <p> Now you are able to compile and run C programs at the command line: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8845d664f688'><button class='copyBtn' data-clipboard-target='#id8845d664f688' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mkdir test && cd test <span class='unselectable'>$ </span>cat &gt;test.c &lt;&lt;EOF #include &lt;stdio.h&gt; int main(void) { printf("Hello, World!\n"); } EOF <span class='unselectable'>$ </span>gcc test.c -o test && ./test <span class='unselectable'>Hello, World! </span></pre> </div> <!-- endregion --> <h3 id="vscode">Visual Studio Code</h3> <p> Visual Studio Code is a high-quality (and free) IDE. To work with C and C++ in Visual Studio Code, install the <a href='https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools' target='_blank' rel="nofollow">Microsoft C/C++</a> extension. </p> <!-- endregion --> Don't Poke the Bear 2023-03-03T00:00:00-05:00 https://mslinn.github.io/blog/2023/03/03/lego-tm <!-- #region --> <h2 style="margin-bottom: 2em; margin-top: 2em;"> To the developers of the excellent &lsquo;<a href='https://github.com/go-acme/lego/' class='code>lego</span>' target='_blank' rel="nofollow"><span</a>&rsquo; GitHub project </h2> <div class='imgWrapper imgFlex inline' style=' '> <a href='https://github.com/go-acme/lego' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/ddns/lego-logo.svg" type="image/svg"> <!---<source srcset="/blog/images/ddns/lego-logo.avif" type="image/avif">--> <source srcset="/blog/images/ddns/lego-logo.webp" type="image/webp"> <source srcset="/blog/images/ddns/lego-logo.apng" type="image/apng"> <source srcset="/blog/images/ddns/lego-logo.png" type="image/png"> <source srcset="/blog/images/ddns/lego-logo.jpg" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.jpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.jfif" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.pjpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.pjp" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.gif" type="image/gif"> <source srcset="/blog/images/ddns/lego-logo.tif" type="image/tiff"> <source srcset="/blog/images/ddns/lego-logo.tiff" type="image/tiff"> <source srcset="/blog/images/ddns/lego-logo.bmp" type="image/bmp"> <source srcset="/blog/images/ddns/lego-logo.ico" type="image/x-icon"> <source srcset="/blog/images/ddns/lego-logo.cur" type="image/x-icon"> <img class="imgImg " src="/blog/images/ddns/lego-logo.png" style='width: 100%; ' /> </picture> </a> </div> <p> The name of this project is problematic, and you <a href='https://github.com/go-acme/lego/issues/470' target='_blank' rel="nofollow">publicly dismissed</a> the issue when it was brought to your attention. While ignorance of the law is no defense; demonstrably flaunting another party&rsquo;s rights never has positive consequences. </p> <p> <a href='https://www.lego.com/en-sg/service/help/fun-for-fans/behind-the-scenes/our-company/commercial-use-of-lego-products-bltb4ad5090ec5e4b08' target='_blank' rel="nofollow">The LEGO Group</a> has not taken legal action yet, but the laws of most countries (including the US) are that <a href='https://www.forbes.com/sites/oliverherzfeld/2013/02/28/failure-to-enforce-trademarks-if-you-snooze-do-you-lose/?sh=1d67ec326c22' target='_blank' rel="nofollow">if a trademark is not defended, then the trademark is at risk of being lost</a>. </p> <p> You should expect to receive a legal notice at some point. Not because the trademark owners are evil, but because they have a duty to their company, and the well-being of their employees, and a duty to their shareholders. </p> <p> Your disregard for the LEGO Group's rights constituted a real and present danger to them, their employees and shareholders. They are compelled to address threats against their well-being. </p> <div class='imgWrapper imgBlock right fullsize' style=' '> <figure> <a href='https://www.flickr.com/photos/angietigger/6894972004/' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/bearpokelego.svg" type="image/svg"> <!---<source srcset="/blog/images/bearpokelego.avif" type="image/avif">--> <source srcset="/blog/images/bearpokelego.webp" type="image/webp"> <source srcset="/blog/images/bearpokelego.apng" type="image/apng"> <source srcset="/blog/images/bearpokelego.png" type="image/png"> <source srcset="/blog/images/bearpokelego.jpg" type="image/jpeg"> <source srcset="/blog/images/bearpokelego.jpeg" type="image/jpeg"> <source srcset="/blog/images/bearpokelego.jfif" type="image/jpeg"> <source srcset="/blog/images/bearpokelego.pjpeg" type="image/jpeg"> <source srcset="/blog/images/bearpokelego.pjp" type="image/jpeg"> <source srcset="/blog/images/bearpokelego.gif" type="image/gif"> <source srcset="/blog/images/bearpokelego.tif" type="image/tiff"> <source srcset="/blog/images/bearpokelego.tiff" type="image/tiff"> <source srcset="/blog/images/bearpokelego.bmp" type="image/bmp"> <source srcset="/blog/images/bearpokelego.ico" type="image/x-icon"> <source srcset="/blog/images/bearpokelego.cur" type="image/x-icon"> <img alt='&lsquo;lego bear&rsquo; by angie <br> CC BY-NC-SA 2.0' class="imgImg rounded shadow" src="/blog/images/bearpokelego.png" style='width: 100%; ' title='&lsquo;lego bear&rsquo; by angie <br> CC BY-NC-SA 2.0' /> </picture> </a> <figcaption class='imgFigCaption fullsize'> <a href="https://www.flickr.com/photos/angietigger/6894972004/" target='_blank' > &lsquo;lego bear&rsquo; by angie <br> CC BY-NC-SA 2.0 </a> </figcaption> </figure> </div> <h2 id="think">Think</h2> <p> Not only must a trademark holder defend their trademark, they must be seen to defend it. That means taking action that is publicly visible. For example, they might include GitHub and maybe even Microsoft in the legal action against you. </p> <div class="pullQuote"> No-one would have reason to defend you </div> <p> The LEGO Group would likely include GitHub/Microsoft in the case, even though they know that <a href='https://www.eff.org/issues/cda230' target='_blank' rel="nofollow">Section 230</a> would probably be found to apply (in due course), and GitHub/Microsoft would not be found to be liable &ndash; after they spent several million dollars in a successful legal defense. </p> <p> It is easy to understand the motivation for the LEGO Group deliberately &lsquo;losing&rsquo; at the attempt to ensnare GitHub/Microsoft in this court case. You would have caused GitHub/Microsoft an unnecessary and pointless waste of time and resources, and it would be shown that you knew better. Remember, you have already been publicly been warned that there would likely be consequences &ndash; yet you chose to ignore the warning. </p> <p> This would likely motivate GitHub/Microsoft to protect themselves against similar misguided behavior of other GitHub users. Restrictive policies and procedures would likely get implemented, and the entire world would know that the resulting degraded GitHub experience was your fault. You would be vilified by every other GitHub user. Not a great thing for your resume. </p> <p> ... And GitHub/Microsoft might ask you to pay their legal expenses. </p> <h2 id="family">Do You Have Families?</h2> <p> Please understand that because you are acting as an informal collection of individuals, everyone who ever contributed to this project is exposed to signficant financial risk. The LEGO Group would probably not attempt to narrow down the guilty individuals &ndash; identifying the guilty and assigning punishment would be the outcomes of the court case. Instead, they would likely include every contributor in the litigation; that means every committer. </p> <p> You are risking the financial well-being of your families ... just because you like a cute play on words? </p> <p class="pullQuoteFull"> Rational people would not expose themselves to significant risk without any prospect of signficant benefit, and when not faced with dire necessity. </p> <p> If you presently balk at paying a lawyer to advise you on this matter, that cost will be as nothing compared to the world of hurt you are likely to find yourselves in if you do not change the name of your project. </p> <p> From a technical aspect, you have done excellent work on your project, and you are freely providing the world a valuable benefit. Unfortunately, you trampled the rights of another in doing so... and for no reason. Please act on this before the inevitable negative consequences arise. Do not poke the bear because you are curious to see what it will do. </p> <div class='imgWrapper imgFlex right' style='width: 45%; '> <picture class='imgPicture'> <source srcset="/blog/images/bearpoke.svg" type="image/svg"> <!---<source srcset="/blog/images/bearpoke.avif" type="image/avif">--> <source srcset="/blog/images/bearpoke.webp" type="image/webp"> <source srcset="/blog/images/bearpoke.apng" type="image/apng"> <source srcset="/blog/images/bearpoke.png" type="image/png"> <source srcset="/blog/images/bearpoke.jpg" type="image/jpeg"> <source srcset="/blog/images/bearpoke.jpeg" type="image/jpeg"> <source srcset="/blog/images/bearpoke.jfif" type="image/jpeg"> <source srcset="/blog/images/bearpoke.pjpeg" type="image/jpeg"> <source srcset="/blog/images/bearpoke.pjp" type="image/jpeg"> <source srcset="/blog/images/bearpoke.gif" type="image/gif"> <source srcset="/blog/images/bearpoke.tif" type="image/tiff"> <source srcset="/blog/images/bearpoke.tiff" type="image/tiff"> <source srcset="/blog/images/bearpoke.bmp" type="image/bmp"> <source srcset="/blog/images/bearpoke.ico" type="image/x-icon"> <source srcset="/blog/images/bearpoke.cur" type="image/x-icon"> <img class="imgImg " src="/blog/images/bearpoke.png" style='width: 100%; ' /> </picture> </div> <p class="left alert rounded shadow" style="width: 50%;"> The wheels of corporate law grind slowly, and huge amounts of money are spent. According to the path you are now on, the committers of this project may be subjected to immense financial and mental burdens. Broken families and suicides may result. <br><br> Please heed this warning. If nothing bad has ever happened to you yet in life, rest assured that <a href='https://brewminate.com/clotho-lachesis-and-atropos-the-three-sisters-of-fate-in-ancient-greek-mythology/' target='_blank' rel="nofollow">Lachesis, Atropos and Clotho</a> will inevitably make themselves known. Do not let arrogance and pride ruin you. </p> <div class="clear pullQuoteFull" style="margin-top: 2.5em;"> I am trying to help you people.<br> Because you have done good work,<br> and it would be a shame to see you suffer. </div> <h2 id="disclaim" class="clear">Disclaimer</h2> <p> I have no relationship with the LEGO Group, or any individuals associated with them. Please consider this as a notice freely offered for public service. </p> <p> Although <a href='/softwareexpert/'>I have experience working as an expert witness in the US Federal legal system as a software expert</a>, I am not a lawyer and I have no legal training. However, I have seen corporate legal machinery at work many times, up close. I have spent thousands of hours working with corporate attorneys in the US and Europe. My clients have included Adobe, Amazon/AWS, Apple, and SAP. </p> <!-- endregion --> Letsencrypt/ACME Wildcard SSL Certificates by Lego 2023-03-02T00:00:00-05:00 https://mslinn.github.io/blog/2023/03/02/lego <style> .redButton { color:white; background-color:#e04433; padding: 2px; font-family: Helvetica, Arial, Sans-Serif; font-weight: bold; } </style> <!-- #region intro --> <div class='imgWrapper imgFlex right' style='width: 40%; '> <a href='https://letsencrypt.org/' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/ddns/letsencrypt-logo.svg" type="image/svg"> <!---<source srcset="/blog/images/ddns/letsencrypt-logo.avif" type="image/avif">--> <source srcset="/blog/images/ddns/letsencrypt-logo.webp" type="image/webp"> <source srcset="/blog/images/ddns/letsencrypt-logo.apng" type="image/apng"> <source srcset="/blog/images/ddns/letsencrypt-logo.png" type="image/png"> <source srcset="/blog/images/ddns/letsencrypt-logo.jpg" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.jpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.jfif" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.pjpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.pjp" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.gif" type="image/gif"> <source srcset="/blog/images/ddns/letsencrypt-logo.tif" type="image/tiff"> <source srcset="/blog/images/ddns/letsencrypt-logo.tiff" type="image/tiff"> <source srcset="/blog/images/ddns/letsencrypt-logo.bmp" type="image/bmp"> <source srcset="/blog/images/ddns/letsencrypt-logo.ico" type="image/x-icon"> <source srcset="/blog/images/ddns/letsencrypt-logo.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/ddns/letsencrypt-logo.png" style='width: 100%; ' /> </picture> </a> </div> <p> Letsencrypt certificates are only valid for <a href='https://letsencrypt.org/docs/faq/#what-is-the-lifetime-for-let-s-encrypt-certificates-for-how-long-are-they-valid' target='_blank' rel="nofollow">90 days</a>. It is common practice to renew them every 60 days, a task that one should automate when the website is first published. </p> <p> I wrote about <a href='/blog/2022/06/15/certbot.html'>setting up wildcard SSL certificates with Nginx</a> 9 months ago. That article included a demonstration of how to create an SSL certificate manually, but did not discuss how to maintain it. </p> <p> This article discusses how to automatically generate and maintain the SSL certificate using <a href='https://github.com/go-acme/lego' target='_blank' rel="nofollow"><code>lego</code></a>, a Letsencrypt/ACME client and library written in Go that has become popular since I last wrote about this topic. </p> <p> <code>Lego</code> handles many moving parts transparently. It is <i>much</i> simpler than using DNS delegates with <a href='https://github.com/joohoi/acme-dns' target='_blank' rel="nofollow"><code>acme-dns</code></a>. </p> <p> We&rsquo;ll start by briefly discussing some background information. </p> <!-- endregion --> <!-- #region Letsencrypt certbot and wildcard ssl certs --> <h2 id="certbot" class="clear">Letsencrypt&rsquo;s <span class="code">Certbot</span> and Wildcard SSL Certificates</h2> <p> You must prove to Letsencrypt that you control the DNS for a domain before it issues a wildcard SSL certificate for that domain. <a href='https://certbot.eff.org/' target='_blank' rel="nofollow">Letsencrypt&rsquo;s <code>certbot</code></a> currently uses the <a href='https://cert-manager.io/docs/configuration/acme/dns01/' target='_blank' rel="nofollow"><code>DNS-01</code> challenge</a> for this purpose. </p> <h3 id="DNS-01">DNS-01 Challenge</h3> <p> The DNS-01 challenge requires that DNS <code>TXT</code> records for the domain be created with specific values as part of the authentication mechanism. Several of these <code>TXT</code> records can co-exist among the DNS records for a domain. The name of the <code>TXT</code> records is always <code>_acme-challenge</code>. </p> <p> When <code>certbot</code> requests that a new wildcard SSL certificate be created by Letsencrypt, it sends DNS-01 <code>TXT</code> queries to the primary DNS. The Letsencrypt service verifies that the required <code>_acme-challenge</code> <code>TXT</code> records are available with the correct values. </p> <p> <a href='https://www.eff.org/deeplinks/2018/02/technical-deep-dive-securing-automation-acme-dns-challenge-validation' target='_blank' rel="nofollow">A Technical Deep Dive: Securing the Automation of ACME DNS Challenge Validation</a>, published by the Electronic Frontier Foundation, explains how this works in detail. </p> <p> <code>Certbot</code> plugins have been created for various DNS providers to make this process easier. <code>Lego</code> is a higher-level program that makes the entire process even easier. </p> <!-- endregion --> <!-- #region agenda --> <h2 id="agenda">Agenda</h2> <p> The high-level outline of the remainder of this blog is: </p> <ol> <li><a href='#go'>Install <code>go</code> language support.</a></li> <li><a href='#install'>Install <code>lego</code>.</a></li> <li><a href='#gen'>Generate the wildcard SSL certificate.</a></li> <li><a href='#provide'>Provide the certificate</a> to your production web server.</li> <li><a href='#restart'>Restart the webserver</a> and verify the new certificate is served.</li> <li><a href='#crontab'>Add a new entry to <code>crontab</code></a> to automate the SSL certificate generation.</li> </ol> <!-- endregion --> <!-- #region Install Go Language Support --> <h2 id="go">Install Go Language Support</h2> <p> <code>Lego</code> is written in Go. Ensure that <a href='https://go.dev/doc/install' target='_blank' rel="nofollow">Go language support</a> is installed. For Ubuntu Linux (and the default WSL distro), type: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id78051f496c07'><button class='copyBtn' data-clipboard-target='#id78051f496c07' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install golang-go <span class='unselectable'>The following NEW packages will be installed: golang-1.18-go golang-1.18-src golang-go golang-src 0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded. Need to get 82.2 MB of archives. After this operation, 436 MB of additional disk space will be used. Do you want to continue? [Y/n] Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 golang-1.18-src all 1.18.1-1ubuntu1 [16.2 MB] Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 golang-1.18-go amd64 1.18.1-1ubuntu1 [66.0 MB] Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 golang-src all 2:1.18~0ubuntu2 [4438 B] Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 golang-go amd64 2:1.18~0ubuntu2 [41.8 kB] Fetched 82.2 MB in 2s (33.6 MB/s) Selecting previously unselected package golang-1.18-src. (Reading database ... 173763 files and directories currently installed.) Preparing to unpack .../golang-1.18-src_1.18.1-1ubuntu1_all.deb ... Unpacking golang-1.18-src (1.18.1-1ubuntu1) ... Selecting previously unselected package golang-1.18-go. Preparing to unpack .../golang-1.18-go_1.18.1-1ubuntu1_amd64.deb ... Unpacking golang-1.18-go (1.18.1-1ubuntu1) ... Selecting previously unselected package golang-src. Preparing to unpack .../golang-src_2%3a1.18~0ubuntu2_all.deb ... Unpacking golang-src (2:1.18~0ubuntu2) ... Selecting previously unselected package golang-go:amd64. Preparing to unpack .../golang-go_2%3a1.18~0ubuntu2_amd64.deb ... Unpacking golang-go:amd64 (2:1.18~0ubuntu2) ... Setting up golang-1.18-src (1.18.1-1ubuntu1) ... Setting up golang-src (2:1.18~0ubuntu2) ... Setting up golang-1.18-go (1.18.1-1ubuntu1) ... Setting up golang-go:amd64 (2:1.18~0ubuntu2) ... Processing triggers for man-db (2.10.2-1) ... Scanning processes... Scanning processor microcode... Scanning linux images... Failed to retrieve available kernel versions. Failed to check for processor microcode upgrades. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. </span></pre> </div> <p> View the version of <code>go</code> that was installed: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddfd09a343bf8'><button class='copyBtn' data-clipboard-target='#iddfd09a343bf8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>go version <span class='unselectable'>go version go1.19.2 linux/amd64 </span></pre> </div> <!-- endregion --> <!-- #region Installing Lego --> <h2 id="install">Installing Lego</h2> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/ddns/lego-logo.svg" type="image/svg"> <!---<source srcset="/blog/images/ddns/lego-logo.avif" type="image/avif">--> <source srcset="/blog/images/ddns/lego-logo.webp" type="image/webp"> <source srcset="/blog/images/ddns/lego-logo.apng" type="image/apng"> <source srcset="/blog/images/ddns/lego-logo.png" type="image/png"> <source srcset="/blog/images/ddns/lego-logo.jpg" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.jpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.jfif" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.pjpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.pjp" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.gif" type="image/gif"> <source srcset="/blog/images/ddns/lego-logo.tif" type="image/tiff"> <source srcset="/blog/images/ddns/lego-logo.tiff" type="image/tiff"> <source srcset="/blog/images/ddns/lego-logo.bmp" type="image/bmp"> <source srcset="/blog/images/ddns/lego-logo.ico" type="image/x-icon"> <source srcset="/blog/images/ddns/lego-logo.cur" type="image/x-icon"> <img class="imgImg " src="/blog/images/ddns/lego-logo.png" style='width: 100%; ' /> </picture> </div> <p> The official <code>lego</code> installation instructions are <a href='https://go-acme.github.io/lego/installation/' target='_blank' rel="nofollow">here</a>. They do not mention Ubuntu instructions, but the <a href='https://ubuntu.pkgs.org/22.04/ubuntu-universe-amd64/lego_4.1.3-3ubuntu1_amd64.deb.html' target='_blank' rel="nofollow"><code>lego</code> package for Ubuntu</a> can be installed in the usual way: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2712262e30f0'><button class='copyBtn' data-clipboard-target='#id2712262e30f0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install lego <span class='unselectable'>Reading package lists... Done Building dependency tree... Done Reading state information... Done The following NEW packages will be installed: lego 0 upgraded, 1 newly installed, 0 to remove and 7 not upgraded. Need to get 5322 kB of archives. After this operation, 19.7 MB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 lego amd64 4.1.3-3ubuntu1.22.04.1 [5322 kB] Fetched 5322 kB in 1s (10.2 MB/s) Selecting previously unselected package lego. (Reading database ... 86044 files and directories currently installed.) Preparing to unpack .../lego_4.1.3-3ubuntu1.22.04.1_amd64.deb ... Unpacking lego (4.1.3-3ubuntu1.22.04.1) ... Setting up lego (4.1.3-3ubuntu1.22.04.1) ... </span></pre> </div> <p> Here is the <code>lego</code> help message: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf79aaa2ffdf6'><button class='copyBtn' data-clipboard-target='#idf79aaa2ffdf6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>lego -h <span class='unselectable'>NAME: lego - Let&#39;s Encrypt client written in Go<br/> USAGE: lego [global options] command [command options] [arguments...]<br/> VERSION: dev<br/> COMMANDS: run Register an account, then create and install a certificate revoke Revoke a certificate renew Renew a certificate dnshelp Shows additional help for the &#39;--dns&#39; global option list Display certificates and accounts information. help, h Shows a list of commands or help for one command<br/> GLOBAL OPTIONS: --domains value, -d value Add a domain to the process. Can be specified multiple times. --server value, -s value CA hostname (and optionally :port). The server certificate must be trusted in order to avoid further modifications to the client. (default: &quot;https://acme-v02.api.letsencrypt.org/directory&quot;) --accept-tos, -a By setting this flag to true you indicate that you accept the current Let&#39;s Encrypt terms of service. --email value, -m value Email used for registration and recovery contact. --csr value, -c value Certificate signing request filename, if an external CSR is to be used. --eab Use External Account Binding for account registration. Requires --kid and --hmac. --kid value Key identifier from External CA. Used for External Account Binding. --hmac value MAC key from External CA. Should be in Base64 URL Encoding without padding format. Used for External Account Binding. --key-type value, -k value Key type to use for private keys. Supported: rsa2048, rsa4096, rsa8192, ec256, ec384. (default: &quot;ec256&quot;) --filename value (deprecated) Filename of the generated certificate. --path value Directory to use for storing the data. (default: &quot;/mnt/_/work/lego/.lego&quot;) [$LEGO_PATH] --http Use the HTTP challenge to solve challenges. Can be mixed with other types of challenges. --http.port value Set the port and interface to use for HTTP based challenges to listen on.Supported: interface:port or :port. (default: &quot;:80&quot;) --http.proxy-header value Validate against this HTTP header when solving HTTP based challenges behind a reverse proxy. (default: &quot;Host&quot;) --http.webroot value Set the webroot folder to use for HTTP based challenges to write directly in a file in .well-known/acme-challenge. This disables the built-in server and expects the given directory to be publicly served with access to .well-known/acme-challenge --http.memcached-host value Set the memcached host(s) to use for HTTP based challenges. Challenges will be written to all specified hosts. --tls Use the TLS challenge to solve challenges. Can be mixed with other types of challenges. --tls.port value Set the port and interface to use for TLS based challenges to listen on. Supported: interface:port or :port. (default: &quot;:443&quot;) --dns value Solve a DNS challenge using the specified provider. Can be mixed with other types of challenges. Run &#39;lego dnshelp&#39; for help on usage. --dns.disable-cp By setting this flag to true, disables the need to wait the propagation of the TXT record to all authoritative name servers. --dns.resolvers value Set the resolvers to use for performing recursive DNS queries. Supported: host:port. The default is to use the system resolvers, or Google&#39;s DNS resolvers if the system&#39;s cannot be determined. --http-timeout value Set the HTTP timeout value to a specific value in seconds. (default: 0) --dns-timeout value Set the DNS timeout value to a specific value in seconds. Used only when performing authoritative name servers queries. (default: 10) --pem Generate a .pem file by concatenating the .key and .crt files together. --cert.timeout value Set the certificate timeout value to a specific value in seconds. Only used when obtaining certificates. (default: 30) --help, -h show help --version, -v print the version </span></pre> </div> <!-- endregion --> <!-- #region Generating the Wildcard SSL Certificate --> <h2 id="gen">Generating the Wildcard SSL Certificate</h2> <!-- #region /etc/environment --> <h3 id="env"><span class="code">/etc/environment</span></h3> <p> My projects are defined in a directory tree, which is pointed to by an environment variable called <code>work</code>. This is helpful when working across many machines, each of which has a different directory layout. </p> <p> I define the <code>work</code> environment variable in the <a href='https://askubuntu.com/a/866240/58760' target='_blank' rel="nofollow">system-wide <code>/etc/environment</code></a> file, where it affects all users, all scripts, and crontab entries: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>/etc/environment</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id33c45eaa5c45'><button class='copyBtn' data-clipboard-target='#id33c45eaa5c45' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin" work=/mnt/_/work sites=/mnt/_/www</pre> </div> <p> <code>$work</code> is used in the script shown <a href='#certGenScalaCourses'>below</a>, and that script will be launched via <code>crontab</code>, which is why <code>/etc/environment</code> is used instead of <code>$HOME/.bashrc</code>. </p> <p class="alert rounded shadow"> WSL does not run <code>systemd</code> services, and WSL2 only runs them if they are enabled. You can easily convert WSL instances into WSL2 instances. <br><br> Unless <code>systemd</code> starts automatically at boot time, <code>/etc/environment</code> will not be processed. <br><br> Read Microsoft&rsquo;s announcement to learn more: <a href='https://devblogs.microsoft.com/commandline/systemd-support-is-now-available-in-wsl/' target='_blank' rel="nofollow"><code>Systemd</code> support is now available in WSL!</a> </p> <!-- endregion --> <!-- #region generation --> <h3 id="dir">Generation</h3> <p> When working with the <code>lego</code> CLI with DNS authentication, a directory is needed to hold all the files that are required. I created a directory for this purpose at <code>$work/lego</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc0db70cc11da'><button class='copyBtn' data-clipboard-target='#idc0db70cc11da' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mkdir $work/lego<br> <span class='unselectable'>$ </span>cd $work/lego</pre> </div> <p> The <code>lego --dns namecheap</code> option connects to Namecheap, which is my DNS provider, and uses the Namecheap DNS for the <code>DNS-01</code> challenge. Lego supports <a href='https://go-acme.github.io/lego/dns/#dns-providers' target='_blank' rel="nofollow">many other DNS providers</a>. </p> <!-- #region Namecheap Users --> <div class="alert rounded shadow"> <h2>Namecheap Users</h2> <p> The Namecheap API documentation is <a href='https://www.namecheap.com/support/api/intro/' target='_blank' rel="nofollow">here</a>. For some reason, that page reloads to a different page, which is annoying. Press the <kbd>Esc</kbd> key right after the page opens, so you can read it. </p> <p> Please also read the <a href='https://www.namecheap.com/support/knowledgebase/article.aspx/9739/63/api-faq/#t' target='_blank' rel="nofollow">FAQ</a>.<br> Enable the Namecheap API <a href='https://ap.www.namecheap.com/settings/tools/apiaccess/' target='_blank' rel="nofollow">here</a>.<br> You could enable the Namecheap sandbox API <a href='https://ap.www.sandbox.namecheap.com/settings/tools/apiaccess/' target='_blank' rel="nofollow">here</a>. </p> <p> Here is a short script that displays your current public IP address: </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,whatismyip' download='whatismyip' title='Click on the file name to download the file'>whatismyip</a> </div> <pre data-lt-active="false" class="pre_tag maxOneScreenHigh copyContainer" id="id156d6dba46ce"><button class='copyBtn' data-clipboard-target='#id156d6dba46ce'title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash dig +short myip.opendns.com @resolver1.opendns.com </pre> <p style="margin-top: 1em;"> If your modem goes offline, Namecheap&rsquo;s DNS will assign a new IP address the next time the modem reconnects. That will cause the scripts shown below to stop working until you manually provide the new address to the Namecheap API. Annoying! Be sure to plug your modem into a UPS to maintain the IP address even during short outages, power brownouts and voltage dropouts. A sine-wave UPS is strongly preferred, since modems are sensitive to low-quality power. </p> <p> The Namecheap API needs 2 environment variables for authentication (<code>NAMECHEAP_API_USER</code> and <code>NAMECHEAP_API_KEY</code>). Here is how I defined those environment variables so they were available for the <code>lego</code> process launched by <code>sudo</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id69a2d78614d8'><button class='copyBtn' data-clipboard-target='#id69a2d78614d8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>export NAMECHEAP_API_USER=asdf<br> <span class='unselectable'>$ </span>export NAMECHEAP_API_KEY=asdfasdf</pre> </div> </div> <!-- endregion --> <p> To generate a wildcard certificate, you must specify the <code>--domains</code> option twice: once with the domain name, and once for all subdomains, like this: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8ab7a07eecf8'><button class='copyBtn' data-clipboard-target='#id8ab7a07eecf8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>lego \ --accept-tos \ --dns namecheap \ --domains="scalacourses.com" \ --domains="*.scalacourses.com" \ --email="mslinn@scalacourses.com" \ <span class="bg_yellow">run</span> <span class='unselectable'>2023/03/01 13:40:03 [INFO] [*.scalacourses.com] acme: Obtaining bundled SAN certificate 2023/03/01 13:40:03 [INFO] [*.scalacourses.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/207409481036 2023/03/01 13:40:03 [INFO] [*.scalacourses.com] acme: use dns-01 solver 2023/03/01 13:40:03 [INFO] [*.scalacourses.com] acme: Preparing to solve DNS-01 2023/03/01 13:40:04 [INFO] [*.scalacourses.com] acme: Trying to solve DNS-01 2023/03/01 13:40:04 [INFO] [*.scalacourses.com] acme: Checking DNS record propagation using [172.19.176.1:53] 2023/03/01 13:40:19 [INFO] Wait for propagation [timeout: 1h0m0s, interval: 15s] 2023/03/01 13:40:20 [INFO] [*.scalacourses.com] acme: Waiting for DNS record propagation. 2023/03/01 13:40:35 [INFO] [*.scalacourses.com] acme: Waiting for DNS record propagation. 2023/03/01 13:40:56 [INFO] [*.scalacourses.com] The server validated our request 2023/03/01 13:40:56 [INFO] [*.scalacourses.com] acme: Cleaning DNS-01 challenge 2023/03/01 13:40:56 [INFO] [*.scalacourses.com] acme: Validations succeeded; requesting certificates 2023/03/01 13:40:57 [INFO] [*.scalacourses.com] Server responded with a certificate. </span></pre> </div> <!-- endregion --> <p> The wildcard SSL certificate and associated files are generated into the <code>.lego/certificates</code> directory. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id19b653c20850'><button class='copyBtn' data-clipboard-target='#id19b653c20850' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls -alF .lego/certificates <span class='unselectable'>total 12 drwx------ 1 mslinn mslinn 4096 Mar 1 13:40 ./ drwx------ 1 mslinn mslinn 4096 Mar 1 13:15 ../ -rw------- 1 mslinn mslinn 5325 Mar 1 13:40 scalacourses.com.crt -rw------- 1 mslinn mslinn 3751 Mar 1 13:40 scalacourses.com.issuer.crt -rw------- 1 mslinn mslinn 239 Mar 1 13:40 scalacourses.com.json -rw------- 1 mslinn mslinn 227 Mar 1 13:40 scalacourses.com.key </span></pre> </div> <!-- endregion --> <ul> <li> <code>scalacourses.com.crt</code> is the server certificate that nginx needs, already combined with the CA certificate. </li> <li><code>scalacourses.com.key</code> is the private key for the server certificate.</li> <li><code>scalacourses.com.issuer.crt</code> is the issuing Certificate Authority&apos;s certificate.</li> <li> <code>scalacourses.com.json</code> contains JSON encoded meta information. It looked like this: <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>&dollar;work/lego/.lego/certificates/scalacourses.com.json</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida5f699db615d'><button class='copyBtn' data-clipboard-target='#ida5f699db615d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{ "domain": "*.scalacourses.com", "certUrl": "https://acme-v02.api.letsencrypt.org/acme/cert/3939493849340275024024", "certStableUrl": "https://acme-v02.api.letsencrypt.org/acme/cert/738378373923492384932874932" }</pre> </div> </li> </ul> <p> Using the value for <code>certStableUrl</code> in the above JSON file, we can view the generated (and combined) certificate that was saved on <code>letsencrypt.org</code>. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6989c396f55a'><button class='copyBtn' data-clipboard-target='#id6989c396f55a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>wget -O aw.crt \ https://acme-v02.api.letsencrypt.org/acme/cert/03b099f52b4841db17e035c5c2b390f0219b</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region certGenScalaCourses script --> <h2 id="certGenScalaCourses"><span class="code">certGenScalaCourses</span> Script</h2> <p> 60 days from now, when the certificate should be renewed, the last parameter passed to <code>lego</code> in the above command line should be changed from <code class="bg_yellow">run</code> to <code>renew</code>. Here is a bash script that you can edit for this purpose; at the end of this article this script is incorporated into a new <code>crontab</code> entry: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>&dollar;work/lego/certGenScalaCourses</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id882d7d31f2ba'><button class='copyBtn' data-clipboard-target='#id882d7d31f2ba' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash export NAMECHEAP_API_USER=asdf export NAMECHEAP_API_KEY=asdfasdf ACTION=run if [ -f .lego/certificates/scalacourses.com.crt ]; then ACTION=renew fi cd $work/lego lego \ --accept-tos \ --dns namecheap \ --domains="scalacourses.com" \ --domains="*.scalacourses.com" \ --email="mslinn@scalacourses.com" \ "$ACTION" sudo systemctl reload nginx</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Viewing a Certificate --> <h2 id="view">Viewing a Certificate</h2> <p> For Ubuntu 22.10, the filetype associations are defined in a hierarchy of files called <code>mimeapps.list</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idba9c2c5b720a'><button class='copyBtn' data-clipboard-target='#idba9c2c5b720a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>locate mimeapps.list <span class='unselectable'>~/.config/mimeapps.list ~/.local/share/applications/mimeapps.list /snap/core/14447/usr/share/applications/mimeapps.list /snap/core/14784/usr/share/applications/mimeapps.list /snap/core18/2679/usr/share/applications/mimeapps.list /snap/core18/2697/usr/share/applications/mimeapps.list /snap/core20/1778/usr/share/applications/mimeapps.list /snap/core20/1822/usr/share/applications/mimeapps.list /snap/core22/509/usr/share/applications/mimeapps.list /snap/core22/522/usr/share/applications/mimeapps.list /usr/share/gdm/greeter/applications/mimeapps.list </span></pre> </div> <!-- endregion --> <p> <code>/snap/<wbr>core22/<wbr>509/<wbr>usr/<wbr>share/<wbr>applications/<wbr>mimeapps.list</code> contains the default association for <code>*.crt</code> files: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>/snap/core22/509/usr/share/applications/mimeapps.list</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida2b9b3c6b293'><button class='copyBtn' data-clipboard-target='#ida2b9b3c6b293' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[Default Applications] x-scheme-handler/http=xdg-open.desktop x-scheme-handler/https=xdg-open.desktop x-scheme-handler/mailto=xdg-open.desktop x-scheme-handler/help=xdg-open.desktop</pre> </div> <!-- endregion --> <p> The above associates a program called <a href='https://linux.die.net/man/1/xdg-open' target='_blank' rel="nofollow"><code>xdg-open</code></a> with CRT files. </p> <p> Double-clicking on a <code>crt</code> file displayed by an Ubuntu file manager such as Nautilus causes <code>xdg-open</code> to open the file. You can also use the command line to open the file in <code>xdg-open</code>. For example, typing the following causes <code>aw.crt</code> to open: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id17f7dbb914d0'><button class='copyBtn' data-clipboard-target='#id17f7dbb914d0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>xdg-open aw.crt</pre> </div> <p> This is what <code>xdg-open</code> displays: </p> <div class='imgWrapper imgFlex center' style='width: 478px; '> <picture class='imgPicture'> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.svg" type="image/svg"> <!---<source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.avif" type="image/avif">--> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.webp" type="image/webp"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.apng" type="image/apng"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.png" type="image/png"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.jpg" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.jpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.jfif" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.pjpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.pjp" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.gif" type="image/gif"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.tif" type="image/tiff"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.tiff" type="image/tiff"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.bmp" type="image/bmp"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.ico" type="image/x-icon"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/ddns/ubuntuViewFileOfSslCert_2.png" style='width: 100%; ' /> </picture> </div> <p> As you can see, this is a combined certificate, containing not just one certificate, but an entire certificate chain. Clicking on any of the red <kbd class="redButton">&gt; Details</kbd> buttons causes more information to be displayed about the corresponding certificate in the chain. </p> <!-- endregion --> <!-- #region warning --> <div class="alert rounded shadow"> <h2>Warning</h2> <p> If you forget to include the domain, and just specify the wildcard for subdomains, the certificate will not be valid for the domain. For example, if above I had only specified one <code>--domains</code> option, like this: <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida3933230d011'><button class='copyBtn' data-clipboard-target='#ida3933230d011' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>--domains="scalacourses.com"</pre> </div> ... instead of <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id27839e73ed4c'><button class='copyBtn' data-clipboard-target='#id27839e73ed4c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>--domains="scalacourses.com" \<br>--domains="*.scalacourses.com"</pre> </div> ... then that would be an error. You can notice your error three ways: </p> <ol> <li> The name of the generated certificate will start with an underscore (<code>_</code>), for example <code>_.scalacourses.<wbr>com.<wbr>crt</code>, instead of being named <code>scalacourses.<wbr>com.<wbr>crt</code>. </li> <li> When you examine the certificate, the domain listed will show the subdomain wildcard, as shown below, with a leading underscore asterisk (<code>*</code>).<br> <div class='imgWrapper imgFlex center' style='width: 478px; '> <picture class='imgPicture'> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.svg" type="image/svg"> <!---<source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.avif" type="image/avif">--> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.webp" type="image/webp"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.apng" type="image/apng"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.png" type="image/png"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.jpg" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.jpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.jfif" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.pjpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.pjp" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.gif" type="image/gif"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.tif" type="image/tiff"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.tiff" type="image/tiff"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.bmp" type="image/bmp"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.ico" type="image/x-icon"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/ddns/ubuntuViewFileOfSslCert_1.png" style='width: 100%; margin-top: 2em;' /> </picture> </div> </li> <li style="margin-bottom: 0; padding-bottom: 0;"> When you open the certificate, the section entitled <b>Subject Alternative Names</b> will just contain the subdomain wildcard, like this:<br><br> <b>Subject Alternative Names</b><br> <code>DNS: *.scalacourses.com</code> <br><br>Or just the domain, without a wildcard:<br><br> <b>Subject Alternative Names</b><br> <code>DNS: scalacourses.com</code><br><br> Instead of both being present, like this:<br><br> <b>Subject Alternative Names</b><br> <code>DNS: *.scalacourses.com<br> DNS: scalacourses.com</code> </li> </ol> </div> <!-- endregion --> <!-- #region Provide the certificate to Nginx --> <h2 id="provide">Provide the certificate to Nginx</h2> <p> <a href='/blog/2022/07/08/reverse-proxy.html#vs'>Previously</a>, before working with <code>lego</code>, I used raw <code>certbot</code> to generate the SSL wildcard certificates. That process generated files in <code>~/.certbot/<wbr>scalacourses.com/<wbr>config/<wbr>live/<wbr>scalacourses.com</code>: </p> <ul> <li> <code>fullchain.pem</code> (the <a href='http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate' target='_blank' rel="nofollow"><code>ssl_certificate</code></a>) </li> <li> <code>privkey.pem</code> (the <a href='https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate_key' target='_blank' rel="nofollow"><code>ssl_certificate_key</code></a>) </li> </ul> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>From /etc/nginx/sites-enabled/scalacourses.com</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id37eed1c977d5'><button class='copyBtn' data-clipboard-target='#id37eed1c977d5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>ssl_certificate /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/fullchain.pem; ssl_certificate_key /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/privkey.pem;</pre> </div> <p> Now I need to modify <code>/etc/<wbr>nginx/<wbr>sites-enabled/<wbr>scalacourses.com</code> to reference the files generated by <code>lego</code>: </p> <ul> <li> <code>scalacourses.com.crt</code> (the <a href='http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate' target='_blank' rel="nofollow"><code>ssl_certificate</code></a>) </li> <li> <code>scalacourses.com.key</code> (the <a href='https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate_key' target='_blank' rel="nofollow"><code>ssl_certificate_key</code></a>) </li> </ul> <p> If the webserver had access to the directory containing the new wildcard certificate, it would be simplest to provide the full path to the certificate and its key, like this: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Modified portion of /etc/nginx/sites-enabled/scalacourses.com</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0ff6ddeb2e88'><button class='copyBtn' data-clipboard-target='#id0ff6ddeb2e88' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>ssl_certificate /mnt/_/work/lego/.lego/certificates/scalacourses.com.crt; ssl_certificate_key /mnt/_/work/lego/.lego/certificates/scalacourses.com.key;</pre> </div> <!-- endregion --> <!-- #region Restart Nginx --> <h2 id="restart">Restart Nginx</h2> <p> I tested the <code>nginx</code> configuration: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd3f945e30004'><button class='copyBtn' data-clipboard-target='#idd3f945e30004' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo nginx -t <span class='unselectable'>nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful </span></pre> </div> <p> Then I reloaded nginx: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id76e36e176141'><button class='copyBtn' data-clipboard-target='#id76e36e176141' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl reload nginx</pre> </div> <p> The new SSL certificate needs to be checked to make sure it was installed properly. I just visually examine the start and expire dates: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id26a4bc831ec7'><button class='copyBtn' data-clipboard-target='#id26a4bc831ec7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -Lvs https://www.scalacourses.com \ 2>&1 1>/dev/null | \ grep '\(start\|expire\) date:' <span class='unselectable'>* <span style="color:red">start date</span>: Mar 1 17:40:56 2023 GMT * <span style="color:red">expire date</span>: May 30 17:40:55 2023 GMT </span></pre> </div> <span style='font-size: 3em;;'>&#x1F601;</span> <!-- endregion --> <!-- #region Add a New Entry to crontab --> <h2 id="crontab" class="clear">Add a New Entry to <span class="code">Crontab</span></h2> <p> Automating the regeneration of the SSL wildcard certificate is easy. Edit your <code>crontab</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id51c51b01e59c'><button class='copyBtn' data-clipboard-target='#id51c51b01e59c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>crontab -e</pre> </div> <p> Add the following reference to the <a href='#certGenScalaCourses'><code>certGenScalaCourses</code> script</a> that I gave you earlier to <code>crontab</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>crontab Entry</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2db80bd244b7'><button class='copyBtn' data-clipboard-target='#id2db80bd244b7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># Runs every 2 months / 60 days (more or less) 30 0 1 */2 * $work/lego/certGenScalaCourses</pre> </div> <p> The above only works because <a href='#env'>we defined the <code>work</code> environment variable</a> in <code>/etc/<wbr>environment</code>. </p> <!-- endregion --> <!-- #region checking with With A Web Browser --> <h2 id="chrome">Checking the Certificate With A Web Browser</h2> <p> Microsoft Windows caches SSL certificates, which can prevent your web browser from detecting newly updated SSL certificates. To clear the Windows SSL cache: </p> <ol> <li>Press the <kbd>Windows</kbd> key, type <code>Internet Options</code>, and press <kbd>Enter</kbd>.</li> <li>Select the <b>Content</b> tab.</li> <li>Click the <kbd>Clear SSL state</kbd> button.</li> <li>Click the <kbd>OK</kbd> button.</li> </ol> <!-- endregion --> HTML Hyphens 2023-01-23T00:00:00-05:00 https://mslinn.github.io/blog/2023/01/23/html-hyphen <!-- #region intro --> <p> Web browsers now have built-in automatic hyphenation support, but it is turned off by default for compatibity with older browsers. You can enable hyphenation with two words of CSS, like this: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>your_stylesheet.css</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id83ad56a7f88d'><button class='copyBtn' data-clipboard-target='#id83ad56a7f88d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>body { <span class='bg_yellow'>hyphens: auto;</span> }</pre> </div> <!-- endregion --> <!-- #region W3 CSS3 Hyphenation Standard --> <h2 id="w3c">W3 CSS3 Hyphenation Standard</h2> <div class='quote'> <div class='quoteText clearfix'> Hyphenation occurs when the line breaks at a valid hyphenation opportunity, which is a type of soft wrap opportunity that exists within a word where hyphenation is allowed. <br><br> In CSS, hyphenation opportunities are controlled with the <code>hyphens</code> property. CSS Text Level 3 does not define the exact rules for hyphenation; however, UAs are strongly encouraged to optimize their choice of break points and to chose language-appropriate hyphenation points. <div class="jekyll_pre" > <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id81c058af48ed'><button class='copyBtn' data-clipboard-target='#id81c058af48ed' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>Possible hyphens values: none | manual | auto Initial value: manual</pre> </div> </div><div class='quoteAttribution'> &nbsp;&ndash; <a href='https://www.w3.org/TR/css-text-3/#hyphenation' rel='nofollow' target='_blank'>W3 CSS3 Standard</a></div> </div> <p> However, the W3 CSS3 hyphenation standard also says: </p> <div class='quote'> <div class='quoteText clearfix'> Correct automatic hyphenation requires a hyphenation resource appropriate to the language of the text being broken. The UA must therefore only automatically hyphenate text for which the content language is known and for which it has an appropriate hyphenation resource. <br><br> Authors should correctly tag their content’s language (e.g. using the HTML <code>lang</code> attribute or XML <code>xml:lang</code> attribute) in order to obtain correct automatic hyphenation. </div><div class='quoteAttribution'> &nbsp;&ndash; <a href='https://www.w3.org/TR/css-text-3/#hyphenation' rel='nofollow' target='_blank'>W3 CSS3 Standard</a></div> </div> <p> Let&rsquo;s put this into practice now. </p> <!-- endregion --> <!-- #region Your Website Stylesheet --> <h2 id="css">Your Website Stylesheet</h2> <p> You may not want certain passages to be hyphenated. Defining a CSS style that disables automatic hyphenation for portions of a document is easy; you can still use <code>&ampshy;</code> in those portions of the document to manually define optional hyphenation points. </p> <p> Here is a suggestion for your stylesheet, so it has the hyphenation support you need. Note that hypenation for <kbd>kbd</kbd> tags is completely disabled. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>your_stylesheet.css</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2ae2217b77d2'><button class='copyBtn' data-clipboard-target='#id2ae2217b77d2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>body, .hyphen { hyphens: auto; } .nohyphen { hyphens: manual; } kbd { hyphens: none; }</pre> </div> <!-- endregion --> <!-- #region Complete Example --> <h2 id="example">Complete Example</h2> <p> The following example enables automatic hyphenation of the entire HTML <code>body</code>, according to US English rules. The HTML document contains nested tags, some of which disable or enable automatic hyphenation. One line demonstrates manual hyphenation, through the use of <a href='https://developer.mozilla.org/en-US/docs/Web/CSS/hyphens' target='_blank' rel="nofollow"><code>&amp;shy;</code> entities</a>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>index.html</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' style='margin-bottom: 1em;' id='id7c8a4eb24dc6'><button class='copyBtn' data-clipboard-target='#id7c8a4eb24dc6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>&lt;html <span class="bg_yellow">lang="en-US"</span>> &lt;head> &lt;style> body, .hyphen { hyphens: auto; } .nohyphen { hyphens: manual; } &lt;/style> &lt;/head> &lt;body> &lt;p>This element is automatically hyphenated.&lt;/p> &lt;p <span class="bg_yellow">class="nohyphen"</span>>This element is not hyphenated.&lt;/p> &lt;ol <span class="bg_yellow">class="nohyphen"</span>> &lt;li>This elem<span class="bg_yellow">&amp;shy;</span>ent is manual<span class="bg_yellow">&amp;shy;</span>ly hyphen<span class="bg_yellow">&amp;shy;</span>ated.&lt;/li> &lt;li <span class="bg_yellow">class="hyphen"</span>>This element is automatically hyphenated.&lt;/li> &lt;/ol> &lt;/body> &lt;/html></pre> </div> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <p>Easy!</p> <!-- endregion --> <!-- #region Alternative Manual Hyphenation --> <h2 id="alt" class="clear">Alternative Manual Hyphenation</h2> <!-- #region implicit --> <p> Invisible hypenation is sometimes desirable. For example, if I have a long directory path, like <code>$HOME/.rbenv/versions/$RUBY_VERSION/lib/ruby/gems/$RUBY_VERSION/$GEM_NAME-$GEM_VERSION</code> I might want to tell the web browser where the string could be broken. </p> <p> You have two options &mdash; they are actually different syntaxes for the same <a href='https://en.wikipedia.org/wiki/Zero-width_space' target='_blank' rel="nofollow"><i>zero-width space</i></a>. </p> <!-- endregion --> <!-- #region Zero-Width Space HTML Entity --> <h3 id="alt">Zero-Width Space HTML Entity</h3> <p> <code>&amp;#8203;</code> is the HTML entity for a unicode character called the zero-width space (<a href='https://unicode-table.com/en/200B/' target='_blank' rel="nofollow"><code>ZWSP</code></a>). </p> <p> For example, <code>$HOME/.rbenv/<span class="bg_yellow">&amp;#8203;</span>versions/<span class="bg_yellow">&amp;#8203;</span>$RUBY_VERSION/<span class="bg_yellow">&amp;#8203;</span>lib/<span class="bg_yellow">&amp;#8203;</span>ruby/<span class="bg_yellow">&amp;#8203;</span>gems/<span class="bg_yellow">&amp;#8203;</span>$RUBY_VERSION/<span class="bg_yellow">&amp;#8203;</span>$GEM_NAME-<span class="bg_yellow">&amp;#8203;</span>$GEM_VERSION</code> <br><br>renders as:<br><br> <code>$HOME/.rbenv/versions/$RUBY_VERSION/&#8203;lib/&#8203;ruby/&#8203;gems/&#8203;$RUBY_VERSION/&#8203;$GEM_NAME-&#8203;$GEM_VERSION</code> </p> <!-- endregion --> <!-- #region Line Break Opportunity Tag --> <h3 id="alt">Line Break Opportunity Tag</h3> <p> You could also use the <a href='https://developer.mozilla.org/en-US/docs/Web/HTML/Element/wbr' target='_blank' rel="nofollow">line break opportunity tag</a>, <code>&lt;wbr&gt;</code> . </p> <p> For example, <code>$HOME/.rbenv/<span class="bg_yellow">&lt;wbr&gt;</span>versions/<span class="bg_yellow">&lt;wbr&gt;</span>$RUBY_VERSION/<span class="bg_yellow">&lt;wbr&gt;</span>lib/<span class="bg_yellow">&lt;wbr&gt;</span>ruby/<span class="bg_yellow">&lt;wbr&gt;</span>gems/<span class="bg_yellow">&lt;wbr&gt;</span>$RUBY_VERSION/<span class="bg_yellow">&lt;wbr&gt;</span>$GEM_NAME-<span class="bg_yellow">&lt;wbr&gt;</span>$GEM_VERSION</code> <br><br>renders as:<br><br> <code>$HOME/.rbenv/versions/$RUBY_VERSION/<wbr>lib/<wbr>ruby/<wbr>gems/<wbr>$RUBY_VERSION/<wbr>$GEM_NAME-<wbr>$GEM_VERSION</code> </p> <!-- endregion --> <!-- #region Both Options Yield the Same Rendered Text --> <h3 id="same">Both Options Yield the Same Rendered Text</h3> <p> I prefer to use &lt;wbr&gt;. <a href='https://dictionary.cambridge.org/dictionary/english/ymmv' target='_blank' rel="nofollow">YMMV</a>. </p> <!-- endregion --> <!-- endregion --> <!-- #region Non-Breaking Hyphen --> <h2 id="nbh">Non-Breaking Hyphen</h2> <p> The non-breaking hyphen HTML entity (<code>&amp;#8209;</code>) displays a hyphen character that does not break. </p> <p> For CSS, use <code>content: "\2011"</code> to produce a non-breaking hyphen. </p> <!-- endregion --> A Curmudgeon’s Social Networking 2023-01-07T00:00:00-05:00 https://mslinn.github.io/blog/2023/01/07/curmudgeon <p> I am an unapologetic curmudgeon. I am not here to please anyone else, I am only here to earnestly be myself to the best of my ability. </p> <p> If we do not continuously demonstrate that we endeavor to treat each other right, then we should be dismissed and forgotten. </p> <p> Bullshit walks; money drives away, unsatisfied. </p> <p> Fairness and equity are essential in all things without reservation. Otherwise, go away, life is short and you are just noise. </p> <h2 id="social">Facebook and LinkedIn</h2> <p> Twitter is now a circus. I don't need or want another circus in my communication. Do not look for me on Twitter any longer. </p> <p> If someone connects with me on Facebook or LinkedIn, I expect them to interact with me in person, on the phone, and to want to engage with me. Failing that, I disconnect from them without hesitation. </p> <p> Yes, please connect with me. Tell me your truth. Listen to me. I will listen to you. </p> <h2 id="action">Action Over Empty Words</h2> <p> Even better, we might endeavor to do things together. Imagining, conceptualizing, and building something real with others gives me joy. I want to share that joy. </p> <p> People who talk about what &lsquo;others&rsquo; should do, but are unwilling to act themselves, are a waste of time. They irritate me. Only approach me if you are an action-oriented individual. </p> <div class='quote'> The most difficult thing is the decision to act, the rest is merely tenacity. <span class='quoteAttribution'> &nbsp;&ndash; Amelia Earhart </span> </div> <p> Peace and love. <span style="font-size: 36pt;"><a href='https://en.wiktionary.org/wiki/Sigma' target='_blank' rel="nofollow">&#963;</a></span> </p> Working With Volumes and Directories Under Ubuntu 2022-12-01T00:00:00-05:00 https://mslinn.github.io/blog/2022/12/01/ubuntu-files <p> I've been reorganizing directories across several volumes on an Ubuntu server. This article documents how I used the commands I found to be most useful: <a href='https://linux.die.net/man/1/rsync' target='_blank' rel="nofollow"><code>rsync</code></a>, <a href='https://linux.die.net/man/1/ncdu' target='_blank' rel="nofollow"><code>ncdu</code></a>, and <a href='https://linux.die.net/man/1/fdupes' target='_blank' rel="nofollow"><code>fdupes</code></a>. The latter two programs are not installed by default. You can install them this way: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7b1c2e6c33ea'><button class='copyBtn' data-clipboard-target='#id7b1c2e6c33ea' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install fdupes ncdu</pre> </div> <h2 id="ncdu">Display Directory Sizes</h2> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id61e3b687d982'><button class='copyBtn' data-clipboard-target='#id61e3b687d982' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ncdu /directory</pre> </div> <h2 id="fdupes">Find Duplicate Files</h2> <p> This displays duplicate files and interactively asks which duplicate files to delete. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb97bd7e13adc'><button class='copyBtn' data-clipboard-target='#idb97bd7e13adc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>fdupes -r1Sd /directory</pre> </div> <h2 id="comp">Compare Two Folders For Missing Files</h2> <p> This tip was inspired by an answer on <a href='https://unix.stackexchange.com/questions/524074/compare-two-folders-for-missing-files' target='_blank' rel="nofollow"><code>unix.stackexchange.com</code></a>. </p> <h3 id="dry">Dry Run</h3> <p> The <code>-n</code> option displays the names of missing files in the destination directory, but makes no changes to the destination file system. If that option is not specified, then the missing files are copied. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf57ae4c108fa'><button class='copyBtn' data-clipboard-target='#idf57ae4c108fa' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sync -ri --ignore-existing -n /srcDir<span class="bg_yellow">/</span> /destDir</pre> </div> <p> There are a few things to note: </p> <ul> <li>The first directory is the source, and it must end with a slash (<kbd class="bg_yellow">/</kbd>).</li> <li>The second directory is the target, and it must NOT end with a slash.</li> </ul> <h3 id="doit">Copy Missing Files</h3> <p> Simply do not provide the <code>-n</code> (dry run) option: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idddcf670b6e2d'><button class='copyBtn' data-clipboard-target='#idddcf670b6e2d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>rsync -ri --ignore-existing /srcDir<span class="bg_yellow">/</span> /destDir</pre> </div> JiraCLI, a Feature-rich Interactive Jira Command Line 2022-08-12T00:00:00-04:00 https://mslinn.github.io/blog/2022/08/12/jiracli <!-- #region intro --> <p> When first released in 2002, <a href='https://www.atlassian.com/software/jira' target='_blank' rel="nofollow">Jira</a> was merely a bug tracker. Since then, it has gained features and is now used as a project management tool. Jira only provides a web user interface. </p> <p> <a href='https://github.com/ankitpokhrel/jira-cli' target='_blank' rel="nofollow">JiraCLI</a>, an independent F/OSS project, provides command-line explorers for Jira issues, epics, and sprints. </p> <!-- endregion --> <!-- #region Installation --> <h2 id="install">Installation</h2> <p> JiraCLI is a Go program, so it is quick and easy to install on Ubuntu once Go is installed. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb03d07d3db10'><button class='copyBtn' data-clipboard-target='#idb03d07d3db10' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install golang-go<br> <span class='unselectable'>$ </span>go install github.com/ankitpokhrel/jira-cli/cmd/jira@latest <span class='unselectable'>go: downloading github.com/ankitpokhrel/jira-cli v1.0.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/spf13/cobra v1.5.0 go: downloading github.com/spf13/viper v1.12.0 go: downloading github.com/zalando/go-keyring v0.2.1 go: downloading github.com/briandowns/spinner v1.18.1 go: downloading github.com/fatih/color v1.13.0 go: downloading github.com/mitchellh/go-homedir v1.1.0 go: downloading github.com/AlecAivazis/survey/v2 v2.3.5 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 go: downloading github.com/cpuguy83/go-md2man/v2 v2.0.2 go: downloading github.com/spf13/pflag v1.0.5 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/atotto/clipboard v0.1.4 go: downloading github.com/charmbracelet/glamour v0.5.0 go: downloading github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d go: downloading github.com/cli/safeexec v1.0.0 go: downloading github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 go: downloading github.com/kentaro-m/blackfriday-confluence v0.0.0-20220126124413-8e85477b49b3 go: downloading github.com/russross/blackfriday/v2 v2.1.0 go: downloading github.com/mattn/go-colorable v0.1.12 go: downloading github.com/mattn/go-isatty v0.0.14 go: downloading github.com/gdamore/tcell/v2 v2.5.1 go: downloading github.com/rivo/tview v0.0.0-20220610163003-691f46d6f500 go: downloading github.com/fsnotify/fsnotify v1.5.4 go: downloading github.com/mitchellh/mapstructure v1.5.0 go: downloading github.com/spf13/afero v1.8.2 go: downloading github.com/spf13/cast v1.5.0 go: downloading github.com/spf13/jwalterweatherman v1.1.0 go: downloading github.com/godbus/dbus/v5 v5.1.0 go: downloading golang.org/x/text v0.3.7 go: downloading golang.org/x/term v0.0.0-20220526004731-065cf7ba2467 go: downloading golang.org/x/sys v0.0.0-20220615213510-4f61da869c0c go: downloading github.com/subosito/gotenv v1.4.0 go: downloading github.com/hashicorp/hcl v1.0.0 go: downloading gopkg.in/ini.v1 v1.66.6 go: downloading github.com/magiconair/properties v1.8.6 go: downloading github.com/pelletier/go-toml/v2 v2.0.2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/gdamore/encoding v1.0.0 go: downloading github.com/pelletier/go-toml v1.9.5 go: downloading github.com/lucasb-eyer/go-colorful v1.2.0 go: downloading github.com/mattn/go-runewidth v0.0.13 go: downloading github.com/muesli/termenv v0.12.0 go: downloading github.com/yuin/goldmark v1.4.12 go: downloading github.com/yuin/goldmark-emoji v1.0.1 go: downloading github.com/rivo/uniseg v0.2.0 go: downloading github.com/alecthomas/chroma v0.10.0 go: downloading github.com/microcosm-cc/bluemonday v1.0.18 go: downloading github.com/muesli/reflow v0.3.0 go: downloading github.com/olekukonko/tablewriter v0.0.5 go: downloading github.com/aymerick/douceur v0.2.0 go: downloading golang.org/x/net v0.0.0-20220621193019-9d032be2e588 go: downloading github.com/dlclark/regexp2 v1.4.0 go: downloading github.com/gorilla/css v1.0.0 </span></pre> </div> <!-- endregion --> <p> Add <code>$HOME/go/bin</code> to the <code>PATH</code> so <code>jira</code> can be found: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc69ea89092f2'><button class='copyBtn' data-clipboard-target='#idc69ea89092f2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo 'PATH=$HOME/go/bin:$PATH' >> ~/.bashrc<br> <span class='unselectable'>$ </span>source ~/.bashrc<br> <span class='unselectable'>$ </span>which jira <span class='unselectable'>/home/mslinn/go/bin/jira </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Usage --> <h2 id="usage">Usage</h2> <p>This is the <code>jira </code> help message:</p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0a47c78df698'><button class='copyBtn' data-clipboard-target='#id0a47c78df698' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira <span class='unselectable'>Interactive Jira CLI.<br/> USAGE jira [flags]<br/> MAIN COMMANDS board Board manages Jira boards in a project epic Epic manage epics in a project issue Issue manage issues in a project open Open issue in a browser project Project manages Jira projects sprint Sprint manage sprints in a project board<br/> OTHER COMMANDS completion Output shell completion code for the specified shell (bash or zsh) help Help about any command init Init initializes jira config man Help generate man(7) pages for Jira CLI. me Displays configured jira user version Print the app version information<br/> FLAGS -c, --config string Config file (default is /home/mslinn/.config/.jira/.config.yml) --debug Turn on debug output -h, --help help for jira -p, --project string Jira project to look into (defaults to /home/mslinn/.config/.jira/.config.yml)<br/> LEARN MORE Use &#39;jira &lt;command&gt; &lt;subcommand&gt; --help&#39; for more information about a command. </span></pre> </div> <!-- endregion --> <p> You need an Jira API token before you can use JiraCLI. Get it from <a href='https://id.atlassian.com/manage-profile/security/api-tokens' target='_blank' rel="nofollow">here</a>. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/atlassian_api_token.svg" type="image/svg"> <!---<source srcset="/blog/images/atlassian_api_token.avif" type="image/avif">--> <source srcset="/blog/images/atlassian_api_token.webp" type="image/webp"> <source srcset="/blog/images/atlassian_api_token.apng" type="image/apng"> <source srcset="/blog/images/atlassian_api_token.png" type="image/png"> <source srcset="/blog/images/atlassian_api_token.jpg" type="image/jpeg"> <source srcset="/blog/images/atlassian_api_token.jpeg" type="image/jpeg"> <source srcset="/blog/images/atlassian_api_token.jfif" type="image/jpeg"> <source srcset="/blog/images/atlassian_api_token.pjpeg" type="image/jpeg"> <source srcset="/blog/images/atlassian_api_token.pjp" type="image/jpeg"> <source srcset="/blog/images/atlassian_api_token.gif" type="image/gif"> <source srcset="/blog/images/atlassian_api_token.tif" type="image/tiff"> <source srcset="/blog/images/atlassian_api_token.tiff" type="image/tiff"> <source srcset="/blog/images/atlassian_api_token.bmp" type="image/bmp"> <source srcset="/blog/images/atlassian_api_token.ico" type="image/x-icon"> <source srcset="/blog/images/atlassian_api_token.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/atlassian_api_token.png" style='width: 100%; ' /> </picture> </div> <p> You need to configure the program before you use it. Set the <code>JIRA_API_TOKEN</code> environment variable before running <code>jira init</code>. If you do not, then you will get an error like <code>Received unexpected response '401 Unauthorized' from jira.</code> in the next step. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7e97ed005966'><button class='copyBtn' data-clipboard-target='#id7e97ed005966' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>export JIRA_API_TOKEN=asdfasdfasdf <span class='unselectable'>$ </span>jira init <span class='unselectable'>? Installation type: Cloud ? Link to Jira server: https://xxx.atlassian.net/ ? Login email: mslinn@mslinn.com ⠼ Verifying login details... ? Default project: My ? Default board: [Use arrows to move, type to filter, ? for more help] > [Search...] ---------- My Scrum Board None ✓ Configuration generated: /home/mslinn/.config/.jira/.config.yml </span></pre> </div> <!-- endregion --> <p> This is the configuration file that was generated: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>/home/mslinn/.config/.jira/.config.yml/</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id89db4d502b6c'><button class='copyBtn' data-clipboard-target='#id89db4d502b6c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>board: id: 350 name: My Scrum Board type: scrum epic: name: customfield_10009 link: customfield_10008 installation: Cloud issue: fields: custom: - name: Epic Link key: customfield_10008 schema: datatype: any - name: Epic Name key: customfield_10009 schema: datatype: string types: - id: "3" name: Task handle: Task subtask: false - id: "5" name: Sub-task handle: Sub-task subtask: true - id: "7" name: Story handle: Story subtask: false - id: "1" name: Bug handle: Bug subtask: false - id: "6" name: Epic handle: Epic subtask: false login: mslinn@mslinn.com project: key: My type: classic server: https://xxx.atlassian.net</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Example Commands --> <h2 id="ex">Example Commands</h2> <p> Help for the <code>jira issue</code> subcommand: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6a9277e1afaf'><button class='copyBtn' data-clipboard-target='#id6a9277e1afaf' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue -h <span class='unselectable'>Issue manage issues in a given project. See available commands below.<br/> USAGE jira issue [flags]<br/> MAIN COMMANDS assign Assign issue to a user clone Clone duplicates an issue comment Manage issue comments create Create an issue in a project delete Delete an issue edit Edit an issue in a project link Link connects two issues list List lists issues in a project move Transition an issue to a given state unlink Unlink disconnects two issues from each other view View displays contents of an issue worklog Manage issue worklog<br/> FLAGS -h, --help help for issue<br/> INHERITED FLAGS -c, --config string Config file (default is /home/mslinn/.config/.jira/.config.yml) --debug Turn on debug output -p, --project string Jira project to look into (defaults to /home/mslinn/.config/.jira/.config.yml)<br/> ALIASES issues<br/> LEARN MORE Use &#39;jira &lt;command&gt; &lt;subcommand&gt; --help&#39; for more information about a command. </span></pre> </div> <!-- endregion --> <p> Help for the <code>jira issue list</code> sub-subcommand: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf7ad8e385044'><button class='copyBtn' data-clipboard-target='#idf7ad8e385044' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue list -h <span class='unselectable'>List lists issues in a given project.<br/> You can combine different flags to create a unique query. For instance,<br/> # Issues that are of high priority, is in progress, was created this month, and has given labels jira issue list -yHigh -s&quot;In Progress&quot; --created month -lbackend -l&quot;high prio&quot;<br/> Issues are displayed in an interactive list view by default. You can use a --plain flag to display output in a plain text mode. A --no-headers flag will hide the table headers in plain view. A --no-truncate flag will display all available fields in plain mode.<br/> USAGE jira issue list [flags]<br/> FLAGS -t, --type string Filter issues by type -R, --resolution string Filter issues by resolution type -s, --status string Filter issues by status -y, --priority string Filter issues by priority -r, --reporter string Filter issues by reporter (email or display name) -a, --assignee string Filter issues by assignee (email or display name) -C, --component string Filter issues by component -l, --label stringArray Filter issues by label -P, --parent string Filter issues by parent --history Issues you accessed recently -w, --watching Issues you are watching --created string Filter issues by created date Accepts: today, week, month, year, or a date in yyyy-mm-dd and yyyy/mm/dd format, or a period format using w = weeks, d = days, h = hours, m = minutes. eg: -10d Created filter will have precedence over created-after and created-before filter --updated string Filter issues by updated date Accepts: today, week, month, year, or a date in yyyy-mm-dd and yyyy/mm/dd format, or a period format using w = weeks, d = days, h = hours, m = minutes. eg: -10d Updated filter will have precedence over updated-after and updated-before filter --created-after string Filter by issues created after certain date --updated-after string Filter by issues updated after certain date --created-before string Filter by issues created before certain date --updated-before string Filter by issues updated before certain date -q, --jql string Run a raw JQL query in a given project context --order-by string Field to order the list with (default &quot;created&quot;) --reverse Reverse the display order (default &quot;DESC&quot;) --paginate string Paginate the result. Max 100 at a time, format: &lt;from&gt;:&lt;limit&gt; where &lt;from&gt; is optional (default &quot;0:100&quot;) --plain Display output in plain mode --no-headers Don&#39;t display table headers in plain mode. Works only with --plain --no-truncate Show all available columns in plain mode. Works only with --plain --columns string Comma separated list of columns to display in the plain mode. Accepts: TYPE, KEY, SUMMARY, STATUS, ASSIGNEE, REPORTER, PRIORITY, RESOLUTION, CREATED, UPDATED -h, --help help for list<br/> INHERITED FLAGS -c, --config string Config file (default is /home/mslinn/.config/.jira/.config.yml) --debug Turn on debug output -p, --project string Jira project to look into (defaults to /home/mslinn/.config/.jira/.config.yml)<br/> EXAMPLES $ jira issue list<br/> # Limit list to 20 items $ jira issue list --paginate 20<br/> # Get 50 items starting from 10 $ jira issue list --paginate 10:50<br/> # List issues in a plain table view without headers $ jira issue list --plain --no-headers<br/> # List some columns of the issue in a plain table view $ jira issue list --plain --columns key,assignee,status<br/> # List issues in a plain table view and show all fields $ jira issue list --plain --no-truncate<br/> # List issues of type &quot;Epic&quot; in status &quot;Done&quot; $ jira issue list -tEpic -sDone<br/> # List issues in status other than &quot;Open&quot; and is assigned to no one $ jira issue list -s~Open -ax<br/> # List issues from all projects $ jira issue list -q&quot;project IS NOT EMPTY&quot;<br/> ALIASES lists ls<br/> LEARN MORE Use &#39;jira &lt;command&gt; &lt;subcommand&gt; --help&#39; for more information about a command. </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Issues Updated on a Certain Date --> <h3 id="updated">Issues Updated on a Certain Date</h3> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id603f7cb11d98'><button class='copyBtn' data-clipboard-target='#id603f7cb11d98' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue list --updated 2021-11-17</pre> </div> <!-- endregion --> <!-- #region Issues Selected by Complex Criteria --> <h3 id="updated">Issues Selected by Complex Criteria</h3> <p> The following command will yield a list of high-priority issues, created this month, with status <code>To Do</code>, that are assigned to you, and have the label <code>backend</code>. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id99d1d4fc1258'><button class='copyBtn' data-clipboard-target='#id99d1d4fc1258' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue list -yHigh -s"To Do" --created month -lbackend -a$(jira me)</pre> </div> <p> Output might look something like the following, which was redacted: </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/jira_issues.svg" type="image/svg"> <!---<source srcset="/blog/images/jira_issues.avif" type="image/avif">--> <source srcset="/blog/images/jira_issues.webp" type="image/webp"> <source srcset="/blog/images/jira_issues.apng" type="image/apng"> <source srcset="/blog/images/jira_issues.png" type="image/png"> <source srcset="/blog/images/jira_issues.jpg" type="image/jpeg"> <source srcset="/blog/images/jira_issues.jpeg" type="image/jpeg"> <source srcset="/blog/images/jira_issues.jfif" type="image/jpeg"> <source srcset="/blog/images/jira_issues.pjpeg" type="image/jpeg"> <source srcset="/blog/images/jira_issues.pjp" type="image/jpeg"> <source srcset="/blog/images/jira_issues.gif" type="image/gif"> <source srcset="/blog/images/jira_issues.tif" type="image/tiff"> <source srcset="/blog/images/jira_issues.tiff" type="image/tiff"> <source srcset="/blog/images/jira_issues.bmp" type="image/bmp"> <source srcset="/blog/images/jira_issues.ico" type="image/x-icon"> <source srcset="/blog/images/jira_issues.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/jira_issues.png" style='width: 100%; ' /> </picture> </div> <!-- endregion --> <!-- #region Issue Ids Only --> <h3 id="ids"><!-- #region Issue Ids Only --></h3> <p> The <code>--plain</code> and <code>--no-headers</code> options are useful for driving scripts. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8a1a1d52bb65'><button class='copyBtn' data-clipboard-target='#id8a1a1d52bb65' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue list --updated 2021-11-17 --plain --no-headers --columns KEY</pre> </div> <!-- endregion --> <!-- #region View Issue --> <h3 id="detail">View Issue</h3> <p> The following returns the available information about an issue, in a plain-text format. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd74aba049933'><button class='copyBtn' data-clipboard-target='#idd74aba049933' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue view ISSUE-1 --comments 9999 --plain</pre> </div> <!-- endregion --> <!-- #region Verdict --> <h2 id="verdict">Verdict</h2> <p> JiraCLI is useful project! <span style='font-size: 3em;;'>&#x1F601;</span> </p> <!-- endregion --> ImageMagick Slicing on Ubuntu/WSL 2022-07-28T00:00:00-04:00 https://mslinn.github.io/blog/2022/07/28/imagemagick-slicing <!-- #region intro --> <p> Lawyers like the Microsoft Office software suite; so when I am working on a court case as an expert, I endeavor to provide my clients with Word documents that contain necessary information. I like working in WSL/WSL2 because I can use Windows programs and Ubuntu programs together effectively. </p> <!-- endregion --> <!-- #region Grab Image, Then Slice --> <h2 id="steps">Grab Image, Then Slice</h2> <p> Recently, I used <a href='https://www.techsmith.com/screen-capture.html' target='_blank' rel="nofollow">SnagIt</a>, a Windows program, to capture large web pages as single images. Some of these images were quite tall. </p> <p> The CSS for the web pages made some content invisible on the printed page. Yes, I could have injected CSS using a Chrome plugin like <a href='https://chrome.google.com/webstore/detail/my-style/ljdhjpmbnkbengahefamnhmegbdifhlb' target='_blank' rel="nofollow">My Style</a> to ensure that all content will be printed, like this: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Injected Style</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1f0851edcd33'><button class='copyBtn' data-clipboard-target='#id1f0851edcd33' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>@media print { * { display: initial; visibility: visible; } }</pre> </div> <!-- endregion --> <p> I decided to use screen grabs, which would guarantee that the contents of my report would exactly match what had been displayed on the screen, without injecting anything into the web pages. </p> <p> <a href='https://imagemagick.org/index.php' target='_blank' rel="nofollow">ImageMagick</a> is preinstalled on Ubuntu Desktop. I used ImageMagick to slice the image captures into smaller page-sized images, so they could be inserted into a Word document. </p> <!-- endregion --> <!-- #region The Computer Worked Hard --> <h2 id="grunt">The Computer Worked Hard</h2> <p> Grabbing such large web pages was a lot of work for my desktop computer. The only programs active during the screen grab process were the Google Chrome browser and SnagIt. I found that 10GB RAM and 30% of the GPU capability (an NVidia GTX 1660 Super) was used. </p> <p> The screen grab failed if I did not start scrolling from the top of the web page; while it is possible to scrub up and down smaller web pages in order to grab portions of interest, this fails for large pages. </p> <p> I also found that scrolling too fast caused the screen grabbing process to fail. Clicking and holding the bottom scroll arrowhead at the bottom right of the screen seemed to result in a smooth and optimal scrolling speed. This meant that grabbing large web pages took a few minutes as the page slowly scrolled downward. </p> <!-- endregion --> <!-- #region Setting Up the Conversion --> <h2 id="setup">Setting Up the Conversion</h2> <p> The Word documents I usually work with are formatted for North American standards. This means one-inch margins on letter-sized paper (8.5" x 11"), which gives a working area of 6.5" x 9", yielding an aspect ratio of 0.72. </p> <p> The tall captured images needed to be sliced into rectangles that fit efficiently into Word documents. The computations are as follows. </p> <ol> <li> Determine the width of a screen grab and save it into <code>W</code>. The ImageMagick <code>identify</code> command does not provide a newline after its output, however I have inserted one for readability: <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd02de3489c38'><button class='copyBtn' data-clipboard-target='#idd02de3489c38' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>identify -ping -format '%w' ../IMG2005.png <span class='unselectable'>1536 </span> <span class='unselectable'>$ </span>export W="$( identify -ping -format '%w' ../IMG2005.png )"</pre> </div> </li> <li> Determine the height of a screen grab and save it into <code>H</code>: <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide01e33d12c2f'><button class='copyBtn' data-clipboard-target='#ide01e33d12c2f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>identify -ping -format '%h' ../IMG2005.png <span class='unselectable'>$ </span>export H="$( identify -ping -format '%h' ../IMG2005.png )"</pre> </div> </li> <li> The width can be divided by the aspect ratio to obtain the desired height of each slice so they can be inserted optimally into the Word documents. I used the <a href='https://linux.die.net/man/1/bc' target='_blank' rel="nofollow"><code>bc</code> calculator</a> provided with <code>Bash</code> to divide <code>W / ASPECT_RATIO</code>. The <code>H2</code> integer variable contains the computed height for the images. <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id906f3b2720b1'><button class='copyBtn' data-clipboard-target='#id906f3b2720b1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>export ASPECT_RATIO=0.72 <span class='unselectable'>$ </span>export H2="$( echo "scale=0 ; $W / $ASPECT_RATIO" | bc )"</pre> </div> </li> <li> Now the image called <code>IMG2005.png</code> can be sliced using ImageMagick’s convert command. The slices are stored into a subdirectory called <code>slices</code>, with file names like <code>IMG2005-1.jpg</code>, <code>IMG2005-2.jpg</code>, etc. <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id39a3f02a03c9'><button class='copyBtn' data-clipboard-target='#id39a3f02a03c9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>convert IMG2005.png -crop ${W}x${H2} \ -quality 100% -scene 0 slices/IMG2005-%d.jpg</pre> </div> </li> </ol> <!-- endregion --> <!-- #region Automating the Conversion --> <h2 id="script">Automating the Conversion</h2> <p> I wrote the following bash script, which incorporates the above computations. It slices all the images in a directory and saves the results to a second directory. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>sliceImages</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9bd7ed037acb'><button class='copyBtn' data-clipboard-target='#id9bd7ed037acb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash<br> function help { if [ "$1" ]; then echo "Error: $1"; fi echo " $(basename $0): slice all images in the given directory and place them into a specified directory, which will be created if required. " exit 1 }<br> function setup { export ASPECT_RATIO=0.72 export W="$( identify -ping -format '%w' "$1" )" export H="$( identify -ping -format '%h' "$1" )" export H2="$( echo "scale=0 ; $W / $ASPECT_RATIO" | bc )" }<br> function convert1 { FULLNAME=$(basename -- "$1") FILENAME="${FULLNAME%.*}" FILETYPE="${FULLNAME##*.}"<br> convert "$1" \ -crop "${W}x${H2}" \ -quality 100% \ -scene 0 \ "$DIR_OUTPUT/$FILENAME-%d.png" }<br><br> if [ -z "$1" ]; then help "No directory path for images to be converted was provided."; fi export DIR_INPUT="$( realpath $1 )"<br> if [ -z "$2" ]; then help "No directory path for the image slices to be saved into was provided."; fi export DIR_OUTPUT="$( realpath $2 )"<br> mkdir -p "$DIR_OUTPUT"<br> find $DIR_INPUT -type f -exec file --mime-type {} \+ | awk -F: '{if ($2 ~/image\//) print $1}' | while read FILE; do setup "$FILE" echo "Slicing $FILE into ${W}x${H2} pixels" convert1 "$FILE" done</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Overcoming ImageMagick Processing Limits --> <h2 id="limits">Overcoming ImageMagick Processing Limits</h2> <p> Some of the web pages that I needed to grab were quite long, which resulted in those images requiring more computational resources than the default Imagemagick configuration allows. This caused errors such as the following to appear: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc6cc5facb93d'><button class='copyBtn' data-clipboard-target='#idc6cc5facb93d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>convert-im6.q16: no images defined `/mnt/c/images/slices/IMG1466-%d.png' @ error/convert.c/ConvertImageCommand/3229. convert-im6.q16: cache resources exhausted `/mnt/c/images/IMG1091.png' @ error/cache.c/OpenPixelCache/4095.</pre> </div> <p> Imagemagick defines computational resources limits in <code>/etc/ImageMagick-6/policy.xml</code>. The default maximum memory is 256 KB, the default maximum allowable height is 16,000 pixels (16KP), and the default maximum area is 128M pixels. These values are defined by the following entries: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>/etc/ImageMagick-6/policy.xml</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide8c6897cac22'><button class='copyBtn' data-clipboard-target='#ide8c6897cac22' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>&lt;policy domain="resource" name="memory" value="256MiB"/> &lt;policy domain="resource" name="height" value="16KP"/> &lt;policy domain="resource" name="area" value="128MP"/></pre> </div> <p> I changed the maximum memory limit to 2 GB RAM, the maximum height limit to 10,000,000 pixels (10MP), and the maximum area limit to 2G pixels with these entries: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>/etc/ImageMagick-6/policy.xml</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id124153a5c09d'><button class='copyBtn' data-clipboard-target='#id124153a5c09d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>&lt;policy domain="resource" name="memory" value="2GiB"/> &lt;policy domain="resource" name="height" value="10MP"/> &lt;policy domain="resource" name="area" value="2GP"/></pre> </div> <p> Alternatively, I could have simply commented out the limits, as shown in highlighted text below. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>/etc/ImageMagick-6/policy.xml</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf1f534bd5fb6'><button class='copyBtn' data-clipboard-target='#idf1f534bd5fb6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class="bg_yellow">&lt;!--</span> &lt;policy domain="resource" name="memory" value="256MiB"/> &lt;policy domain="resource" name="height" value="16KP"/> &lt;policy domain="resource" name="area" value="128MP"/> <span class="bg_yellow">--></span></pre> </div> <p> The largest web page to be sliced was converted to a very tall image, which was 83,703 pixels high. It was sliced into 40 images. </p> <!-- endregion --> <!-- #region Word Macro --> <h2 id="macro">Word Macro</h2> <p> A Word macro is also needed to insert the images into the currently open Word document in alphabetical order. I modified <a href='https://software-solutions-online.com/word-vba-insert-images/' target='_blank' rel="nofollow">this one</a>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft Word Macro</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide2bac796915e'><button class='copyBtn' data-clipboard-target='#ide2bac796915e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>Sub insertImages() Dim intResult As Integer Dim strPath As String Dim strFolderPath As String Dim objFSO As Object Dim objFolder As Object Dim objFile As Object Dim i As Integer<br> intResult = Application.FileDialog(msoFileDialogFolderPicker).Show 'Check if user canceled the dialog If intResult <> 0 Then 'dispaly message box strFolderPath = Application.FileDialog(msoFileDialogFolderPicker).SelectedItems(1) 'Create an instance of the FileSystemObject Set objFSO = CreateObject("Scripting.FileSystemObject") 'Get the folder object Set objFolder = objFSO.GetFolder(strFolderPath) i = 1 'loops through each file in the directory and prints their names and path For Each objFile In objFolder.Files 'get file path strPath = objFile.Path 'insert the image Selection.InlineShapes.AddPicture FileName:= _ strPath, LinkToFile:=False, _ SaveWithDocument:=True Next objFile End If End Sub</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Done! --> <h2 id="done">Done!</h2> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <p> Thanks to the above automation, I was able to deliver the Word documents containing the sliced web pages to my client soon after they were requested. </p> <!-- endregion --> Uncomplicated Firewall on Ubuntu 2022-07-16T00:00:00-04:00 https://mslinn.github.io/blog/2022/07/16/ufw <p> All websites, especially ecommerce sites, need to be secure. A properly set up firewall is an essential component for a secure server. </p> <p> Ubuntu 22.04 uses the <a href='https://wiki.ubuntu.com/UncomplicatedFirewall' target='_blank' rel="nofollow">Uncomplicated Firewall <code>ufw</code> firewall</a> frontend by default. <code>Ufw</code> has been provided for Ubuntu since v8.04 (Hardy Heron). </p> <h2 id="tldr">Quick Setup</h2> <p> Enable <code>ufw</code> as follows: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id407102ea525c'><button class='copyBtn' data-clipboard-target='#id407102ea525c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo ufw enable <span class='unselectable'>Firewall is active and enabled on system startup </span></pre> </div> <p> Enable the <a href='https://github.com/ageis/ufw-application-profiles' target='_blank' rel="nofollow"><code>ufw</code> application profiles</a> for <code>ssh</code> and nginx (HTTP/HTTPS) like this: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0cbfff3e8be8'><button class='copyBtn' data-clipboard-target='#id0cbfff3e8be8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo ufw allow OpenSSH <span class='unselectable'>Output Rule added Rule added (v6) </span> <span class='unselectable'>$ </span>sudo ufw allow 'Nginx Full' <span class='unselectable'>Output Rule added Rule added (v6) </span></pre> </div> <h3 id="ufwDetails">Diving Deeper</h3> <p> The following is mostly true: </p> <div class='quote'> <div class='quoteText clearfix'> The default firewall on Ubuntu 22.04 Jammy Jellyfish is <code>ufw</code>, which is short for “uncomplicated firewall.” <code>Ufw</code> is a frontend for the typical Linux <code>iptables</code> commands, but it is developed in such a way that basic firewall tasks can be performed without the knowledge of <code>iptables</code>. <br><br> Additionally, <code>ufw</code> can be managed from a graphical interface. In this tutorial, you will learn how to enable and disable the <code>ufw</code> firewall on Ubuntu 22.04 Jammy Jellyfish from both command line and GUI. </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://linuxconfig.org/how-to-enable-disable-firewall-on-ubuntu-22-04-lts-jammy-jellyfish-linux' rel='nofollow' target='_blank'>How to enable/disable firewall on Ubuntu 22.04 LTS Jammy Jellyfish Linux</a></div> </div> <p> The above makes no mention of how Ubuntu 22.04 replaced <code>iptables</code> with <code>nftables</code>, as described below. </p> <div class='quote'> <div class='quoteText clearfix'> <h2><span class="code">nftables</span> as the default firewall backend</h2> Firewalling on Linux consists of two components &ndash; the firewall mechanism within the Linux kernel, and the tools used to configure this from userspace. The Linux kernel has traditionally supported two different subsystems for firewall policies – <code>iptables</code> / <code>xtables</code> and the newer <code>nftables</code>. <br><br> <code>Nftables</code> brings significant benefits both in terms of performance and flexibility when creating and deploying firewall rules, particularly for dual stack IPv4/IPv6 systems. <br><br> The traditional <code>iptables</code> userspace management tool now configures the <code>nftables</code> kernel backend, whilst the new <code>nft</code> userspace tool is also present to allow the creation of more flexible rules not supported by the traditional iptables paradigm. </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://ubuntu.com/blog/whats-new-in-security-for-ubuntu-22-04-lts/docs-internal-guid-117e0493-7fff-ea13-3537-bde70965f89d' rel='nofollow' target='_blank'>What’s new in Security for Ubuntu 22.04 LTS?</a></div> </div> <p> Digital Ocean has a <a href='https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-18-04' target='_blank' rel="nofollow">good <code>ufw</code> tutorial</a>. </p> Using Nginx As a Reverse Proxy With SSL 2022-07-08T00:00:00-04:00 https://mslinn.github.io/blog/2022/07/08/reverse-proxy <div class="right"> <div class='imgWrapper imgFlex inline' style=' '> <figure> <a href='https://www.ScalaCourses.com' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/svg"> <!---<source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/avif">--> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/webp"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/apng"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/png"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/jpeg"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/jpeg"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/jpeg"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/jpeg"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/jpeg"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/gif"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/tiff"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/tiff"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/bmp"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/x-icon"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/x-icon"> <img alt='ScalaCourses.com' class="imgImg rounded shadow" src="/assets/images/ScalaCoursesLogo207x207.png" style='width: 100%; ' title='ScalaCourses.com' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://www.ScalaCourses.com" target='_blank' > ScalaCourses.com </a> </figcaption> </figure> </div> </div> <p> Recently I <a href='https://mslinn.com/blog/2022/05/26/aws-hijacking.html' target='_blank' rel="nofollow">migrated</a> ScalaCourses.com from AWS EC2/S3/CloudFront to a server in my apartment, which has fiber optic internet service. The server&rsquo;s motherboard is an ASUS Sabertooth x79 with an Intel i7 4820, 32 GB DDR3 RAM, and a 4TB SATA SSD. This motherboard is unable to boot from NVMe drives, so SATA is necessary. At the time of this writing, the server runs Ubuntu 22.04. </p> <p> The backup server is currently being set up. I am repurposing an old Hackintosh as a fallback to the Ubuntu server. That motherboard is another ASUS Sabertooth x79, with 32 GB DDR3 RAM and two 2 TB SATA drives. </p> <h2 id="why">Why Am I Doing This?</h2> <p> Once again, I control my hardware, my software, my network, and all ancillary services. After several years of using PaaS vendor servers, I am now reverting to running ScalaCourses.com on my own hardware and software, using my own network. </p> <div class="pullQuote"> No longer will I subject myself to the unlimited financial liability that current PaaS vendors expose their customers to. </div> <h2 id="defs">Definitions</h2> <p> A <i>proxy</i> is a person or process serving as an authorized agent or substitute for another. In computer science, a more specific term is <i>forward proxy</i>. <br><br> A <i>proxy server</i> is a server process that acts as an intermediary between a client requesting a resource, and the process that provides the resource. <br><br> A <i>reverse proxy</i> is a process that sits in front of other processes, and forwards client requests to them. The term <i>forwarding process</i> is similar to <i>reverse proxy</i>. <br><br> The definitions for <i>proxy</i>, <i>forward proxy</i> and <i>reverse proxy</i> all sound identical. The key difference between a reverse proxy and a forward proxy is that a <b>forward proxy enables computers isolated on a private network</b> to connect to the public internet, while a <b>reverse proxy enables computers on the internet</b> to access a private subnet. </p> <h2 id="details">Dealing With Details</h2> <p> The old <a href='https://en.wikipedia.org/wiki/Pound_(networking)' target='_blank' rel="nofollow">Pound</a> v2.8-2 <a href='https://www.cloudflare.com/en-ca/learning/cdn/glossary/reverse-proxy/' target='_blank' rel="nofollow">reverse proxy</a> that was the front end for the old Play Framework app that runs <a href='https://scalacourses.com' target='_blank' rel="nofollow">ScalaCourses.com</a> is no longer viable, and the new version 3 of Pound is incomplete. Depending on configuration, reverse proxies can provide extra security from external attacks on a website, decrypt https requests to http, and act as stream editors for the content. </p> <p> This site still uses AWS CloudFront, for the moment. I am researching alternatives, with the goal of closing my AWS account... unless they introduce a way to cap financial liability before I fully migrate off AWS. </p> <h2 id="vs">Apache httpd vs. Nginx</h2> <p> Two popular reverse proxies are <a href='https://httpd.apache.org/docs/trunk/mod/mod_proxy_http2.html' target='_blank' rel="nofollow">Apache <code>mod_proxy_http2</code></a> and <a href='https://www.nginx.com/' target='_blank' rel="nofollow">nginx</a>. Anything Pound can do, both nginx and Apache httpd/2 can do. While they both work well, Apache httpd has an older code base, in fact, many of my websites ran on Apache httpd at the turn of the century. </p> <p> Nginx is relatively newer than Apache httpd. It is performant, well supported, well documented, and widely available. I decided to use nginx as the reverse proxy because I had found that nginx worked well on previous projects. </p> <p> Installing nginx and the Letsencrypt software is easy on Debian distros such as Ubuntu: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9b009641e9d8'><button class='copyBtn' data-clipboard-target='#id9b009641e9d8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install nginx certbot python3-certbot-nginx</pre> </div> <h2 id="vs">Verifying the Nginx Build</h2> <p> When acting as a proxy server, nginx requires the <code>http_sub_module</code> to translate HTTP content. This allows the website content to be stream edited, so various links and paths that are not translated properly can be fixed up. </p> <p> The nginx package provided by Ubuntu includes the <code>&#8209;&#8209;with&#8209;http_sub_module</code> option. You can verify that your instance of nginx was built with the <code>&#8209;&#8209;with&#8209;http_sub_module</code> option as follows: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbd30340806af'><button class='copyBtn' data-clipboard-target='#idbd30340806af' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>nginx -V <span class='unselectable'>nginx version: nginx/1.18.0 (Ubuntu) built with OpenSSL 3.0.2 15 Mar 2022 TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -ffile-prefix-map=/build/nginx-9P0wNJ/nginx-1.18.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --add-dynamic-module=/build/nginx-9P0wNJ/nginx-1.18.0/debian/modules/http-geoip2 --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module <span class="bg_yellow">--with-http_sub_module</span> </span></pre> </div> <h2 id="vs">Nginx SSL Configuration</h2> <p> Two files are needed for Letsencrypt to be able to create SSL certificates for nginx: </p> <ol> <li> The contents of <a href='https://github.com/certbot/certbot/blob/master/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf' target='_blank' rel="nofollow"><code>/etc/letsencrypt/options-ssl-nginx.conf</code></a> need to be stored into <code>/etc/letsencrypt/options-ssl-nginx.conf</code>. I later discovered this file was also available locally at <code>/usr/lib/python3/dist-packages/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf</code>. </li> <li> The contents of <a href='https://github.com/certbot/certbot/blob/master/certbot/certbot/ssl-dhparams.pem' target='_blank' rel="nofollow"><code>/etc/letsencrypt/ssl-dhparams.pem</code></a> need to be stored into <code>/etc/letsencrypt/ssl-dhparams.pem</code>. I later discovered that this file was also available locally at <code>/usr/lib/python3/dist-packages/certbot/ssl-dhparams.pem</code>. </li> </ol> <h2 id="ssl">Making a Wildcard SSL Certificate</h2> <p> I knew an SSL wildcard certificate would be needed, so I made one using Letsencrypt. Please read <a href='/blog/2022/06/15/certbot.html'>Creating and Renewing Letsencrypt Wildcard SSL Certificates</a> for details. </p> <p class="alert rounded shadow"> Update &ndash; 8 months after writing this article, I wrote about a better way to generate wildcard SSL certificates: <a href='/blog/2023/03/02/lego.html'>Wildcard SSL Certificates for Let&apos;s Encrypt with Lego</a>. </p> <h2 id="vs">Defining the Nginx Website Reverse Proxy</h2> <p> Please see my article entitled <a href='/blog/2021/03/20/cors.html'>Cross-Origin Resource Sharing (CORS)</a> for a discussion of how to configure servers whose content needs to be proxied. </p> <p> I saved the following configuration in a new file called <code>/etc/nginx/sites-available/scalacourses.com</code>. Note that a single <code>server</code> block answers on ports 80 and 443, using IPv4 and IPv6, for SSL and non-SSL requests, for the apex domain (<code>scalacourses.com</code>) and the <code>www.scalacourses.com</code> subdomain. No redirects are used. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id518017b205dc'><button class='copyBtn' data-clipboard-target='#id518017b205dc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>server { listen 80; listen [::]:80; listen 443 ssl; listen [::]:443 ssl; server_name scalacourses.com www.scalacourses.com; ssl_certificate /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/fullchain.pem; ssl_certificate_key /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/privkey.pem; ssl_trusted_certificate /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/chain.pem; include /etc/letsencrypt/options-ssl-nginx.conf; ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; root /var/www/html; index index.html; # This gets served if the proxied website is down location / { proxy_pass http://localhost:9000/; proxy_set_header Accept-Encoding ""; # sub_filter requires this sub_filter 'https://localhost:9000/authenticate/' 'https://www.scalacourses.com/authenticate/'; sub_filter_once off; } }</pre> </div> <p> I disabled the <code>default</code> site: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idad3ff27ca76d'><button class='copyBtn' data-clipboard-target='#idad3ff27ca76d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo rm /etc/nginx/sites-enabled/default</pre> </div> <p> I enabled the new <code>scalacourses.com</code> site: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc5d5a6c1b604'><button class='copyBtn' data-clipboard-target='#idc5d5a6c1b604' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo ln /etc/nginx/sites-{available,enabled}/scalacourses.com</pre> </div> <p> The <code>nginx</code> configuration was tested for syntax: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3b1c7b1a735e'><button class='copyBtn' data-clipboard-target='#id3b1c7b1a735e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo nginx -t <span class='unselectable'>nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful </span></pre> </div> <p> The nginx configuration was reloaded: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb471a6496f4e'><button class='copyBtn' data-clipboard-target='#idb471a6496f4e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl reload nginx</pre> </div> <h2 id="flush">Flush DNS Cache</h2> <p> Ensure that DNS requests receive up-to-date values by flushing the DNS cache. The following works on Ubuntu, but not when running on WSL/WSL2. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5578feee6d72'><button class='copyBtn' data-clipboard-target='#id5578feee6d72' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo resolvectl flush-caches</pre> </div> <p> The following works on Windows 10: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Command and PowerShell consoles</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6781fd4782cc'><button class='copyBtn' data-clipboard-target='#id6781fd4782cc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>ipconfig /flushdns</pre> </div> <h2 id="verify">Verifying nginx Works</h2> <p> I verified that <code>nginx</code> was listening on ports 80 and 443. I highlighted the <code>nginx</code> process number &ndash; you might need to scroll the output below to the right to see it. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id94c66c01764e'><button class='copyBtn' data-clipboard-target='#id94c66c01764e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo netstat -tulpn | grep ':\(443\|80\)' <span class='unselectable'>tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN - tcp6 0 0 :::80 :::* LISTEN - tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN <span class="bg_yellow">87487</span>/nginx: master </span></pre> </div> <span> <p> The executable for process 87487 can be found by: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id010bdddd8009'><button class='copyBtn' data-clipboard-target='#id010bdddd8009' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls -l /proc/87487/exe <span class='unselectable'>lrwxrwxrwx 1 mslinn mslinn 0 Jul 7 09:13 /proc/5166/exe -> <span class="bg_yellow">/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java*</span> </span></pre> </div> <p> If more detail is desired, get it from the process list. Enclosing a character of the process id within square brackets is an old trick for only showing the desired process. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7efff55cf3fd'><button class='copyBtn' data-clipboard-target='#id7efff55cf3fd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ps aux | grep [8]7487 <span class='unselectable'>mslinn 87487 2.4 1.3 6596476 439456 ? Sl 09:13 0:20 java -Xms1024m -Xmx1024m -Dhttp.port=9000 /path/to/jar </span></pre> </div> <p> If there is any problem getting things to work, it is often helpful to monitor the logs as you click on the web pages: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7f1e6edc202e'><button class='copyBtn' data-clipboard-target='#id7f1e6edc202e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo tail -f /var/log/nginx/*.log</pre> </div> <h2 id="persist">Finishing Up</h2> <p> Make nginx start each time the system starts as follows. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id75525715cfcc'><button class='copyBtn' data-clipboard-target='#id75525715cfcc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl enable nginx <span class='unselectable'>Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable nginx </span> <span class='unselectable'>$ </span>sudo update-rc.d nginx defaults</pre> </div> <h3 id="performance">Performance Test</h3> <p> <a href='https://tools.keycdn.com/performance?url=https://www.scalacourses.com' target='_blank' rel="nofollow">KeyCDN</a> offfers free website performance statistics. The result column labeled <b>TTFB</b> means &ldquo;time to first byte&rdquo;, which is the length of time required for the website content to begin being received by a user's web browser. </p> <p> I am in Montreal, Canada. Response time for TTFB reported by KeyCDN varies, from a minimum of 68ms in New York, to a maximum of 815ms in Bangalore. This range of response times is typical for websites that have a centralized process. The speed of light is finite, after all. </p> <h3 id="monitor">Free Availability Monitoring</h3> <p> <a href='https://hetrixtools.com/' target='_blank' rel="nofollow">HetrixTools</a> offers free availability monitoring for up to 10 websites. It was quick and easy to set up. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/hetrix.svg" type="image/svg"> <!---<source srcset="/blog/images/hetrix.avif" type="image/avif">--> <source srcset="/blog/images/hetrix.webp" type="image/webp"> <source srcset="/blog/images/hetrix.apng" type="image/apng"> <source srcset="/blog/images/hetrix.png" type="image/png"> <source srcset="/blog/images/hetrix.jpg" type="image/jpeg"> <source srcset="/blog/images/hetrix.jpeg" type="image/jpeg"> <source srcset="/blog/images/hetrix.jfif" type="image/jpeg"> <source srcset="/blog/images/hetrix.pjpeg" type="image/jpeg"> <source srcset="/blog/images/hetrix.pjp" type="image/jpeg"> <source srcset="/blog/images/hetrix.gif" type="image/gif"> <source srcset="/blog/images/hetrix.tif" type="image/tiff"> <source srcset="/blog/images/hetrix.tiff" type="image/tiff"> <source srcset="/blog/images/hetrix.bmp" type="image/bmp"> <source srcset="/blog/images/hetrix.ico" type="image/x-icon"> <source srcset="/blog/images/hetrix.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/hetrix.png" style='width: 100%; ' /> </picture> </div> <h3 id="live">We Are Live!</h3> <p> <a href='https://www.scalacourses.com' target='_blank' rel="nofollow"><code>ScalaCourses.com</code></a> now serves Scala students from its newly refurbished server! <span style='font-size: 3em;;'>&#x1F601;</span> </p> Trialing mslinn.com on Linode Storage 2022-07-01T00:00:00-04:00 https://mslinn.github.io/blog/2022/07/01/trialing-linode-storage <!-- #region intro --> <p> This is another article in my ongoing saga of moving off AWS, which <a href='/blog/2022/05/26/aws-hijacking.html'>does not integrate security with real-time billing</a>. I recently learned this the hard way: when my AWS account was hijacked, in less than 15 minutes, a huge bill was incurred. </p> <div class="pullQuote"> &ldquo;Pay-as-you-go&rdquo; is shorthand for &ldquo;there is nothing you can do to limit your financial liability&rdquo; </div> <p> The world of pain that I experienced after the breach was inflicted by broken and wasteful AWS remedial processes, and an ineffective AWS management structure. This type of issue only is enabled because of deficiencies in the AWS architecture. Those AWS architectural deficiencies feel like the result of an exploitive mindset: </p> <div class='quote'> <div class='quoteText clearfix'> Unlimited financial liability is our customer&rsquo;s problem, not ours &ndash; as a result, exploits are quite profitable for us. </div><div class='quoteAttribution'> &nbsp;&ndash; From a mythical retrospective discussion at an AWS offsite. </div> </div> <div class='quote'> <div class='quoteText clearfix'> At this moment that feature of setting limits does not exist, Azure is not able to safeguard customers from unlimited financial liability. </div><div class='quoteAttribution'> &nbsp;&ndash; From an email sent to me from Microsoft Azure support staff on 2022-06-22. </div> </div> <!-- endregion --> <!-- #region Demand Limits to Financial Liability From PaaS Vendors --> <div class="alert rounded shadow"> <h2>Demand Limits to Financial Liability From PaaS Vendors</h2> <p> PaaS vendors currently provide accounts with all services ready to go, without limit. That is good for the vendor's bottom line, but highly dangerous for their customers. </p> <p> All PaaS customers should demand the ability for themselves to be able to set firm limits on budgeted expenses, along with the ability to deny all services not explicitly authorized. </p> <p> You can buy insurance against losses resulting from various calamities, but you cannot limit your financial liability with PaaS vendors. </p> <p> <span style="font-size:larger; font-weight: bold;">Yet.</span> </p> </div> <!-- endregion --> <!-- #region Website Hosting Market --> <h2 id="market">Website Hosting Market</h2> <p> Recently, I have spent a lot of time looking at options for hosting websites. I found the following types of products: </p> <style> .row th { text-align: right; } </style> <table class="table"> <tr> <td></td> <th>WordPress</th> <th>General Web Server</th> <th>VM</th> <th>S3 Compatible</th> </tr> <tr class="row"> <th>CDN</th> <td>No</td> <td>No</td> <td>No</td> <td>Yes</td> </tr> <tr class="row"> <th>Speed</th> <td>Slow to Medium</td> <td>Slow to Medium</td> <td>Slow to Fast</td> <td>Fast</td> </tr> <tr class="row"> <th>Reliability</th> <td>Fair</td> <td>Fair</td> <td>Depends on you</td> <td>Good</td> </tr> <tr class="row"> <th>Financial Liability</th> <td>Fixed monthly cost</td> <td>Fixed monthly cost</td> <td class="bg_yellow">Depends</td> <td class="bg_yellow">Unlimited</td> </tr> <tr class="row"> <th>Storage</th> <td>Small</td> <td>Small</td> <td>Depends</td> <td>Unlimited</td> </tr> <tr class="row"> <th>CLI & API</th> <td>No</td> <td>No</td> <td>Depends</td> <td>Yes</td> </tr> </table> <p> This website requires about 220 GB of storage. It has numerous images, in 2 versions: <a href='/blog/2020/08/15/converting-all-images-to-webp-format.html'><code>webp</code></a> and <code>png</code>. So for this website, &lsquo;small&rsquo; means less than 250 GB storage. </p> <p> I decided to give the S3-compatible product <a href='https://www.linode.com/docs/products/storage/object-storage/' target='_blank' rel="nofollow">Linode Storage</a> a try. </p> <!-- endregion --> <!-- #region Why Linode --> <h2 id="why">Why Linode?</h2> <p> First, the positive: <a href='https://www.linode.com/' target='_blank' rel="nofollow">Linode</a> is a pioneer in virtual computing. It has been on my short list of places for hosting my pet projects for many years. Prices have always been quite competitive, products solid, with good support ... and smart people have answered the phone when I have called. Being able to easily speak with a capable human provides huge value. </p> <p> On the other hand, as I shall demonstrate in this article, Linode&rsquo;s documentation presents as a significant barrier to customers considering adopting their services. I had to work a lot harder than I should have to get my evaluation website up and running. Hopefully, this document is complete enough, so others can follow along and host their static websites on Linode Storage. </p> <!-- endregion --> <!-- #region Acquired by Akamai --> <h2 id="akamai">Acquired by Akamai</h2> <p> <a href='https://www.akamai.com/newsroom/press-release/akamai-completes-acquisition-of-linode' target='_blank' rel="nofollow">Linode was acquired by Akamai</a> 3 months ago. Akamai is the original CDN, and their network is gigantic. </p> <p> Linode does not yet offer a CDN product, but the person I spoke to at Linode when I started writing this article suggested that a CDN product from Linode based on Akamai&rsquo;s network might be available soon. He also said that they were planning to working on a mechanism for limiting financial liability to their customers in 2022. </p> <p> This was music to my financially risk-averse ears! </p> <div class="notepaper shadow" style="width: 80%;"> <p> The remainder of this article will take you through all the steps necessary to: </p> <ol> <li>Install and configure software tools on a WSL / WSL2 / Ubuntu computer</li> <li>Make an S3-compatible bucket and set it up to hold a website</li> <li>Upload the website, and easily handle mimetype issues</li> <li>Generate and install a free 4096-bit SSL wildcard certificate</li> <li>Get a security rating from Qualys / SSL Labs</li> </ol> </div> <!-- endregion --> <!-- #region Trialing Linode Storage --> <h2 id="setup">Trialing Linode Storage</h2> <!-- #region Installing s3cmd --> <h3 id="s3cmd">Installing <span class="code">s3cmd</span></h3> <p> I installed the recommended S3-compatible command-line program <code>s3cmd</code> on WSL2 / Ubuntu like this: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida565bb031a04'><button class='copyBtn' data-clipboard-target='#ida565bb031a04' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install s3cmd <span class='unselectable'>Reading package lists... Done Building dependency tree... Done Reading state information... Done The following additional packages will be installed: python3-magic The following NEW packages will be installed: python3-magic s3cmd 0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 133 kB of archives. After this operation, 584 kB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-magic all 2:0.4.24-2 [12.6 kB] Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 s3cmd all 2.2.0-1 [120 kB] Fetched 133 kB in 0s (278 kB/s) Selecting previously unselected package python3-magic. (Reading database ... 169821 files and directories currently installed.) Preparing to unpack .../python3-magic_2%3a0.4.24-2_all.deb ... Unpacking python3-magic (2:0.4.24-2) ... Selecting previously unselected package s3cmd. Preparing to unpack .../archives/s3cmd_2.2.0-1_all.deb ... Unpacking s3cmd (2.2.0-1) ... Setting up python3-magic (2:0.4.24-2) ... Setting up s3cmd (2.2.0-1) ... Processing triggers for man-db (2.10.2-1) ... Scanning processes... Scanning processor microcode... Scanning linux images... Failed to retrieve available kernel versions. Failed to check for processor microcode upgrades. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. </span></pre> </div> <!-- endregion --> <p> Here is the <code>s3cmd</code> help message: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id72b744a0a73c'><button class='copyBtn' data-clipboard-target='#id72b744a0a73c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd <span class='unselectable'>Usage: s3cmd [options] COMMAND [parameters]<br/> S3cmd is a tool for managing objects in Amazon S3 storage. It allows for making and removing &quot;buckets&quot; and uploading, downloading and removing &quot;objects&quot; from these buckets.<br/> Options: -h, --help show this help message and exit --configure Invoke interactive (re)configuration tool. Optionally use as &#39;--configure s3://some-bucket&#39; to test access to a specific bucket instead of attempting to list them all. -c FILE, --config=FILE Config file name. Defaults to $HOME/.s3cfg --dump-config Dump current configuration after parsing config files and command line options and exit. --access_key=ACCESS_KEY AWS Access Key --secret_key=SECRET_KEY AWS Secret Key --access_token=ACCESS_TOKEN AWS Access Token -n, --dry-run Only show what should be uploaded or downloaded but don&#39;t actually do it. May still perform S3 requests to get bucket listings and other information though (only for file transfer commands) -s, --ssl Use HTTPS connection when communicating with S3. (default) --no-ssl Don&#39;t use HTTPS. -e, --encrypt Encrypt files before uploading to S3. --no-encrypt Don&#39;t encrypt files. -f, --force Force overwrite and other dangerous operations. --continue Continue getting a partially downloaded file (only for [get] command). --continue-put Continue uploading partially uploaded files or multipart upload parts. Restarts parts/files that don&#39;t have matching size and md5. Skips files/parts that do. Note: md5sum checks are not always sufficient to check (part) file equality. Enable this at your own risk. --upload-id=UPLOAD_ID UploadId for Multipart Upload, in case you want continue an existing upload (equivalent to --continue- put) and there are multiple partial uploads. Use s3cmd multipart [URI] to see what UploadIds are associated with the given URI. --skip-existing Skip over files that exist at the destination (only for [get] and [sync] commands). -r, --recursive Recursive upload, download or removal. --check-md5 Check MD5 sums when comparing files for [sync]. (default) --no-check-md5 Do not check MD5 sums when comparing files for [sync]. Only size will be compared. May significantly speed up transfer but may also miss some changed files. -P, --acl-public Store objects with ACL allowing read for anyone. --acl-private Store objects with default ACL allowing access for you only. --acl-grant=PERMISSION:EMAIL or USER_CANONICAL_ID Grant stated permission to a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all --acl-revoke=PERMISSION:USER_CANONICAL_ID Revoke stated permission for a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all -D NUM, --restore-days=NUM Number of days to keep restored file available (only for &#39;restore&#39; command). Default is 1 day. --restore-priority=RESTORE_PRIORITY Priority for restoring files from S3 Glacier (only for &#39;restore&#39; command). Choices available: bulk, standard, expedited --delete-removed Delete destination objects with no corresponding source file [sync] --no-delete-removed Don&#39;t delete destination objects [sync] --delete-after Perform deletes AFTER new uploads when delete-removed is enabled [sync] --delay-updates *OBSOLETE* Put all updated files into place at end [sync] --max-delete=NUM Do not delete more than NUM files. [del] and [sync] --limit=NUM Limit number of objects returned in the response body (only for [ls] and [la] commands) --add-destination=ADDITIONAL_DESTINATIONS Additional destination for parallel uploads, in addition to last arg. May be repeated. --delete-after-fetch Delete remote objects after fetching to local file (only for [get] and [sync] commands). -p, --preserve Preserve filesystem attributes (mode, ownership, timestamps). Default for [sync] command. --no-preserve Don&#39;t store FS attributes --exclude=GLOB Filenames and paths matching GLOB will be excluded from sync --exclude-from=FILE Read --exclude GLOBs from FILE --rexclude=REGEXP Filenames and paths matching REGEXP (regular expression) will be excluded from sync --rexclude-from=FILE Read --rexclude REGEXPs from FILE --include=GLOB Filenames and paths matching GLOB will be included even if previously excluded by one of --(r)exclude(-from) patterns --include-from=FILE Read --include GLOBs from FILE --rinclude=REGEXP Same as --include but uses REGEXP (regular expression) instead of GLOB --rinclude-from=FILE Read --rinclude REGEXPs from FILE --files-from=FILE Read list of source-file names from FILE. Use - to read from stdin. --region=REGION, --bucket-location=REGION Region to create bucket in. As of now the regions are: us-east-1, us-west-1, us-west-2, eu-west-1, eu- central-1, ap-northeast-1, ap-southeast-1, ap- southeast-2, sa-east-1 --host=HOSTNAME HOSTNAME:PORT for S3 endpoint (default: s3.amazonaws.com, alternatives such as s3-eu- west-1.amazonaws.com). You should also set --host- bucket. --host-bucket=HOST_BUCKET DNS-style bucket+hostname:port template for accessing a bucket (default: %(bucket)s.s3.amazonaws.com) --reduced-redundancy, --rr Store object with &#39;Reduced redundancy&#39;. Lower per-GB price. [put, cp, mv] --no-reduced-redundancy, --no-rr Store object without &#39;Reduced redundancy&#39;. Higher per- GB price. [put, cp, mv] --storage-class=CLASS Store object with specified CLASS (STANDARD, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER or DEEP_ARCHIVE). [put, cp, mv] --access-logging-target-prefix=LOG_TARGET_PREFIX Target prefix for access logs (S3 URI) (for [cfmodify] and [accesslog] commands) --no-access-logging Disable access logging (for [cfmodify] and [accesslog] commands) --default-mime-type=DEFAULT_MIME_TYPE Default MIME-type for stored objects. Application default is binary/octet-stream. -M, --guess-mime-type Guess MIME-type of files by their extension or mime magic. Fall back to default MIME-Type as specified by --default-mime-type option --no-guess-mime-type Don&#39;t guess MIME-type and use the default type instead. --no-mime-magic Don&#39;t use mime magic when guessing MIME-type. -m MIME/TYPE, --mime-type=MIME/TYPE Force MIME-type. Override both --default-mime-type and --guess-mime-type. --add-header=NAME:VALUE Add a given HTTP header to the upload request. Can be used multiple times. For instance set &#39;Expires&#39; or &#39;Cache-Control&#39; headers (or both) using this option. --remove-header=NAME Remove a given HTTP header. Can be used multiple times. For instance, remove &#39;Expires&#39; or &#39;Cache- Control&#39; headers (or both) using this option. [modify] --server-side-encryption Specifies that server-side encryption will be used when putting objects. [put, sync, cp, modify] --server-side-encryption-kms-id=KMS_KEY Specifies the key id used for server-side encryption with AWS KMS-Managed Keys (SSE-KMS) when putting objects. [put, sync, cp, modify] --encoding=ENCODING Override autodetected terminal and filesystem encoding (character set). Autodetected: UTF-8 --add-encoding-exts=EXTENSIONs Add encoding to these comma delimited extensions i.e. (css,js,html) when uploading to S3 ) --verbatim Use the S3 name as given on the command line. No pre- processing, encoding, etc. Use with caution! --disable-multipart Disable multipart upload on files bigger than --multipart-chunk-size-mb --multipart-chunk-size-mb=SIZE Size of each chunk of a multipart upload. Files bigger than SIZE are automatically uploaded as multithreaded- multipart, smaller files are uploaded using the traditional method. SIZE is in Mega-Bytes, default chunk size is 15MB, minimum allowed chunk size is 5MB, maximum is 5GB. --list-md5 Include MD5 sums in bucket listings (only for &#39;ls&#39; command). -H, --human-readable-sizes Print sizes in human readable form (eg 1kB instead of 1234). --ws-index=WEBSITE_INDEX Name of index-document (only for [ws-create] command) --ws-error=WEBSITE_ERROR Name of error-document (only for [ws-create] command) --expiry-date=EXPIRY_DATE Indicates when the expiration rule takes effect. (only for [expire] command) --expiry-days=EXPIRY_DAYS Indicates the number of days after object creation the expiration rule takes effect. (only for [expire] command) --expiry-prefix=EXPIRY_PREFIX Identifying one or more objects with the prefix to which the expiration rule applies. (only for [expire] command) --progress Display progress meter (default on TTY). --no-progress Don&#39;t display progress meter (default on non-TTY). --stats Give some file-transfer stats. --enable Enable given CloudFront distribution (only for [cfmodify] command) --disable Disable given CloudFront distribution (only for [cfmodify] command) --cf-invalidate Invalidate the uploaded filed in CloudFront. Also see [cfinval] command. --cf-invalidate-default-index When using Custom Origin and S3 static website, invalidate the default index file. --cf-no-invalidate-default-index-root When using Custom Origin and S3 static website, don&#39;t invalidate the path to the default index file. --cf-add-cname=CNAME Add given CNAME to a CloudFront distribution (only for [cfcreate] and [cfmodify] commands) --cf-remove-cname=CNAME Remove given CNAME from a CloudFront distribution (only for [cfmodify] command) --cf-comment=COMMENT Set COMMENT for a given CloudFront distribution (only for [cfcreate] and [cfmodify] commands) --cf-default-root-object=DEFAULT_ROOT_OBJECT Set the default root object to return when no object is specified in the URL. Use a relative path, i.e. default/index.html instead of /default/index.html or s3://bucket/default/index.html (only for [cfcreate] and [cfmodify] commands) -v, --verbose Enable verbose output. -d, --debug Enable debug output. --version Show s3cmd version (2.2.0) and exit. -F, --follow-symlinks Follow symbolic links as if they are regular files --cache-file=FILE Cache FILE containing local source MD5 values -q, --quiet Silence output on stdout --ca-certs=CA_CERTS_FILE Path to SSL CA certificate FILE (instead of system default) --ssl-cert=SSL_CLIENT_CERT_FILE Path to client own SSL certificate CRT_FILE --ssl-key=SSL_CLIENT_KEY_FILE Path to client own SSL certificate private key KEY_FILE --check-certificate Check SSL certificate validity --no-check-certificate Do not check SSL certificate validity --check-hostname Check SSL certificate hostname validity --no-check-hostname Do not check SSL certificate hostname validity --signature-v2 Use AWS Signature version 2 instead of newer signature methods. Helpful for S3-like systems that don&#39;t have AWS Signature v4 yet. --limit-rate=LIMITRATE Limit the upload or download speed to amount bytes per second. Amount may be expressed in bytes, kilobytes with the k suffix, or megabytes with the m suffix --no-connection-pooling Disable connection re-use --requester-pays Set the REQUESTER PAYS flag for operations -l, --long-listing Produce long listing [ls] --stop-on-error stop if error in transfer --content-disposition=CONTENT_DISPOSITION Provide a Content-Disposition for signed URLs, e.g., &quot;inline; filename=myvideo.mp4&quot; --content-type=CONTENT_TYPE Provide a Content-Type for signed URLs, e.g., &quot;video/mp4&quot;<br/> Commands: Make bucket s3cmd mb s3://BUCKET Remove bucket s3cmd rb s3://BUCKET List objects or buckets s3cmd ls [s3://BUCKET[/PREFIX]] List all object in all buckets s3cmd la Put file into bucket s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX] Get file from bucket s3cmd get s3://BUCKET/OBJECT LOCAL_FILE Delete file from bucket s3cmd del s3://BUCKET/OBJECT Delete file from bucket (alias for del) s3cmd rm s3://BUCKET/OBJECT Restore file from Glacier storage s3cmd restore s3://BUCKET/OBJECT Synchronize a directory tree to S3 (checks files freshness using size and md5 checksum, unless overridden by options, see below) s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR or s3://BUCKET[/PREFIX] s3://BUCKET[/PREFIX] Disk usage by buckets s3cmd du [s3://BUCKET[/PREFIX]] Get various information about Buckets or Files s3cmd info s3://BUCKET[/OBJECT] Copy object s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2] Modify object metadata s3cmd modify s3://BUCKET1/OBJECT Move object s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2] Modify Access control list for Bucket or Files s3cmd setacl s3://BUCKET[/OBJECT] Modify Bucket Policy s3cmd setpolicy FILE s3://BUCKET Delete Bucket Policy s3cmd delpolicy s3://BUCKET Modify Bucket CORS s3cmd setcors FILE s3://BUCKET Delete Bucket CORS s3cmd delcors s3://BUCKET Modify Bucket Requester Pays policy s3cmd payer s3://BUCKET Show multipart uploads s3cmd multipart s3://BUCKET [Id] Abort a multipart upload s3cmd abortmp s3://BUCKET/OBJECT Id List parts of a multipart upload s3cmd listmp s3://BUCKET/OBJECT Id Enable/disable bucket access logging s3cmd accesslog s3://BUCKET Sign arbitrary string using the secret key s3cmd sign STRING-TO-SIGN Sign an S3 URL to provide limited public access with expiry s3cmd signurl s3://BUCKET/OBJECT &lt;expiry_epoch|+expiry_offset&gt; Fix invalid file names in a bucket s3cmd fixbucket s3://BUCKET[/PREFIX] Create Website from bucket s3cmd ws-create s3://BUCKET Delete Website s3cmd ws-delete s3://BUCKET Info about Website s3cmd ws-info s3://BUCKET Set or delete expiration rule for the bucket s3cmd expire s3://BUCKET Upload a lifecycle policy for the bucket s3cmd setlifecycle FILE s3://BUCKET Get a lifecycle policy for the bucket s3cmd getlifecycle s3://BUCKET Remove a lifecycle policy for the bucket s3cmd dellifecycle s3://BUCKET List CloudFront distribution points s3cmd cflist Display CloudFront distribution point parameters s3cmd cfinfo [cf://DIST_ID] Create CloudFront distribution point s3cmd cfcreate s3://BUCKET Delete CloudFront distribution point s3cmd cfdelete cf://DIST_ID Change CloudFront distribution point parameters s3cmd cfmodify cf://DIST_ID Display CloudFront invalidation request(s) status s3cmd cfinvalinfo cf://DIST_ID[/INVAL_ID]<br/> For more information, updates and news, visit the s3cmd website: http://s3tools.org </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Signup --> <h3 id="signup">Signup</h3> <p> I went to the <a href='https://login.linode.com/signup' target='_blank' rel="nofollow">Linode signup page</a> and signed up with my email. Using third parties for authentication means they track you more easily, and introduces an unnecessary dependency. </p> <p> The email signup procedure requires a mobile phone for SMS-based MFA. I would rather use a <a href='https://datatracker.ietf.org/doc/html/rfc6238' target='_blank' rel="nofollow">TOTP authenticator app</a> instead of SMS for MFA. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/linode/signup.svg" type="image/svg"> <!---<source srcset="/blog/images/linode/signup.avif" type="image/avif">--> <source srcset="/blog/images/linode/signup.webp" type="image/webp"> <source srcset="/blog/images/linode/signup.apng" type="image/apng"> <source srcset="/blog/images/linode/signup.png" type="image/png"> <source srcset="/blog/images/linode/signup.jpg" type="image/jpeg"> <source srcset="/blog/images/linode/signup.jpeg" type="image/jpeg"> <source srcset="/blog/images/linode/signup.jfif" type="image/jpeg"> <source srcset="/blog/images/linode/signup.pjpeg" type="image/jpeg"> <source srcset="/blog/images/linode/signup.pjp" type="image/jpeg"> <source srcset="/blog/images/linode/signup.gif" type="image/gif"> <source srcset="/blog/images/linode/signup.tif" type="image/tiff"> <source srcset="/blog/images/linode/signup.tiff" type="image/tiff"> <source srcset="/blog/images/linode/signup.bmp" type="image/bmp"> <source srcset="/blog/images/linode/signup.ico" type="image/x-icon"> <source srcset="/blog/images/linode/signup.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/linode/signup.png" style='width: 100%; ' /> </picture> </div> <p> I want to ensure that I limit my financial liability. One way of protecting myself is to limit the maximum amount that can be charged to the payment mechanism. Because PayPal has no maximum transaction limit, it is not a suitable choice for a service with unlimited financial liability. However, credit cards can have transaction limits set, and Google Pay has the following limits: </p> <div class='quote'> If you <a href='https://support.google.com/googlepay/answer/10623602' target='_blank' rel="nofollow">set up your Google Pay balance to make contactless payments</a>, there are some transaction limits: <br><br> Maximum single transaction amount: $2,000 USD.<br> Daily maximum total transaction amount: $2,500 USD.<br> Up to 15 transactions per day.<br> Additional limits on the dollar amount or frequency of transactions may be imposed in accordance with the Google Pay Terms of Service. <span class='quoteAttribution'> &nbsp;&ndash; From <a href='https://support.google.com/googlepay/answer/10187490#:~:text=If%20you%20set%20up%20your,to%2015%20transactions%20per%20day.' rel='nofollow' target='_blank'>Google Pay Help</a></span> </div> <p> <a href='https://support.google.com/googlepay/answer/10187490' target='_blank' rel="nofollow">Here</a> are additional limits for Google Pay. Because of prior good experiences with how credit card processors handled fraud, I decided to use a credit card with a low limit instead of Google Pay. </p> <!-- endregion --> <!-- #region Generating Linode Access Keys --> <h3 id="keys">Generating Linode Access Keys</h3> <p> I followed the <a href='https://www.linode.com/docs/products/storage/object-storage/guides/access-keys/' target='_blank' rel="nofollow">directions</a>: </p> <ol> <li>Logged into the <a href='https://cloud.linode.com/' target='_blank' rel="nofollow">Cloud Manager</a>.</li> <li>Selected the <b>Object Storage</b> menu item in the sidebar and clicked on the <b>Access Keys</b> tab.</li> <li>Clicked on the <b>Create Access</b> Key button, which displays the <b>Create Access Key</b> panel.</li> <li>Typed in the name <code>mslinn</code> as the label for the new access key.</li> <li>Clicked the <kbd>Submit</kbd> button to create the access key.</li> <li>The new access key and its secret key are displayed. This is the only time that the secret key is visible. I stored the access key and the secret key in my password manager, <a href='https://lastpass.com'><code>lastpass.com</code></a>.</li> </ol> <!-- endregion --> <!-- #region Configuring s3cmd --> <h3 id="s3cmd_cfg">Configuring <span class="code">s3cmd</span></h3> <p> <code>s3cmd</code> is a Python program that works with all S3-compatible APIs, such as those provided by AWS, Linode Storage, Google Cloud Storage and DreamHost DreamObjects. I intend to use its <a href='https://s3tools.org/s3cmd-sync' target='_blank' rel="nofollow"><code>sync</code></a> subcommand to synchronize the <code>www.mslinn.com</code> bucket contents in Linode Storage with the most recently generated version of this website by Jekyll. </p> <p> <code>s3cmd</code> needs to be <a href='https://s3tools.org/kb/item14.htm' target='_blank' rel="nofollow">configured</a> for <a href='https://www.linode.com/docs/products/storage/object-storage/guides/s3cmd' target='_blank' rel="nofollow">Linode</a> before it can be used. Some configuration settings for Linode are non-obvious. The following should work for most users, with the caution that the access key and secret key are of course unique for every user. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida0935a8b0e7a'><button class='copyBtn' data-clipboard-target='#ida0935a8b0e7a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd --configure <span class='unselectable'><br>Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options.<br/> Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. Access Key [asdfasdf]: </span>asdfasdfasdf <span class='unselectable'>Secret Key [asdfasdfasdf]: </span>asdfasdfasdf <span class='unselectable'>Default Region [US]:<br/> Use &quot;s3.amazonaws.com&quot; for S3 Endpoint and not modify it to the target Amazon S3. S3 Endpoint [s3.amazonaws.com]: </span>us-east-1.linodeobjects.com<br/> <span class='unselectable'>Use &quot;%(bucket)s.s3.amazonaws.com&quot; to the target Amazon S3. &quot;%(bucket)s&quot; and &quot;%(location)s&quot; vars can be used if the target S3 system supports dns based buckets. DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: </span>%(bucket)s.us-east-1.linodeobjects.com<br/> <span class='unselectable'>Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: Path to GPG program [/usr/bin/gpg]:<br/> When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP, and can only be proxied with Python 2.7 or newer Use HTTPS protocol [Yes]:<br/> On some networks all internet access must go through a HTTP proxy. Try setting it here if you can&#39;t connect to S3 directly HTTP Proxy server name:<br/> New settings: Access Key: </span>asdfasdfasdf <span class='unselectable'>Secret Key: </span>asdfasdfasdf <span class='unselectable'>Default Region: US S3 Endpoint: </span>https://us-east-1.linodeobjects.com <span class='unselectable'>DNS-style bucket+hostname:port template for accessing a bucket: </span>%(bucket)s.us-east-1.linodeobjects.com <span class='unselectable'>Encryption password: Path to GPG program: /usr/bin/gpg Use HTTPS protocol: True HTTP Proxy server name: HTTP Proxy server port: 0<br/> Test access with supplied credentials? [Y/n] n Save settings? [y/N] y Configuration saved to '/home/mslinn/.s3cfg' </span></pre> </div> <!-- endregion --> <p> I was unable to successfully test access. Two different errors appeared at different times: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3c56aedfef14'><button class='copyBtn' data-clipboard-target='#id3c56aedfef14' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>Please wait, attempting to list all buckets... ERROR: Test failed: 403 (SignatureDoesNotMatch) ERROR: Test failed: [Errno -2] Name or service not known<br/></pre> </div> <!-- endregion --> <p> Eventually I saved the configuration without testing as shown above. This created a file called <code>$HOME/.s3cfg</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>~/.s3cfg</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id30e1c8ff4b57'><button class='copyBtn' data-clipboard-target='#id30e1c8ff4b57' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[default] access_key = asdfasdfasdf access_token = add_encoding_exts = add_headers = bucket_location = US ca_certs_file = cache_file = check_ssl_certificate = True check_ssl_hostname = True cloudfront_host = cloudfront.amazonaws.com connection_max_age = 5 connection_pooling = True content_disposition = content_type = default_mime_type = binary/octet-stream delay_updates = False delete_after = False delete_after_fetch = False delete_removed = False dry_run = False enable_multipart = True encoding = UTF-8 encrypt = False expiry_date = expiry_days = expiry_prefix = follow_symlinks = False force = False get_continue = False gpg_command = /usr/bin/gpg gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_passphrase = guess_mime_type = True host_base = us-east-1.linodeobjects.com host_bucket = %(bucket)s.us-east-1.linodeobjects.com human_readablesizes = False invalidate_default_index_on_cf = False invalidate_default_index_root_on_cf = True invalidate_on_cf = False kms_key = limit = -1 limitrate = 0 list_md5 = False log_target_prefix = long_listing = False max_delete = -1 mime_type = multipart_chunksize_mb = 15 multipart_copy_chunksize_mb = 1024 multipart_max_chunks = 10000 preserve_attrs = True progress_meter = True proxy_host = proxy_port = 0 public_url_use_https = False put_continue = False recursive = False recv_chunk = 65536 reduced_redundancy = False requester_pays = False restore_days = 1 restore_priority = Standard secret_key = asdfasdfasdf send_chunk = 65536 server_side_encryption = False signature_v2 = False signurl_use_https = False simpledb_host = sdb.amazonaws.com skip_existing = False socket_timeout = 300 ssl_client_cert_file = ssl_client_key_file = stats = False stop_on_error = False storage_class = throttle_max = 100 upload_id = urlencoding_mode = normal use_http_expect = False use_https = True use_mime_magic = True verbosity = INFO website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/ website_error = website_index = index.html</pre> </div> <!-- endregion --> <p> According to the docs (<a href='https://www.linode.com/docs/products/storage/object-storage/guides/urls/' target='_blank' rel="nofollow">Access Buckets and Files through URLs</a>), the configuration file needs <code>website_endpoint: http://%(bucket)s.website-[cluster-url]/</code>. However, this is an error; instead of <code>cluster-url</code>, which might be <code>https://us-east-1.linodeobjects.com</code>, <code>cluster-id</code> should be used, which for me was simply <code>us-east-1</code>. <span class="bg_yellow">Thus the endpoint for me, and everyone with a bucket at <code>us-east-1</code> in Newark, New Jersey should be:<br> <code>http://%(bucket)s.<span class="bg_yellow">website-</span>us-east-1.linodeobjects.com/</code></span> </p> <p> After editing the file to manually change the value of <code>website_endpoint</code>, I tried to list the buckets, and that worked: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6d22155af193'><button class='copyBtn' data-clipboard-target='#id6d22155af193' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd ls <span class='unselectable'>2022-07-01 17:32 s3://mslinn </span></pre> </div> <!-- endregion --> <!-- #region linode-cli --> <h3 id="linode-cli"><span class="code">linode-cli</span></h3> <p> Next I tried <a href='https://github.com/linode/linode-cli' target='_blank' rel="nofollow"><code>linode-cli</code></a>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id62316e391b34'><button class='copyBtn' data-clipboard-target='#id62316e391b34' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pip install linode-cli <span class='unselectable'>Collecting linode-cli Downloading linode_cli-5.21.0-py2.py3-none-any.whl (204 kB) &#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473; 204.2/204.2 kB 4.8 MB/s eta 0:00:00 Collecting terminaltables Downloading terminaltables-3.1.10-py2.py3-none-any.whl (15 kB) Collecting PyYAML Downloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB) &#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473; 682.2/682.2 kB 16.8 MB/s eta 0:00:00 Requirement already satisfied: requests in /home/mslinn/venv/default/lib/python3.10/site-packages (from linode-cli) (2.28.0) Requirement already satisfied: charset-normalizer~=2.0.0 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests-&gt;linode-cli) (2.0.12) Requirement already satisfied: idna&lt;4,&gt;=2.5 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests-&gt;linode-cli) (3.3) Requirement already satisfied: urllib3&lt;1.27,&gt;=1.21.1 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests-&gt;linode-cli) (1.26.9) Requirement already satisfied: certifi&gt;=2017.4.17 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests-&gt;linode-cli) (2022.6.15) Installing collected packages: terminaltables, PyYAML, linode-cli Successfully installed PyYAML-6.0 linode-cli-5.21.0 terminaltables-3.1.10 </span></pre> </div> <!-- endregion --> <p> The documentation fails to mention that the first time <code>linode-cli</code> is executed, and every time it is invoked with the <code>configure</code> subcommand, it attempts to open a web browser. WSL and WSL2 will not respond appropriately unless an X client is already running, for example <a href='https://apps.microsoft.com/store/detail/gwsl/9NL6KD1H33V3' target='_blank' rel="nofollow">GWSL</a>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4ee7343fd8af'><button class='copyBtn' data-clipboard-target='#id4ee7343fd8af' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>linode-cli configure <span class='unselectable'><br>Welcome to the Linode CLI. This will walk you through some initial setup.<br/> The CLI will use its web-based authentication to log you in. If you prefer to supply a Personal Access Token, use <code>linode-cli configure --token</code><br/> Press enter to continue. This will open a browser and proceed with authentication. A browser should open directing you to this URL to authenticate:<br/> https://login.linode.com/oauth/authorize?client_id=asdfasdfasdf&amp;response_type=token&amp;scopes=*&amp;redirect_uri=http://localhost:45789<br/> If you are not automatically directed there, please copy/paste the link into your browser to continue..<br/><br/> Configuring mslinn<br/><br/> Default Region for operations. Choices are: 1 - ap-west 2 - ca-central 3 - ap-southeast 4 - us-central 5 - us-west 6 - us-southeast 7 - us-east 8 - eu-west 9 - ap-south 10 - eu-central 11 - ap-northeast<br/> Default Region (Optional): 7<br/> Default Type of Linode to deploy. Choices are: 1 - g6-nanode-1 2 - g6-standard-1 3 - g6-standard-2 4 - g6-standard-4 5 - g6-standard-6 6 - g6-standard-8 7 - g6-standard-16 8 - g6-standard-20 9 - g6-standard-24 10 - g6-standard-32 11 - g7-highmem-1 12 - g7-highmem-2 13 - g7-highmem-4 14 - g7-highmem-8 15 - g7-highmem-16 16 - g6-dedicated-2 17 - g6-dedicated-4 18 - g6-dedicated-8 19 - g6-dedicated-16 20 - g6-dedicated-32 21 - g6-dedicated-48 22 - g6-dedicated-50 23 - g6-dedicated-56 24 - g6-dedicated-64 25 - g1-gpu-rtx6000-1 26 - g1-gpu-rtx6000-2 27 - g1-gpu-rtx6000-3 28 - g1-gpu-rtx6000-4<br/> Default Type of Linode (Optional):<br/> Default Image to deploy to new Linodes. Choices are: 1 - linode/almalinux8 2 - linode/almalinux9 3 - linode/alpine3.12 4 - linode/alpine3.13 5 - linode/alpine3.14 6 - linode/alpine3.15 7 - linode/alpine3.16 8 - linode/arch 9 - linode/centos7 10 - linode/centos-stream8 11 - linode/centos-stream9 12 - linode/debian10 13 - linode/debian11 14 - linode/debian9 15 - linode/fedora34 16 - linode/fedora35 17 - linode/fedora36 18 - linode/gentoo 19 - linode/kali 20 - linode/debian11-kube-v1.20.15 21 - linode/debian9-kube-v1.20.7 22 - linode/debian9-kube-v1.21.1 23 - linode/debian11-kube-v1.21.12 24 - linode/debian9-kube-v1.22.2 25 - linode/debian11-kube-v1.22.9 26 - linode/debian11-kube-v1.23.6 27 - linode/opensuse15.3 28 - linode/opensuse15.4 29 - linode/rocky8 30 - linode/slackware14.2 31 - linode/slackware15.0 32 - linode/ubuntu16.04lts 33 - linode/ubuntu18.04 34 - linode/ubuntu20.04 35 - linode/ubuntu21.10 36 - linode/ubuntu22.04 37 - linode/centos8 38 - linode/slackware14.1 39 - linode/ubuntu21.04<br/> Default Image (Optional): Active user is now mslinn<br/> Config written to /home/mslinn/.config/linode-cli </span></pre> </div> <!-- endregion --> <p> <code>linode-cli</code> configuration created this file: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>~/.config/linode-cli</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida589b7f4a907'><button class='copyBtn' data-clipboard-target='#ida589b7f4a907' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[DEFAULT] default-user = mslinn [mslinn] token = asdfasdfasdfasdfasdfasdf region = us-east</pre> </div> <!-- endregion --> <p> Right after I did the above, I was surprised to get the following email from Linode: </p> <div class="quote"> Hello,<br><br> This is a notification to inform you that a device has been trusted to skip authentication on your Linode Account (mslinn). The request came from the following IP address: 70.53.179.203. <br><br> The device will not be prompted for a username or password for 30 days. <br><br> If this action did not originate from you, we recommend logging in and changing your password immediately. Also, as an extra layer of security, we highly recommend enabling two-factor authentication on your account. Please see the link below for more information on how to enable two-factor authentication: <br><br> <a href='https://www.linode.com/docs/security/linode-manager-security-controls/#enable-two-factor-authentication' target='_blank' rel="nofollow">Two-Factor Authentication</a> <br><br> If you have any questions or concerns, please do not hesitate to contact us 24/7 by opening a support ticket from within the Linode Manager, giving us a call, or emailing <a href='mailto:support@linode.com'>support@linode.com</a>. <br><br> <a href='https://www.linode.com/contact' target='_blank' rel="nofollow">Contact</a> <br><br> Thank you,<br> The Linode.com Team<br> <br><br> <a href='mailto:support@linode.com'>Contact Us</a> </div> <p class="alert rounded shadow"> I began to suspect that Linode&rsquo;s documentation had major deficiencies at this point. Instead of merely configuring a command-line client, my IP was whitelisted unexpectedly. I have nothing polite to say about this. </p> <p> After poking around a bit, I discovered my <a href='https://cloud.linode.com/profile/tokens' target='_blank' rel="nofollow">API Tokens page</a> on Linode. It seems that the <code>linode-cli</code> created a personal access token with a label containing the name of the computer used: <code>Linode CLI @ camille</code>. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/linode/apiTokens.svg" type="image/svg"> <!---<source srcset="/blog/images/linode/apiTokens.avif" type="image/avif">--> <source srcset="/blog/images/linode/apiTokens.webp" type="image/webp"> <source srcset="/blog/images/linode/apiTokens.apng" type="image/apng"> <source srcset="/blog/images/linode/apiTokens.png" type="image/png"> <source srcset="/blog/images/linode/apiTokens.jpg" type="image/jpeg"> <source srcset="/blog/images/linode/apiTokens.jpeg" type="image/jpeg"> <source srcset="/blog/images/linode/apiTokens.jfif" type="image/jpeg"> <source srcset="/blog/images/linode/apiTokens.pjpeg" type="image/jpeg"> <source srcset="/blog/images/linode/apiTokens.pjp" type="image/jpeg"> <source srcset="/blog/images/linode/apiTokens.gif" type="image/gif"> <source srcset="/blog/images/linode/apiTokens.tif" type="image/tiff"> <source srcset="/blog/images/linode/apiTokens.tiff" type="image/tiff"> <source srcset="/blog/images/linode/apiTokens.bmp" type="image/bmp"> <source srcset="/blog/images/linode/apiTokens.ico" type="image/x-icon"> <source srcset="/blog/images/linode/apiTokens.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/linode/apiTokens.png" style='width: 100%; ' /> </picture> </div> <p> Clicking on the <kbd>View Scopes</kbd> button above displays the permissions granted to the token: </p> <div class='imgWrapper imgFlex center halfsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/linode/cliPermissions.svg" type="image/svg"> <!---<source srcset="/blog/images/linode/cliPermissions.avif" type="image/avif">--> <source srcset="/blog/images/linode/cliPermissions.webp" type="image/webp"> <source srcset="/blog/images/linode/cliPermissions.apng" type="image/apng"> <source srcset="/blog/images/linode/cliPermissions.png" type="image/png"> <source srcset="/blog/images/linode/cliPermissions.jpg" type="image/jpeg"> <source srcset="/blog/images/linode/cliPermissions.jpeg" type="image/jpeg"> <source srcset="/blog/images/linode/cliPermissions.jfif" type="image/jpeg"> <source srcset="/blog/images/linode/cliPermissions.pjpeg" type="image/jpeg"> <source srcset="/blog/images/linode/cliPermissions.pjp" type="image/jpeg"> <source srcset="/blog/images/linode/cliPermissions.gif" type="image/gif"> <source srcset="/blog/images/linode/cliPermissions.tif" type="image/tiff"> <source srcset="/blog/images/linode/cliPermissions.tiff" type="image/tiff"> <source srcset="/blog/images/linode/cliPermissions.bmp" type="image/bmp"> <source srcset="/blog/images/linode/cliPermissions.ico" type="image/x-icon"> <source srcset="/blog/images/linode/cliPermissions.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/linode/cliPermissions.png" style='width: 100%; ' /> </picture> </div> <p class="alert rounded shadow"> This personal access token is all-powerful. The <code>linode-cli</code> quietly created a personal access token with all permissions enabled, and did not warn the user, and did not say how to reduce or limit the permissions. <br><br> Security is enhanced when the minimum permission necessary is provided to accomplish necessary tasks. Instead, this token silently maximizes the financial risk to the person or entity who pays for the account. </p> <!-- endregion --> <!-- #region Creating the Website Bucket --> <h3 id="bucket">Creating the Website Bucket</h3> <p class="alert rounded shadow"> The name of the bucket must exactly match the name of the subdomain that the website is served from. If you try to serve content from a misnamed bucket, you will not be able to apply an SSL certificate; instead, the certificate will be issued by <code>linodeobjects.com</code>. </p> <p> I wanted to test using the subdomain <code>linode.mslinn.com</code>, and then if all was well, switch <code>www.mslinn.com</code> over to Linode Storage. Because buckets cannot be renamed, I had to make 2 buckets with identical contents, one called <code>linode.mslinn.com</code> and the other called <code>www.mslinn.com</code>. If I decide to stay with Linode, I will delete the bucket I am using for testing, called <code>linode.mslinn.com</code>, and run this website from the <code>www.mslinn.com</code> bucket. </p> <p> I created a website-enabled bucket called <code>www.mslinn.com</code> as follows: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide3b86bc7c25a'><button class='copyBtn' data-clipboard-target='#ide3b86bc7c25a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd mb --acl-public s3://www.mslinn.com <span class='unselectable'>Bucket 's3://www.mslinn.com/' created </span> <span class='unselectable'>$ </span>s3cmd ls <span class='unselectable'>2022-07-01 17:32 s3://www.mslinn.com </span> <span class='unselectable'>$ </span>s3cmd ws-create \ --ws-index=index.html \ --ws-error=404.html \ s3://www.mslinn.com <span class='unselectable'>Bucket 's3://www.mslinn.com/': website configuration created. </span></pre> </div> <!-- endregion --> <p> The <code>s3cmd ws-info</code> subcommand displays the Linode Object Storage bucket URL as the <b>Website endpoint</b>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3c327817b2d6'><button class='copyBtn' data-clipboard-target='#id3c327817b2d6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd ws-info s3://www.mslinn.com <span class='unselectable'>Bucket s3://www.mslinn.com/: Website configuration Website endpoint: http://www.mslinn.com.website-us-east-1.linodeobjects.com/ Index document: index.html Error document: 404.html </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Defining a CNAME for the Bucket --> <h3 id="subdomain">Defining a CNAME for the Bucket</h3> <p> Although Linode provides a <a href='https://www.linode.com/docs/guides/dns-manager/' target='_blank' rel="nofollow">DNS Manager</a>, while I am testing out Linode I wanted to continue using Namecheap for DNS, so I navigated to <a href='https://ap.www.namecheap.com/Domains/DomainControlPanel/mslinn.com/advancedns'><code>ap.www.namecheap.com/Domains/DomainControlPanel/mslinn.com/advancedns</code></a> and created CNAMEs like this: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id74ceb10798ba'><button class='copyBtn' data-clipboard-target='#id74ceb10798ba' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>linode.mslinn.com CNAME linode.mslinn.com.<span class="bg_yellow">website-</span>us-east-1.linodeobjects.com www.mslinn.com CNAME www.mslinn.com.<span class="bg_yellow">website-</span>us-east-1.linodeobjects.com</pre> </div> <p class="alert rounded shadow"> Warning: if the <code>CNAME</code> does not point to a URL with the word <code>website</code> in it, for example <code>www.mslinn.com.us-east-1.linodeobjects.com</code>, the default index file and the error file will not automatically be served when expected. </p> <!-- endregion --> <!-- #region Making a Free SSL Certificate --> <h3 id="cert">Making a Free SSL Certificate</h3> <p> <a href='https://www.ssllabs.com/ssltest/' target='_blank' rel="nofollow">Qualys / SSL Labs</a> rates the SSL security provided by AWS CloudFront using the AWS-generated SSL certificates with a &ldquo;B&rdquo; grade. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/sslReport.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/sslReport.avif" type="image/avif">--> <source srcset="/blog/images/aws/sslReport.webp" type="image/webp"> <source srcset="/blog/images/aws/sslReport.apng" type="image/apng"> <source srcset="/blog/images/aws/sslReport.png" type="image/png"> <source srcset="/blog/images/aws/sslReport.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/sslReport.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/sslReport.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/sslReport.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/sslReport.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/sslReport.gif" type="image/gif"> <source srcset="/blog/images/aws/sslReport.tif" type="image/tiff"> <source srcset="/blog/images/aws/sslReport.tiff" type="image/tiff"> <source srcset="/blog/images/aws/sslReport.bmp" type="image/bmp"> <source srcset="/blog/images/aws/sslReport.ico" type="image/x-icon"> <source srcset="/blog/images/aws/sslReport.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/sslReport.png" style='width: 100%; ' /> </picture> </div> <p> Lets see if Linode Object Storage can host a more secure website when provided with a 4096-bit certificate generated by <code>certbot</code>/<code>letsencrypt</code>. The conclusion of this article will reveal the answer. </p> <p> Please read <a href='/blog/2022/06/15/certbot.html'>Creating and Renewing Letsencrypt Wildcard SSL Certificates</a> for details. </p> <!-- endregion --> <!-- #region Linode DNS Authentication --> <h4 id="linodeDns">Linode DNS Authentication</h4> <p> Eventually, if I stick with Linode, I could use the <a href='https://certbot-dns-linode.readthedocs.io/en/stable/' target='_blank' rel="nofollow"><code>certbot</code> DNS plugin for Linode</a>, but I will not use it right now because I am trialing Linode. Use of this plugin requires a configuration file containing Linode API credentials, obtained from your Linode account&rsquo;s <a href='https://cloud.linode.com/profile/tokens' target='_blank' rel="nofollow">Applications &amp; API Tokens</a> page. I stored the credentials in <code>~/.certbot/linode.ini</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>~/.certbot/linode.ini</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0311a91b2479'><button class='copyBtn' data-clipboard-target='#id0311a91b2479' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># Linode API credentials used by Certbot dns_linode_key = 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ64 dns_linode_version = [<blank>|3|4]</pre> </div> <!-- endregion --> <p> Here is how to create an SSL certificate <code>certbot</code> using Linode DNS authentication: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id55464fe97507'><button class='copyBtn' data-clipboard-target='#id55464fe97507' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><!-- #region --> <span class='unselectable'>$ </span>certbot certonly \ --dns-linode \ --dns-linode-credentials ~/.certbot/linode.ini \ --rsa-key-size 4096 \ -d mslinn.com \ -d *.mslinn.com \ --config-dir ~/.certbot/mslinn.com/config \ --logs-dir ~/.certbot/mslinn.com/logs \ --work-dir ~/.certbot/mslinn.com/work</pre> </div> <!-- endregion --> <!-- endregion --> <!-- endregion --> <!-- #region Installing the Certificate --> <h2 id="certUp">Installing the Certificate</h2> <!-- #region implicit --> <p> Now I created 2 certificates: one with the full chain of responsibility (<code>fullchain.pem</code>) and one containing my private key (<code>privkey.pem</code>). I used bash environment variables called <code>CERT_DIR</code>, <code>CERT</code> and <code>KEY</code> to make the incantation convenient to type in. <code>CERT</code> and <code>KEY</code> contain the contents of the files <code>fullchain.pem</code> and <code>privkey.pem</code>, respectively. As you can see, the result was a valid (wildcard) SSL certificate. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6e11ed466150'><button class='copyBtn' data-clipboard-target='#id6e11ed466150' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>CERT_DIR=/home/mslinn/.certbot/mslinn.com/config/live/mslinn.com <span class='unselectable'>$ </span>CERT="$( cat $CERT_DIR/fullchain.pem )" <span class='unselectable'>$ </span>KEY="$( cat $CERT_DIR/privkey.pem )" <span class='unselectable'>$ </span>linode-cli object-storage ssl-upload \ us-east-1 www.mslinn.com \ --certificate "$CERT" \ --private_key "$KEY" <div class="tree"><span class='unselectable'>┌──────┐ │ ssl │ ├──────┤ │ True │ └──────┘ </span></div></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Syncing the Website Bucket --> <h3 id="sync">Syncing the Website Bucket</h3> <p> I wrote a bash script to build this Jekyll-generated website. Jekyll places the generated website in a directory called <code>_site</code>. I <a href='https://s3tools.org/s3cmd-sync' target='_blank' rel="nofollow">synchronized</a> the contents of the freshly generated <code>mslinn.com</code> website, stored in <code>_site/</code>, with the bucket as follows. Note the trailing slash on <code>_site<span class="bg_yellow">/</span></code>, <a href='https://s3tools.org/s3cmd-sync' target='_blank' rel="nofollow">it is significant</a>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3d393b6ef2ae'><button class='copyBtn' data-clipboard-target='#id3d393b6ef2ae' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd sync \ --acl-public --delete-removed --guess-mime-type --quiet \ _site<span class="bg_yellow">/</span> \ s3://www.mslinn.com <span class='unselectable'>INFO: No cache file found, creating it. INFO: Compiling list of local files... INFO: Running stat() and reading/calculating MD5 values on 2157 files, this may take some time... INFO: [1000/2157] INFO: [2000/2157] INFO: Retrieving list of remote files for s3://www.mslinn.com/_site ... INFO: Found 2157 local files, 0 remote files INFO: Verifying attributes... INFO: Summary: 2137 local files to upload, 20 files to remote copy, 0 remote files to delete ... lots more output ... Done. Uploaded 536630474 bytes in 153.6 seconds, 3.33 MB/s. </span></pre> </div> <!-- endregion --> <p> Originally, I did not provide the <code>--acl-public</code> option to <code>s3cmd sync</code>. That meant the files in the bucket were all private &ndash; the opposite of what is required for a website. Here is the incantation for making all the files in the bucket publicly readable: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id18089b2cc5b0'><button class='copyBtn' data-clipboard-target='#id18089b2cc5b0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd setacl s3://www.mslinn.com/ \ --acl-public --recursive --quiet</pre> </div> <p> Linode&rsquo;s S3 implementation faithfully mirrors a problem that AWS S3 has. CSS files are automatically assigned the MIME type <code>text/plain</code>, instead of <code>text/css</code>. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ideecd9436a428'><button class='copyBtn' data-clipboard-target='#ideecd9436a428' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -sI http://linode.mslinn.com/assets/css/style.css | \ grep Content-Type <span class='unselectable'>Content-Type: text/plain </span></pre> </div> <p> I re-uploaded the CSS files with the proper mime type using this incantation: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc5fcec8482e4'><button class='copyBtn' data-clipboard-target='#idc5fcec8482e4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cd _site <span class='unselectable'>$ </span>find . -name *.css -exec \ s3cmd put -P --mime-type=text/css {} s3://www.mslinn.com/{} \;</pre> </div> <p> It should be possible to provide a MIME-type mapping somehow! Linode, if you are reading this, please provide an extension to the S3 functionality, so customers could go beyond AWS's bugs. </p> <!-- endregion --> <!-- endregion --> <!-- #region The Result --> <h2 id="result">The Result</h2> <p> Success! The same content can be fetched with 3 different URLs: </p> <ul> <li> <a href='https://https://linode.mslinn.com' target='_blank' rel="nofollow"><code>https://linode.mslinn.com</code></a> uses the free wildcard certificate that was created above. </li> <li> <a href='http://linode.mslinn.com.website-us-east-1.linodeobjects.com' target='_blank' rel="nofollow"><code>http://linode.mslinn.com.website-us-east-1.linodeobjects.com</code></a> works without an SSL certificate. </li> <li> <a href='https://linode.mslinn.com.website-us-east-1.linodeobjects.com' target='_blank' rel="nofollow"><code>https://linode.mslinn.com.website-us-east-1.linodeobjects.com</code></a> uses a certificate from <code>us-east-1.linodeobjects.com</code>, which shows as an error, as it should. The error would be prevented if the certificate was a wildcard certificate issued by <code>linodeobjects.com</code> instead. </li> </ul> <p> Qualys / SSL Labs rated the site on Linode Storage with the Let&rsquo;s Encrypt SSL certificate: and gave it the same rating as when the site was hosted on AWS S3 / CloudFront. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/linode/sslReportLinode.svg" type="image/svg"> <!---<source srcset="/blog/images/linode/sslReportLinode.avif" type="image/avif">--> <source srcset="/blog/images/linode/sslReportLinode.webp" type="image/webp"> <source srcset="/blog/images/linode/sslReportLinode.apng" type="image/apng"> <source srcset="/blog/images/linode/sslReportLinode.png" type="image/png"> <source srcset="/blog/images/linode/sslReportLinode.jpg" type="image/jpeg"> <source srcset="/blog/images/linode/sslReportLinode.jpeg" type="image/jpeg"> <source srcset="/blog/images/linode/sslReportLinode.jfif" type="image/jpeg"> <source srcset="/blog/images/linode/sslReportLinode.pjpeg" type="image/jpeg"> <source srcset="/blog/images/linode/sslReportLinode.pjp" type="image/jpeg"> <source srcset="/blog/images/linode/sslReportLinode.gif" type="image/gif"> <source srcset="/blog/images/linode/sslReportLinode.tif" type="image/tiff"> <source srcset="/blog/images/linode/sslReportLinode.tiff" type="image/tiff"> <source srcset="/blog/images/linode/sslReportLinode.bmp" type="image/bmp"> <source srcset="/blog/images/linode/sslReportLinode.ico" type="image/x-icon"> <source srcset="/blog/images/linode/sslReportLinode.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/linode/sslReportLinode.png" style='width: 100%; ' /> </picture> </div> <p> BTW, you can test the date range of a certificate with this incantation: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0b362d060ae5'><button class='copyBtn' data-clipboard-target='#id0b362d060ae5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl https://linode.mslinn.com -vI --stderr - | grep 'date:' <span class='unselectable'>* start date: Nov 25 17:46:07 2022 GMT * expire date: Feb 23 17:46:06 2023 GMT </span></pre> </div> <p> The site feels quite responsive. <span style='font-size: 3em;;'>&#x1F601;</span> </p> <!-- endregion --> Considering Microsoft Azure for Static Websites 2022-06-20T00:00:00-04:00 https://mslinn.github.io/blog/2022/06/20/azure <p> After realizing that <a href='/blog/2022/06/02/azure-security.html'>AWS does not integrate security with real-time billing</a>, and being presented with a huge bill after my account was hijacked, I decided to see if Microsoft Azure offered me better financial security. </p> <p> In particular, I am looking for two things: </p> <ol> <li> <b>A more secure authentication mechanism</b> &ndash; If AWS credentials are compromised, they can be used from anywhere in the world. </li> <li> <b>Restricting scope and scale of authorized services</b> &ndash; With AWS, if an IAM user has a role that allows EC2 instances to be launched, any number of any size EC2 instances can be launched. There is no way to cap the number or type of EC2 instances, and AWS real-time billing is not integrated with the authentication mechanism. Furthermore, AWS has a peculiar set of busy work tasks for victims of account hijacking that appears to be targeted more towards increasing support revenue than address root issues. This means that the bad guys have no limit to the financial damage they can incur on their victims. </li> </ol> <p class="alert rounded shadow"> For PaaS vendors such as AWS, Azure, Digital Ocean, Cloudflare, ScaleWay, etc: &ldquo;pay-as-you-go&rdquo; is shorthand for &ldquo;there is nothing you can do to limit your financial liability&rdquo;. </p> <p> I set out to discover if Azure also suffers from these same problems. </p> <h2 id="policies_roles">Azure Roles and Policies</h2> <div class='quote'> <h2>Policies</h2> <p> <a href='https://docs.microsoft.com/en-us/azure/governance/policy/overview' target='_blank' rel="nofollow">Azure policies</a> can be used to define the desired behavior for your organization's <a href='https://docs.microsoft.com/en-us/azure/virtual-machines/windows/policy' target='_blank' rel="nofollow">Windows VMs</a> and <a href='https://docs.microsoft.com/en-us/azure/virtual-machines/linux/policy' target='_blank' rel="nofollow">Linux VMs</a>. By using policies, an organization can enforce various conventions and rules throughout the enterprise. Enforcement of the desired behavior can help mitigate risk while contributing to the success of the organization. </p> <h3>Azure role-based access control</h3> <p> Using <a href='https://docs.microsoft.com/en-us/azure/role-based-access-control/overview' target='_blank' rel="nofollow">Azure role-based access control (Azure RBAC)</a>, you can segregate duties within your team and grant only the amount of access to users on your VM that they need to perform their jobs. Instead of giving everybody unrestricted permissions on the VM, you can allow only certain actions. You can configure access control for the VM in the <a href='https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal' target='_blank' rel="nofollow">Azure portal</a>, using the <a href='https://docs.microsoft.com/en-us/cli/azure/role' target='_blank' rel="nofollow">Azure CLI</a>, or <a href='https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-powershell' target='_blank' rel="nofollow">Azure PowerShell</a>. </p> <span class='quoteAttribution'> &nbsp;&ndash; From <a href='https://docs.microsoft.com/en-us/azure/virtual-machines/security-policy' rel='nofollow' target='_blank'>Azure Virtual Machines Documentation</a></span> </div> <p> Ekran Systems publishes a <a href='https://www.ekransystem.com/en/blog/rbac-vs-abac' target='_blank' rel="nofollow">good description</a> of various flavors of RBAC, and the more complex ABAC. Okta, Inc. also publishes a <a href='https://www.okta.com/identity-101/role-based-access-control-vs-attribute-based-access-control/' target='_blank' rel="nofollow">good overview of RBAC and ABAC</a>. I have no connection with either company. </p> <p> I soon discovered the following passage in the Azure documentation, which appeared to exactly match the second item in the wish list at the top of this article: </p> <div class='quote'> Azure RBAC focuses on managing user <a href='https://docs.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations' target='_blank' rel="nofollow">actions</a> at different scopes. If control of an action is required, then Azure RBAC is the correct tool to use. Even if an individual has access to perform an action, if the result is a non-compliant resource, Azure Policy still blocks the create or update. <br><br> The combination of Azure RBAC and Azure Policy provides full scope control in Azure. <span class='quoteAttribution'> &nbsp;&ndash; From <a href='https://docs.microsoft.com/en-us/azure/governance/policy/overview#azure-policy-and-azure-rbac' rel='nofollow' target='_blank'>Azure documentation</a></span> </div> <p class="alert rounded shadow"> <a href='https://docs.aws.amazon.com/prescriptive-guidance/latest/saas-multitenant-api-access-authorization/access-control-types.html' target='_blank' rel="nofollow">AWS RBAC</a> does not provide functionality equivalent to that provided by Microsoft Azure RBAC plus Azure Policy, even when combined with other AWS functionality. </p> <h2 id="rbacAgain">No Limits On Other Azure Services</h2> <p> The financial limitations that Azure allows you to impose are only available for virtual machines. Users are subject to unlimited financial liability from other Azure services. It seems I have <strike>two</strike>three choices: </p> <ol> <li>Figure out how to set up RBAC and accept that other services might run up a huge bill.</li> <li>Look for a fixed-price hosting service</li> <li>Give <a href="#spendLimit">Azure Spending Limit</a> a try</li> </ol> <p> Before I sign off today, I encountered a blocking issue with Azure that I'd like to mention. </p> <h2 id="ad">Active Directory?</h2> <p> I want to update my website by running a command-line command that runs something analogous to <a href='https://linux.die.net/man/1/rsync' target='_blank' rel="nofollow"><code>rsync</code></a>. I spent quite some time looking for how this might be done with Azure. </p> <p> Before long, it seemed that Active Directory was the best way forward, however setting it up is daunting. Apparently a <a href='https://docs.microsoft.com/en-us/azure/active-directory/develop/app-objects-and-service-principals' target='_blank' rel="nofollow">service principal</a> is better suited for scripts that run on-premises, like my home office. I got the impression that I need to register an app; unsure what app might be required for hosting a web site on Azure Blob Storage with Azure CDN. </p> <p> These Active Directory docs look like a lot of abstract, generalized information that is way more complex than I need. I found a <a href='https://docs.microsoft.com/en-us/azure/storage/blobs/authorize-access-azure-active-directory' target='_blank' rel="nofollow">streamlined article</a> for my use case. Not sure what this means: &ldquo;Only storage accounts created with the Azure Resource Manager deployment model support Azure AD authorization.&rdquo; This streamlined document is not very streamlined. For me, this issue is a significant barrier to adoption. </p> <h2 id="spendLimit">Update: Azure Spending Limit</h2> <p> I just bumped into an article entitled <a href='https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/spending-limit' target='_blank' rel="nofollow">Azure spending limit</a>. Perhaps this might be useful. I tried to find out more from Microsoft, but at first they said I would have to subscribe to an expensive support plan before anyone would speak to me. That is not how I wish to proceed. Later, I opened a ticket inquiring about Azure customers facing unlimited financial liability. </p> <h2>Update 2022-06-29</h2> <p> The ticket was escalated twice, then Microsoft sent me the following official response (emphasis is mine): </p> <div class='quote'> Correct, at this moment that feature of setting limits does not exist, <b class="bg_yellow">Azure is not able to safeguard customers from unlimited financial liability</b>. <br><br> We are a support team that we provide help with current account or billing issues, in addition to that, I would like to mention that Azure products are constantly being updated, also Azure is constantly improving to offer better services to our customers. For this reason, I would like to invite you to leave your feedback in the Azure feedback forum (<a href='https://feedback.azure.com/d365community/forum/79b1327d-d925-ec11-b6e6-000d3a4f06a4' target='_blank' rel="nofollow">General Feedback · Community (azure.com)</a>), so your feedback is forwarded through the right channel and to the Microsoft internal resources that work on it. </div> <p> I responded by indicating that I would close my Azure account. </p> <p class="alert rounded shadow"> While the practice of imposing unlimited financial liability on customers is common today for PaaS vendors, it is unnecessary and predatory. If in the future I need to use a product or service that incurs unlimited financial liability for a client, I will require the client to accept the liability. </p> Creating and Renewing Letsencrypt Wildcard SSL Certificates 2022-06-15T00:00:00-04:00 https://mslinn.github.io/blog/2022/06/15/certbot <!-- #region intro --> <p> The process of making wildcard SSL certificates can be hard to get right at first. However, I use wildcard SSL certificates for most of my internet domains, so it is important to me. This article distills the essence of what I have found to be important when creating these certificates. </p> <p class="alert rounded shadow"> Update &ndash; 9 months after writing this article, I wrote about a better way to generate wildcard SSL certificates: <a href='/blog/2023/03/02/lego.html'>Wildcard SSL Certificates for Letsencrypt with Lego</a>. </p> <!-- endregion --> <!-- #region cert --> <div class='imgWrapper imgFlex right quartersize' style=' '> <a href='https://letsencrypt.org/' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/ddns/letsencrypt-logo.svg" type="image/svg"> <!---<source srcset="/blog/images/ddns/letsencrypt-logo.avif" type="image/avif">--> <source srcset="/blog/images/ddns/letsencrypt-logo.webp" type="image/webp"> <source srcset="/blog/images/ddns/letsencrypt-logo.apng" type="image/apng"> <source srcset="/blog/images/ddns/letsencrypt-logo.png" type="image/png"> <source srcset="/blog/images/ddns/letsencrypt-logo.jpg" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.jpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.jfif" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.pjpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.pjp" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.gif" type="image/gif"> <source srcset="/blog/images/ddns/letsencrypt-logo.tif" type="image/tiff"> <source srcset="/blog/images/ddns/letsencrypt-logo.tiff" type="image/tiff"> <source srcset="/blog/images/ddns/letsencrypt-logo.bmp" type="image/bmp"> <source srcset="/blog/images/ddns/letsencrypt-logo.ico" type="image/x-icon"> <source srcset="/blog/images/ddns/letsencrypt-logo.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/ddns/letsencrypt-logo.png" style='width: 100%; ' /> </picture> </a> </div> <h2 id="cert">Making a Free Wildcard SSL Certificate</h2> <p> <a href='https://certbot.eff.org/' target='_blank' rel="nofollow"><code>Certbot</code></a> powers <a href='https://www.eff.org/' target='_blank' rel="nofollow">EFF</a>&rsquo;s <a href='https://letsencrypt.org/' target='_blank' rel="nofollow">Letsencrypt</a> capability. Source code is on <a href='https://github.com/certbot/certbot' target='_blank' rel="nofollow">GitHub</a>. </p> <p> Install <code>certbot</code> on Debian distros such as Ubuntu as follows: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable clear' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id84dc7d59bbe8'><button class='copyBtn' data-clipboard-target='#id84dc7d59bbe8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install certbot <span class='unselectable'>Preparing to unpack .../06-python3-zope.event_4.4-3_all.deb ... Unpacking python3-zope.event (4.4-3) ... Selecting previously unselected package python3-zope.component. Preparing to unpack .../07-python3-zope.component_4.3.0-3_all.deb ... Unpacking python3-zope.component (4.3.0-3) ... Selecting previously unselected package python3-certbot. Preparing to unpack .../08-python3-certbot_1.21.0-1build1_all.deb ... Unpacking python3-certbot (1.21.0-1build1) ... Selecting previously unselected package python3-icu. Preparing to unpack .../09-python3-icu_2.8.1-0ubuntu2_amd64.deb ... Unpacking python3-icu (2.8.1-0ubuntu2) ... Selecting previously unselected package certbot. Preparing to unpack .../10-certbot_1.21.0-1build1_all.deb ... Unpacking certbot (1.21.0-1build1) ... Setting up python3-configargparse (1.5.3-1) ... Setting up python3-requests-toolbelt (0.9.1-1) ... Setting up python3-parsedatetime (2.6-2) ... Setting up python3-icu (2.8.1-0ubuntu2) ... Setting up python3-zope.event (4.4-3) ... Setting up python3-zope.hookable (5.1.0-1build1) ... Setting up python3-josepy (1.10.0-1) ... Setting up python3-zope.component (4.3.0-3) ... Setting up python3-acme (1.21.0-1) ... Setting up python3-certbot (1.21.0-1build1) ... Setting up certbot (1.21.0-1build1) ... Created symlink /etc/systemd/system/timers.target.wants/certbot.timer → /lib/systemd/system/certbot.timer. Processing triggers for man-db (2.10.2-1) ... Scanning processes... Scanning processor microcode... Scanning linux images... Failed to retrieve available kernel versions. Failed to check for processor microcode upgrades. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. </span></pre> </div> <!-- endregion --> <p> <code>certbot</code> has 2 subcommands of interest: <code>certonly</code> (used when creating a certificate for the first time), and <code>renew</code> (used when updating a pre-existing certificate). The creation process requires the user to do things before it can complete, so it can only be run interactively. </p> <p class="alert shadow rounded"> The files and directories creating by the process of creating a new SSL certificate should not be deleted. Contrary to what the Letsencrypt <code>certbot</code> documentation says, I have found that these files and directories allow the renewal process to proceed without requiring user interaction, so it can be scripted. </p> <!-- endregion --> <!-- #region help --> <h2 id="help"><span class='code'>Certbot</span> Help Messages</h2> <p> Here is the help message for the <code>certbot certonly</code> subcommand: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3b303de1943c'><button class='copyBtn' data-clipboard-target='#id3b303de1943c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>certbot certonly --help <span class='unselectable'>- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...<br/> Certbot can obtain and install HTTPS/TLS/SSL certificates. By default, it will attempt to use a webserver both for obtaining and installing the certificate. The most common SUBCOMMANDS and flags are:<br/> obtain, install, and renew certificates: (default) run Obtain &amp; install a certificate in your current webserver certonly Obtain or renew a certificate, but do not install it renew Renew all previously obtained certificates that are near expiry enhance Add security enhancements to your existing configuration -d DOMAINS Comma-separated list of domains to obtain a certificate for<br/> (the certbot apache plugin is not installed) --standalone Run a standalone webserver for authentication (the certbot nginx plugin is not installed) --webroot Place files in a server&#39;s webroot folder for authentication --manual Obtain certificates interactively, or using shell script hooks<br/> -n Run non-interactively --test-cert Obtain a test certificate from a staging server --dry-run Test &quot;renew&quot; or &quot;certonly&quot; without saving any certificates to disk<br/> manage certificates: certificates Display information about certificates you have from Certbot revoke Revoke a certificate (supply --cert-name or --cert-path) delete Delete a certificate (supply --cert-name)<br/> manage your account: register Create an ACME account unregister Deactivate an ACME account update_account Update an ACME account --agree-tos Agree to the ACME server&#39;s Subscriber Agreement -m EMAIL Email address for important account notifications<br/> More detailed help:<br/> -h, --help [TOPIC] print this message, or detailed help on a topic; the available TOPICS are:<br/> all, automation, commands, paths, security, testing, or any of the subcommands or plugins (certonly, renew, install, register, nginx, apache, standalone, webroot, etc.) -h all print a detailed help page including all topics --version print the version number - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </span></pre> </div> <!-- endregion --> <p> Here is the help message for the <code>certbot certonly</code> subcommand: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9e3eaf937427'><button class='copyBtn' data-clipboard-target='#id9e3eaf937427' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>certbot renew --help <span class='unselectable'>- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -<br/> certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...<br/> Certbot can obtain and install HTTPS/TLS/SSL certificates. By default, it will attempt to use a webserver both for obtaining and installing the certificate. The most common SUBCOMMANDS and flags are:<br/> obtain, install, and renew certificates: (default) run Obtain &amp; install a certificate in your current webserver certonly Obtain or renew a certificate, but do not install it renew Renew all previously obtained certificates that are near expiry enhance Add security enhancements to your existing configuration -d DOMAINS Comma-separated list of domains to obtain a certificate for<br/> (the certbot apache plugin is not installed) --standalone Run a standalone webserver for authentication (the certbot nginx plugin is not installed) --webroot Place files in a server&#39;s webroot folder for authentication --manual Obtain certificates interactively, or using shell script hooks<br/> -n Run non-interactively --test-cert Obtain a test certificate from a staging server --dry-run Test &quot;renew&quot; or &quot;certonly&quot; without saving any certificates to disk<br/> manage certificates: certificates Display information about certificates you have from Certbot revoke Revoke a certificate (supply --cert-name or --cert-path) delete Delete a certificate (supply --cert-name)<br/> manage your account: register Create an ACME account unregister Deactivate an ACME account update_account Update an ACME account show_account Display account details --agree-tos Agree to the ACME server&#39;s Subscriber Agreement -m EMAIL Email address for important account notifications<br/> More detailed help:<br/> -h, --help [TOPIC] print this message, or detailed help on a topic; the available TOPICS are:<br/> all, automation, commands, paths, security, testing, or any of the subcommands or plugins (certonly, renew, install, register, nginx, apache, standalone, webroot, etc.) -h all print a detailed help page including all topics --version print the version number - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </span></pre> </div> <!-- endregion --> <p> Complete documentation for <code>certbot</code> is <a href='https://eff-certbot.readthedocs.io/en/stable/intro.html' target='_blank' rel="nofollow">here</a>. </p> <!-- endregion --> <!-- #region dns --> <h2 id="genericDns">Generic DNS Authentication</h2> <p> I used <code>certbot</code> to create a special file within the website as follows, without using <code>sudo</code>. I want a <a href='https://en.wikipedia.org/wiki/Wildcard_certificate' target='_blank' rel="nofollow">wildcard SSL certificate</a>, which <a href='https://certbot.eff.org/faq#does-let-s-encrypt-issue-wildcard-certificates' target='_blank' rel="nofollow">requires certbot to use DNS authentication</a>. </p> <p> BTW, the following options include two <code>-d</code> options, one for the domain apex (<code>mslinn.com</code>) and one for the subdomains (<code>*.mslinn.com</code>) </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide873c2a468c7'><button class='copyBtn' data-clipboard-target='#ide873c2a468c7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>certbot certonly \ --manual \ --agree-tos \ --preferred-challenges dns-01 \ --rsa-key-size 4096 \ -d mslinn.com -d *.mslinn.com \ --config-dir ~/.certbot/mslinn.com/config \ --logs-dir ~/.certbot/mslinn.com/logs \ --work-dir ~/.certbot/mslinn.com/work <span class='unselectable'>Saving debug log to /home/mslinn/.certbot/mslinn.com/logs/letsencrypt.log Enter email address (used for urgent renewal and security notices) (Enter &#39;c&#39; to cancel): </span> mslinn@mslinn.com <br/><span class='unselectable'>- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Would you be willing, once your first certificate is successfully issued, to share your email address with the Electronic Frontier Foundation, a founding partner of the Let&#39;s Encrypt project and the non-profit organization that develops Certbot? We&#39;d like to send you email about our work encrypting the web, EFF news, campaigns, and ways to support digital freedom. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (Y)es/(N)o: </span>n <span class='unselectable'>Account registered. Requesting a certificate for mslinn.com and *.mslinn.com<br/> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Please deploy a DNS TXT record under the name:<br/> _acme-challenge.mslinn.com.<br/> with the following value:<br/> n6o3qMw5N7qxAzV4uNQePKxJjaw0f-Bo32CybWr-lhE<br/> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Press Enter to Continue (This must be set up in addition to the previous challenges; do not remove, replace, or undo the previous challenge tasks yet. Note that you might be asked to create multiple distinct TXT records with the same name. This is permitted by DNS standards.) Before continuing, verify the TXT record has been deployed. Depending on the DNS provider, this may take some time, from a few seconds to multiple minutes. You can check if it has finished deploying with aid of online tools, such as the Google Admin Toolbox: https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.mslinn.com. Look for one or more bolded line(s) below the line ';ANSWER'. It should show the value(s) you&rsquo;ve just added. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Press Enter to Continue Successfully received certificate. Certificate is saved at: /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/fullchain.pem Key is saved at: /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/privkey.pem This certificate expires on 2022-09-29. These files will be updated when the certificate renews. NEXT STEPS: - This certificate will not be renewed automatically. Autorenewal of --manual certificates requires the use of an authentication hook script (--manual-auth-hook) but one was not provided. To renew this certificate, repeat this same certbot command before the certificate&rsquo;s expiry date. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - If you like Certbot, please consider supporting our work by: * Donating to ISRG / Let&rsquo;s Encrypt: https://letsencrypt.org/donate * Donating to EFF: https://eff.org/donate-le - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </span></pre> </div> <!-- endregion --> <p> I made a bash script (<code>~/.local/bin/sslCert</code>) to create SSL certificates, which performs the same steps as the above, but it works for any site, and can renew certificates as well as create them: </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,sslCert' download='sslCert' title='Click on the file name to download the file'>~/.local/bin/sslCert</a> </div> <pre data-lt-active="false" class="pre_tag maxOneScreenHigh copyContainer" id="id81179c404557">#!/bin/bash function help &#123; echo "Creates or updates an wildcard SSL certificate Usage: $(basename $0) DOMAIN Example: $(basename $0) scalacourses.com The relevant files are stored in ~/.certbot/DOMAIN/ Creating a new certificate requires an interactive user session; updating a certificate can be done in a cron job. Currently, the renew verb is capable of either renewing all installed certificates that are due to be renewed or renewing a single certificate specified by its name. If you would like to renew specific certificates by their domains, use the certonly command instead. The renew verb may provide other options for selecting certificates to renew in the future. Ask for help or search for solutions at https://community.letsencrypt.org See the logfile /home/mslinn/.certbot/ancientwarmth.com/logs/letsencrypt.log or re-run Certbot with -v for more details. BTW, you can test the date range of the certificate with this incantation: curl https://domain.com -vI --stderr - | grep 'date:' " exit 1 &#125; if [ -z "$1" ]; then help; fi DOMAIN="$1" if [ -d "$HOME/.certbot/$DOMAIN" ]; then # Renew existing certificate CMD=certonly # renew OPTIONS=--force-renew else # Make new certificate (interactively) CMD=certonly unset OPTIONS fi certbot "$CMD" $OPTIONS \ --agree-tos \ --config-dir "$HOME/.certbot/$DOMAIN/config" \ -d "$DOMAIN" -d "*.$DOMAIN" \ --email mslinn@mslinn.com \ --logs-dir "$HOME/.certbot/$DOMAIN/logs" \ --manual \ --preferred-challenges dns-01 \ --rsa-key-size 4096 \ --work-dir "$HOME/.certbot/$DOMAIN/work" </pre> <p> Here is the help message for the script: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id382c86b87a52'><button class='copyBtn' data-clipboard-target='#id382c86b87a52' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sslCert <span class='unselectable'>Creates or updates an wildcard SSL certificate Usage: sslCert DOMAIN Example: sslCert scalacourses.com The relevant files are stored in ~/.certbot/DOMAIN/ Creating a new certificate requires an interactive user session; updating a certificate can be done in a cron job. </span></pre> </div> <!-- endregion --> <p> The following <code>crontab</code> entry causes the script to run every 60 days. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>crontab</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddeb47bc3801f'><button class='copyBtn' data-clipboard-target='#iddeb47bc3801f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>0 0 1 */2 * .local/bin/sslCert mslinn.com</pre> </div> <p> The next time the script was run, output looked like: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb4ad8da92da2'><button class='copyBtn' data-clipboard-target='#idb4ad8da92da2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sslCert mslinn.com <span class='unselectable'>Saving debug log to /home/mslinn/.certbot/mslinn.com/logs/letsencrypt.log Renewing an existing certificate for mslinn.com and *.mslinn.com<br/> Successfully received certificate. Certificate is saved at: /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/fullchain.pem Key is saved at: /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/privkey.pem This certificate expires on 2022-12-13. These files will be updated when the certificate renews.<br/> NEXT STEPS: - This certificate will not be renewed automatically. Autorenewal of --manual certificates requires the use of an authentication hook script (--manual-auth-hook) but one was not provided. To renew this certificate, repeat this same certbot command before the certificate&#39;s expiry date.<br/> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - If you like Certbot, please consider supporting our work by: * Donating to ISRG / Let&#39;s Encrypt: https://letsencrypt.org/donate * Donating to EFF: https://eff.org/donate-le - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </span></pre> </div> <!-- endregion --> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <!-- endregion --> <!-- #region checking with With A Web Browser --> <h2 id="chrome">Checking the Certificate With A Web Browser</h2> <p> Microsoft Windows caches SSL certificates, which can prevent your web browser from detecting newly updated SSL certificates. To clear the Windows SSL cache: </p> <ol> <li>Press the <kbd>Windows</kbd> key, type <code>Internet Options</code>, and press <kbd>Enter</kbd>.</li> <li>Select the <b>Content</b> tab.</li> <li>Click the <kbd>Clear SSL state</kbd> button.</li> <li>Click the <kbd>OK</kbd> button.</li> </ol> <!-- endregion --> Considering Cloudflare R2 for Static Websites 2022-06-09T00:00:00-04:00 https://mslinn.github.io/blog/2022/06/09/cloudflare-r2 <p> After deciding to <a href='/blog/2022/05/26/aws-hijacking.html'>close my AWS account</a>, I started looking for alternatives to host my static websites. Without getting into my selection criteria, my short list of possible hosting companies was <a href='https://azure.microsoft.com/' target='_blank' rel="nofollow">Microsoft Azure</a>, <a href='https://www.cloudflare.com' target='_blank' rel="nofollow">Cloudflare</a>, <a href='https://www.digitalocean.com/' target='_blank' rel="nofollow">Digital Ocean</a>, <a href='https://www.linode.com' target='_blank' rel="nofollow">Linode</a>, and <a href='https://www.netlify.com/' target='_blank' rel="nofollow">Netlify</a>. Many other options exist. </p> <p> Cloudflare has a free tier that has no time limit. It includes an S3 work-alike called R2, SSL and a built-in world-wide CDN that works automatically. 250 GB of storage and 1 TB/month of transfer are provided at no charge, forever. There are also no ingress or egress charges. <a href='https://www.fool.com/investing/2022/05/02/this-tumbling-cloud-stock-is-still-way-too-expensi/' target='_blank' rel="nofollow">Cloudflare&rsquo;s edge network</a> now spans 275 cities around the world, with nearly all Internet users within 50 milliseconds of a Cloudflare server. </p> <p> This article documents my experience with Cloudflare. If you don't care about details, and want to know my verdict on Cloudflare R2, <a href='#verdict'>skip to the end</a>. </p> <p class="alert rounded shadow"> For PaaS vendors such as AWS, Azure, Digital Ocean, Cloudflare, ScaleWay, etc.: &ldquo;pay-as-you-go&rdquo; is shorthand for &ldquo;there is nothing you can do to limit your financial liability&rdquo;. </p> <h2 id="history">This is what I did</h2> <p> After creating an account, I followed the directions at <a href='https://developers.cloudflare.com/r2/get-started/' target='_blank' rel="nofollow">R2 get started guide</a>. </p> <h2 id="wrangler">Wrangler</h2> <p> The directions told me to set up <a href='https://github.com/cloudflare/wrangler2' target='_blank' rel="nofollow">Wrangler v2</a>, a command-line interface for transferring files to R2. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6fcfa6df8142'><button class='copyBtn' data-clipboard-target='#id6fcfa6df8142' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>npm install -g wrangler <span class='unselectable'>npm WARN config global `--global`, `--local` are deprecated. Use `--location=global` instead. npm WARN deprecated rollup-plugin-inject@3.0.2: This package has been deprecated and is no longer maintained. Please use @rollup/plugin-inject.<br/> added 56 packages, and audited 57 packages in 8s<br/> 10 high severity vulnerabilities<br/> To address issues that do not require attention, run: npm audit fix<br/> To address all issues, run: npm audit fix --force<br/> Run `npm audit` for details. </span></pre> </div> <p> The above messages do not inspire confidence. Wrangler is written using Node. I believe that Node has more security issues than any other computer language, particularly in <a href='https://turingpoint.de/en/blog/node-package-manager-security-everything-about-npm-package-security/' target='_blank' rel="nofollow">package management</a>. <a href='https://cheatsheetseries.owasp.org/cheatsheets/NPM_Security_Cheat_Sheet.html' target='_blank' rel="nofollow">The OWasp recommendations</a> do not address the <a href='https://news.ycombinator.com/item?id=29245080' target='_blank' rel="nofollow">fundamental security vulnerabilities</a> in the Node package management infrastructure. </p> <h2 id="rclone">RClone</h2> <p> I looked for alternatives to Wrangler and found <a href='https://rclone.org/docs/' target='_blank' rel="nofollow">RClone</a>. RClone is a command-line program to manage files on cloud storage. It has <a href='https://rclone.org/docs/#subcommands' target='_blank' rel="nofollow">many subcommands</a>, including two types of sync. Although the RClone documentation does not mention Cloudflare, the <a href='https://developers.cloudflare.com/r2/examples/rclone/' target='_blank' rel="nofollow">Cloudflare docs described how to set up RClone</a>. </p> <h2 id="limits">Limits</h2> <p> <a href='https://developers.cloudflare.com/workers/platform/limits/' target='_blank' rel="nofollow">Platform limits</a> are important. Not only are the technical limits important for defining inputs and outputs, users should be particularly interested in spend limits, so they are not subject to unlimited financial liability. </p> <div class='quote'> <div class='quoteText clearfix'> Workers Paid plan is separate from any other Cloudflare plan (Free, Professional, Business) you may have. If you are an Enterprise customer, reach out to your account team to confirm pricing details. <br><br> Only requests that hit a Worker will count against your limits and your bill. </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://developers.cloudflare.com/workers/platform/pricing/#fine-print' rel='nofollow' target='_blank'>Cloudflare Workers Pricing Fine Print</a></div> </div> <h2 id="setup">Setup</h2> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/cloudflare/bucket0.svg" type="image/svg"> <!---<source srcset="/blog/images/cloudflare/bucket0.avif" type="image/avif">--> <source srcset="/blog/images/cloudflare/bucket0.webp" type="image/webp"> <source srcset="/blog/images/cloudflare/bucket0.apng" type="image/apng"> <source srcset="/blog/images/cloudflare/bucket0.png" type="image/png"> <source srcset="/blog/images/cloudflare/bucket0.jpg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/bucket0.jpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/bucket0.jfif" type="image/jpeg"> <source srcset="/blog/images/cloudflare/bucket0.pjpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/bucket0.pjp" type="image/jpeg"> <source srcset="/blog/images/cloudflare/bucket0.gif" type="image/gif"> <source srcset="/blog/images/cloudflare/bucket0.tif" type="image/tiff"> <source srcset="/blog/images/cloudflare/bucket0.tiff" type="image/tiff"> <source srcset="/blog/images/cloudflare/bucket0.bmp" type="image/bmp"> <source srcset="/blog/images/cloudflare/bucket0.ico" type="image/x-icon"> <source srcset="/blog/images/cloudflare/bucket0.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/cloudflare/bucket0.png" style='width: 100%; ' /> </picture> </div> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/cloudflare/analytics0.svg" type="image/svg"> <!---<source srcset="/blog/images/cloudflare/analytics0.avif" type="image/avif">--> <source srcset="/blog/images/cloudflare/analytics0.webp" type="image/webp"> <source srcset="/blog/images/cloudflare/analytics0.apng" type="image/apng"> <source srcset="/blog/images/cloudflare/analytics0.png" type="image/png"> <source srcset="/blog/images/cloudflare/analytics0.jpg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/analytics0.jpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/analytics0.jfif" type="image/jpeg"> <source srcset="/blog/images/cloudflare/analytics0.pjpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/analytics0.pjp" type="image/jpeg"> <source srcset="/blog/images/cloudflare/analytics0.gif" type="image/gif"> <source srcset="/blog/images/cloudflare/analytics0.tif" type="image/tiff"> <source srcset="/blog/images/cloudflare/analytics0.tiff" type="image/tiff"> <source srcset="/blog/images/cloudflare/analytics0.bmp" type="image/bmp"> <source srcset="/blog/images/cloudflare/analytics0.ico" type="image/x-icon"> <source srcset="/blog/images/cloudflare/analytics0.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/cloudflare/analytics0.png" style='width: 100%; ' /> </picture> </div> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/cloudflare/billing0.svg" type="image/svg"> <!---<source srcset="/blog/images/cloudflare/billing0.avif" type="image/avif">--> <source srcset="/blog/images/cloudflare/billing0.webp" type="image/webp"> <source srcset="/blog/images/cloudflare/billing0.apng" type="image/apng"> <source srcset="/blog/images/cloudflare/billing0.png" type="image/png"> <source srcset="/blog/images/cloudflare/billing0.jpg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/billing0.jpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/billing0.jfif" type="image/jpeg"> <source srcset="/blog/images/cloudflare/billing0.pjpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/billing0.pjp" type="image/jpeg"> <source srcset="/blog/images/cloudflare/billing0.gif" type="image/gif"> <source srcset="/blog/images/cloudflare/billing0.tif" type="image/tiff"> <source srcset="/blog/images/cloudflare/billing0.tiff" type="image/tiff"> <source srcset="/blog/images/cloudflare/billing0.bmp" type="image/bmp"> <source srcset="/blog/images/cloudflare/billing0.ico" type="image/x-icon"> <source srcset="/blog/images/cloudflare/billing0.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/cloudflare/billing0.png" style='width: 100%; ' /> </picture> </div> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/cloudflare/workers0.svg" type="image/svg"> <!---<source srcset="/blog/images/cloudflare/workers0.avif" type="image/avif">--> <source srcset="/blog/images/cloudflare/workers0.webp" type="image/webp"> <source srcset="/blog/images/cloudflare/workers0.apng" type="image/apng"> <source srcset="/blog/images/cloudflare/workers0.png" type="image/png"> <source srcset="/blog/images/cloudflare/workers0.jpg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers0.jpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers0.jfif" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers0.pjpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers0.pjp" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers0.gif" type="image/gif"> <source srcset="/blog/images/cloudflare/workers0.tif" type="image/tiff"> <source srcset="/blog/images/cloudflare/workers0.tiff" type="image/tiff"> <source srcset="/blog/images/cloudflare/workers0.bmp" type="image/bmp"> <source srcset="/blog/images/cloudflare/workers0.ico" type="image/x-icon"> <source srcset="/blog/images/cloudflare/workers0.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/cloudflare/workers0.png" style='width: 100%; ' /> </picture> </div> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/cloudflare/workers1.svg" type="image/svg"> <!---<source srcset="/blog/images/cloudflare/workers1.avif" type="image/avif">--> <source srcset="/blog/images/cloudflare/workers1.webp" type="image/webp"> <source srcset="/blog/images/cloudflare/workers1.apng" type="image/apng"> <source srcset="/blog/images/cloudflare/workers1.png" type="image/png"> <source srcset="/blog/images/cloudflare/workers1.jpg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers1.jpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers1.jfif" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers1.pjpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers1.pjp" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers1.gif" type="image/gif"> <source srcset="/blog/images/cloudflare/workers1.tif" type="image/tiff"> <source srcset="/blog/images/cloudflare/workers1.tiff" type="image/tiff"> <source srcset="/blog/images/cloudflare/workers1.bmp" type="image/bmp"> <source srcset="/blog/images/cloudflare/workers1.ico" type="image/x-icon"> <source srcset="/blog/images/cloudflare/workers1.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/cloudflare/workers1.png" style='width: 100%; ' /> </picture> </div> <h2 id="cache">Caching</h2> <p> <a href='https://dash.cloudflare.com/de87d0e0cbc74c70e8feb87e9671cdd0/mslinn.com/caching/configuration' target='_blank' rel="nofollow">Cache control</a>. </p> <h2 id="pages">Pages</h2> <div class='quote'> <div class='quoteText clearfix'>> Cloudflare Pages is a JAMstack platform for frontend developers to collaborate and deploy websites. </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://pages.cloudflare.com' rel='nofollow' target='_blank'>pages.cloudflare.com</a></div> </div> <p> Cloudflare Pages supports <a href='https://developers.cloudflare.com/pages/framework-guides/deploy-a-jekyll-site/' target='_blank' rel="nofollow">Jekyll sites</a>. However, it looks like Cloudflare Pages builds the site in the cloud. While this might be a useful mechanism for many, my Jekyll builds need to access my local machine, and use my <a href="/jekyll/3000-jekyll-plugins.html">Jekyll plugins</a>. </p> <p> <a href='https://developers.cloudflare.com/workers/platform/sites/start-from-existing' target='_blank' rel="nofollow">Cloudflare Workers Sites</a> suits my use case. The <a href='https://developers.cloudflare.com/workers/platform/sites/start-from-existing' target='_blank' rel="nofollow">Start From Existing</a> documentation looks appropriate, except it is written for using Wrangler, which I view as a security threat. </p> <h2 id="verdict">The Verdict: No to Cloudflare</h2> <p class="alert rounded shadow"> CloudFlare does not offer a spend limit for accounts on paid plans. This is unacceptable. I tried to remove my credit card, but found I could not. I then deleted my user account, and saw: </p> <p class="warning shadow rounded"> It could take up to 12 months to delete your information completely. </p> <p> There is no way to frame this as an example of how Cloudflare is looking out for the best interests of their customers. </p> Upgrading PostgreSQL Ubuntu 2022-05-28T00:00:00-04:00 https://mslinn.github.io/blog/2022/05/28/psql-upgrade <!-- #region intro --> <p> This article describes my recommended steps to follow when upgrading a Postgres database forced upon you when upgrading Ubuntu, and the new OS includes a new release of Postgres. </p> <p> I use PostgreSQL as the database for <a href='https://ScalaCourses.com'><code>ScalaCourses.com</code></a>. The site is served from an Ubuntu instance, and there is a backup instance. </p> <p> When upgrading to Ubuntu 22.04, PostgreSQL upgraded from v13 to v14; I used <code>pg_upgrade</code>, which required temporarily swapping the ports that Postgres used. </p> <p> However, I used <code>pg_upgradecluster</code> when upgrading to Ubuntu 23.04; and PostgreSQL upgraded from v14 to v15. Unfortunately, the documented procedure to use <code>pg_upgradecluster</code> has no consideration for <code>systemd</code>, and this makes the upgrade slightly more awkward than it might otherwise be. </p> <p> While both approaches have issues, I found that it is easier and faster to upgrade a Postgres installation using <code>pg_upgradecluster</code> than using <code>pg_upgrade</code>. </p> <!-- endregion --> <!-- #region Upgrade Backup System First --> <h2 id="upgradeubuntu">Upgrade Backup System First</h2> <p> One of the benefits of a backup system is that you can work through upgrade problems on the backup, before doing the same with the live instance. </p> <p> <a href='https://gorails.com/guides/upgrading-postgresql-version-on-ubuntu-server' target='_blank' rel="nofollow">This article</a> suggests to upgrade the entire system to Ubuntu 23.04, then upgrade the active Postgres database cluster. The alternative would be to add a new PPA for Postgres, and just update that program before upgrading the entire system; this merely substitutes one set of potential issues for another. The article also demonstrates renaming the extra database cluster that the Postgres upgrade creates, but that just creates cruft since it is not required for anything. </p> <p> I decided to delete the new database cluster instead, since upgrading the old database cluster is all that is required. As you will read in the <a href='#ug'>Upgrade Procedure</a> section, this is also the advice given during the upgrade procedure. </p> <p> I decided to first <a href='https://help.ubuntu.com/community/LunarUpgrades' target='_blank' rel="nofollow">upgrade the Ubuntu OS</a>, then migrate the Postgres database cluster. </p> <!-- endregion --> <!-- #region Upgrade Removed NVidia Support --> <h2 id="nvidia">Upgrade Removed NVidia Support</h2> <p> After running <code>do-release upgrade</code> on the production system, the system worked fine, except the console had no graphics. I fixed it by automatically installing all video drivers: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id95d5313bdc64'><button class='copyBtn' data-clipboard-target='#id95d5313bdc64' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo ubuntu-drivers autoinstall</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region pg_lsclusters --> <h2 id="pg_lsclusters">Examining the Postgres Databases</h2> <p> <code>pg_lsclusters</code> displays the available PostgreSQL database clusters. Prior to upgrading to Postgres 15, I had never deleted the old versions. They had quietly accumulated over the 11 years that I had been running ScalaCourses: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbac8665338d1'><button class='copyBtn' data-clipboard-target='#idbac8665338d1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pg_lsclusters <span class='unselectable'>Ver Cluster Port Status Owner Data directory Log file 9.5 main 5431 online postgres /var/lib/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log 9.6 main 5433 online postgres /var/lib/postgresql/9.6/main /var/log/postgresql/postgresql-9.6-main.log 10 main 5434 online postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log 12 main 5432 online postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log 13 main 5435 online postgres /var/lib/postgresql/13/main /var/log/postgresql/postgresql-13-main.log 14 main 5436 online postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log </span></pre> </div> <p> Listing the databases in the active cluster showed an unnecessary database: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6ebeb9d021ca'><button class='copyBtn' data-clipboard-target='#id6ebeb9d021ca' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo -iu postgres psql -l <span class='unselectable'>List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges --------------+----------+----------+-------------+-------------+----------------------- bix_dev | postgres | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | postgres | postgres | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | scalacourses | postgres | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | template0 | postgres | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres template1 | postgres | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres (5 rows) </span></pre> </div> <p> I do not care about the <code>bix_dev</code> database, so I decided to delete it. The database that I care about is called <code>scalacourses</code>. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf95d3345dda2'><button class='copyBtn' data-clipboard-target='#idf95d3345dda2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo -iu postgres psql -c 'drop database bix_dev;' <span class='unselectable'>psql (14.7 (Ubuntu 14.7-0ubuntu0.22.10.1)) Type "help" for help. DROP DATABASE </span></pre> </div> <!-- endregion --> <!-- #region Upgrade Procedure --> <h2 id="ug">Upgrade Procedure</h2> <p> Ubuntu is normally upgraded by running <code>do-release-upgrade</code>. Near the end of that process, the following message appeared. </p> <p> At this point <code>postgresql-15</code> and <code>postgresql-client-15</code> had just been installed. The message did not reflect the true state of the system, which caused me some consternation for a moment. </p> <p class="quote shadow rounded"> The PostgreSQL version 14 is obsolete, but the server or client packages are still installed. Please install the latest packages (<code>postgresql-15</code> and <code>postgresql-client-15</code>) and upgrade the existing clusters with pg_upgradecluster (<a href='https://manpages.ubuntu.com/manpages/trusty/man8/pg_upgradecluster.8.html' target='_blank' rel="nofollow">see manpage</a>). <br><br> Please be aware that the installation of postgresql-15 will automatically create a default cluster 15/main. If you want to upgrade the 14/main cluster, you need to remove the already existing 15 cluster (<code>pg_dropcluster --stop 15 main</code>, see manpage for details). <br><br> The old server and client packages are no longer supported. After the existing clusters are upgraded, the <code>postgresql-14</code> and <code>postgresql-client-14</code> packages should be removed. <br><br> Please see /usr/share/doc/postgresql-common/README.Debian.gz for details. </p> <p> Following is <code>/usr/share/doc/postgresql-common/README.Debian.gz</code>, which was referenced by the above message. Most of that file is uninteresting, however two sections are important, and we will look at them. </p> <!-- #region README.Debian.gz --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>/usr/share/doc/postgresql-common/README.Debian.gz</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc8769d30b134'><button class='copyBtn' data-clipboard-target='#idc8769d30b134' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>PostgreSQL for Debian =====================<br/> PostgreSQL is a fully featured object-relational database management system. It supports a large part of the SQL standard and is designed to be extensible by users in many aspects. Its features include ACID transactions, foreign keys, views, sequences, subqueries, triggers, outer joins, multiversion concurrency control, and user-defined types and functions.<br/> Since the on-disk data format of all major PostgreSQL versions (like 9.6, 11, etc.) is incompatible to each other, Debian&#39;s PostgreSQL packaging architecture is designed to maintain clusters of different major versions in parallel.<br/> This postgresql-common package provides the common infrastructure and all frontend programs that users and administrators use. The version specific server and client programs are shipped in postgresql-*-&lt;version&gt; packages.<br/> For a detailed description of the architecture, please see<br/> /usr/share/doc/postgresql-common/README.md.gz<br/> First steps for the impatient ----------------------------- Eventually you will not get around reading at least some parts of the manual, but if you want to get straight into playing SQL, here are the steps to create a database user and a database for the Unix user &#39;joe&#39;:<br/> 1. Install a database server with the major version of your choice (&#39;postgresql-XY&#39;, e. g. &#39;postgresql-11&#39;). Preferably the latest version, which you can get by installing the metapackage &#39;postgresql&#39;. This will automatically create a default cluster &#39;main&#39; with the database superuser &#39;postgres&#39;.<br/> 2. Get a shell for the database superuser &#39;postgres&#39;. If your system has an active root user, use su:<br/> # su -s /bin/bash postgres<br/> If your system uses sudo to get administrative rights, use sudo instead:<br/> joe$ sudo -u postgres bash<br/> 3. In this postgres shell, create a database user with the same name as your Unix login:<br/> $ createuser -DRS joe<br/> For details about the options, see createuser(1).<br/> 4. Create a database &quot;joework&quot; which is owned by &quot;joe&quot;:<br/> $ createdb -O joe joework<br/> For details about the options, see createdb(1).<br/> 5. Exit the postgres shell.<br/> 6. As user joe, you should now be able to connect to your database with<br/> $ psql joework<br/> Cluster management ------------------ For managing clusters, the following commands are provided (each with its own manual page):<br/> pg_createcluster - Create a new cluster or integrate an existing one into the postgresql-common architecture. pg_dropcluster - Completely remove a cluster. pg_ctlcluster - Control the server process of a cluster (start, stop, restart). pg_lsclusters - Show a list of all existing clusters and their status. pg_upgradecluster - Migrate a cluster from one major version to another one. pg_renamecluster - Rename a cluster.<br/> Please note that you can of course also use the upstream tools for creating clusters, such as initdb(1). However, please note that in this case you cannot expect *any* of above pg_* tools to work, since they use different configuration settings (SSL, data directories, etc.) and file locations (e. g. /etc/postgresql/11/main/postgresql.conf). If in doubt, then do *not* use initdb, but only pg_createcluster. Since merely installing postgresql-NN will already set up a default cluster which is ready to work, most people do not need to bother about initdb or pg_createcluster at all.<br/> Port assignment --------------- Please note that the pg_* tools automatically manage the server ports unless you specify them manually. The first cluster which is ever created (by any major version) will run on the default port 5432, and each new cluster will use the next higher free one.<br/> E. g. if you first install &quot;postgresql-11&quot; on a clean system, the default 11/main cluster will run on port 5432. If you then create another 11 cluster, or install the &quot;postgresql-12&quot; package, that new one will run on 5433.<br/> Please use &quot;pg_lsclusters&quot; for displaying the cluster &lt;-&gt; port mapping, and please have a look at the pg_createcluster manpage (the --port option) for details.<br/> Default clusters and upgrading ------------------------------ When installing a postgresql-NN package from scratch, a default cluster &#39;main&#39; will automatically be created. This operation is equivalent to doing &#39;pg_createcluster NN main --start&#39;.<br/> Due to this default cluster, an immediate attempt to upgrade an earlier &#39;main&#39; cluster to a new version will fail and you need to remove the newer default cluster first. E. g., if you have postgresql-9.6 installed and want to upgrade to 11, you first install postgresql-11:<br/> apt-get install postgresql-11<br/> Then drop the default 11 cluster that was just created:<br/> pg_dropcluster 11 main --stop<br/> And then upgrade the 9.6 cluster to the latest installed version (e. g. 11):<br/> pg_upgradecluster 9.6 main<br/> SSL --- The PostgreSQL server packages support SSL, which provides encrypted and authenticated network communication. SSL should be used if you have an untrusted network between a database server and a client and these exchange security sensitive data like passwords or confidential database contents.<br/> When a cluster is created with pg_createcluster, SSL support will automatically be enabled. postgresql-common makes use of the &#39;snakeoil&#39; SSL certificate that is generated by the ssl-cert package, so that SSL works out of the box (ssl_cert_file, ssl_key_file). In addition, if /etc/postgresql-common/root.crt exists, it will be used as CA certificate file (ssl_ca_file).<br/> /etc/postgresql-common/root.crt is a dummy file by default, so that client-side authentication is not performed. To enable it, you should add some root certificates to it. A reasonable choice is to just symlink the file to /etc/ssl/certs/ssl-cert-snakeoil.pem; in this case, client certificates need to be signed by the snakeoil certificate, which might be desirable in many cases. See<br/> /usr/share/doc/postgresql-doc-11/html/ssl-tcp.html<br/> for details (in package postgresql-doc).<br/> Further documentation --------------------- All commands shipped by postgresql-common have detailed manpages. See postgresql-common(7) for the documentation of the database client program wrapping, and user_clusters(5) and postgresqlrc(5) for the cluster configuration.<br/> The documentation of the database server and client functions, SQL commands, modules, etc. documented is shipped in the per-version packages postgresql-doc-&lt;version&gt;.</pre> </div> <!-- endregion --> <p> The <b>Cluster management</b> section in the above file provides some background information for our purposes, so I repeat it here: </p> <!-- #region Cluster management --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Cluster management</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7e02e3cc688e'><button class='copyBtn' data-clipboard-target='#id7e02e3cc688e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>For managing clusters, the following commands are provided (each with its own manual page): pg_createcluster - Create a new cluster or integrate an existing one into the postgresql-common architecture. pg_dropcluster - Completely remove a cluster. pg_ctlcluster - Control the server process of a cluster (start, stop, restart). pg_lsclusters - Show a list of all existing clusters and their status. pg_upgradecluster - Migrate a cluster from one major version to another one. pg_renamecluster - Rename a cluster. Please note that you can of course also use the upstream tools for creating clusters, such as initdb(1). However, please note that in this case you cannot expect *any* of above pg_* tools to work, since they use different configuration settings (SSL, data directories, etc.) and file locations (e. g. /etc/postgresql/11/main/postgresql.conf). If in doubt, then do *not* use initdb, but only pg_createcluster. Since merely installing postgresql-NN will already set up a default cluster which is ready to work, most people do not need to bother about initdb or pg_createcluster at all.</pre> </div> <!-- endregion --> <p> The <b>Default clusters and upgrading</b> section is more important, however it contained errors. I fixed the errors in the following instructions. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Default clusters and upgrading</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc9865dd5cace'><button class='copyBtn' data-clipboard-target='#idc9865dd5cace' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>When installing a postgresql-NN package from scratch, a default cluster 'main' will automatically be created. This operation is equivalent to doing 'pg_createcluster NN main --start'. Due to this default cluster, an immediate attempt to upgrade an earlier 'main' cluster to a new version will fail and you need to remove the newer default cluster first. E. g., if you have postgresql-9.6 installed and want to upgrade to 11, you first install postgresql-11: yes | sudo apt install postgresql-11 Then drop the default 11 cluster that was just created: sudo -iu postgres pg_dropcluster 11 main --stop And then upgrade the 9.6 cluster to the latest installed version (e. g. 11): sudo -iu postgres pg_upgradecluster 9.6 main</pre> </div> <!-- endregion --> <p> <a href='https://freedesktop.org/wiki/Software/systemd/' target='_blank' rel="nofollow">Systemd</a> can significantly affect the upgrade process, and unfortunately it is completely ignored by the above instructions. I muddled through, as you will see. </p> <!-- endregion --> <!-- #region transcript --> <h2 id="trans">Transcript</h2> <p> The following is a transcript of what I actually did: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfb9f7fc462bd'><button class='copyBtn' data-clipboard-target='#idfb9f7fc462bd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo -iu postgres pg_dropcluster --stop 15 main <span class='unselectable'>Warning: stopping the cluster using pg_ctlcluster will mark the systemd unit as failed. Consider using systemctl: sudo systemctl stop postgresql@15-main Warning: systemd was not informed about the removed cluster yet. Operations like "service postgresql start" might fail. To fix, run: sudo systemctl daemon-reload </span> <span class='unselectable'>$ </span>sudo -iu postgres pg_upgradecluster 14 main <span class='unselectable'>Stopping old cluster... Warning: stopping the cluster using pg_ctlcluster will mark the systemd unit as failed. Consider using systemctl: sudo systemctl stop postgresql@14-main Restarting old cluster with restricted connections... Notice: extra pg_ctl/postgres options given, bypassing systemctl for start operation Creating new PostgreSQL cluster 15/main ... /usr/lib/postgresql/15/bin/initdb -D /var/lib/postgresql/15/main --auth-local peer --auth-host scram-sha-256 --no-instructions --encoding UTF8 --lc-collate en_CA.UTF-8 --lc-ctype en_CA.UTF-8 The files belonging to this database system will be owned by user "postgres". This user must also own the server process. The database cluster will be initialized with locale "en_CA.UTF-8". The default text search configuration will be set to "english". Data page checksums are disabled. fixing permissions on existing directory /var/lib/postgresql/15/main ... ok creating subdirectories ... ok selecting dynamic shared memory implementation ... posix selecting default max_connections ... 100 selecting default shared_buffers ... 128MB selecting default time zone ... America/Toronto creating configuration files ... ok running bootstrap script ... ok performing post-bootstrap initialization ... ok syncing data to disk ... ok Warning: systemd does not know about the new cluster yet. Operations like "service postgresql start" will not handle it. To fix, run: sudo systemctl daemon-reload Copying old configuration files... Copying old start.conf... Copying old pg_ctl.conf... Starting new cluster... Notice: extra pg_ctl/postgres options given, bypassing systemctl for start operation Roles, databases, schemas, ACLs... set_config ------------ (1 row) set_config ------------ (1 row) set_config ------------ (1 row) set_config ------------ (1 row) Fixing hardcoded library paths for stored procedures... Upgrading database template1... Analyzing database template1... Fixing hardcoded library paths for stored procedures... Upgrading database postgres... Analyzing database postgres... Fixing hardcoded library paths for stored procedures... Upgrading database scalacourses... Analyzing database scalacourses... Stopping target cluster... Stopping old cluster... Disabling automatic startup of old cluster... Starting upgraded cluster on port 5432... Warning: the cluster will not be running as a systemd service. Consider using systemctl: sudo systemctl start postgresql@15-main Success. Please check that the upgraded cluster works. If it does, you can remove the old cluster with pg_dropcluster 14 main Ver Cluster Port Status Owner Data directory Log file 14 main 5433 down postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log Ver Cluster Port Status Owner Data directory Log file 15 main 5432 online postgres /var/lib/postgresql/15/main /var/log/postgresql/postgresql-15-main.log </span></pre> </div> <!-- endregion --> <p> When I attempted to following the above instructions for systemd, errors appeared: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide4a744226502'><button class='copyBtn' data-clipboard-target='#ide4a744226502' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl daemon-reload <span class='unselectable'>$ </span>sudo systemctl start postgresql@15-main <span class='unselectable'>Job for postgresql@15-main.service failed because the service did not take the steps required by its unit configuration. See "systemctl status postgresql@15-main.service" and "journalctl -xeu postgresql@15-main.service" for details. </span></pre> </div> <!-- endregion --> <p> Great, an obscure failure message. <a href='https://askubuntu.com/questions/1311254/postgres-12-on-ubuntu-20-04-replication-checkpoint-has-wrong-magic-539122744-in' target='_blank' rel="nofollow">Others have fussed with this problem.</a> I expected that rebooting would be much faster than trying to fix it, and that did prove to be the case: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide1157b8dfbdb'><button class='copyBtn' data-clipboard-target='#ide1157b8dfbdb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo reboot</pre> </div> <!-- endregion --> <p> After rebooting, the upgraded Postgres cluster worked fine: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4a7c9742f2e0'><button class='copyBtn' data-clipboard-target='#id4a7c9742f2e0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pg_lsclusters <span class='unselectable'>Ver Cluster Port Status Owner Data directory Log file 14 main 5433 down postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log 15 main 5432 online postgres /var/lib/postgresql/15/main /var/log/postgresql/postgresql-15-main.log </span> <span class='unselectable'>$ </span>sudo -iu postgres psql -l <span class='unselectable'>List of databases Name | Owner | Encoding | Collate | Ctype | ICU Locale | Locale Provider | Access privileges --------------+--------------+----------+-------------+-------------+------------+-----------------+----------------------- postgres | postgres | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | | libc | scalacourses | scalacourses | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | | libc | template0 | postgres | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | | libc | =c/postgres + | | | | | | | postgres=CTc/postgres template1 | postgres | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | | libc | =c/postgres + | | | | | | | postgres=CTc/postgres (4 rows) </span></pre> </div> <!-- endregion --> <p> Here is how I deleted the older database clusters: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id84ae3f2865e8'><button class='copyBtn' data-clipboard-target='#id84ae3f2865e8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>for X in 9.5 9.6 10 12 13; do sudo -iu postgres pg_dropcluster --stop $X main end</pre> </div> <!-- endregion --> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <p> All done! </p> <!-- endregion --> Montréal International vs. Bill 96 2022-05-27T00:00:00-04:00 https://mslinn.github.io/blog/2022/05/27/mi-bill96 <p> Last night I attended the 25th Annual General Meeting of <a href='https://www.montrealinternational.com/en/' target='_blank' rel="nofollow">Montréal International</a>, an energetic organization dedicated to attracting foreign business to establish a presence in the greater Montréal region. My key takeaways were: </p> <ol> <li> I spoke with about 20 Montréal International employees; all of them were bilingual or trilingual, intelligent, motivated and well-informed. Their talent and commitment is palpable. Clearly, the hiring process is successful. </li> <li> The event was marred by long-winded, self-congratulatory speeches by senior management, and former senior managers who were invited to a panel discussion. This lack of discipline caused the event to significantly run overtime. </li> <li> The proceedings were conducted entirely in French. For an organization that calls itself &ldquo;Montréal International&rdquo;, this is unforgivable. </li> <li> I asked every employee I spoke with if <a href='https://qcgn.ca/bill-96/' target='_blank' rel="nofollow">Quebec&rsquo;s new Bill 96</a> impacted their goals. All of them expressed grave concern, and said that their mandate to attract foreign business had become extremely difficult as a result. However, not one mention was made by anyone on stage about Bill 96. </li> </ol> <p> Montréal International&rsquo;s mandate is severely impacted by Quebec's Bill 96. If any organization has an intrinsic mandate to demonstrate public leadership towards repealing or modifying Bill 96 so it respects basic human rights, it would be Montréal International. Instead, Montréal International&rsquo;s leadership, which consists entirely of old white French-speaking alpha males, is complicit in their silence, and by their actions. </p> Limit Your Financial Vulnerability From AWS Account Hijacking 2022-05-26T00:00:00-04:00 https://mslinn.github.io/blog/2022/05/26/aws-hijacking <p class="alert shadow rounded"> I am a free agent, independent and critical. <a href='/softwareexpert/index.html'>You can buy my time as a software expert</a>, but not my opinion. Just because Amazon / AWS hired me in the past to help them against a patent troll does not mean they own me. Quite the contrary. </p> <p> In the time you take a quick shower, your AWS account can be highjacked and tens of thousands of dollars in fees can be incurred. EC2 is the service that provides the greatest financial risk. You can take steps to limit your liability, and this article shows you how. </p> <h2 id="backlash">Out-of-Control Cloud Costs</h2> <p> Out-of-control costs are causing cloud customers to reduce or eliminate their cloud use. <a href='https://techcrunch.com/2023/03/20/the-cloud-backlash-has-begun-why-big-data-is-pulling-compute-back-on-premises/' target='_blank' rel="nofollow">The cloud backlash has begun: Why big data is pulling compute back on premises</a> &ndash; Thomas Robinson, published by TechCrunch 2023-03-20. </p> <p> <a href='https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47e0' target='_blank' rel="nofollow">Why we&rsquo;re leaving the cloud</a>, by David Heinemeier Hansson, creator of Ruby on Rails and CTO of 37signals. Mr. Hansson also wrote <a href='https://world.hey.com/dhh/we-stand-to-save-7m-over-five-years-from-our-cloud-exit-53996caa' target='_blank' rel="nofollow">We stand to save $7m over five years from our cloud exit</a>. </p> <h2 id="nolimits">AWS IAM Users and Roles Have No Budget Limitations</h2> <div class="pullQuote"> AWS customers are exposed to unlimited financial liability </div> <p> I encountered two main issues when attempting to secure against AWS account hijacking: </p> <ol> <li> <b>AWS does not provide a mechanism to guard against launching expensive services.</b> For example, if an IAM user has a role that allows EC2 instances to be launched, then that user can launch an unlimited number of EC2 instances of any size. <br><br> The most expensive Amazon EC2 is currently the <a href='https://aws.amazon.com/ec2/instance-types/p4/' target='_blank' rel="nofollow"><code>p4de.24xlarge</code></a>, which costs $40.96 USD per hour on-demand in the US East (N. Virginia) region. The instance comes with 96 vCPUs, 1152 GiB of RAM and eight 1TB NVMe SSDs, and has eight NVIDIA A100 Tensor Core GPUs. <br><br> Using stolen credentials that allow EC2 instances to be launched, an attacker using scripts could spin up an armada of <code>p4de.24xlarge</code> instances and incur eye-popping costs in minutes. </li> <li> AWS Budgets provides a convenient way to shut down pre-designated services when a total budgetary amount is exceeded. <b>However, the AWS AMI security model is not integrated into real-time cost monitoring.</b> Thus, there is no way to automatically shut down newly launched service instances that exceed a pre-authorized budgetary amount. </li> </ol> <div class="alert rounded shadow"> <h2 id="vercel"><a href='https://vercel.com/pricing' target='_blank' rel="nofollow">Vercel&rsquo;s Pricing Policy Incorporates Limits</a></h2> <p> <i>We provide customers with tools to observe, control, and alert on their infrastructure spend with Spend Management. You can define a spend amount (e.g. $40) and receive email, web, and SMS notifications as you reach that amount. When reaching 100%, you can optionally automatically pause all projects with a hard limit. Over the past year, we've added features to the platform based on your feedback to help automatically prevent runaway spend, including recursion protection, improved function defaults, hard spend limits, Attack Challenge Mode, and more.</i> </p> <p> AWS, Azure, Google App Engine, IBM Cloud and Linode all need a pricing policy like Vercel&rsquo;s. </p> </div> <h2 id="orElse">Shared Security Responsibility</h2> <p> The <a href='https://aws.amazon.com/agreement/' target='_blank' rel="nofollow">AWS Customer Agreement</a> and <a href='https://aws.amazon.com/compliance/shared-responsibility-model/' target='_blank' rel="nofollow">Shared Responsibility Model</a> describe how AWS shares responsibility for security with users. If you follow the AWS security recommendations, described next, you will not be held accountable for extra charges if your AWS account is hijacked. </p> <p> The AWS Security Blog published a nice overview, entitled <a href='https://aws.amazon.com/blogs/security/getting-started-follow-security-best-practices-as-you-configure-your-aws-resources/' target='_blank' rel="nofollow">Getting Started: Follow Security Best Practices as You Configure Your AWS Resources</a>. Some of the same information is also provided in <a href='https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html' target='_blank' rel="nofollow">Security best practices in IAM</a>. </p> <p class="alert rounded shadow"> Even though the AWS instructions do not prevent bad guys taking over your account, follow them anyway so you won't be held liable for the costs incurred if your AWS account is hijacked. </p> <h2 id="recommendations">AWS Security Recommendations</h2> <p> Security needs to have depth. Any single measure can fail, and given enough scale, all measures will eventually fail. Layer your security measures so that one failure, no matter how grave, will not be fatal. </p> <p> AWS recommends the following. I have highlighted what AWS personnel have described to me as being the quickest and easiest path; this article walks through those steps to the extent that I have been able. </p> <ol> <li> Set up at least two of the following services to monitor cost and usage: <ol> <li> <a href='https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-managing-costs.html' class='bg_yellow>Managing' target='_blank' rel="nofollow"><span your costs with AWS Budgets </span></a> </li> <li> <a href='https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/gs_monitor_estimated_charges_with_cloudwatch.html#gs_creating_billing_alarm' class='bg_yellow>Create' target='_blank' rel="nofollow"><span a billing alarm Using CloudWatch</span></a>. </li> <li> <a href='https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html' target='_blank' rel="nofollow">CloudTrail User Guide</a>. </li> <li> <a href='https://docs.aws.amazon.com/waf/latest/developerguide/what-is-aws-waf.html' target='_blank' rel="nofollow">Web Application Firewall (WAF)</a>. </li> <li> <a href='https://docs.aws.amazon.com/awssupport/latest/user/get-started-with-aws-trusted-advisor.html' target='_blank' rel="nofollow">Trusted Advisor</a>. <span class="bg_yellow">I found that Trusted Advisor was somewhat easier to use than CloudWatch.</span> </li> </ol> For more information about managing your AWS cost and usage, see the <a href='https://docs.aws.amazon.com/cost-management/latest/userguide/what-is-costmanagement.html' target='_blank' rel="nofollow">AWS Cost Management User Guide</a>. </li> <li>Set up at least one of the following security best practices: <ol> <li> <a href='https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa_enable.html' class='bg_yellow>Using' target='_blank' rel="nofollow"><span multi-factor authentication (MFA) in AWS</span></a>. </li> <li> <a href='https://docs.aws.amazon.com/securityhub/index.html' target='_blank' rel="nofollow">AWS Security Hub</a>. </li> <li> <a href='https://docs.aws.amazon.com/guardduty/index.html' target='_blank' rel="nofollow">Amazon GuardDuty</a>. </li> </ol> </ol> <h2 id="easiestFirst">Do The Easiest Things First</h2> <p> I usually prefer to do the easiest things first. <a href='https://www.lifehack.org/articles/productivity/easy-tasks-difficult-tasks-first-which-one-more-productive.html' target='_blank' rel="nofollow">Not everyone agrees.</a> Doing even one highlighted item above provides a significant security improvement, so why wait? </p> <div class='quote'> <div class='quoteText clearfix'> The best is the enemy of the good. </div><div class='quoteAttribution'> &nbsp;&ndash; Voltaire </div> </div> <div class='quote'> <div class='quoteText clearfix'> Better a diamond with a flaw than a pebble without. </div><div class='quoteAttribution'> &nbsp;&ndash; Confucius </div> </div> <div class='quote'> <div class='quoteText clearfix'> Striving to better, oft we mar what&rsquo;s well. </div><div class='quoteAttribution'> &nbsp;&ndash; Shakespeare </div> </div> <h2 id="root">Root Credentials</h2> <p> If a bad guy has your root credentials, they can change budget limits. For greatest security, <a href='https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/security_credentials' target='_blank' rel="nofollow">delete your root credentials</a> by selecting <b>Access Keys (access key ID and secret access key)</b> as shown in the image below. However, if you do so, <a href='https://docs.aws.amazon.com/general/latest/gr/root-vs-iam.html#aws_tasks-that-require-root' target='_blank' rel="nofollow">many things become impossible</a>. </p> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/rootKeys.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/rootKeys.avif" type="image/avif">--> <source srcset="/blog/images/aws/rootKeys.webp" type="image/webp"> <source srcset="/blog/images/aws/rootKeys.apng" type="image/apng"> <source srcset="/blog/images/aws/rootKeys.png" type="image/png"> <source srcset="/blog/images/aws/rootKeys.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/rootKeys.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/rootKeys.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/rootKeys.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/rootKeys.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/rootKeys.gif" type="image/gif"> <source srcset="/blog/images/aws/rootKeys.tif" type="image/tiff"> <source srcset="/blog/images/aws/rootKeys.tiff" type="image/tiff"> <source srcset="/blog/images/aws/rootKeys.bmp" type="image/bmp"> <source srcset="/blog/images/aws/rootKeys.ico" type="image/x-icon"> <source srcset="/blog/images/aws/rootKeys.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/rootKeys.png" style='width: 100%; ' /> </picture> </div> <h3 id="mfa">Enabling MFA</h3> <div class='imgWrapper imgBlock right' style='width: 10em; '> <figure> <a href='https://www.amazon.com/Yubico-YubiKey-NFC-Authentication-USB/dp/B07HBD71HL' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/aws/yubikey.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/yubikey.avif" type="image/avif">--> <source srcset="/blog/images/aws/yubikey.webp" type="image/webp"> <source srcset="/blog/images/aws/yubikey.apng" type="image/apng"> <source srcset="/blog/images/aws/yubikey.png" type="image/png"> <source srcset="/blog/images/aws/yubikey.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/yubikey.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/yubikey.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/yubikey.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/yubikey.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/yubikey.gif" type="image/gif"> <source srcset="/blog/images/aws/yubikey.tif" type="image/tiff"> <source srcset="/blog/images/aws/yubikey.tiff" type="image/tiff"> <source srcset="/blog/images/aws/yubikey.bmp" type="image/bmp"> <source srcset="/blog/images/aws/yubikey.ico" type="image/x-icon"> <source srcset="/blog/images/aws/yubikey.cur" type="image/x-icon"> <img alt='Yubikey NFC' class="imgImg rounded shadow" src="/blog/images/aws/yubikey.png" style='width: 100%; ' title='Yubikey NFC' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://www.amazon.com/Yubico-YubiKey-NFC-Authentication-USB/dp/B07HBD71HL" target='_blank' > Yubikey NFC </a> </figcaption> </figure> </div> <p> The easiest highlighted item above is to enable multifactor authentication (MFA). </p> <p> I use a virtual MFA device, <a href='https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa.html#enable-virt-mfa-for-root' target='_blank' rel="nofollow">enabled for my root account</a> Google Authenticator for <a href='https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2' target='_blank' rel="nofollow">Android</a> and <a href='https://apps.apple.com/us/app/google-authenticator/id388497605' target='_blank' rel="nofollow">iOS</a>. Google Authenticator features a <a href='https://tools.ietf.org/html/rfc6238' target='_blank' rel="nofollow">RFC 6238</a> standards-based TOTP (time-based one-time password) algorithm. </p> <p> I use a <a href='https://www.amazon.com/Yubico-YubiKey-NFC-Authentication-USB/dp/B07HBD71HL' target='_blank' rel="nofollow">YubiKey</a> to provide MFA for the AMI user ID that I use to log into the AWS console for everyday work. </p> <p class="pullQuote"> Only use root credentials when absolutely necessary. </p> <h2 id="trustedAdvisor">Trusted Advisor</h2> <p> After adding MFA to root credentials, I found that using <a href='https://docs.aws.amazon.com/awssupport/latest/user/get-started-with-aws-trusted-advisor.html' target='_blank' rel="nofollow">AWS Trusted Advisor</a> was the next easiest thing to try to make my AWS account secure. </p> <p> Once you visit the <a href='https://console.aws.amazon.com/trustedadvisor/home' target='_blank' rel="nofollow">Trusted Advisor Console</a>, and agree to enable Trusted Advisor, <a href='https://docs.aws.amazon.com/awssupport/latest/user/security-trusted-advisor.html' target='_blank' rel="nofollow">IAM permissions are added to the AWS IAM user</a> that you are logged in as. </p> <p> Usage seems straightforward; however, I found the information for securing S3 buckets puzzling at first: </p> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/trustedAdvisor1.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/trustedAdvisor1.avif" type="image/avif">--> <source srcset="/blog/images/aws/trustedAdvisor1.webp" type="image/webp"> <source srcset="/blog/images/aws/trustedAdvisor1.apng" type="image/apng"> <source srcset="/blog/images/aws/trustedAdvisor1.png" type="image/png"> <source srcset="/blog/images/aws/trustedAdvisor1.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor1.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor1.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor1.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor1.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor1.gif" type="image/gif"> <source srcset="/blog/images/aws/trustedAdvisor1.tif" type="image/tiff"> <source srcset="/blog/images/aws/trustedAdvisor1.tiff" type="image/tiff"> <source srcset="/blog/images/aws/trustedAdvisor1.bmp" type="image/bmp"> <source srcset="/blog/images/aws/trustedAdvisor1.ico" type="image/x-icon"> <source srcset="/blog/images/aws/trustedAdvisor1.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/trustedAdvisor1.png" style='width: 100%; ' /> </picture> </div> <p> The column labeled <b>ACL Allows List</b> means that buckets flagged with <b>Yes</b> have an ACL that allows objects to be listed. AWS recommends that S3 buckets not allow objects to be listed by the public. To correct this: </p> <ol> <li>Click on the bucket name.</li> <li>Click on the <b>Permissions</b> tab in the page that opens next.</li> <li>Scroll down to <b>Access Control List</b>.</li> <li>Click on the <kbd>Edit</kbd> button.</li> <li>Disable the items marked in red, as shown below.</li> </ol> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/trustedAdvisor2.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/trustedAdvisor2.avif" type="image/avif">--> <source srcset="/blog/images/aws/trustedAdvisor2.webp" type="image/webp"> <source srcset="/blog/images/aws/trustedAdvisor2.apng" type="image/apng"> <source srcset="/blog/images/aws/trustedAdvisor2.png" type="image/png"> <source srcset="/blog/images/aws/trustedAdvisor2.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor2.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor2.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor2.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor2.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor2.gif" type="image/gif"> <source srcset="/blog/images/aws/trustedAdvisor2.tif" type="image/tiff"> <source srcset="/blog/images/aws/trustedAdvisor2.tiff" type="image/tiff"> <source srcset="/blog/images/aws/trustedAdvisor2.bmp" type="image/bmp"> <source srcset="/blog/images/aws/trustedAdvisor2.ico" type="image/x-icon"> <source srcset="/blog/images/aws/trustedAdvisor2.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/trustedAdvisor2.png" style='width: 100%; ' /> </picture> </div> <p> Referring back to the previous screenshot, the <b>Policy Allows Access</b> column flags all S3 buckets used to serve webpages as insecure. To serve HTML pages and assets stored in an S3 bucket, the permissions must be set to allow objects to be read by everyone. Trusted Advisor flags these buckets as insecure, which does not make sense to me. </p> <p> Perhaps there is a better security policy for when webpages are served via CloudFront, but as is often the case with AWS documentation, it is difficult to piece together how to optimally configure S3 permissions with CloudFront options. </p> <h3 id="evolution">Best Practices Evolve With Technology</h3> <p> The AWS documentation is written as if the products described have always existed in their present state. In the course of researching this article, I discovered that AWS provides revision histories for their services. For example, the revision history of S3 is documented in the <a href='https://docs.aws.amazon.com/AmazonS3/latest/userguide/WhatsNew.html' target='_blank' rel="nofollow">AWS S3 User Guide</a>, and the CloudFront revision history is documented in the <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/WhatsNew.html' target='_blank' rel="nofollow">AWS CloudFront Developer Guide</a>. Revision histories are useful for experienced technologists to review periodically, so they can keep up with best practices as they evolve. </p> <p> I started using AWS S3 to serve websites in 2013, and at that time, CloudFront did not exist yet. As the two services matured, best practices, which were at first unknown, evolved and interactions between them became more complex. In particular, CloudFront <code>http</code>/<code>https</code> promotion, how origins are specified, and S3 permissions, all interact in ways that were not well documented for several years. This was a source of security problems and unwanted downtime for me. </p> <div class='imgWrapper imgFlex right' style='width: 12em; margin-right: 1em;'> <picture class='imgPicture'> <source srcset="/blog/images/aws/roadToHell.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/roadToHell.avif" type="image/avif">--> <source srcset="/blog/images/aws/roadToHell.webp" type="image/webp"> <source srcset="/blog/images/aws/roadToHell.apng" type="image/apng"> <source srcset="/blog/images/aws/roadToHell.png" type="image/png"> <source srcset="/blog/images/aws/roadToHell.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/roadToHell.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/roadToHell.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/roadToHell.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/roadToHell.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/roadToHell.gif" type="image/gif"> <source srcset="/blog/images/aws/roadToHell.tif" type="image/tiff"> <source srcset="/blog/images/aws/roadToHell.tiff" type="image/tiff"> <source srcset="/blog/images/aws/roadToHell.bmp" type="image/bmp"> <source srcset="/blog/images/aws/roadToHell.ico" type="image/x-icon"> <source srcset="/blog/images/aws/roadToHell.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/roadToHell.png" style='width: 100%; ' /> </picture> </div> <p class="alert rounded shadow"> I have 15 buckets that are used to serve web pages. On my <a href='https://us-east-1.console.aws.amazon.com/support/home?region=us-east-1&skipRegion=true#/' target='_blank' rel="nofollow">AWS Support Center console</a>, Trusted Advisor shows that &ldquo;You have a yellow check affecting 15 of 24 resources&rdquo;; however, the Trusted Advisor console displays those items as red triangles with exclamation marks. <br><br> This is confusing, and became frustrating when an AWS security support technician suggested I disregard anything that seemed annoying. For an IT/security person, that advice is the quickest way to hell that I know. <br><br><br> </p> <h3 id="rough">Rough Spots That Need Love</h3> <p> While researching this article I found: </p> <div class='quote'> <div class='quoteText clearfix'> Under <b>Origin Settings</b>, for <b>Origin Domain Name</b>, choose the Amazon S3 bucket that you created previously. <span class="bg_yellow">For <b>S3 bucket access</b>, select <b>Yes, use OAI (bucket can restrict access to only CloudFront)</b></span>. For the <b>Origin access identity</b>, you can choose from the list, or choose <b>Create new OAI</b> (both will work). For <b>Bucket policy</b>, select <b>Yes, update the bucket policy</b>. </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/getting-started-cloudfront-overview.html#getting-started-cloudfront-distribution' rel='nofollow' target='_blank'>Use an Amazon CloudFront distribution to serve a static website</a></div> </div> <p> However, I found conflicting information near the top of this article: </p> <div class='quote'> <div class='quoteText clearfix'> <h2>Important</h2> <p> If you use an Amazon S3 bucket configured as a website endpoint, you must set it up with CloudFront as a custom origin. <span class="bg_yellow">You can&rsquo;t use the origin access identity feature described in this topic.</span> However, you can restrict access to content on a custom origin by setting up custom headers and configuring your origin to require them. For more information, see <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-overview.html#forward-custom-headers-restrict-access' target='_blank' rel="nofollow">Restricting access to files on custom origins</a>. </p> </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3.html' rel='nofollow' target='_blank'>Restricting access to Amazon S3 content by using an origin access identity (OAI)</a></div> </div> <p> As if this is not confusing enough, <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/DownloadDistS3AndCustomOrigins.html#concept_S3Origin_website' target='_blank' rel="nofollow">Using various origins with CloudFront distributions</a> states that in order for Amazon S3 redirects to work, a non-standard format for the origin must be used:<br><br> <code>http://<span style="color: red;">bucket-name</span>.s3-website-<span style="color: red;">region</span>.amazonaws.com</code> <br><br> As I discussed in a <a href='/jekyll/10500-redirects.html'>previous article</a>, AWS S3 buckets support two types of redirects, but there is no indication of which type might or might not work with the non-standard origin format. <br><br> Also, there is no mention of whether <code>https</code> would work or not, or if these disparate instructions even work together. As usual, there is lots of uncertainty about how things actually work on AWS, and for how long before things change. </p> <p> As per standard operating procedure with AWS, one would have to muck about extensively to determine which of the conflicting statements above is generally prevalent, and under what circumstances. I expect that one article is newer than the other, and it likely supercedes the older article somewhat to some degree under as-yet unknown circumstances. If I get lucky, the Trusted Advisor warnings will go away. Trusted Advisor should reference whichever article is currently correct. </p> <h3 id="2022-05-31">Update 2022-05-31</h3> <p> I received the following email from Trusted Advisor. Unfortunately, the contents of the email were different from what I found on the Trusted Advisor console. Perhaps this is because a higher-level, and more expensive, service contract would be required to see those messages. If so, then this is not the way to gain happy customers. The cost of the higher service level would be 4 times more than my total AWS costs at present, clearly not something that I would want to do. </p> <div class="quote"> Dear AWS customer:<br><br> AWS Trusted Advisor currently shows alerts for 3 checks (1 red and 2 yellow) and $0 of potential monthly savings based on your usage.<br><br> Here is a summary of status changes for this week:<br><br> The alert severity of these checks has improved:<br><br> From red to yellow:<br> &nbsp;&nbsp;&nbsp;Amazon S3 Bucket Permissions<br><br> From red to green:<br> &nbsp;&nbsp;&nbsp;VPC<br> &nbsp;&nbsp;&nbsp;Security Groups - Specific Ports Unrestricted<br><br> From yellow to green:<br> &nbsp;&nbsp;&nbsp;VPC Internet Gateways<br><br> The alert status of 5 checks has not changed. For details and recommendations, visit AWS Trusted Advisor.<br><br> Sign in to the Trusted Advisor console to view your check results.<br><br> Best,<br><br> AWS Support<br> Was this report helpful? <a href='https://www.amazon.com/gp/f.html' target='_blank' rel="nofollow">Send us your feedback.</a> </div> <p> To add insult to injury, the final "Send us your feedback" just links to a generic form for AWS sales, where one can <a href='https://writingexplained.org/idiom-dictionary/go-pound-sand' target='_blank' rel="nofollow">pound sand</a>. </p> <p class="alert rounded shadow"> There is abundant evidence to demonstrate that whereas <code>amazon.com</code> is consumer-centric, AWS marches to a very different drummer. Perhaps small businesses are not presently viewed by AWS as a significant growth opportunity. My personal experience is that the current risk/value proposition provided by AWS is no longer attractive for that demographic. </p> <h3 id="2022-05-31">Update 2022-06-03</h3> <p> The credit card I used to pay for my AWS usage had a very high limit. In retrospect that was unwise. </p> <p> AWS took weeks to agree to refund the charge on my credit card, and then they said it would take another week for the reversed charges to appear. This meant that if I only paid for the other charges on that card, the month-to-month credit card interest on the fraudulent AWS charges would have been problematic. </p> <p> Just before the credit card billing period ended, I called Visa and asked what my options were. Apparently I am not the first person who had this problem with AWS. The answer was immediate: report the charges as fraud. OK, that made sense. This meant my card was immediately cancelled, the charges were forgiven by Visa, and a new card with a new number would arrive in about a week. </p> <h2 id="budgets">Managing Costs With AWS Budgets</h2> <p> Short-lived hijacking is the most serious financial threat, and AWS Budgets was seemingly not designed to guard against that. Ticking off this item will enable AWS to absolve you of financial responsibility for being hacked, but you should not expect that it will do much to slow down an attacker. </p> <p> AWS Budgets are easy to set up in a passive manner, but they are clumsy to work with and not effective for preventing new services from being launched that would exceed the budget. The cost limiting functionality is not automatic, it must be set up manually. AWS documentation does not provide detailed how-to instructions for common scenarios. Tens of thousands of words of documentation must be digested before the full capabilities can be harnessed. </p> <p> Another significant issue is that only three types of actions are supported: </p> <ol> <li> <b>IAM Policy</b> &ndash; I am unclear on how to set this up so the bad guys cannot launch new services, and I am not certain this is possible. </li> <li> <a href='https://us-east-1.console.aws.amazon.com/organizations/v2/home/policies/service-control-policy' target='_blank' rel="nofollow"><b>Service Control Policy</b></a> &ndash; Hopefully this could prevent new services from being launched that would exceed the budget; however, my head exploded when I researched this. <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/serviceControlPolicies1.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/serviceControlPolicies1.avif" type="image/avif">--> <source srcset="/blog/images/aws/serviceControlPolicies1.webp" type="image/webp"> <source srcset="/blog/images/aws/serviceControlPolicies1.apng" type="image/apng"> <source srcset="/blog/images/aws/serviceControlPolicies1.png" type="image/png"> <source srcset="/blog/images/aws/serviceControlPolicies1.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies1.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies1.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies1.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies1.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies1.gif" type="image/gif"> <source srcset="/blog/images/aws/serviceControlPolicies1.tif" type="image/tiff"> <source srcset="/blog/images/aws/serviceControlPolicies1.tiff" type="image/tiff"> <source srcset="/blog/images/aws/serviceControlPolicies1.bmp" type="image/bmp"> <source srcset="/blog/images/aws/serviceControlPolicies1.ico" type="image/x-icon"> <source srcset="/blog/images/aws/serviceControlPolicies1.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/serviceControlPolicies1.png" style='width: 100%; ' /> </picture> </div> Attempting to use this option to prevent new, expensive services from being launched would seem to introduce lots of complexity. Perhaps AWS did not consider this use case, or deemed it unimportant. </li> <li> <b>Automate Instances to Stop for EC2 or RDS</b> &ndash; If you run a <a href='https://aws.amazon.com/ec2/instance-types/' target='_blank' rel="nofollow"><code>t3.nano</code></a> instance, for example, bad guys can only incur a certain amount of financial damage. Instead of shutting down the EC2 or RDS instances that I want to run, my main concern is to prevent bad guys from launching expensive new services. This option is therefore unsuitable for my use case. </li> </ol> <p> If AWS Budgets provides an easy way of preventing new services to be launched that would exceed the budget, I could not find it after several hours of reading. </p> <h3 id="passive">Passively Monitoring Budget Spend</h3> <p> Simply follow <a href='https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-create.html' target='_blank' rel="nofollow">the directions</a> and accept the defaults. Daily budgets do not support enabling forecasted alerts, or daily budget planning, so select <b>Monthly Budgets</b>. There is no need to set up SNS alerts or AWS Chatbot alerts, email notification works just as well for most users. </p> <h3 id="passive">Attaching Actions</h3> <p> Following are my notes from my attempt to enable cost limiting. I was unable to attach an action that prevented expensive new services from being started, and it is unclear if resolving the problem will, in fact, provide the desired benefit. Perhaps someone reading these notes will be able to tell me about an approach that might work for a reasonable amount of effort. </p> <p class="alert rounded shadow"> Because AWS will absolve you of financial liability if you create a Budget without any actions, don't stress about the pointless exercise involved in making a Budget that cannot protect you. Just do it and give AWS more time to figure out a better way of preventing account hijacking. <br><br> You can stop reading this article now, unless the details interest you. </p> <p class="alert rounded shadow"> For PaaS vendors such as AWS, Azure, Digital Ocean, Cloudflare, ScaleWay, etc: &ldquo;pay-as-you-go&rdquo; is shorthand for &ldquo;there is nothing you can do to limit your financial liability&rdquo;. </p> <ol> <li>When you get to the step labeled <b>Attach actions - Optional</b>, choose <b>Add Action</b>.</li> <li> You must choose between using an existing IAM role for AWS Budgets to use, or you can create a new IAM role expressly for this purpose. IAM roles are free. I like the idea of a dedicated IAM role, so I clicked on <a href='https://us-east-1.console.aws.amazon.com/iamv2/home?region=us-east-1#/roles' target='_blank' rel="nofollow">manually create an IAM role</a>.<br> <div class='imgWrapper imgFlex inline' style='width: 75%; '> <picture class='imgPicture'> <source srcset="/blog/images/aws/budgetIamRole1.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/budgetIamRole1.avif" type="image/avif">--> <source srcset="/blog/images/aws/budgetIamRole1.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole1.apng" type="image/apng"> <source srcset="/blog/images/aws/budgetIamRole1.png" type="image/png"> <source srcset="/blog/images/aws/budgetIamRole1.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole1.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole1.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole1.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole1.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole1.gif" type="image/gif"> <source srcset="/blog/images/aws/budgetIamRole1.tif" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole1.tiff" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole1.bmp" type="image/bmp"> <source srcset="/blog/images/aws/budgetIamRole1.ico" type="image/x-icon"> <source srcset="/blog/images/aws/budgetIamRole1.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/budgetIamRole1.png" style='width: 100%; margin-left: 1.2em;' /> </picture> </div> </li> <li> A new browser tab opened, and I clicked on the blue <kbd>Create role</kbd> button. </li> <li> Now a confusing page appeared: <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/budgetIamRole2.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/budgetIamRole2.avif" type="image/avif">--> <source srcset="/blog/images/aws/budgetIamRole2.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole2.apng" type="image/apng"> <source srcset="/blog/images/aws/budgetIamRole2.png" type="image/png"> <source srcset="/blog/images/aws/budgetIamRole2.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole2.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole2.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole2.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole2.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole2.gif" type="image/gif"> <source srcset="/blog/images/aws/budgetIamRole2.tif" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole2.tiff" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole2.bmp" type="image/bmp"> <source srcset="/blog/images/aws/budgetIamRole2.ico" type="image/x-icon"> <source srcset="/blog/images/aws/budgetIamRole2.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/budgetIamRole2.png" style='width: 100%; ' /> </picture> </div> </li> <li> I selected the default <b>Trusted entity type</b>, AWS Service, then from the pulldown menu labeled <b>Use cases for other AWS services:</b> I selected <b>Budgets</b>. </li> <li> A new radio button appeared underneath the pull-down, also labeled <b>Budgets</b>, and I clicked on that. This does not win any usability awards. </li> <li> I clicked on the button labeled <kbd>Next</kbd>. </li> <li> <div class='quote'> <div class='quoteText clearfix'> Managed policy name: <code><a href='https://docs.aws.amazon.com/cost-management/latest/userguide/billing-permissions-ref.html#budget-managedIAM-SSM' target='_blank' rel="nofollow"><code>AWSBudgetsActionsRolePolicyForResourceAdministrationWithSSM</code></a></code><br><br> This managed policy is focused on specific actions that AWS Budgets takes on your behalf when completing a specific action. This policy gives AWS Budgets broad permission to control AWS resources. For example, starts and stops Amazon EC2 or Amazon RDS instances by running AWS Systems Manager (SSM) scripts. </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://docs.aws.amazon.com/cost-management/latest/userguide/billing-permissions-ref.html#budget-managedIAM-SSM' rel='nofollow' target='_blank'>Using identity-based policies (IAM policies) for AWS Cost Management</a></div> </div> On the page, I applied <code>AWSBudgetsActionsRolePolicyForResourceAdministrationWithSSM</code>, then I clicked on <kbd>Next</kbd>. <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/budgetIamRole3.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/budgetIamRole3.avif" type="image/avif">--> <source srcset="/blog/images/aws/budgetIamRole3.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole3.apng" type="image/apng"> <source srcset="/blog/images/aws/budgetIamRole3.png" type="image/png"> <source srcset="/blog/images/aws/budgetIamRole3.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole3.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole3.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole3.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole3.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole3.gif" type="image/gif"> <source srcset="/blog/images/aws/budgetIamRole3.tif" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole3.tiff" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole3.bmp" type="image/bmp"> <source srcset="/blog/images/aws/budgetIamRole3.ico" type="image/x-icon"> <source srcset="/blog/images/aws/budgetIamRole3.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/budgetIamRole3.png" style='width: 100%; ' /> </picture> </div> </li> <li> On the final page, I named the IAM role <b>Budget</b> and clicked on <kbd>Create role</kbd>. <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/budgetIamRole4.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/budgetIamRole4.avif" type="image/avif">--> <source srcset="/blog/images/aws/budgetIamRole4.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole4.apng" type="image/apng"> <source srcset="/blog/images/aws/budgetIamRole4.png" type="image/png"> <source srcset="/blog/images/aws/budgetIamRole4.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole4.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole4.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole4.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole4.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole4.gif" type="image/gif"> <source srcset="/blog/images/aws/budgetIamRole4.tif" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole4.tiff" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole4.bmp" type="image/bmp"> <source srcset="/blog/images/aws/budgetIamRole4.ico" type="image/x-icon"> <source srcset="/blog/images/aws/budgetIamRole4.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/budgetIamRole4.png" style='width: 100%; ' /> </picture> </div> </li> <li> Back in the <b>Billing Management Console</b>, I refreshed the list of available IAM roles, then selected the new role called <b>Budget</b>. <div class='imgWrapper imgFlex inline' style='width: 75%; '> <picture class='imgPicture'> <source srcset="/blog/images/aws/budgetIamRole5.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/budgetIamRole5.avif" type="image/avif">--> <source srcset="/blog/images/aws/budgetIamRole5.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole5.apng" type="image/apng"> <source srcset="/blog/images/aws/budgetIamRole5.png" type="image/png"> <source srcset="/blog/images/aws/budgetIamRole5.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole5.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole5.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole5.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole5.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole5.gif" type="image/gif"> <source srcset="/blog/images/aws/budgetIamRole5.tif" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole5.tiff" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole5.bmp" type="image/bmp"> <source srcset="/blog/images/aws/budgetIamRole5.ico" type="image/x-icon"> <source srcset="/blog/images/aws/budgetIamRole5.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/budgetIamRole5.png" style='width: 100%; ' /> </picture> </div> </li> <li> I selected <b>Service Control Policy</b> from the pull-down that appeared next, labeled <b>Which action type should be applied when the budget threshold has been exceeded?</b>. I have no idea how to proceed, or even if going in this direction might provide the desired results. <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/serviceControlPolicies2.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/serviceControlPolicies2.avif" type="image/avif">--> <source srcset="/blog/images/aws/serviceControlPolicies2.webp" type="image/webp"> <source srcset="/blog/images/aws/serviceControlPolicies2.apng" type="image/apng"> <source srcset="/blog/images/aws/serviceControlPolicies2.png" type="image/png"> <source srcset="/blog/images/aws/serviceControlPolicies2.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies2.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies2.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies2.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies2.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies2.gif" type="image/gif"> <source srcset="/blog/images/aws/serviceControlPolicies2.tif" type="image/tiff"> <source srcset="/blog/images/aws/serviceControlPolicies2.tiff" type="image/tiff"> <source srcset="/blog/images/aws/serviceControlPolicies2.bmp" type="image/bmp"> <source srcset="/blog/images/aws/serviceControlPolicies2.ico" type="image/x-icon"> <source srcset="/blog/images/aws/serviceControlPolicies2.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/serviceControlPolicies2.png" style='width: 100%; ' /> </picture> </div> </li> </ol> Microsoft Clarity Lets Me Watch You Click and Scroll 2022-03-31T00:00:00-04:00 https://mslinn.github.io/blog/2022/03/31/clarity <!-- #region intro --> <p> I spent a fair bit of time designing the plugins for this Jekyll-powered website for <a href='https://moz.com/beginners-guide-to-seo' target='_blank' rel="nofollow">SEO</a>. These plugins are published as open source; click on the word Jekyll in the sentence above this paragraph to learn more. You are welcome! </p> <p> I do not use Google Analytics because it slows down page load time dramatically. I do use <a href='https://search.google.com' target='_blank' rel="nofollow">Google Search Console</a>, however, and recently started trying out <a href='https://www.bing.com/webmasters/' target='_blank' rel="nofollow">Microsoft Bing Webmaster Tools</a>. </p> <!-- endregion --> <!-- #region clarity --> <h2 id="clarity">Microsoft Clarity and Hotjar</h2> <p> The day before April Fool's Day in 2022, I stumbled across <a href='https://clarity.microsoft.com/' target='_blank' rel="nofollow">Microsoft Clarity</a>, a free product that I knew nothing about. I was curious to know what benefit it might provide. </p> <p> I learned that one of the things it can do is provide videos of actual user sessions as they interact with the website. Those videos are fantastic. </p> <p> As a writer of a lot of online content, I have spent hours watching you, dear readers. Individually yet anonymously. Watching people read teaches you a lot about what is important to them. As a writer, knowing your audience allows you to be more relevant. BTW, I do not know who or where my readers are, just the country they are in. </p> <!-- endregion --> <!-- #region update 1 --> <h2 id="upd">Update 2023-07-10</h2> <p class="pullQuoteFull"> I no longer need to watch the videos.<br> Microsoft Clarity's AI generates summaries.<br> Incredibly actionable information. </p> <p> Today I noticed the <b>insights</b> tab, visible when viewing an entry for a &lsquo;hit&rsquo;. Seems like Microsoft has employed AI to generate the following impressive summary of the user's behavior. I added punctuation, a light edit and a whimsical pad of notepaper. </p> <div class="notepaper"> <p style="text-align: left; "> Visited URL matches regex: <code>^https://www\.mslinn\.com/softwareexpert/index\.html(\?.*)?$</code><br> <b>Last 7 days<br><br> Session insights</b><br> Some key takeaways from this session are:<br><br> The user was interested in the software expert witness and computer expert services offered by Mike Slinn, as they visited the homepage from Google and spent about four minutes there. <br><br> The user was also curious about the technology expert article series, especially the ones related to Git and version control systems. They visited the article index page multiple times and clicked on several articles, spending about 15 minutes in total on this topic. <br><br> The user was most engaged with the article on libgit2, a library that provides low-level access to Git operations. They spent about one and a half minutes on this article and clicked on a link to git-fame, a tool that shows the contribution statistics of a Git repository. <br><br> The user frequently resized their browser window, which may indicate that they were using a mobile device or adjusting their screen for better readability. The user also switched between tabs or applications often, as indicated by the page hidden and page visible events. This may suggest that they were multitasking or comparing information from different sources. <br><br> 01:47 / 47:31 </p> </div> <!-- endregion --> <!-- #region update 2 --> <h2 id="update2">Update 2023-07-18</h2> <p> Today I stumbled across the <a href='https://chrome.google.com/webstore/detail/microsoft-clarity-live/cjfdbemmaeeohgibnhdhlakiahifjjcf/related' target='_blank' rel="nofollow">Clarity Live</a> plugin for the Google Chrome web browser. There is no such plugin for Firefox, sadly. </p> <div class='quote'> <div class='quoteText clearfix'> View instant heatmaps right on your live site and watch recent session recordings for any page you are on with our extension. <br><br> <ul> <li>GDPR & CCPA ready</li> <li>No sampling</li> <li>Built on open source</li> </ul> </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://clarity.microsoft.com/live-extension' rel='nofollow' target='_blank'>Microsoft Clarity</a></div> </div> <!-- endregion --> <!-- #region update 3 --> <h2 id="update3">Update 2023-10-04</h2> <p> I just stumbled across the <a href='https://learn.microsoft.com/en-us/clarity/setup-and-installation/clarity-api#customize-your-identifiers' target='_blank' rel="nofollow">Clarity Client API</a>. </p> <div class='quote'> <p> You can quickly get started with Clarity without coding but by interacting with the Clarity client API. This API can help you access advanced features as described in this reference. Add the following calls to Clarity APIs to the HTML or JavaScript of your webpage to access these features. </p> <h2>Note</h2> <p> Your Clarity ID serves as your API key. No other client API key is necessary, and there is no cost for using Clarity client APIs. </p> </div> <!-- endregion --> <!-- #region video --> <h2 id="video">Check out this video!</h2> <p> This is one of the first user sessions that Microsoft Clarity recorded for this website. </p> <style>.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }</style><div class='embed-container'> <iframe title="YouTube video player" width="640" height="390" src="//www.youtube.com/embed/N-Tip0Y4GMU" frameborder="0" allowfullscreen></iframe></div> <p> As you can see, Microsoft Clarity lets me watch movies of users clicking and scrolling through my website; spooky yet very informative. </p> <p> Clarity is open source. <a href='https://github.com/microsoft/clarity' target='_blank' rel="nofollow">Here</a> is the GitHub project. </p> <p> I have since learned that <a href='https://www.hotjar.com/' target='_blank' rel="nofollow">Hotjar</a> is similar to Microsoft Clarity. </p> <!-- endregion --> <!-- #region behavior --> <h2 id="behavior">Online Behavior Matches Real-World Behavior</h2> <p> The user in the above video read a bit about my experience as a software expert witness, then straightaway downloaded my resume. They were on the website for just over one minute. Although they did call me, their online behavior showed a lack of urgency, and their &lsquo;real-world&rsquo; behavior mirrored what I saw in the movie. </p> <p> A week later another party visited my site, and spent 80 minutes carefully reading 3 pages, among others. Their &lsquo;real-world&rsquo; behavior also matched their online behavior, in that they exhibited a sense of urgency towards engaging a software expert. </p> <!-- endregion --> <!-- #region downloads --> <h2 id="downloads">Tracking Downloads And Other Behavior</h2> <p> Microsoft Clarity does not consider a download as a click. It does not even notice downloads. For me, downloads are what I care about most. If you never download my resume, you probably are not a candidate for hiring me as a <a href='/softwareexpert/index.html'>software expert</a>. I want to track downloads, not clicks. Clicks are nice, but there is a direct relationship between resume downloads and signing contracts with legal firms who represent new clients. </p> <p> AWS <a href='https://aws.amazon.com/premiumsupport/knowledge-center/view-iam-history/' target='_blank' rel="nofollow">CloudTrail</a> and <a href='https://aws.amazon.com/aws-cost-management/aws-cost-optimization/monitor-track-and-analyze/' target='_blank' rel="nofollow">CloudWatch</a> can provide download details and much more. For example, user IP addresses and geographic location can be captured when a monitored event occurs. </p> <!-- endregion --> <!-- #region surveil --> <h2 id="others">Many Websites Perform Surveillance</h2> <p> Wired Magazine published an article on a similar type of surveillance (keyloggers) on May 11, 2022. I paraphrased two sentences from that article: </p> <div class='quote'> <ul> <li>1.8% of websites studied gathered an EU user's email address without their consent, and a staggering 2.95% logged a US user's email.</li> <li>For US users, 8.4% of sites may have been leaking data to Meta, Facebook&rsquo;s parent company, and 7.4% of sites may be impacted for EU users.</li> </ul> <span class='quoteAttribution'> &nbsp;&ndash; From <a href='https://www.wired.com/story/leaky-forms-keyloggers-meta-tiktok-pixel-study/' rel='nofollow' target='_blank'>Thousands of Popular Websites See What You Type—Before You Hit Submit</a></span> </div> <p> Yours truly does nothing of the sort. </p> <!-- endregion --> <!-- #region next --> <h2 id="next">Looking Ahead</h2> <p> I would very much like to receive a continuous data feed of the above AI-generated per-visit summary, in real time as it becomes available. An email stream would be a quick way to start. A streaming api for data would be a logical next step. </p> <!-- endregion --> Make a Visual Studio Code Extension 2022-03-01T00:00:00-05:00 https://mslinn.github.io/blog/2022/03/01/make-vscode-extension <p> I want to be able to move pages in my Jekyll-powered website without breaking links from the outside. The <a href='https://github.com/jekyll/jekyll-redirect-from' target='_blank' rel="nofollow"><code>jekyll-redirect-from</code></a> Jekyll plugin generates little HTML files that contain <code>http-meta-refresh</code> client-side redirects as desired. I wanted an easy way to inject the names of those redirect pages into the Jekyll front matter. Since I usually author my website using Visual Studio Code, a custom Visual Studio Code extension seems like a good way to create the redirects. </p> <h2 id="requirements">Extension Requirements</h2> <p> All the extension has to do is discover the URL path component of the page currently being edited, and write an entry into the front matter. For example, I have set up Jekyll such that given a page at <code>collections/_posts/2022/2022-02-21-jekyll-debugging.html</code>, it would deploy to <code>/blog/2022/02/21/jekyll-debugging.html</code>. The required front matter would be: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida06988bdbd81'><button class='copyBtn' data-clipboard-target='#ida06988bdbd81' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>redirect_from: - /blog/2022/02/21/jekyll-debugging.html</pre> </div> <p> With that in mind, if I want to move a published article to another location without breaking it, the extension should: </p> <ol> <li>Present a new menu option when a right-click on a file in the sidebar, or a right-click on a file name tab in the editor.</li> <li> When the menu item is selected, obtain the relative path to the file within the project directory (for example: <code>collections/_posts/2022/2022-02-21-jekyll-debugging.html</code>) </li> <li> Convert the relative path to the deployed relative path (for example, <code>/blog/2022/02/21/jekyll-debugging.html</code>) </li> <li> Write the entry into the front matter of the file. For example: <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida64824b1f895'><button class='copyBtn' data-clipboard-target='#ida64824b1f895' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>redirect_from:<br> - /blog/2022/02/21/jekyll-debugging.html</pre> </div> </li> </ol> <h2 id="background">Background</h2> <p> The <a href='https://code.visualstudio.com/api/get-started/your-first-extension' target='_blank' rel="nofollow">Microsoft documentation</a> describes how to write a Visual Studio Code extension. </p> <h2 id="setup">Setup</h2> <p> Visual Studio Code extensions are written in JavaScript (actually, node.js) or TypeScript. Ensure that a version of node.js has been installed: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9d628eb1b793'><button class='copyBtn' data-clipboard-target='#id9d628eb1b793' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>which node <span class='unselectable'>/home/mslinn/.nvm/versions/node/v17.3.1/bin/node </span></pre> </div> <p> Ensure you have <a href='properly'>set up your global package manager</a> for node.js, then type: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4eecad2c664a'><button class='copyBtn' data-clipboard-target='#id4eecad2c664a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>npm install -g <a href='https://github.com/yeoman/yo/' target='_blank' rel="nofollow">yo</a> <a href='https://github.com/microsoft/vscode-generator-code' target='_blank' rel="nofollow">generator-code</a> <span class='unselectable'>&nbsp; npm WARN deprecated har-validator@5.1.5: this library is no longer supported npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142<br> added 872 packages, and audited 873 packages in 57s<br> 15 vulnerabilities (13 moderate, 2 high)<br> To address issues that do not require attention, run: npm audit fix<br> To address all issues (including breaking changes), run: npm audit fix --force<br> Run `npm audit` for details. </span></pre> </div> <p> The <code>yo</code> module is the culprit; lets address its vulnerabilities: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1f9de8e28ad4'><button class='copyBtn' data-clipboard-target='#id1f9de8e28ad4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>npm install -g npm-check-updates <span class='unselectable'>added 270 packages, and audited 271 packages in 9s<br> found 0 vulnerabilities </span></pre> </div> <h2 id="generate">Generating the Skeleton</h2> <p> I decided to use <a href='https://www.typescriptlang.org/' target='_blank' rel="nofollow">TypeScript</a> instead of JavaScript for the extension. TypeScript code converts to JavaScript, however it uses type inference and integrates better with some editors. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8a627e537950'><button class='copyBtn' data-clipboard-target='#id8a627e537950' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mkdir redirect_generator<br> <span class='unselectable'>$ </span>cd redirect_generator/<br> <span class='unselectable'>$ </span>$ yo code <span class='unselectable'>? ========================================================================== We&rsquo;re constantly looking for ways to make yo better! May we anonymously report usage statistics to improve the tool over time? More info: https://github.com/yeoman/insight &amp; http://yeoman.io ========================================================================== </span> No <span class='unselectable'><br> _-----_ ╭──────────────────────────╮ | | │ Welcome to the Visual │ |--(o)--| │ Studio Code Extension │ `---------´ │ generator! │ ( _´U`_ ) ╰──────────────────────────╯ /___A___\ / | ~ | __'.___.'__ ´ ` |° ´ Y ` ? What type of extension do you want to create? </span> New Extension (TypeScript) <span class='unselectable'>? What&apos;s the name of your extension? </span> redirect_generator <span class='unselectable'>? What&apos;s the identifier of your extension? </span> redirect-generator ? What's the description of your extension? Injects the URL of a redirect page into Jekyll front matter <span class='unselectable'>? Initialize a git repository? </span>Yes <span class='unselectable'>? Bundle the source code with webpack? </span>No <span class='unselectable'>? Which package manager to use? </span>npm <span class='unselectable'>Writing in /mnt/f/work/jekyll/redirect_generator/redirect-generator... create redirect-generator/.vscode/extensions.json create redirect-generator/.vscode/launch.json create redirect-generator/.vscode/settings.json create redirect-generator/.vscode/tasks.json create redirect-generator/package.json create redirect-generator/tsconfig.json create redirect-generator/.vscodeignore create redirect-generator/vsc-extension-quickstart.md create redirect-generator/README.md create redirect-generator/CHANGELOG.md create redirect-generator/src/extension.ts create redirect-generator/src/test/runTest.ts create redirect-generator/src/test/suite/extension.test.ts create redirect-generator/src/test/suite/index.ts create redirect-generator/.eslintrc.json<br> Changes to package.json were detected.<br> Running npm install for you to install the required dependencies.<br> added 203 packages, and audited 204 packages in 29s<br> found 0 vulnerabilities<br> Your extension redirect-generator has been created!<br> To start editing with Visual Studio Code, use the following commands:<br> code redirect-generator<br> Open vsc-extension-quickstart.md inside the new extension for further instructions on how to modify, test and publish your extension.<br> For more information, also visit http://code.visualstudio.com and follow us @code.<br><br> ? Do you want to open the new folder with Visual Studio Code? Open with `code`<br> _-----_ ╭───────────────────────╮ | | │ Bye from us! │ |--(o)--| │ Chat soon. │ `---------´ │ Yeoman team │ ( _´U`_ ) │ http://yeoman.io │ /___A___\ /╰───────────────────────╯ | ~ | __'.___.'__ ´ ` |° ´ Y ` </span></pre> </div> <p> The instructions in the web page assume the <a href='https://dictionary.cambridge.org/dictionary/english/happy-path' target='_blank' rel="nofollow">happy path</a> is the only possibility. Instead: </p> <ol> <li>Do not work on the extension in a workspace that has any other projects in it.</li> <li>The online instructions assume that TypeScript was used, not JavaScript, so the file types are assumed to be <code>.ts</code>. </ol> <p> The instructions in <code>vsc-extension-quickstart.md</code> do not make assumptions about TypeScript. </p> <h2 id="debug">Debugging the Extension</h2> <p> I found the procedure awkward at first: </p> <ol> <li>Press <kbd>F5</kbd> to activate the first defined launcher</li> <li> Once the new debug VSCode instance settles down, click in it and type <kbd>CTRL</kbd>-<kbd>Shift</kbd>-<kbd>P</kbd>, then type <code>redirect</code>. </li> <li>Any breakpoints in the original VSCode instance will trigger.</li> <li> Once the breakpoint is cleared, a message saying <b>Add redirecct from redirect_generator!</b> will appear in the debugged instance of VSCode. </li> </ol> <h2 id="anatomy">Extension Anatomy</h2> <p> I moved on to the next part of the tutorial, <a href='https://code.visualstudio.com/api/get-started/extension-anatomy' target='_blank' rel="nofollow">Extension Anatomy</a>. </p> Node.js, NVM, NPM and Yarn 2022-03-01T00:00:00-05:00 https://mslinn.github.io/blog/2022/03/01/node-package-managers <!-- #region intro --> <p> <a href='https://nodejs.dev/learn/a-brief-history-of-nodejs' target='_blank' rel="nofollow"><code>Node.js</code></a>, a JavaScript runtime, is not my favorite programming environment. Originally released 14 years ago by <a href='https://en.wikipedia.org/wiki/Ryan_Dahl' target='_blank' rel="nofollow">Ryan Dahl</a>, a software engineer at Google, it&rsquo;s <a href='https://www.shiftleft.io/blog/node.js-vulnerability-cheatsheet/' target='_blank' rel="nofollow">historical disregard</a> for <a href='https://cheatsheetseries.owasp.org/cheatsheets/Nodejs_Security_Cheat_Sheet.html' target='_blank' rel="nofollow">security</a> and stability is problematic. </p> <p> However, <code>node.js</code> has significant traction, especially amongst Millennial software technologists, and is used by many Ruby and Scala projects for HTML <a href='http://graphics.cs.cmu.edu/courses/15-466-f19/notes/asset-pipelines.html' target='_blank' rel="nofollow">asset pipelines</a>. </p> <p> I put together these notes for installing and maintaining <code>node.js</code> on Ubuntu/WSL using a virtualized version manager. </p> <!-- endregion --> <!-- #region Why Virtualize Node.js --> <h2 id="nvm">Why Virtualize <span class="code">Node.js</span>?</h2> <div class='imgWrapper imgFlex right' style='width: 40%; '> <picture class='imgPicture'> <source srcset="/blog/images/node/pushback.svg" type="image/svg"> <!---<source srcset="/blog/images/node/pushback.avif" type="image/avif">--> <source srcset="/blog/images/node/pushback.webp" type="image/webp"> <source srcset="/blog/images/node/pushback.apng" type="image/apng"> <source srcset="/blog/images/node/pushback.png" type="image/png"> <source srcset="/blog/images/node/pushback.jpg" type="image/jpeg"> <source srcset="/blog/images/node/pushback.jpeg" type="image/jpeg"> <source srcset="/blog/images/node/pushback.jfif" type="image/jpeg"> <source srcset="/blog/images/node/pushback.pjpeg" type="image/jpeg"> <source srcset="/blog/images/node/pushback.pjp" type="image/jpeg"> <source srcset="/blog/images/node/pushback.gif" type="image/gif"> <source srcset="/blog/images/node/pushback.tif" type="image/tiff"> <source srcset="/blog/images/node/pushback.tiff" type="image/tiff"> <source srcset="/blog/images/node/pushback.bmp" type="image/bmp"> <source srcset="/blog/images/node/pushback.ico" type="image/x-icon"> <source srcset="/blog/images/node/pushback.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/node/pushback.png" style='width: 100%; ' /> </picture> </div> <p> It is better to use virtualized user- and project-specific <code>node.js</code> instances, instead of working with a system-wide installation of <code>node.js</code>. This allows you to install and upgrade <code>node.js</code> packages without using supervisor privileges. Also, virtualized instances allows you to work on many different independent <code>node.js</code> projects at the same time, without package version collisions. </p> <p> <code>Docker</code> is over-sold. It adds unnecessary complexity to software projects. Instead of virtualizing the entire software environment, as <code>docker</code> attempts to do, virtualizing the programming environment with <code>node.js nvm</code>, <a href='/blog/2021/04/09/python-venvs.html'>Python <code>venv</code></a>, or <a href='/ruby/1000-ruby-setup.html'>Ruby <code>rbenv</code></a> are much easier and more productive approaches. </p> <p> I think <code>docker</code> has been pushed hard in the media because it is a gateway technology to <a href='https://www.infoworld.com/article/3223434/what-is-paas-platform-as-a-service-a-simpler-way-to-build-software-applications.html' target='_blank' rel="nofollow">PaSS</a>. This is a trend that PaSS vendors like AWS and Azure want to encourage, but <a href='https://www.datacenterdynamics.com/en/news/37signals-spent-more-than-3-million-on-the-cloud-in-2022-for-basecamp-and-hey' target='_blank' rel="nofollow">customers are pushing back</a>. </p> <!-- endregion --> <!-- #region Nvm --> <h2 id="nvm" class="clear"><span class="code">Nvm</span></h2> <p> <code>Nvm</code>, the Node Version Manager, makes it easy to install multiple virtualized <code>node.js</code> instances, and to easily switch between them. <code>Nvm</code> retains a unique set of installed packages for each <code>node.js</code> instance. The <code>node.js</code> version in each instance is distinct. </p> <p> <code>Nvm</code> is installed and updated as follows: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id875e25a3ff39'><button class='copyBtn' data-clipboard-target='#id875e25a3ff39' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash <span class='unselectable'>&nbsp; % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 14926 100 14926 0 0 84806 0 --:--:-- --:--:-- --:--:-- 85291 => nvm is already installed in /home/mslinn/.nvm, trying to update using git => => Compressing and cleaning up git repository => nvm source string already in /home/mslinn/.bashrc => bash_completion source string already in /home/mslinn/.bashrc => Close and reopen your terminal to start using nvm or run the following to use it now: export NVM_DIR="$HOME/.nvm" [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" # This loads nvm bash_completion </span></pre> </div> <p> View the currently installed versions of <code>node.js</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0cdbc02ce4d8'><button class='copyBtn' data-clipboard-target='#id0cdbc02ce4d8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>nvm list <span class='unselectable'>-> system iojs -> N/A (default) node -> stable (-> N/A) (default) unstable -> N/A (default) </span></pre> </div> <!-- #region --> <p> View the very long list of available versions of <code>node.js</code> like this: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idba23d583758f'><button class='copyBtn' data-clipboard-target='#idba23d583758f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>nvm list-remote <span class='unselectable'>&nbsp; v17.7.0 v17.7.1 v17.7.2 v17.8.0 v17.9.0 v17.9.1 v18.0.0 v18.1.0 v18.2.0 v18.3.0 v18.4.0 v18.5.0 v18.6.0 v18.7.0 v18.8.0 v18.9.0 v18.9.1 v18.10.0 v18.11.0 v18.12.0 (LTS: Hydrogen) v18.12.1 (LTS: Hydrogen) v18.13.0 (Latest LTS: Hydrogen) v19.0.0 v19.0.1 v19.1.0 v19.2.0 v19.3.0 v19.4.0 v19.5.0 </span></pre> </div> <!-- endregion --> <h2 id="npm_install_node">Using <span class="code">Nvm</span> to Install <span class="code">Node.js</span></h2> <p> Install the latest release of <code>node.js</code> using <code>nvm</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id435c60154749'><button class='copyBtn' data-clipboard-target='#id435c60154749' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>nvm install <span class="bg_yellow">node</span> <span class='unselectable'>Downloading and installing node v19.5.0... Downloading https://nodejs.org/dist/v19.5.0/node-v19.5.0-linux-x64.tar.xz... ##################################################################### 100.0% Computing checksum with sha256sum Checksums matched! Now using node v19.5.0 (npm v9.3.1) Creating default alias: default -> node (-> v19.5.0) </span></pre> </div> <p> In the above example, <code class="bg_yellow">node</code> is an alias for &ldquo;the latest version of <code>node.js</code>&rdquo;. To install a specific version of <code>node.js</code>, for example 18.13.0: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf2d5a9ee9a75'><button class='copyBtn' data-clipboard-target='#idf2d5a9ee9a75' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>nvm install 18.13.0</pre> </div> <!-- endregion --> <!-- #region Yarn --> <h2 id="yarn"><span class="code">Yarn</span></h2> <!-- #region implicit --> <div class='imgWrapper imgBlock right halfsize' style=' '> <figure> <picture class='imgPicture'> <source srcset="/blog/images/node/needle_and_thread.svg" type="image/svg"> <!---<source srcset="/blog/images/node/needle_and_thread.avif" type="image/avif">--> <source srcset="/blog/images/node/needle_and_thread.webp" type="image/webp"> <source srcset="/blog/images/node/needle_and_thread.apng" type="image/apng"> <source srcset="/blog/images/node/needle_and_thread.png" type="image/png"> <source srcset="/blog/images/node/needle_and_thread.jpg" type="image/jpeg"> <source srcset="/blog/images/node/needle_and_thread.jpeg" type="image/jpeg"> <source srcset="/blog/images/node/needle_and_thread.jfif" type="image/jpeg"> <source srcset="/blog/images/node/needle_and_thread.pjpeg" type="image/jpeg"> <source srcset="/blog/images/node/needle_and_thread.pjp" type="image/jpeg"> <source srcset="/blog/images/node/needle_and_thread.gif" type="image/gif"> <source srcset="/blog/images/node/needle_and_thread.tif" type="image/tiff"> <source srcset="/blog/images/node/needle_and_thread.tiff" type="image/tiff"> <source srcset="/blog/images/node/needle_and_thread.bmp" type="image/bmp"> <source srcset="/blog/images/node/needle_and_thread.ico" type="image/x-icon"> <source srcset="/blog/images/node/needle_and_thread.cur" type="image/x-icon"> <img alt='Needle and thread under an electron microscope' class="imgImg rounded shadow" src="/blog/images/node/needle_and_thread.png" style='width: 100%; ' title='Needle and thread under an electron microscope' /> </picture> <figcaption class='imgFigCaption halfsize'> Needle and thread under an electron microscope </figcaption> </figure> </div> <p> Whether you use <code>nvm</code> as your <code>node.js</code> virtualization mechanism, you also need a <code>node.js</code> package manager to install and maintain project dependencies. </p> <p> <a href='https://yarnpkg.com/' target='_blank' rel="nofollow"><code>Yarn</code></a> supposedly stands for Yet Another Resource Negotiator, and it is a package manager like <code>npm</code>, described <a href="#npm">next</a>. It was developed by Facebook and is now open-source. <code>Yarn</code> was developed to address <code>npm</code>&rsquo;s performance and security issues. </p> <p> The name actually suggests the major advantage <code>yarn</code> has over its predecessor, <code>npm</code>: multi-threading instead of single-threading. </p> <p> Yarn generates a <code>yarn.lock</code> file, which helps easy merges. The merges are predictable. <code>Yarn</code> caches every package it has downloaded, so it never needs to download the same package again. It also does almost everything concurrently to maximize resource utilization. This means faster installs. <code>Yarn</code> guarantees that any installation that works on one system will work exactly the same on another system, unlike <code>npm</code>. </p> <p> The notes for the <a href='https://www.npmjs.com/package/yarn' target='_blank' rel="nofollow"><code>yarn</code> package</a> state that <code>Yarn</code> was inspired by <a href='https://bundler.io/' target='_blank' rel="nofollow">Bundler</a> and <a href='https://doc.rust-lang.org/cargo/' target='_blank' rel="nofollow">Cargo</a>, and is nearly command-line compatible with <code>npm</code>. </p> <p> <code>Yarn</code> introduces the zero-install concept, which means that a project should be able to be used as soon as it is cloned. It uses Plug&rsquo;n&rsquo;Play to resolve dependencies via the <code>yarn</code> cache folder and not from <code>node_modules</code>. The cache folder is by default stored within your project folder, in <code>.yarn/cache</code>. </p> <!-- endregion --> <!-- #region Installing Yarn --> <h3 id="install_yarn">Installing <span class="code">Yarn</span></h3> <p> To install <code>yarn</code> on Ubuntu: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3a49235368d9'><button class='copyBtn' data-clipboard-target='#id3a49235368d9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | \ sudo apt-key add - <span class='unselectable'>$ </span>echo "deb https://dl.yarnpkg.com/debian/ stable main" | \ sudo tee /etc/apt/sources.list.d/yarn.list <span class='unselectable'>$ </span>sudo apt update && sudo apt install yarn</pre> </div> <!-- endregion --> <!-- #region Configuring a Project With Yarn --> <h3 id="yarn_config">Configuring a Project With Yarn</h3> <p> The <a href='https://classic.yarnpkg.com/lang/en/docs/cli/init/' target='_blank' rel="nofollow"><code>yarn init</code></a> command initiates an interactive session that creates a <code>package.json</code> file. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id45f99457df55'><button class='copyBtn' data-clipboard-target='#id45f99457df55' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cd /path/to/my/node/project<br> <span class='unselectable'>$ </span>yarn init <span class='unselectable'>yarn init v1.22.19 question name (node): blah question version (1.0.0): 0.1.0 question description: Just a little song I wrote question entry point (index.js): question repository url: https://github.com/mslinn/blah question author: Mike Slinn question license (MIT): question private: success Saved package.json Done in 48.29s. </span></pre> </div> <!-- endregion --> <p> The above dialog creates <code>package.json</code> in the current directory: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>package.json</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id07548422647c'><button class='copyBtn' data-clipboard-target='#id07548422647c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{ "name": "blah", "version": "0.1.0", "description": "Just a little song I wrote", "main": "index.js", "repository": "https://github.com/mslinn/blah", "author": "Mike Slinn", "license": "MIT" }</pre> </div> <!-- endregion --> <!-- #region Adding a Dependency to a Project --> <h3 id="yarn_config">Adding a Dependency to a Project</h3> <p> Yarn stores dependencies locally. If the proper version is present locally, it is fetched from the disk during a <code>yarn add</code> command, otherwise it is downloaded. This is the general format of the command to add a dependency to a <code>node.js</code> project using <code>yarn</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7bee12ba4641'><button class='copyBtn' data-clipboard-target='#id7bee12ba4641' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yarn add package_name</pre> </div> <p> To add a specific version of a package: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9834cfea8eb5'><button class='copyBtn' data-clipboard-target='#id9834cfea8eb5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yarn add package_name<span class="bg_yellow">@version_number</span></pre> </div> <p> To install a global package, the general formats are: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9adaeb562386'><button class='copyBtn' data-clipboard-target='#id9adaeb562386' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yarn <span class="bg_yellow">global</span> add package_name <span class='unselectable'>$ </span>yarn <span class="bg_yellow">global</span> add package_name@version_number</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Yarn and Git --> <h3 id="git_yarn"><span class="code">Yarn</span> and <span class="code">Git</span></h3> <p> The <a href='https://yarnpkg.com/getting-started/install' target='_blank' rel="nofollow">official Yarn docs</a> say to add the following to <code>.gitignore</code> for git projects that use <code>yarn</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>.gitignore</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide1c70c501701'><button class='copyBtn' data-clipboard-target='#ide1c70c501701' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>.yarn/* !.yarn/cache !.yarn/patches !.yarn/plugins !.yarn/releases !.yarn/sdks !.yarn/versions</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Installing Dependencies --> <h3 id="yarn_install">Installing Dependencies</h3> <p> To install all the dependencies in a <code>node.js</code> project, type <a href='https://yarnpkg.com/getting-started/usage#installing-all-the-dependencies' target='_blank' rel="nofollow"><code>yarn</code> or <code>yarn install</code></a>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcf92f1bd3ce6'><button class='copyBtn' data-clipboard-target='#idcf92f1bd3ce6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cd /path/to/my/project <span class='unselectable'>$ </span>yarn</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Updating Dependencies --> <h3 id="yarn_update">Updating Dependencies</h3> <p> To upgrade all the dependencies in a <code>node.js</code> project, use <a href='https://classic.yarnpkg.com/lang/en/docs/cli/upgrade/' target='_blank' rel="nofollow"><code>yarn upgrade</code></a>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id127884294a8b'><button class='copyBtn' data-clipboard-target='#id127884294a8b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cd /path/to/my/project <span class='unselectable'>$ </span>yarn upgrade</pre> </div> <!-- endregion --> <!-- endregion --> <!-- endregion --> <!-- #region Npm --> <h2 id="npm"><span class="code">Npm</span></h2> <!-- #region implicit --> <p> <code>Npm</code> is the original package manager for the family of JavaScript programming languages, and it is still the default package manager for <code>node.js</code>. This will likely change as <code>yarn</code> matures. <code>Npm</code> helps install libraries, plugins, frameworks and applications. It consists of a command-line client, also called <code>npm</code>, and an online database of public and paid-for private packages called the <code>npm</code> registry. </p> <div class="pullQuote"> If you are using <span class="code">yarn</span> then you do not need to use <span class="code">npm</span>. </div> <p> <code>Npm</code> fetches dependencies from the <code>npm</code> registry for every <code>npm install</code> command. </p> <p> <code>Npm</code> generates a <code>package-lock.json</code> file. The layout of this file is a trade-off between determinism and simplicity. The same <code>node_modules/</code> folder will be generated by every version of <code>npm</code>. Every dependency will have a version number associated with it in the <code>package-lock</code> file. </p> <!-- endregion --> <!-- #region Npm vs. Yarn --> <h3 id="npm_yarn"><span class="code">Npm</span> vs. <span class="code">Yarn</span></h3> <p> When using <code>npm install</code>, dependencies are installed sequentially, one after another. The output logs in the terminal are informative but a bit hard to read. To install the packages with Yarn, run the <code>yarn</code> command. Yarn installs packages in parallel, which is one of the reasons it is quicker than npm. </p> <p> For more information, see <a href='https://www.geeksforgeeks.org/difference-between-npm-and-yarn/' target='_blank' rel="nofollow">Difference between npm and yarn</a>. </p> <!-- endregion --> <!-- #region Install Npm With a Node Version Manager --> <h3 id="npm_install">Install <span class="code">Npm</span> With a Node Version Manager</h3> <p> Follow <a href='https://docs.npmjs.com/downloading-and-installing-node-js-and-npm#using-a-node-version-manager-to-install-nodejs-and-npm' target='_blank' rel="nofollow">these instructions</a>. In summary: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id93526a663e6a'><button class='copyBtn' data-clipboard-target='#id93526a663e6a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mkdir ~/.npm-global<br> <span class='unselectable'>$ </span>npm config set prefix ~/.npm-global<br> <span class='unselectable'>$ </span>cat >> ~/.bashrc &lt;&lt;EOF export PATH="$HOME/.npm-global/bin:$PATH"<br> export NVM_DIR="$HOME/.nvm" [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" 1>&2 # Loads nvm EOF<br> <span class='unselectable'>$ </span>source ~/.bashrc<br> <span class='unselectable'>$ </span>curl -fsSL https://deb.nodesource.com/setup_21.x | \ sudo -E bash - && \ sudo apt-get install -y nodejs</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Install A Global Package --> <h3 id="global_install">Install A Global Package</h3> <p> To install a global package, the command template for <code>npm</code> is: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8d3afbdaccd3'><button class='copyBtn' data-clipboard-target='#id8d3afbdaccd3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>npm install -g package_name[@version_number]</pre> </div> <p> For example: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id220b7845af11'><button class='copyBtn' data-clipboard-target='#id220b7845af11' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>npm install -g broken-link-checker <span class='unselectable'>npm WARN deprecated calmcard@0.1.1: no longer maintained npm WARN deprecated nopter@0.3.0: try optionator npm WARN deprecated uuid@2.0.3: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. npm WARN deprecated urlobj@0.0.11: use universal-url, minurl, relateurl, url-relation added 104 packages, and audited 105 packages in 5s </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- endregion --> Fun With Python Enums 2022-02-10T00:00:00-05:00 https://mslinn.github.io/blog/2022/02/10/python-3.4-enums <!-- #region intro --> <p> This article demonstrates how to define additional properties for <a href='https://docs.python.org/3/library/enum.html' target='_blank' rel="nofollow">Python 3 enums</a>. Defining an additional property in a Python enum can provide a simple way to provide string values. The concept is then expanded to demonstrate composition, an important concept for functional programming. This article concludes with a demonstration of dynamic dispatch in Python, by further extending an enum. </p> <!-- endregion --> <!-- #region Adding Properties to Python Enums --> <h2 id="enum">Adding Properties to Python Enums</h2> <p> Searching for <a href='https://www.google.com/search?q=python+enum+string+value' target='_blank' rel="nofollow"><code>python enum string value</code></a> yields some complex and arcane ways to approach the problem. </p> <p> Below is a short example of a Python enum that demonstrates a simple way to provide lower-case string values for enum constants: a new property, <code>to_s</code>, is defined. This property provides the string representation that is required. You could define other properties and methods to suit the needs of other projects. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cad_enums.py</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5bcfd5dfbf06'><button class='copyBtn' data-clipboard-target='#id5bcfd5dfbf06' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>"""Defines enums"""<br/> from enum import Enum, auto<br/> class EntityType(Enum): """Types of entities""" SITE = auto() GROUP = auto() COURSE = auto() SECTION = auto() LECTURE = auto()<br/> <span class="bg_yellow"> @property def to_s(self) -> str: """:return: lower-case name of this instance""" return self.name.lower()</span></pre> </div> <!-- endregion --> <p> Adding the following to the bottom of the program allows us to demonstrate it: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cad_enums.py (part 2)</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id43219a5b20b1'><button class='copyBtn' data-clipboard-target='#id43219a5b20b1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>if __name__ == "__main__": # Just for demonstration print("Specifying individual values:") print(f" {EntityType.SITE.value}: {EntityType.SITE.to_s}") print(f" {EntityType.GROUP.value}: {EntityType.GROUP.to_s}") print(f" {EntityType.COURSE.value}: {EntityType.COURSE.to_s}") print(f" {EntityType.SECTION.value}: {EntityType.SECTION.to_s}") print(f" {EntityType.LECTURE.value}: {EntityType.LECTURE.to_s}")<br> print("\nIterating through all values:") for entity_type in EntityType: print(f" {entity_type.value}: {entity_type.to_s}")</pre> </div> <!-- endregion --> <p> Running the program produces this output: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id83640e04896e'><button class='copyBtn' data-clipboard-target='#id83640e04896e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cad_enums.py <span class='unselectable'>Specifying individual values: 1: site 2: group 3: course 4: section 5: lecture<br> Iterating through all values: 1: site 2: group 3: course 4: section 5: lecture </span></pre> </div> <!-- endregion --> <div class="right" style="font-size: 3em;">&#128513;</div> <p> Easy! </p> <!-- endregion --> <!-- #region Constructing Enums --> <h2 id="constructor">Constructing Enums</h2> <p> Enum constructors work the same as other Python class constructors. There are several ways to make a new instance of a Python enum. Let's try two ways by using the <a href='https://docs.python.org/3/tutorial/interpreter.html' target='_blank' rel="nofollow">Python interpreter</a>. Throughout this article I've inserted a blank line between Python interpreter prompts for readability. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Python</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id65483c057720'><button class='copyBtn' data-clipboard-target='#id65483c057720' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>python <span class='unselectable'>Python 3.9.7 (default, Sep 10 2021, 14:59:43) [GCC 11.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.<br> >>> </span>from cad_enums import EntityType<br> <span class='unselectable'>>>> </span># Specify the desired enum constant value symbolically <span class='unselectable'>>>> </span><span class="bg_yellow">gtype = EntityType.GROUP</span> <span class='unselectable'>>>> </span>print(gtype) <span class='unselectable'>EntityType.GROUP </span><br> <span class='unselectable'>>>> </span># Specify the desired enum constant value numerically <span class='unselectable'>>>> </span><span class="bg_yellow">stype = EntityType(1)</span> <span class='unselectable'>>>> </span>print(stype) <span class='unselectable'>EntityType.SITE </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Enum Ordering --> <h2 id="ordering">Enum Ordering</h2> <p> A program I am working on needs to obtain the parent <code>EntityType</code>. By 'parent' I mean the <code>EntityType</code> with the next lowest numeric value. For example, the parent of <code>EntityType.GROUP</code> is <code>EntityType.SITE</code>. We can obtain a parent enum by computing its numeric value by adding the following method to the <code>EntityType</code> class definition. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Python</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3514a1a84cae'><button class='copyBtn' data-clipboard-target='#id3514a1a84cae' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>@property def parent(self) -> <span class="bg_yellow">'EntityType'</span>: """:return: entity type of parent; site has no parent""" return EntityType(max(self.value - 1, 1))</pre> </div> <!-- endregion --> <p> The <span class="bg_yellow">return type</span> above is enclosed in quotes (<code>'EntityType'</code>) to keep Python's type checker happy, because this is a <a href='https://www.python.org/dev/peps/pep-0484/#forward-references' target='_blank' rel="nofollow">forward reference</a>. This is a forward reference because the type is referenced before it is fully compiled. </p> <p> The complete enum class definition is now: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cad_enums.py</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfa92ca8697c4'><button class='copyBtn' data-clipboard-target='#idfa92ca8697c4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>"""Defines enums"""<br/> from enum import Enum, auto<br/> class EntityType(Enum): """Types of entities""" SITE = auto() GROUP = auto() COURSE = auto() SECTION = auto() LECTURE = auto()<br/> @property def to_s(self) -> str: """:return: lower-case name of this instance""" return self.name.lower()<br> @property def parent(self) -> 'EntityType': """:return: entity type of parent; site has no parent""" return EntityType(max(self.value - 1, 1))</pre> </div> <!-- endregion --> <p> Lets try out the new <code>parent</code> property in the Python interpreter. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Python</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1ddce15dfca8'><button class='copyBtn' data-clipboard-target='#id1ddce15dfca8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>EntityType.LECTURE.parent <span class='unselectable'>&lt;EntityType.SECTION: 4> </span><br> <span class='unselectable'>>>> </span>EntityType.SECTION.parent <span class='unselectable'>&lt;EntityType.COURSE: 3> </span><br> <span class='unselectable'>>>> </span>EntityType.COURSE.parent <span class='unselectable'>&lt;EntityType.GROUP: 2> </span><br> <span class='unselectable'>>>> </span>EntityType.GROUP.parent <span class='unselectable'>&lt;EntityType.SITE: 1> </span><br> <span class='unselectable'>>>> </span>EntityType.SITE.parent <span class='unselectable'>&lt;EntityType.SITE: 1> </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Enum Composition --> <h2 id="compose">Enum Composition</h2> <p> Like methods and properties in all other Python classes, enum methods and properties compose if they return an instance of the class. Composition is also known as <a href='https://en.wikipedia.org/wiki/Method_chaining' target='_blank' rel="nofollow">method chaining</a>, and also can apply to class properties. Composition is an essential practice of a functional programming style. </p> <p> The <code>parent</code> property returns an instance of the <code>EntityType</code> enum class, so it can be composed with any other property or method of that class, for example the <code>to_s</code> property shown earlier. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Python</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4ec179c113ed'><button class='copyBtn' data-clipboard-target='#id4ec179c113ed' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>EntityType.LECTURE.parent.to_s <span class='unselectable'>'section' </span><br> <span class='unselectable'>>>> </span>EntityType.SECTION.parent.to_s <span class='unselectable'>'course' </span><br> <span class='unselectable'>>>> </span>EntityType.COURSE.parent.to_s <span class='unselectable'>'group' </span><br> <span class='unselectable'>>>> </span>EntityType.GROUP.parent.to_s <span class='unselectable'>'site' </span><br> <span class='unselectable'>>>> </span>EntityType.SITE.parent.to_s <span class='unselectable'>'site' </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Dynamic Dispatch --> <h2 id="dispatch">Dynamic Dispatch</h2> <p> The <a href='https://docs.python.org/3/library/typing.html#callable' target='_blank' rel="nofollow">Python documentation</a> might lead someone to assume that writing <a href='https://en.wikipedia.org/wiki/Dynamic_dispatch' target='_blank' rel="nofollow">dynamic dispatch</a> code is more complex than it actually is. </p> <p> To summarize the documentation, all Python classes, methods and instances are callable. <a href='https://www.tutorialsteacher.com/python/callable-method' target='_blank' rel="nofollow"><code>Callable</code> functions</a> have type <code>Callable[[InputArg1Type, InputArg2Type], ReturnType]</code>. If you do not want any type checking, write <code>Callable[..., Any]</code>. However, this is not very helpful information for dynamic dispatch. Fortunately, working with <code>Callable</code> is very simple. </p> <p class="notepaper"> You can pass around any Python class, constructor, function or method, and later provide it with the usual arguments. Invocation just works. </p> <p> Let me show you how easy it is to write dynamic dispatch code in Python, let's construct one of five classes, depending on the value of an enum. First, we need a class definition for each enum value: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Python</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idccb0b38c7f68'><button class='copyBtn' data-clipboard-target='#idccb0b38c7f68' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># pylint: disable=too-few-public-methods<br> class BaseClass(): """Demo only"""<br> class TestLecture(BaseClass): """This constructor has type Callable[[int, str], TestLecture]""" def __init__(self, id_: int, action: str): print(f"CadLecture constructor called with id {id_} and action {action}")<br> class TestSection(BaseClass): """This constructor has type Callable[[int, str], TestSection]""" def __init__(self, id_: int, action: str): print(f"CadSection constructor called with id {id_} and action {action}")<br> class TestCourse(BaseClass): """This constructor has type Callable[[int, str], TestCourse]""" def __init__(self, id_: int, action: str): print(f"CadCourse constructor called with id {id_} and action {action}")<br> class TestGroup(BaseClass): """This constructor has type Callable[[int, str], TestGroup]""" def __init__(self, id_: int, action: str): print(f"CadGroup constructor called with id {id_} and action {action}")<br> class TestSite(BaseClass): """This constructor has type Callable[[int, str], TestSite]""" def __init__(self, id_: int, action: str): print(f"CadSite constructor called with id {id_} and action {action}")</pre> </div> <!-- endregion --> <p> Now lets add another method, called <code>construct</code>, to <code>EntityType</code> that invokes the appropriate constructor according to the value of an <code>EntityType</code> instance: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Python</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6f4651c7b5af'><button class='copyBtn' data-clipboard-target='#id6f4651c7b5af' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>@property def construct(self) -> Callable: """:return: the appropriate Callable for each enum value""" if self == EntityType.LECTURE: return TestLecture if self == EntityType.SECTION: return TestSection if self == EntityType.COURSE: return TestCourse if self == EntityType.GROUP: return TestGroup return TestSite</pre> </div> <!-- endregion --> <p class="alert right rounded shadow" style="width: 50%;"> Using named arguments makes your code resistant to problems that might sneak in due to parameters changing over time. </p> <p> I favor using named arguments at all times; it avoids many problems. As code evolves, arguments might be added or removed, or even reordered. </p> <p class="clear"> Let's test out dynamic dispatch in the Python interpreter. A class specific to each <code>EntityType</code> value is constructed by invoking the appropriate <code>Callable</code> and passing it named arguments <a href='https://stackoverflow.com/a/28091085/553865' target='_blank' rel="nofollow"><code>id_</code></a> and <code>action</code>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Python</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0915980bfaca'><button class='copyBtn' data-clipboard-target='#id0915980bfaca' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>EntityType.LECTURE.construct(id_=55, action="gimme_lecture") <span class='unselectable'>TestLecture constructor called with id 55 and action gimme_lecture &lt;entity_types.TestLecture object at 0x7f9aac690070> </span><br> <span class='unselectable'>>>> </span>EntityType.SECTION.construct(id_=13, action="gimme_section") <span class='unselectable'>TestSection constructor called with id 13 and action gimme_section &lt;entity_types.TestSection object at 0x7f9aac5c1730> </span><br> <span class='unselectable'>>>> </span>EntityType.COURSE.construct(id_=40, action="gimme_course") <span class='unselectable'>TestCourse constructor called with id 40 and action gimme_course &lt;entity_types.TestCourse object at 0x7f9aac6900a0> </span><br> <span class='unselectable'>>>> </span>EntityType.GROUP.construct(id_=103, action="gimme_group") <span class='unselectable'>TestGroup constructor called with id 103 and action gimme_group &lt;entity_types.TestGroup object at 0x7f9aac4c6b20> </span><br> <span class='unselectable'>>>> </span>EntityType.SITE.construct(id_=1, action="gimme_site") <span class='unselectable'>TestSite constructor called with id 1 and action gimme_site &lt;entity_types.TestSite object at 0x7f9aac5c1730> </span></pre> </div> <!-- endregion --> <p> Because these factory methods return the newly created instance of the desired type, the string representation is printed on the console after the method finishes outputting its processing results, for example: <code>&lt;entity_types.TestLecture object at 0x7f9aac690070></code>. </p> <p> Using enums to construct class instances and/or invoke methods (aka dynamic dispatch) is super powerful. It rather resembles generics, actually, even though <a href='https://docs.python.org/3/library/typing.html#building-generic-types' target='_blank' rel="nofollow">Python&rsquo;s support for generics</a> is still in its infancy. </p> <p> The complete Python program discussed in this article is <a href='/blog/python/entity_types.py'>here</a>. </p> <!-- endregion --> Linking Directories on NTFS and Ext4 Volumes 2022-02-07T00:00:00-05:00 https://mslinn.github.io/blog/2022/02/07/wsl-volumes <!-- #region intro --> <p> Sometimes I need to insert some code into a program that depends on the type of format that a drive volume has. For example, today I need to either make a Windows junction to connect two directories on NTFS volumes. On the other hand, if one or both directories were on other types of volumes, I would have to connect the directories using a Linux symlink. </p> <p> All of the bash scripts shown in this article are meant to run in a Bash shell running on WSL or WSL2. </p> <!-- endregion --> <!-- #region Use Windows Junctions When Possible --> <h2 id="possible">Use Windows Junctions When Possible</h2> <p> When working on WSL, Windows junctions are more desirable than Linux hard links and symlinks because junctions are visible in Windows and also in WSL. Unlike Linux hard links, which only work within a single volume, Windows junctions can span two volumes. Linux symlinks are only visible from WSL; a symlinked directory only appears as a useless file when viewed from Windows. </p> <div class='imgWrapper imgFlex right' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.svg" type="image/svg"> <!---<source srcset="/blog/images/wsl-volumes/windowsFileMgr.avif" type="image/avif">--> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.webp" type="image/webp"> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.apng" type="image/apng"> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.png" type="image/png"> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.jpg" type="image/jpeg"> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.jpeg" type="image/jpeg"> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.jfif" type="image/jpeg"> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.pjpeg" type="image/jpeg"> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.pjp" type="image/jpeg"> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.gif" type="image/gif"> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.tif" type="image/tiff"> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.tiff" type="image/tiff"> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.bmp" type="image/bmp"> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.ico" type="image/x-icon"> <source srcset="/blog/images/wsl-volumes/windowsFileMgr.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/wsl-volumes/windowsFileMgr.png" style='width: 100%; ' /> </picture> </div> <p> Both directories need to be on NTFS volumes to make a Windows junction between them. Junctions are permitted within a single NTFS volume, or between two NTFS volumes. Linux symlinks can be used on all volume types, but only work properly when viewed from Linux. </p> <p> Windows junctions are shown with a small arrow icon in Windows File Manager. In the image above, the <code>curriculum</code> directory is a junction. </p> <!-- endregion --> <!-- #region Determining a Volume Type --> <h2 id="volumeType">Determining a Volume Type</h2> <p> I wrote the <code>volumeType</code> bash function to obtain the type of the volume that contains a file or directory. Linux <code>ext4</code> volumes have partition type <code>ext4</code>. NTFS volumes have partition type <code>9p</code>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0ff435828149'><button class='copyBtn' data-clipboard-target='#id0ff435828149' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>function volumeType { # Usually returns volume types ext4 or 9p (for NTFS) df -Th "$1" | tail -n 1 | awk '{print $2}' }</pre> </div> <!-- endregion --> <p> Here are examples of using <code>volumeType</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id33aad9590762'><button class='copyBtn' data-clipboard-target='#id33aad9590762' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>volumeType /mnt/c # Under WSL/WSL2 this is usually NTFS <span class='unselectable'>9p </span><br> <span class='unselectable'>$ </span>volumeType / # For Ubuntu this defaults to ext4 <span class='unselectable'>ext4 </span></pre> </div> <!-- endregion --> <p> All of the remaining scripts on this page either return a value (indicating <code>true</code>), or they do not return anything (indicating <code>false</code>). </p> <p> Two more bash functions test if a file or directory is part of an NTFS or ext4 volume: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0b8c304aca50'><button class='copyBtn' data-clipboard-target='#id0b8c304aca50' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>function isNTFS { if [ "$( volumeType "$1" )" == 9p ]; then echo yes; fi }<br> function isExt4 { if [ "$( volumeType "$1" )" == ext4 ]; then echo yes; fi }</pre> </div> <!-- endregion --> <p> Here are examples of using <code>isNTFS</code> and <code>isExt4</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3ac77f14b575'><button class='copyBtn' data-clipboard-target='#id3ac77f14b575' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>isNTFS /mnt/c <span class='unselectable'>yes </span><br> <span class='unselectable'>$ </span>isExt4 /mnt/c<br> <span class='unselectable'>$ </span>isNTFS /<br> <span class='unselectable'>$ </span>isExt4 / <span class='unselectable'>yes </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Windows Junctions --> <h2 id="junctions">Windows Junctions</h2> <p> The <code>bothOnNTFS</code> bash function indicates if both of the paths passed to it are on NTFS volumes. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id785b5a21ee8e'><button class='copyBtn' data-clipboard-target='#id785b5a21ee8e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>function bothOnNTFS { if [ "$( isNTFS "$1" )" ] && [ "$( isNTFS "$2" )" ]; then echo yes; fi }</pre> </div> <p> Let's try out <code>bothOnNTFS</code>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id86f1811a5c07'><button class='copyBtn' data-clipboard-target='#id86f1811a5c07' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>bothOnNTFS /mnt/c /mnt/f <span class='unselectable'>yes </span><br> <span class='unselectable'>$ </span>bothOnNTFS /mnt/c /</pre> </div> <!-- endregion --> <p> <code>bothOnNTFS</code> lets us decide how to connect two directories. If they are both on NTFS volumes, we can connect them using a Windows junction; otherwise we'll need to symlink them. </p> <!-- endregion --> <!-- #region Connecting Via a Windows Junction or Linux Symlink --> <h2 id="connect">Connecting Via a Windows Junction or Linux Symlink</h2> <p> We could either make a Windows junction using the <a href='https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/mklink' target='_blank' rel="nofollow"><code>mklink</code></a> command, or we could make a Linux symlink using the <a href='https://man7.org/linux/man-pages/man1/ln.1.html' target='_blank' rel="nofollow"><code>ln -s</code></a> command. Notice how the order of parameters between <code>mklink</code> is the reverse of the order of the Linux <code>ln</code> command. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd25ec1bdcf24'><button class='copyBtn' data-clipboard-target='#idd25ec1bdcf24' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>if [ $( bothOnNTFS "$cadenzaCurriculum" . ) ]; then WINDOWS_PATH="$( wslpath -w "$cadenzaCurriculum/site_$TITLE" )" echo "Making Windows junction from $cadenzaCurriculum/site_$TITLE to curriculum/" cmd.exe /C mklink /j curriculum "$WINDOWS_PATH" else echo "Symlinking $cadenzaCurriculum/site_$TITLE to curriculum/" ln -s "$cadenzaCurriculum/site_$TITLE" curriculum fi</pre> </div> When the above code ran it produced: <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9f4ee00bfa7d'><button class='copyBtn' data-clipboard-target='#id9f4ee00bfa7d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>Making Windows junction from /mnt/f/work/cadenzaHome/cadenzaCurriculum/site_ScalaCourses.com to curriculum/ Junction created for curriculum <<===>> F:\work\cadenzaHome\cadenzaCurriculum\site_ScalaCourses.com</pre> </div> <!-- endregion --> <!-- #region Windows Junction for Windows Home in Ubuntu --> <h2 id="">Windows Junction for Windows Home in Ubuntu</h2> <p> Neither Linux nor Java appreciate spaces in directories. Spaces in home directories are by nature problematic. If you have a space in your home directory on Windows, you can use the PowerShell <code>New-Item</code> commandlet to create an equivalent junction to a new directory node without spaces in the name. </p> <p> The following uses the PowerShell <a href='https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/new-item?view=powershell-7.3' target='_blank' rel="nofollow"><code>New-Item</code></a> commandlet to create a new directory called <code>C:\<wbr>Users\<wbr>mslinn</code>, accessible from WSL as <code>/mnt/<wbr>c/<wbr>mslinn</code>, which points to the same directory as <code>C:\Users\<wbr>Mike Slinn</code>. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>PowerShell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfcffa2f9fdc2'><button class='copyBtn' data-clipboard-target='#idfcffa2f9fdc2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>PS C:\Users\Mike Slinn> </span>ni "C:\Users\mslinn" -i SymbolicLink -ta "C:\Users\Mike Slinn"</pre> </div> <!-- endregion --> Handcrafted Dynamic DNS for AWS Route53 and Namecheap 2022-01-30T00:00:00-05:00 https://mslinn.github.io/blog/2022/01/30/ddns-route53 <p> Now that I have fiber-optic internet service in my apartment, with 500 GB/s upload and download, I thought I would save money by hosting my <a href='https://scalacourses.com'><code>scalacourses.com</code></a> website on an Ubuntu server that runs here, instead of AWS. My home IP address is quite stable, and only changes when the fiber modem boots up. The modem is branded as a <a href='https://support.bell.ca/internet/products/home-hub-4000-modem' target='_blank' rel="nofollow">Bell Home Hub 4000</a>, but I believe it is actually made by Arris (formerly known as Motorola). </p> <p> It makes little sense to pay the commercial cost of dedicated dynamic DNS services (typically $55 USD / year) when it is so easy to automate, and the operational cost is less than one cent per year. </p> <p> I wrote two little scripts that automatically check my public IP address, and modifies the DNS record for my home IP address whenever the IP address changes. One script is for sites that use DNS provided by AWS Route53, and the other is for DNS provided by Namecheap. </p> <p> The approach shown here could be used for all DNS servers that have a command-line interface. This was originally written for AWS Route53, but when I <a href='/blog/2022/05/26/aws-hijacking.html'>moved off AWS</a> I made a version for <a href='https://www.namecheap.com/support/knowledgebase/article.aspx/29/11/how-to-dynamically-update-the-hosts-ip-with-an-http-request/' target='_blank' rel="nofollow">Namecheap</a>. Alternative DNS providers include <a href='https://docs.microsoft.com/en-us/cli/azure/network/dns?view=azure-cli-latest' target='_blank' rel="nofollow">Azure DNS</a>, <a href='https://developers.cloudflare.com/cloudflare-one/tutorials/cli' target='_blank' rel="nofollow">Cloudflare DNS</a>, <a href='https://support.dnsmadeeasy.com/support/solutions/articles/47001119947-the-ddns-shell-script' target='_blank' rel="nofollow">DNSMadeEasy</a>, <a href='https://developer.dnsimple.com/libraries/' target='_blank' rel="nofollow">DNSimple</a>, <a href='https://cloud.google.com/sdk/gcloud/reference/dns' target='_blank' rel="nofollow">Google Cloud DNS</a>, and <a href='https://github.com/ultradns/dns_sprockets' target='_blank' rel="nofollow">UltraDNS</a>. </p> <h2 id="fwd">Forwarding HTTP Requests</h2> <p> I added some entries to the modem so incoming HTTP traffic on ports 80 and 443 would be forwarded to ports 9000 and 9443 on my home server, <code>gojira</code>. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/ddns/homeHubPortForward.svg" type="image/svg"> <!---<source srcset="/blog/images/ddns/homeHubPortForward.avif" type="image/avif">--> <source srcset="/blog/images/ddns/homeHubPortForward.webp" type="image/webp"> <source srcset="/blog/images/ddns/homeHubPortForward.apng" type="image/apng"> <source srcset="/blog/images/ddns/homeHubPortForward.png" type="image/png"> <source srcset="/blog/images/ddns/homeHubPortForward.jpg" type="image/jpeg"> <source srcset="/blog/images/ddns/homeHubPortForward.jpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/homeHubPortForward.jfif" type="image/jpeg"> <source srcset="/blog/images/ddns/homeHubPortForward.pjpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/homeHubPortForward.pjp" type="image/jpeg"> <source srcset="/blog/images/ddns/homeHubPortForward.gif" type="image/gif"> <source srcset="/blog/images/ddns/homeHubPortForward.tif" type="image/tiff"> <source srcset="/blog/images/ddns/homeHubPortForward.tiff" type="image/tiff"> <source srcset="/blog/images/ddns/homeHubPortForward.bmp" type="image/bmp"> <source srcset="/blog/images/ddns/homeHubPortForward.ico" type="image/x-icon"> <source srcset="/blog/images/ddns/homeHubPortForward.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/ddns/homeHubPortForward.png" style='width: 100%; ' /> </picture> </div> <h2 id="usage">Using the Scripts</h2> <p> The version for AWS Route53 is called <code>dynamicDnsAws</code>, and the version for Namecheap is called <code>dynamicDnsNamecheap</code>. </p> <p> The scripts save the IP address to a file, and periodically compare the saved value to the current value. Then the scripts modify the DNS record for a specified subdomain whenever the value of the public IP address changes. </p> <h3 id="aws">Using the AWS Script</h3> <p> Here is the help information for the script: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc9e7d8e6c445'><button class='copyBtn' data-clipboard-target='#idc9e7d8e6c445' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>dynamicDnsAws <span class='unselectable'>dynamicDnsAws - Maintains a dynamic DNS record in AWS Route53<br> Saves data in '/home/mslinn/.dynamicDnsAws'<br> Syntax: dynamicDnsAws [OPTIONS] SUB_DOMAIN DOMAIN<br> OPTIONS: -v Verbose mode<br> Example usage: dynamicDnsAws www scalacourses.com dynamicDnsAws -v www scalacourses.com </span></pre> </div> <p> Here is a sample usage: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida48f81baa4d3'><button class='copyBtn' data-clipboard-target='#ida48f81baa4d3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>dynamicDnsAws www scalacourses.com <span class='unselectable'>{ "ChangeInfo": { "Id": "/change/C075751811HI18SH4L8L0", "Status": "PENDING", "SubmittedAt": "2022-01-30T21:10:09.261Z", "Comment": "UPSERT a record for www.scalacourses.com" } } </span></pre> </div> <h3 id="aws">Using the Namecheap Script</h3> <p> Here is the help information for the script: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id48a154cbba9d'><button class='copyBtn' data-clipboard-target='#id48a154cbba9d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>dynamicDnsNamecheap <span class='unselectable'>dynamicDnsNamecheap - Maintains two Namecheap dynamic DNS records<br> Saves data in '/home/mslinn/.dynamicDns'<br> Syntax: dynamicDnsNamecheap [OPTIONS] DOMAIN PASSWORD<br> OPTIONS: -v Verbose mode<br> Example usage: dynamicDnsNamecheap mydomain.com asdfasdfasdfasdfasdf dynamicDnsNamecheap -v mydomain.com asdfasdfasdfasdf </span></pre> </div> <p> Here is sample usage: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8e820f36c676'><button class='copyBtn' data-clipboard-target='#id8e820f36c676' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>dynamicDnsNamecheap scalacourses.com asdfasdfasdfasdfasdf <span class='unselectable'>&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-16&quot;?&gt; &lt;interface-response&gt; &lt;Command&gt;SETDNSHOST&lt;/Command&gt; &lt;Language&gt;eng&lt;/Language&gt; &lt;IP&gt;142.126.4.220&lt;/IP&gt; &lt;ErrCount&gt;0&lt;/ErrCount&gt; &lt;errors /&gt; &lt;ResponseCount&gt;0&lt;/ResponseCount&gt; &lt;responses /&gt; &lt;Done&gt;true&lt;/Done&gt; &lt;debug&gt;&lt;![CDATA[]]&gt;&lt;/debug&gt; &lt;/interface-response&gt;&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-16&quot;?&gt; &lt;interface-response&gt; &lt;Command&gt;SETDNSHOST&lt;/Command&gt; &lt;Language&gt;eng&lt;/Language&gt; &lt;IP&gt;142.126.4.220&lt;/IP&gt; &lt;ErrCount&gt;0&lt;/ErrCount&gt; &lt;errors /&gt; &lt;ResponseCount&gt;0&lt;/ResponseCount&gt; &lt;responses /&gt; &lt;Done&gt;true&lt;/Done&gt; &lt;debug&gt;&lt;![CDATA[]]&gt;&lt;/debug&gt; &lt;/interface-response&gt; </span></pre> </div> <h2 id="crontab">Invoking the Scripts from Crontab</h2> <p> A personal <code>crontab</code> can be modified by typing: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd9b9b8d28d9e'><button class='copyBtn' data-clipboard-target='#idd9b9b8d28d9e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>crontab -e</pre> </div> <p> I pasted in the following into <code>crontab</code> on my Ubuntu server, running at home. These lines invoke the <code>dynamicDnsNamecheap</code> script via <code>crontab</code> every 5 minutes. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddb8790ef4b3c'><button class='copyBtn' data-clipboard-target='#iddb8790ef4b3c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>*/5 * * * * /path/to/dynamicDnsNamecheap my_domain.com asdfasdfasdfasdfasdf</pre> </div> <p> One the above is saved, <code>crontab</code> will run the script every 5 minutes. </p> <h2 id="source">Script Source Codes</h2> <p> Here are the bash scripts: </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,dynamicDnsAws' download='dynamicDnsAws' title='Click on the file name to download the file'>dynamicDnsAws</a> </div> <pre data-lt-active="false" class="pre_tag maxOneScreenHigh copyContainer" id="idc2786b9e5c57">#!/bin/bash # Author: Mike Slinn mslinn@mslinn.com # Written 2022-01-30 export SAVE_FILE_NAME="$HOME/.dynamicDns" function help &#123; echo "$( basename $0 ) - Maintains a dynamic DNS record in AWS Route53 Saves data in '$SAVE_FILE_NAME' Syntax: $( basename $0) [OPTIONS] SUB_DOMAIN DOMAIN OPTIONS: -v Verbose mode Example usage: $( basename $0) my_subdomain mydomain.com $( basename $0) -v my_subdomain mydomain.com " exit 1 &#125; function upsert &#123; export HOSTED_ZONES="$( aws route53 list-hosted-zones )" export HOSTED_ZONE_RECORD="$( jq -r ".HostedZones[] | select(.Name == \"$DOMAIN.\")" &lt;&lt;&lt; "$HOSTED_ZONES" )" export HOSTED_ZONE_RECORD_ID="$( jq -r .Id &lt;&lt;&lt; "$HOSTED_ZONE_RECORD" )" aws route53 change-resource-record-sets \ --hosted-zone-id "$HOSTED_ZONE_RECORD_ID" \ --change-batch "&#123; \"Comment\": \"UPSERT a record for $SUBDOMAIN.$DOMAIN\", \"Changes\": [&#123; \"Action\": \"UPSERT\", \"ResourceRecordSet\": &#123; \"Name\": \"$SUBDOMAIN.$DOMAIN\", \"Type\": \"A\", \"TTL\": 300, \"ResourceRecords\": [&#123; \"Value\": \"$IP\"&#125;] &#125; &#125;] &#125;" echo "$IP" > "$SAVE_FILE_NAME" &#125; if [ "$1" == -v ]; then export VERBOSE=true shift fi if [ -z "$2" ]; then help; fi set -e export SUBDOMAIN="$1" export DOMAIN="$2" export IP="$( dig +short myip.opendns.com @resolver1.opendns.com )" if [ ! -f "$SAVE_FILE_NAME" ]; then if [ "$VERBOSE" ]; then echo "Creating $SAVE_FILE_NAME"; fi upsert; elif [ $( cat "$SAVE_FILE_NAME" ) != "$IP" ]; then if [ "$VERBOSE" ]; then echo "Updating $SAVE_FILE_NAME" echo "'$IP' was not equal to '$( cat "$SAVE_FILE_NAME" )'" fi upsert; else if [ "$VERBOSE" ]; then echo "No change necessary for $SAVE_FILE_NAME"; fi fi </pre> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,dynamicDnsNamecheap' download='dynamicDnsNamecheap' title='Click on the file name to download the file'>dynamicDnsNamecheap</a> </div> <pre data-lt-active="false" class="pre_tag maxOneScreenHigh copyContainer" id="id30ea85a6e581">#!/bin/bash # Author: Mike Slinn mslinn@mslinn.com # Modified from AWS version (dynameicDnsAws) 2022-06-30 # See https://www.namecheap.com/support/knowledgebase/article.aspx/36/11/how-do-i-start-using-dynamic-dns/ export SAVE_FILE_NAME="$HOME/.dynamicDns" function help &#123; echo "$( basename $0 ) - Maintains two Namecheap dynamic DNS records Saves data in '$SAVE_FILE_NAME' Syntax: $( basename $0) [OPTIONS] DOMAIN PASSWORD OPTIONS: -v Verbose mode Example usage: $( basename $0) mydomain.com asdfasdfasdfasdfasdf $( basename $0) -v mydomain.com asdfasdfasdfasdf " exit 1 &#125; function upsert &#123; curl "https://dynamicdns.park-your-domain.com/update?host=@&amp;domain=$DOMAIN&amp;password=$PASSWORD&amp;ip=$IP" curl "https://dynamicdns.park-your-domain.com/update?host=www&amp;domain=$DOMAIN&amp;password=$PASSWORD&amp;ip=$IP" echo "$IP" > "$SAVE_FILE_NAME" echo "" &#125; if [ "$1" == -v ]; then export VERBOSE=true shift fi if [ -z "$2" ]; then help; fi set -e export DOMAIN="$1" export PASSWORD="$2" export IP="$( dig +short myip.opendns.com @resolver1.opendns.com )" if [ ! -f "$SAVE_FILE_NAME" ]; then if [ "$VERBOSE" ]; then echo "Creating $SAVE_FILE_NAME"; fi upsert; elif [ "$( cat "$SAVE_FILE_NAME" )" != "$IP" ]; then if [ "$VERBOSE" ]; then echo "Updating $SAVE_FILE_NAME" echo "'$IP' was not equal to '$( cat "$SAVE_FILE_NAME" )'" fi upsert; else if [ "$VERBOSE" ]; then echo "No change necessary for $SAVE_FILE_NAME"; fi fi </pre> Windows Diskpart Cooperates With Diskmgmt 2022-01-14T00:00:00-05:00 https://mslinn.github.io/blog/2022/01/14/diskpart <!-- #region intro --> <p> Recently, I wanted to use some old hard drives as backup media. That meant scrubbing all the partitions off the drives, and installing new partitions, which of course would be empty. </p> <div class="right rounded shadow warning" style="width: 45%"> <p> Warning &ndash; Working with a command line program for system-level operations, without having backed up the system, is like walking on a tightrope without a net. </p> <p style="margin-bottom: 0"> A mistake could inadvertently wipe out a different hard drive on the computer than you intended. Without the ability to restore the system disk from backup, your computer could become inoperable. </p> </div> <p> However, I was unable to repartition one of those old drives using the GUI-based Windows <code>diskmgmt.msc</code> disk manager. The drive had soft errors. For some reason, those errors made it impossible for <code>diskmgmt.msc</code> to scan the drive. </p> <p> The more powerful Windows <a href='https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/diskpart' target='_blank' rel="nofollow"><code>diskpart</code></a>, a command line program, was able to get the job done. </p> <p> This article <a href ="#gui">ends</a> by demonstrating how you can get benefit from the <code>diskmgmt</code> GUI even when you are using the <code>diskpart</code> command line interface. This is possible because every command you type into <code>diskpart</code> causes a Windows system event to be published, and because <code>diskmgmt</code> subscribes to those events, it is able to display the results as they happen. </p> <!-- endregion --> <!-- #region Starting Diskpart --> <h2 id="starting">Starting <span class="code">Diskpart</span></h2> <p> Run <code>diskpart</code> as administrator as follows: </p> <ol> <li> Press the <kbd>Windows</kbd> key. Do not hold it down, just depress it once, as you would do for any other key, and let it go. </li> <li>Type <code>diskpart</code>.</li> <li>Use the mouse or arrow keys to select <b>Run as administrator</b>.</li> </ol> <div class='imgWrapper imgFlex center' style='width: 75%; '> <picture class='imgPicture'> <source srcset="/blog/images/diskpart/diskpartLaunch.svg" type="image/svg"> <!---<source srcset="/blog/images/diskpart/diskpartLaunch.avif" type="image/avif">--> <source srcset="/blog/images/diskpart/diskpartLaunch.webp" type="image/webp"> <source srcset="/blog/images/diskpart/diskpartLaunch.apng" type="image/apng"> <source srcset="/blog/images/diskpart/diskpartLaunch.png" type="image/png"> <source srcset="/blog/images/diskpart/diskpartLaunch.jpg" type="image/jpeg"> <source srcset="/blog/images/diskpart/diskpartLaunch.jpeg" type="image/jpeg"> <source srcset="/blog/images/diskpart/diskpartLaunch.jfif" type="image/jpeg"> <source srcset="/blog/images/diskpart/diskpartLaunch.pjpeg" type="image/jpeg"> <source srcset="/blog/images/diskpart/diskpartLaunch.pjp" type="image/jpeg"> <source srcset="/blog/images/diskpart/diskpartLaunch.gif" type="image/gif"> <source srcset="/blog/images/diskpart/diskpartLaunch.tif" type="image/tiff"> <source srcset="/blog/images/diskpart/diskpartLaunch.tiff" type="image/tiff"> <source srcset="/blog/images/diskpart/diskpartLaunch.bmp" type="image/bmp"> <source srcset="/blog/images/diskpart/diskpartLaunch.ico" type="image/x-icon"> <source srcset="/blog/images/diskpart/diskpartLaunch.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/diskpart/diskpartLaunch.png" style='width: 100%; ' /> </picture> </div> <p> A window should open up labeled <b>Microsoft DiskPart</b>. Let's start by listing all the <code>diskpart</code> commands. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id798b75a50ef7'><button class='copyBtn' data-clipboard-target='#id798b75a50ef7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>Microsoft DiskPart version 10.0.19041.964 Copyright (C) Microsoft Corporation. On computer: BEAR<br> DISKPART> </span>help<br> <span class='unselectable'>Microsoft DiskPart version 10.0.19041.964<br> ACTIVE - Mark the selected partition as active. ADD - Add a mirror to a simple volume. ASSIGN - Assign a drive letter or mount point to the selected volume. ATTRIBUTES - Manipulate volume or disk attributes. ATTACH - Attaches a virtual disk file. AUTOMOUNT - Enable and disable automatic mounting of basic volumes. BREAK - Break a mirror set. CLEAN - Clear the configuration information, or all information, off the disk. COMPACT - Attempts to reduce the physical size of the file. CONVERT - Convert between different disk formats. CREATE - Create a volume, partition or virtual disk. DELETE - Delete an object. DETAIL - Provide details about an object. DETACH - Detaches a virtual disk file. EXIT - Exit DiskPart. EXTEND - Extend a volume. EXPAND - Expands the maximum size available on a virtual disk. FILESYSTEMS - Display current and supported file systems on the volume. FORMAT - Format the volume or partition. GPT - Assign attributes to the selected GPT partition. HELP - Display a list of commands. IMPORT - Import a disk group. INACTIVE - Mark the selected partition as inactive. LIST - Display a list of objects. MERGE - Merges a child disk with its parents. ONLINE - Online an object that is currently marked as offline. OFFLINE - Offline an object that is currently marked as online. RECOVER - Refreshes the state of all disks in the selected pack. Attempts recovery on disks in the invalid pack, and resynchronizes mirrored volumes and RAID5 volumes that have stale plex or parity data. REM - Does nothing. This is used to comment scripts. REMOVE - Remove a drive letter or mount point assignment. REPAIR - Repair a RAID-5 volume with a failed member. RESCAN - Rescan the computer looking for disks and volumes. RETAIN - Place a retained partition under a simple volume. SAN - Display or set the SAN policy for the currently booted OS. SELECT - Shift the focus to an object. SETID - Change the partition type. SHRINK - Reduce the size of the selected volume. UNIQUEID - Displays or sets the GUID partition table (GPT) identifier or master boot record (MBR) signature of a disk. </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Listing Drives, Partitions and Volumes --> <h2 id="listing">Listing Drives, Partitions and Volumes</h2> <p> Listing the drives is generally a good first step. Let's discover the command for that: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9dc9bf02ecce'><button class='copyBtn' data-clipboard-target='#id9dc9bf02ecce' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>list<br> <span class='unselectable'>Microsoft DiskPart version 10.0.19041.964<br> DISK - Display a list of disks. For example, LIST DISK. PARTITION - Display a list of partitions on the selected disk. For example, LIST PARTITION. VOLUME - Display a list of volumes. For example, LIST VOLUME. VDISK - Displays a list of virtual disks. </span></pre> </div> <!-- endregion --> <p> OK, we can list disks, partitions, volumes and virtual disks. Let's list the disk drives. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc9da6d4eb46c'><button class='copyBtn' data-clipboard-target='#idc9da6d4eb46c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>list disk<br> <span class='unselectable'>Disk ### Status Size Free Dyn Gpt -------- ------------- ------- ------- --- --- Disk 0 Online 1863 GB 1024 KB * Disk 1 Online 465 GB 1024 KB Disk 2 Online 1863 GB 1024 KB * Disk 3 Online 1863 GB 0 B * Disk 5 Online 931 GB 931 GB * </span></pre> </div> <!-- endregion --> <div class='imgWrapper imgBlock right quartersize' style=' '> <figure> <a href='https://www.amazon.com/NewerTech-Enclosure-Interface-NWTU3S3HD-hot-swapping/dp/B007TTQQIA' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.svg" type="image/svg"> <!---<source srcset="/blog/images/diskpart/newerTechVoyagerS3.avif" type="image/avif">--> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.webp" type="image/webp"> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.apng" type="image/apng"> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.png" type="image/png"> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.jpg" type="image/jpeg"> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.jpeg" type="image/jpeg"> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.jfif" type="image/jpeg"> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.pjpeg" type="image/jpeg"> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.pjp" type="image/jpeg"> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.gif" type="image/gif"> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.tif" type="image/tiff"> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.tiff" type="image/tiff"> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.bmp" type="image/bmp"> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.ico" type="image/x-icon"> <source srcset="/blog/images/diskpart/newerTechVoyagerS3.cur" type="image/x-icon"> <img alt='NewerTech Voyager S3 caddy' class="imgImg rounded shadow" src="/blog/images/diskpart/newerTechVoyagerS3.png" style='width: 100%; ' title='NewerTech Voyager S3 caddy' /> </picture> </a> <figcaption class='imgFigCaption quartersize'> <a href="https://www.amazon.com/NewerTech-Enclosure-Interface-NWTU3S3HD-hot-swapping/dp/B007TTQQIA" target='_blank' > NewerTech Voyager S3 caddy </a> </figcaption> </figure> </div> <p> At this point, I inserted the old 5.25" SATA drive that I wanted to repurpose into a NewerTech Voyager S3 caddy, connected to my computer via a USB 3 cable, and Windows automatically mounted it. The caddy also accepts 2.5" SATA drives, such as those commonly found in laptops. </p> <p> Now I told <code>diskpart</code> to rescan the drives, and then I listed the volumes on all disks. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer clear' id='id2cb15038b0f1'><button class='copyBtn' data-clipboard-target='#id2cb15038b0f1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>rescan<br> <span class='unselectable'>Please wait while DiskPart scans your configuration...<br> DiskPart has finished scanning your configuration.<br> DISKPART> </span>list volume<br> <span class='unselectable'>Volume ### Ltr Label Fs Type Size Status Info ---------- --- ----------- ----- ---------- ------- --------- -------- Volume 0 D DVD-ROM 0 B No Media Volume 1 C BEAR_C NTFS Partition 1861 GB Healthy Boot Volume 2 FAT32 Partition 100 MB Healthy System Volume 3 NTFS Partition 539 MB Healthy Hidden Volume 4 NTFS Partition 450 MB Healthy Hidden Volume 5 F Work NTFS Partition 1863 GB Healthy Volume 6 E BEAR_E NTFS Partition 1863 GB Healthy <span class="bg_yellow"> Volume 8 FAT32 Partition 512 MB Healthy Hidden</span> </span></pre> </div> <!-- endregion --> <p> <code>Diskpart</code> displayed the hidden volume (#8) in the drive in the caddy. This drive has <code>readonly</code> status set, which prevents its contents from being modified or deleted. That would be good if I wanted to use this drive as an archive, but instead I want to scrub it and write new information on it. The currently existing partitions on this drive cannot be erased until <code>readonly</code> is cleared. Lets remove <code>readonly</code> status from the drive now: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida6672614df63'><button class='copyBtn' data-clipboard-target='#ida6672614df63' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>select volume 8 <span class='unselectable'>Volume 8 is the selected volume.<br> DISKPART> </span>attributes disk clear readonly <span class='unselectable'>Disk attributes cleared successfully.<br> DISKPART> </span>rescan <span class='unselectable'>Please wait while DiskPart scans your configuration... DiskPart has finished scanning your configuration.<br> DISKPART> </span>list volume <span class='unselectable'>Volume ### Ltr Label Fs Type Size Status Info ---------- --- ----------- ----- ---------- ------- --------- -------- Volume 0 D DVD-ROM 0 B No Media Volume 1 C BEAR_C NTFS Partition 1861 GB Healthy Boot Volume 2 FAT32 Partition 100 MB Healthy System Volume 3 NTFS Partition 539 MB Healthy Hidden Volume 4 NTFS Partition 450 MB Healthy Hidden Volume 5 F Work NTFS Partition 1863 GB Healthy Volume 6 E BEAR_E NTFS Partition 1863 GB Healthy Volume 8 FAT32 Partition 512 MB Healthy Hidden </span></pre> </div> <!-- endregion --> <!-- #region Wiping the Drive --> <h2 id="wipe">Wiping the Drive</h2> <p> Now it is time to wipe the drive clean, which removes all partitions. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idacde4c29c31c'><button class='copyBtn' data-clipboard-target='#idacde4c29c31c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>select disk 5 <span class='unselectable'>Disk 5 is now the selected disk.<br> DISKPART> </span>list disk <span class='unselectable'>Disk ### Status Size Free Dyn Gpt -------- ------------- ------- ------- --- --- Disk 0 Online 1863 GB 1024 KB * Disk 1 Online 465 GB 1024 KB Disk 2 Online 1863 GB 1024 KB * Disk 3 Online 1863 GB 0 B * Disk 5 Online 931 GB 931 GB *<br> DISKPART> </span>clean <span class='unselectable'>DiskPart succeeded in cleaning the disk. </span></pre> </div> <!-- endregion --> <!-- #region Archiving a Drive --> <h2 id="setRO">Archiving a Drive</h2> <p> If instead of wiping the drive, I wanted to archive the drive, I would want to set the read-only status. To do that, first select the drive as before, then type: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0c5588970288'><button class='copyBtn' data-clipboard-target='#id0c5588970288' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>list disk <span class='unselectable'># Output as shown above </span><br> <span class='unselectable'>DISKPART> </span>select disk N <span class='unselectable'>Disk N is now the selected disk. </span><br> <span class='unselectable'>DISKPART> </span>attributes disk set readonly</pre> </div> <!-- endregion --> <p> Now the drive's contents could not accidently be erased or modified. </p> <!-- endregion --> <!-- #region Create A New Partition --> <h2 id="partition">Create A New Partition</h2> <p> We need to create a new partition that spans the entire disk on the now-empty selected drive. The <code>create</code> command can do that. Let's look at the help before using the command: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida6ebec7ccc2f'><button class='copyBtn' data-clipboard-target='#ida6ebec7ccc2f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>create <span class='unselectable'>Microsoft DiskPart version 10.0.19041.964<br> PARTITION - Create a partition. VOLUME - Create a volume. VDISK - Creates a virtual disk file.<br> DISKPART> </span>create partition<br> <span class='unselectable'>Microsoft DiskPart version 10.0.19041.964<br> EFI - Create an EFI system partition. EXTENDED - Create an extended partition. LOGICAL - Create a logical drive. MSR - Create a Microsoft Reserved partition. PRIMARY - Create a primary partition. </span></pre> </div> <!-- endregion --> <p> To create a new partition that spans the entire disk on the now-empty selected drive and make it active: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4338336cf9c1'><button class='copyBtn' data-clipboard-target='#id4338336cf9c1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>create partition primary <span class='unselectable'>DiskPart succeeded in creating the specified partition.<br> DISKPART> </span>select partition 1 <span class='unselectable'>Partition 1 is now the selected partition.<br> DISKPART> </span>active <span class='unselectable'>DiskPart marked the current partition as active. </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Format A Volume --> <h2 id="format">Format A Volume</h2> <p> Let's format the entire selected drive as one volume. Like the <code>clean</code> command, the <code>format</code> command operates on the currently selected disk. First, let's look at the help: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7463d0d485e7'><button class='copyBtn' data-clipboard-target='#id7463d0d485e7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>help format<br> <span class='unselectable'>Formats the specified volume for use with Windows.<br> Syntax: FORMAT [[FS=&lt;FS>] [REVISION=&lt;X.XX>] | RECOMMENDED] [LABEL=&lt;"label">] [UNIT=&lt;N>] [QUICK] [COMPRESS] [OVERRIDE] [DUPLICATE] [NOWAIT] [NOERR]<br> FS=<FS> Specifies the type of file system. If no file system is given, the default file system displayed by the FILESYSTEMS command is used.<br> REVISION=&lt;X.XX><br> Specifies the file system revision (if applicable).<br> RECOMMENDED If specified, use the recommended file system and revision instead of the default if a recommendation exists. The recommended file system (if one exists) is displayed by the FILESYSTEMS command.<br> LABEL=&lt;"label"><br> Specifies the volume label.<br> UNIT=&lt;N> Overrides the default allocation unit size. Default settings are strongly recommended for general use. The default allocation unit size for a particular file system is displayed by the FILESYSTEMS command.<br> NTFS compression is not supported for allocation unit sizes above 4096.<br> QUICK Performs a quick format.<br> COMPRESS NTFS only: Files created on the new volume will be compressed by default.<br> OVERRIDE Forces the file system to dismount first if necessary. All opened handles to the volume would no longer be valid.<br> DUPLICATE UDF Only: This flag applies to UDF format, version 2.5 or higher. This flag instructs the format operation to duplicate the file system meta-data to a second set of sectors on the disk. The duplicate meta-data is used by applications, for example repair or recovery applications. If the primary meta-data sectors are found to be corrupted, the file system meta-data will be read from the duplicate sectors.<br> NOWAIT Forces the command to return immediately while the format process is still in progress. If NOWAIT is not specified, DiskPart will display format progress in percentage.<br> NOERR For scripting only. When an error is encountered, DiskPart continues to process commands as if the error did not occur. Without the NOERR parameter, an error causes DiskPart to exit with an error code.<br> A volume must be selected for this operation to succeed.<br> Examples:<br> FORMAT FS=NTFS LABEL="New Volume" QUICK COMPRESS FORMAT RECOMMENDED OVERRIDE </span></pre> </div> <!-- endregion --> <p> <code>Diskpart</code> automatically chooses the optimal file system for the selected drive if you do not specify it. Choices include FAT, FAT32 and NTFS. To quick format the selected drive using the optimal file system: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd7ea4d039a47'><button class='copyBtn' data-clipboard-target='#idd7ea4d039a47' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>format quick</pre> </div> <p> The default is to fully format the selected drive, which is what you want if the selected drive is suspect: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4eb711d58aeb'><button class='copyBtn' data-clipboard-target='#id4eb711d58aeb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>format</pre> </div> <p> You could specify multiple parameters, for example: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd383c4394aca'><button class='copyBtn' data-clipboard-target='#idd383c4394aca' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>format fs=ntfs label="My Drive" quick</pre> </div> <!-- endregion --> <!-- #region Assigning a Drive Letter --> <h2 id="assign">Assigning a Drive Letter</h2> <p> Once the drive was formatted, I assigned the selected drive the letter <code>G</code>. You do not normally need to perform this step, unless you want to <a href='https://www.groovypost.com/howto/assign-permanent-letter-removable-usb-drive-windows/' target='_blank' rel="nofollow">define a default letter for this drive</a>. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5f123e8b750f'><button class='copyBtn' data-clipboard-target='#id5f123e8b750f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>assign letter=G</pre> </div> <!-- endregion --> <!-- #region Diskmgmt GUI Shows Progress --> <h2 id="gui"><span class="code">Diskmgmt</span> GUI Shows Progress</h2> <p> Having a GUI continuously report the current state of the drive maintenance you are performing using a command-line interface is a good practice. </p> <p> The GUI-based Windows disk manager, <code>diskmgmt.msc</code>, can show the instantaneous progress of all <code>diskpart</code> commands, working on every drive, including creating and deleting partitions, volumes, formatting and much more. For example, formatting progress can be seen here: </p> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/diskpart/diskUI.svg" type="image/svg"> <!---<source srcset="/blog/images/diskpart/diskUI.avif" type="image/avif">--> <source srcset="/blog/images/diskpart/diskUI.webp" type="image/webp"> <source srcset="/blog/images/diskpart/diskUI.apng" type="image/apng"> <source srcset="/blog/images/diskpart/diskUI.png" type="image/png"> <source srcset="/blog/images/diskpart/diskUI.jpg" type="image/jpeg"> <source srcset="/blog/images/diskpart/diskUI.jpeg" type="image/jpeg"> <source srcset="/blog/images/diskpart/diskUI.jfif" type="image/jpeg"> <source srcset="/blog/images/diskpart/diskUI.pjpeg" type="image/jpeg"> <source srcset="/blog/images/diskpart/diskUI.pjp" type="image/jpeg"> <source srcset="/blog/images/diskpart/diskUI.gif" type="image/gif"> <source srcset="/blog/images/diskpart/diskUI.tif" type="image/tiff"> <source srcset="/blog/images/diskpart/diskUI.tiff" type="image/tiff"> <source srcset="/blog/images/diskpart/diskUI.bmp" type="image/bmp"> <source srcset="/blog/images/diskpart/diskUI.ico" type="image/x-icon"> <source srcset="/blog/images/diskpart/diskUI.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/diskpart/diskUI.png" style='width: 100%; ' /> </picture> </div> <p> Run <code>diskmgmt.msc</code> by: </p> <ol> <li>Press the <kbd>Windows</kbd> key once and let go.</li> <li>Type <code>diskmgmt</code></li> <li>Press <kbd>Enter</kbd></li> </ol> <!-- endregion --> <!-- #region Exit --> <h2 id="exit"><span class="code">Exit</span></h2> <p> To exit <code>diskpart</code>, either type <code>Exit</code> or press <kbd>CTRL</kbd>-<kbd>C</kbd>. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft&nbsp;DiskPart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc349a1c458fe'><button class='copyBtn' data-clipboard-target='#idc349a1c458fe' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>exit</pre> </div> <p> The drive is ready for its next assignment! </p> <!-- endregion --> WSL / WSL 2 Backup and Restore 2022-01-10T00:00:00-05:00 https://mslinn.github.io/blog/2022/01/10/wsl-backup <!-- #region --> <p> I needed to back up my WSL2 installation before <a href='https://www.microsoft.com/en-us/software-download/windows10' target='_blank' rel="nofollow">reinstalling Windows 10</a>, as one must do every 6 months if you work your machine like a developer. It had been years since I had last refreshed this machine, and it was now bluescreening several times a day. Reboots took forever. </p> <div class="pullQuoteFull liImg"> This article presents the best WSL backup approaches I found </div> <p> I wanted to retain my WSL2 instance. When refreshing Windows you must reinstall all your programs, and you lose all the WSL/WSL2 instances. The refresh decommissions them, then moves them into a hidden directory tree, along with the rest of the stuff stored in your old Windows 10 profile directory tree. </p> <p> I also wanted to be able to replicate my WSL2 instance reliably and easily. It was very important to me to be able to administer WSL instances separately from my Windows 10 profile. </p> <!-- endregion --> <!-- #region --> <h2 id="location">Location, Location, Location</h2> <p> Only recently has it become possible to work with WSL/WSL2 images on non-system drives. Most online documentation is now out of date in this regard. Running Windows off a different drive than the drive that a WSL/WSL2 client OS runs from can provide a dramatic performance boost for some applications. </p> <div class="pullQuoteFull liImg"> Reinstalling Windows is less traumatic if the WSL/WSL2 image elsewhere in the filesystem </div> <!-- endregion --> <!-- #region --> <h2 id="dell">Dell&rsquo;s Ignorance Causes Pain</h2> <p> I use Dell laptops, because I love their onsite warranty price and product features. However, sometimes it feels like Dell&rsquo;s solution to every problem is to replace the motherboard. </p> <p> When Windows 10 encounters a new motherboard, it takes anti-piracy measures, and refuses to do much of anything useful. </p> <p> Refreshing Windows solves that problem, but doing that blows away the standard WSL/WSL2 image. Again. And again. And again! </p> <p> Dell replaced the motherboard 4 times in 2 years for one of my laptops. They simply do not understand how replacing the motherboard creates long-term issues. </p> <!-- endregion --> <!-- #region --> <h2 id="ready">Test Your Backup</h2> <p> Backing up WSL/WSL2 can be problematic. Restoring it is even more delicate. Knowing this, I decided to backup and test the restoration process before refreshing Windows and potentially losing my working system. </p> <p> I have tried several times before to make this work. As I said, it had been years since I allowed the OS in my main workstation to be refreshed. The reason I resisted doing proper maintenance was because I wanted to preserve my WSL2 Ubuntu image. Until today, I met with frustrating failures every time I attempted to use the standard tools. Today I succeeded, via new software, hence the publication of my notes. Who wants to read stories about all the things that do not work? </p> <p class="alert rounded shadow"> Following is my experience. I began by following the directions in <a href='https://www.windowscentral.com/how-backup-windows-subsystem-linux-wsl-distribution' target='_blank' rel="nofollow">How to back up a Windows Subsystem for Linux (WSL) distribution</a>, published 18 Feb 2021 by Windows Central. It is a nice story, but I've tried this stuff on a variety of computers, and this is not Microsoft&rsquo;s most robust code base. Read on and I will tell you of reality as I found it. </p> <!-- endregion --> <!-- #region --> <h2 id="wslOptions">WSL Command-Line Options</h2> <p> First let's look at the command-line options for the <code>wsl</code> command. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id89f5305e381c'><button class='copyBtn' data-clipboard-target='#id89f5305e381c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>Microsoft Windows [Version 10.0.19044.1415] (c) Microsoft Corporation. All rights reserved. </span><br> <span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --help <span class='unselectable'>Copyright (c) Microsoft Corporation. All rights reserved. For privacy information about this product please visit https://aka.ms/privacy.<br/> Usage: wsl.exe [Argument] [Options...] [CommandLine]<br/> Arguments for running Linux binaries:<br/> If no command line is provided, wsl.exe launches the default shell.<br/> --exec, -e &lt;CommandLine&gt; Execute the specified command without using the default Linux shell.<br/> --shell-type &lt;Type&gt; Execute the specified command with the provided shell type.<br/> Types: standard Execute the specified command using the default Linux shell.<br/> login Execute the specified command using the default Linux shell as a log-in shell.<br/> none Execute the specified command without using the default Linux shell.<br/> -- Pass the remaining command line as-is.<br/> Options: --cd &lt;Directory&gt; Sets the specified directory as the current working directory. If ~ is used the Linux user&#39;s home path will be used. If the path begins with a / character, it will be interpreted as an absolute Linux path. Otherwise, the value must be an absolute Windows path.<br/> --distribution, -d &lt;Distro&gt; Run the specified distribution.<br/> --user, -u &lt;UserName&gt; Run as the specified user.<br/> --system Launches a shell for the system distribution.<br/> Arguments for managing Windows Subsystem for Linux:<br/> --help Display usage information.<br/> --debug-shell Open a WSL2 debug shell for diagnostics purposes.<br/> --install [Distro] [Options...] Install a Windows Subsystem for Linux distribution. For a list of valid distributions, use &#39;wsl.exe --list --online&#39;.<br/> Options: --no-launch, -n Do not launch the distribution after install.<br/> --web-download Download the distribution from the internet instead of the Microsoft Store.<br/> --mount &lt;Disk&gt; Attaches and mounts a physical or virtual disk in all WSL 2 distributions.<br/> Options: --vhd Specifies that &lt;Disk&gt; refers to a virtual hard disk.<br/> --bare Attach the disk to WSL2, but don&#39;t mount it.<br/> --name &lt;Name&gt; Mount the disk using a customised name for the mount-point.<br/> --type &lt;Type&gt; Filesystem to use when mounting a disk, if not specified defaults to ext4.<br/> --options &lt;Options&gt; Additional mount options.<br/> --partition &lt;Index&gt; Index of the partition to mount, if not specified defaults to the whole disk.<br/> --set-default-version &lt;Version&gt; Changes the default install version for new distributions.<br/> --shutdown Immediately terminates all running distributions and the WSL 2 lightweight utility virtual machine.<br/> --status Show the status of Windows Subsystem for Linux.<br/> --unmount [Disk] Unmounts and detaches a disk from all WSL2 distributions. Unmounts and detaches all disks if called without argument.<br/> --update Update the Windows Subsystem for Linux package.<br/> Options: --web-download Download the update from the internet instead of the Microsoft Store.<br/> --pre-release Download a pre-release version if available. Implies --web-download.<br/> --version, -v Display version information.<br/> Arguments for managing distributions in Windows Subsystem for Linux:<br/> --export &lt;Distro&gt; &lt;FileName&gt; [Options] Exports the distribution to a tar file. The filename can be - for standard output.<br/> Options: --vhd Specifies that the distribution should be exported as a .vhdx file.<br/> --import &lt;Distro&gt; &lt;InstallLocation&gt; &lt;FileName&gt; [Options] Imports the specified tar file as a new distribution. The filename can be - for standard input.<br/> Options: --version &lt;Version&gt; Specifies the version to use for the new distribution.<br/> --vhd Specifies that the provided file is a .vhdx file, not a tar file. This operation makes a copy of the .vhdx file at the specified install location.<br/> --import-in-place &lt;Distro&gt; &lt;FileName&gt; Imports the specified .vhdx file as a new distribution. This virtual hard disk must be formatted with the ext4 filesystem type.<br/> --list, -l [Options] Lists distributions.<br/> Options: --all List all distributions, including distributions that are currently being installed or uninstalled.<br/> --running List only distributions that are currently running.<br/> --quiet, -q Only show distribution names.<br/> --verbose, -v Show detailed information about all distributions.<br/> --online, -o Displays a list of available distributions for install with &#39;wsl.exe --install&#39;.<br/> --set-default, -s &lt;Distro&gt; Sets the distribution as the default.<br/> --set-version &lt;Distro&gt; &lt;Version&gt; Changes the version of the specified distribution.<br/> --terminate, -t &lt;Distro&gt; Terminates the specified distribution.<br/> --unregister &lt;Distro&gt; Unregisters the distribution and deletes the root filesystem. </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region --> <h2 id="wslBackup">Backing Up With WSL</h2> <p> Now let's use the <code>wsl</code> command to back up the Ubuntu VM in WSL2. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3e637a3af0a9'><button class='copyBtn' data-clipboard-target='#id3e637a3af0a9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --export Ubuntu ubuntuBear_2021-01-10.tar</pre> </div> <p> Well, well, it backed up without any problem in about an hour! Color me <s>surprised</s>happy. File size was about 50 GB. </p> <p> Alright, let's try importing the image now. We'll locate the new image at <code>f:\ubuntuBear</code>. It used to be that WSL did not support moving or installing a distro to non-system drives. No longer! </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7f197f8fc197'><button class='copyBtn' data-clipboard-target='#id7f197f8fc197' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --import Ubuntu f:\ubuntuBear ubuntuBear_2021-01-10.tar <span class='unselectable'>A distribution with the supplied name already exists. </span></pre> </div> <p> The error message <code>A distribution with the supplied name already exists</code> makes sense because I backed up a WSL2 instance called <code>Ubuntu</code> and instance names must be unique. Let's import under the name <code>UbuntuBear</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7edbe1da124e'><button class='copyBtn' data-clipboard-target='#id7edbe1da124e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --import UbuntuBear f:\ubuntuBear ubuntuBear_2021-01-10.tar <span class='unselectable'>Unspecified error </span></pre> </div> <p> Ahh, the dreaded <code>Unspecified error</code> that <code>wsl import</code> is infamous for. I found a few potential solutions, detailed below; I show the most desirable solution first. </p> <!-- endregion --> <!-- #region --> <h2 id="vhdx">Solution 1: Importing the <span class="code">VDHX File</span></h2> <p> Importing the original <code>vhdx</code> file directly instead of creating and importing the <code>tar</code> file has been reported to work by <a href='https://github.com/microsoft/WSL/issues/4735#issuecomment-1370093052' target='_blank' rel="nofollow">several people</a>. This is a more desirable solution because it requires less work and is faster. Note that the <kbd>^</kbd> character is a DOS command-line continuation character, much like the Bash <kbd>\</kbd> character. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc58d5e435a78'><button class='copyBtn' data-clipboard-target='#idc58d5e435a78' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --import Ubuntu-20.04-d^ D:\WSL\Ubuntu-22.04\^ %HOMEDRIVE%%HOMEPATH%\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu20.04onWindows_79rhkp1fndgsc\LocalState\ext4.vhdx^ --vhd</pre> </div> <!-- endregion --> <!-- #region --> <h2 id="vhdx">Solution 2: Updating Linux Kernel</h2> <p> <a href='https://github.com/microsoft/WSL/issues/4735' target='_blank' rel="nofollow">The following</a> was written by <a href='https://github.com/ken-cdit' target='_blank' rel="nofollow"><code>ken-cdit</code></a>: </p> <p> Had the same issue. I exported two WSL 2 distros and then did a clean install of my Windows OS. "Unspecified error" came up when I tried to import them. </p> <p> I fixed it by downloading the Linux kernel update package referred to at <a href='https://https://docs.microsoft.com/en-us/windows/wsl/install-win10' target='_blank' rel="nofollow"><code>https:/<wbr>/<wbr>docs.microsoft.com/<wbr>en-us/<wbr>windows/<wbr>wsl/<wbr>install-win10</code></a> and then running the command <code>wsl --set-default-version 2</code>. </p> <p> After this, my two distros imported without error. </p> <p> I had the whole thing still in the console so here is a screencap... </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/wslBackup/solution.svg" type="image/svg"> <!---<source srcset="/blog/images/wslBackup/solution.avif" type="image/avif">--> <source srcset="/blog/images/wslBackup/solution.webp" type="image/webp"> <source srcset="/blog/images/wslBackup/solution.apng" type="image/apng"> <source srcset="/blog/images/wslBackup/solution.png" type="image/png"> <source srcset="/blog/images/wslBackup/solution.jpg" type="image/jpeg"> <source srcset="/blog/images/wslBackup/solution.jpeg" type="image/jpeg"> <source srcset="/blog/images/wslBackup/solution.jfif" type="image/jpeg"> <source srcset="/blog/images/wslBackup/solution.pjpeg" type="image/jpeg"> <source srcset="/blog/images/wslBackup/solution.pjp" type="image/jpeg"> <source srcset="/blog/images/wslBackup/solution.gif" type="image/gif"> <source srcset="/blog/images/wslBackup/solution.tif" type="image/tiff"> <source srcset="/blog/images/wslBackup/solution.tiff" type="image/tiff"> <source srcset="/blog/images/wslBackup/solution.bmp" type="image/bmp"> <source srcset="/blog/images/wslBackup/solution.ico" type="image/x-icon"> <source srcset="/blog/images/wslBackup/solution.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/wslBackup/solution.png" style='width: 100%; ' /> </picture> </div> <!-- endregion --> <!-- #region --> <h2 id="LxRunOfflineInstall">Solution 3: <span class="code">LxRunOffline</span></h2> <h3 id="LxRunOfflineInstall">Installing <span class="code">LxRunOffline</span></h3> <p> <a href='https://github.com/microsoft/WSL/issues/4735#issuecomment-800103777' target='_blank' rel="nofollow">Google brought me</a> to a potential solution, <a href='https://github.com/DDoSolitary/LxRunOffline' target='_blank' rel="nofollow"><code>LxRunOffline</code></a>, which provided me limited success. </p><p> I downloaded the binaries in <a href='https://github.com/DDoSolitary/LxRunOffline/releases/download/v3.5.0/LxRunOffline-v3.5.0-msvc.zip' target='_blank' rel="nofollow"><code>LxRunOffline-v3.5.0-msvc.zip</code></a> directly from <a href='https://github.com/DDoSolitary/LxRunOffline/releases' target='_blank' rel="nofollow">GitHub</a> into <code>C:\Program Files\LxRunOffline</code>. </p> <p> Now add <code>C:\Program Files\LxRunOffline</code> to the Windows <code>PATH</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id221749f8f435'><button class='copyBtn' data-clipboard-target='#id221749f8f435' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>setx PATH "%PATH%;C:\Program Files\LxRunOffline" SUCCESS: Specified value was saved. %}</pre> </div> <p> I then ran the following in an administrative shell to register the DLL: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc1539e3f97ec'><button class='copyBtn' data-clipboard-target='#idc1539e3f97ec' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>regsvr32 "C:\Program Files\LxRunOffline\LxRunOfflineShellExt.dll"</pre> </div> <!-- endregion --> <!-- #region --> <h3 id="LxRunOffline" class="code">LxRunOffline Help Info</h3> <p> Let's progressively discover how this command-line program can be used. First I'll just type the command name, which causes the program to list its top-level actions. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id95688e3833dc'><button class='copyBtn' data-clipboard-target='#id95688e3833dc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline <span class='unselectable'>[ERROR] No action is specified.<br> Supported actions are: l, list List all installed distributions. gd, get-default Get the default distribution, which is used by bash.exe. sd, set-default Set the default distribution, which is used by bash.exe. i, install Install a new distribution. ui, uninstall Uninstall a distribution. rg, register Register an existing installation directory. ur, unregister Unregister a distribution but not delete the installation directory. m, move Move a distribution to a new directory. d, duplicate Duplicate an existing distribution in a new directory. e, export Export a distribution&quot;s filesystem to a .tar.gz file, which can be imported by the "install" command. r, run Run a command in a distribution. di, get-dir Get the installation directory of a distribution. gv, get-version Get the filesystem version of a distribution. ge, get-env Get the default environment variables of a distribution. se, set-env Set the default environment variables of a distribution. ae, add-env Add to the default environment variables of a distribution. re, remove-env Remove from the default environment variables of a distribution. gu, get-uid Get the UID of the default user of a distribution. su, set-uid Set the UID of the default user of a distribution. gk, get-kernelcmd Get the default kernel command line of a distribution. sk, set-kernelcmd Set the default kernel command line of a distribution. gf, get-flags Get some flags of a distribution. See https://docs.microsoft.com/en-us/previous-versions/windows/desktop/api/wslapi/ne-wslapi-wsl_distribution_flags for details. sf, set-flags Set some flags of a distribution. See https://docs.microsoft.com/en-us/previous-versions/windows/desktop/api/wslapi/ne-wslapi-wsl_distribution_flags for details. s, shortcut Create a shortcut to launch a distribution. ec, export-config Export configuration of a distribution to an XML file. ic, import-config Import configuration of a distribution from an XML file. sm, summary Get general information of a distribution. version Get version information about this LxRunOffline.exe. </span></pre> </div> <!-- endregion --> <p> Alright, let's get information about the <code>i</code> (<code>install</code>) option. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1c6003ec9b80'><button class='copyBtn' data-clipboard-target='#id1c6003ec9b80' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline i <span class='unselectable'>[ERROR] the option '-d' is required but missing<br> Options: -n arg Name of the distribution -d arg The directory to install the distribution into. -f arg The tar file containing the root filesystem of the distribution to be installed. If a file of the same name with a .xml extension exists and "-c" isn&quot;t specified, that file will be imported as a config file. -r arg The directory in the tar file to extract. This argument is optional. -c arg The config file to use. This argument is optional. -v arg (=2) The version of filesystem to use, latest available one if not specified. -s Create a shortcut for this distribution on Desktop. </span></pre> </div> <!-- endregion --> <p> I'll use the <code>-d</code> option to specify the directory to install into, as well as the <code>-f</code> and <code>-n</code> options. </p> <!-- endregion --> <!-- #region --> <h3 id="LxRunOfflineImport"><span class="code">LxRunOffline</span> Import</h3> <p> I feel brave, let's try importing for real now: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id144f64bef595'><button class='copyBtn' data-clipboard-target='#id144f64bef595' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline i -d f:/ubuntuBear -n UbuntuBear -f ubuntuBear_2021-01-10.tar <span class='unselectable'>[ERROR] The distro "UbuntuBear" already exists. </span></pre> </div> <p> Some debris remains from the failed import (remember the <code>Unspecified error</code> a moment ago?). I will delete the b0rked <code>UbuntuBear</code> instance in a moment. Let's call this newly cloned linux instance <code>UbuntuBear2</code>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide53daeebdb45'><button class='copyBtn' data-clipboard-target='#ide53daeebdb45' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline i -d f:/ubuntuBear -n UbuntuBear2 -f ubuntuBear_2021-01-10.tar <span class='unselectable'>[WARNING] Ignoring an unsupported file "var/lib/docker/volumes/backingFsBlockDev" of type 0060000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-17041-8893592-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10716-846285-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1899-75308257-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10151-8749190-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-18789-129803847-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-811-127664322-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12447-115564106-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-24480-65839207-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-23230-107496852-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-23230-107496852-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-24480-65839207-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-14211-5836039-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19778-110293600-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1969-34761241-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-17041-8893592-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1899-75308257-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-7642-17996-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10151-8749190-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19082-8772885-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12857-5829248-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-26692-70009588-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12447-115564106-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-22582-1181861-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-14211-5836039-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-1969-34761241-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-7642-17996-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10716-846285-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-11424-57667328-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-28497-49814437-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-18789-129803847-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-2234-80045-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19082-8772885-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-12857-5829248-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-22582-1181861-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-26692-70009588-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10990-5814132-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-811-127664322-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-10990-5814132-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-11424-57667328-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-2234-80045-out" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-19778-110293600-in" of type 0010000. [WARNING] Ignoring an unsupported file "tmp/clr-debug-pipe-28497-49814437-in" of type 0010000. [WARNING] Ignoring an unsupported file "dev/random" of type 0020000. [WARNING] Ignoring an unsupported file "dev/tty" of type 0020000. [WARNING] Ignoring an unsupported file "dev/full" of type 0020000. [WARNING] Ignoring an unsupported file "dev/urandom" of type 0020000. [WARNING] Ignoring an unsupported file "dev/ptmx" of type 0020000. [WARNING] Ignoring an unsupported file "dev/zero" of type 0020000. [WARNING] Ignoring an unsupported file "dev/console" of type 0020000. [WARNING] Ignoring an unsupported file "dev/null" of type 0020000. [WARNING] Ignoring an unsupported file "dev/mapper/control" of type 0020000. [ERROR] Couldn&quot;t create the file "\\?\f:\ubuntuBear\rootfs\home\mslinn\.atom\packages\markdown-preview-plus\spec\fixtures\subdir\�cc�nt�d.md". Reason: The file exists. </span></pre> </div> <!-- endregion --> <p> Most of the warnings do not seem to be important. I do not care about Docker anyway. Also, now I know that temporary files and <code>dev</code> nodes should all be deleted before running this program. Even better, the program that created the tar should be modified to not attempt to replicate any temporary files. </p> <p> I don't understand the problem with the atom package, but I'll try to delete all of the atom settings from the tar, along with the other problematic directories, and then retry. </p> <!-- endregion --> <!-- #region --> <h3 id="cleanup">Cleaning Up WSL</h3> <p> First let's see the VMs registered with WSL: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3e7a5efa4d57'><button class='copyBtn' data-clipboard-target='#id3e7a5efa4d57' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl -l <span class='unselectable'>Windows Subsystem for Linux Distributions: Ubuntu (Default) UbuntuBear UbuntuBear2 </span></pre> </div> <p> Let's delete the debris remaining from the failed <code>UbuntuBear</code> and <code>UbuntuBear2</code> imports: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide92a21621e0c'><button class='copyBtn' data-clipboard-target='#ide92a21621e0c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --unregister UbuntuBear <span class='unselectable'>Unregistering... </span> <span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --unregister UbuntuBear2 <span class='unselectable'>Unregistering... </span></pre> </div> <p> Attempting to delete the directory created by LxRunOffline hit a corrupted directory. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2a54a01e86de'><button class='copyBtn' data-clipboard-target='#id2a54a01e86de' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>del /s /q f:\ubuntuBear <span class='unselectable'>Deleted file - f:\ubuntuBear\rootfs\init Deleted file - f:\ubuntuBear\rootfs\lib Deleted file - f:\ubuntuBear\rootfs\lib32 Deleted file - f:\ubuntuBear\rootfs\lib64 Deleted file - f:\ubuntuBear\rootfs\libx32 Deleted file - f:\ubuntuBear\rootfs\sbin Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.wget-hsts Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.Xauthority Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zcompdump Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zcompdump-Bear-5.8 Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zcompdump-localhost-5.8 Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zshenv Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zshrc Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zshrc.pre-oh-my-zsh Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.zsh_history Deleted file - f:\ubuntuBear\rootfs\home\mslinn\ancient_warmth_workspace.code-workspace Deleted file - f:\ubuntuBear\rootfs\home\mslinn\bear2.zip Deleted file - f:\ubuntuBear\rootfs\home\mslinn\bear3.zip Deleted file - f:\ubuntuBear\rootfs\home\mslinn\bearDirs.tar Deleted file - f:\ubuntuBear\rootfs\home\mslinn\dead.letter Deleted file - f:\ubuntuBear\rootfs\home\mslinn\django_bash_completion Deleted file - f:\ubuntuBear\rootfs\home\mslinn\jekyll_workspace.code-workspace Deleted file - f:\ubuntuBear\rootfs\home\mslinn\msp.txt Deleted file - f:\ubuntuBear\rootfs\home\mslinn\nodesource_setup.sh Deleted file - f:\ubuntuBear\rootfs\home\mslinn\package-lock.json Deleted file - f:\ubuntuBear\rootfs\home\mslinn\package.json Deleted file - f:\ubuntuBear\rootfs\home\mslinn\worldPeaceMusicCollective.png Deleted file - f:\ubuntuBear\rootfs\home\mslinn\worldPeaceMusicCollectiveBordered.png Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.vscode-server\extensions\wix.vscode-import-cost-2.15.0\package.json Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.vscode-server\extensions\wix.vscode-import-cost-2.15.0\pom.xml Deleted file - f:\ubuntuBear\rootfs\home\mslinn\.vscode-server\extensions\wix.vscode-import-cost-2.15.0\README.md The file or directory is corrupted and unreadable. </span></pre> </div> <!-- endregion --> <p> &ldquo;The file or directory is corrupted and unreadable.&rdquo; That needs to be dealt with right away! I fixed the directory errors in drive F like this: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6871ac2f5c4e'><button class='copyBtn' data-clipboard-target='#id6871ac2f5c4e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Program Files> </span>chkdsk /F F: <span class='unselectable'>The type of the file system is NTFS. Chkdsk cannot run because the volume is in use by another process. Chkdsk may run if this volume is dismounted first. ALL OPENED HANDLES TO THIS VOLUME WOULD THEN BE INVALID. </span>y <span class='unselectable'>Chkdsk cannot dismount the volume because it is a system drive or there is an active paging file on it. Would you like to schedule this volume to be checked the next time the system restarts? (Y/N)</span> y <span class='unselectable'>This volume will be checked the next time the system restarts. </span></pre> </div> <p> I rebooted the system, and it fixed the errors on drive F:. </p> <!-- endregion --> <!-- #region --> <h3 id="prune">Pruning the tar</h3> <p> First lets back up the tar that we want to prune. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6a929a5bf00e'><button class='copyBtn' data-clipboard-target='#id6a929a5bf00e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pushd '/mnt/c/Users/Mike Slinn/'<br> <span class='unselectable'>$ </span>ls -alF *.tar <span class='unselectable'>-rwxr--r-- 1 mslinn mslinn 524482560 Jan 10 18:45 'bear_ubuntu_2021-01-10.tar'* -rwxr--r-- 1 mslinn mslinn 51464028160 Jan 10 21:11 'ubuntuBear_2021-01-10.tar'* </span><br> <span class='unselectable'>$ </span>cp ubuntuBear_2021-01-10.tar ubuntuBearPruned_2021-01-10.tar</pre> </div> <p> Now lets try to prune out all the problematic, and unnecessary, files. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id270e14632a44'><button class='copyBtn' data-clipboard-target='#id270e14632a44' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>tar -f ubuntuBearPruned_2021-01-10.tar \ --delete dev/* \ --delete tmp/* \ --delete home/mslinn/.atom/* \ --delete var/lib/docker/* \ --delete var/sitesUbuntu/* \ --delete var/work/* \ --delete var/tmp/* <span class='unselectable'>tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.fuseoverlayfs.opaque' tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.fuseoverlayfs.opaque' tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: dev/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: tmp/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: home/mslinn/.atom/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: var/lib/docker/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: var/sitesUbuntu/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: var/work/*: Not found in archive tar: Pattern matching characters used in file names tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning tar: var/tmp/*: Not found in archive tar: Exiting with failure status due to previous errors </span></pre> </div> <!-- endregion --> <p> I think the error messages just indicate that the tar was made by <a href='https://www.freebsd.org/cgi/man.cgi?tar(1)' target='_blank' rel="nofollow">BSD-TAR</a>, while Ubuntu uses <a href='https://www.gnu.org/software/tar/' target='_blank' rel="nofollow">GNU-TAR</a>. I installed <code>bsdtar</code> like this: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id841273e808e5'><button class='copyBtn' data-clipboard-target='#id841273e808e5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt-get install <a href='https://packages.ubuntu.com/hirsute/libarchive-tools' target='_blank' rel="nofollow">libarchive-tools</a> <span class='unselectable'>Reading package lists... Done Building dependency tree... Done Reading state information... Done The following NEW packages will be installed: libarchive-tools 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 57.1 kB of archives. After this operation, 207 kB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu hirsute/universe amd64 libarchive-tools amd64 3.4.3-2 [57.1 kB] Fetched 57.1 kB in 0s (167 kB/s) Selecting previously unselected package libarchive-tools. (Reading database ... 244498 files and directories currently installed.) Preparing to unpack .../libarchive-tools_3.4.3-2_amd64.deb ... Unpacking libarchive-tools (3.4.3-2) ... Setting up libarchive-tools (3.4.3-2) ... Processing triggers for man-db (2.9.4-2) ... </span></pre> </div> <!-- endregion --> <p> I <a href='https://stackoverflow.com/a/56031400/553865' target='_blank' rel="nofollow">tried again</a> using <code>bsdtar</code>, which does not support GNU tar's <code>--delete</code> option. Unfortunately, I only got a 1KB file out, no matter what I did. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5f409277a38d'><button class='copyBtn' data-clipboard-target='#id5f409277a38d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>bsdtar -cvf ubuntuBearPruned2_2021-01-10.tar \ --exclude 'dev/*' \ --exclude 'tmp/*' \ --exclude 'home/mslinn/.atom/*' \ --exclude 'var/lib/docker/*' \ --exclude 'var/sitesUbuntu/*' \ --exclude 'var/work/*' \ --exclude 'var/tmp/*' \ @ubuntuBear_2021-01-10.tar</pre> </div> <p> Maybe there is a bug. Maybe there is bad documentation. I do not think I made an error. Life is short. Bugs are rampant. <a href='https://en.wikipedia.org/wiki/Illegitimi_non_carborundum' target='_blank' rel="nofollow">Illegitimi non carborundum!</a> </p> <p> I give up on this direction. Let's try to win some other way! </p> <!-- endregion --> <!-- #region --> <h3 id="duplicate">Success Duplicating the Ubuntu Instance</h3> <p> Let&rsquo;s try duplicating the Ubuntu instance with <code>LxRunOffline</code>. First the <code>Ubuntu</code> VM must be terminated. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id774c7f153d7d'><button class='copyBtn' data-clipboard-target='#id774c7f153d7d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl -t Ubuntu<br> <span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline duplicate -n Ubuntu -N UbuntuWsl2 -d f:\UbuntuWsl2</pre> </div> <p> The above ran for a couple of hours and concluded without error. <code>LxRunOffline duplicate</code> automatically registers the new instance. </p> <p> The <code>F:\UbuntuWsl2</code>directory was 239 GB. That's a fair-sized VM. The directory only contains one file, called <code>ext4.vhdx</code>. </p> <p> Let's see the Ubuntu instances that are currently set: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' style='margin-bottom: 1em;' id='id51a5fc01c6e6'><button class='copyBtn' data-clipboard-target='#id51a5fc01c6e6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --list <span class='unselectable'>Windows Subsystem for Linux Distributions: Ubuntu (Default) UbuntuWsl2 </span></pre> </div> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <p> That is what I wanted to see! I started the new <code>UbuntuWsl2</code> Ubuntu instance like this: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer clear' id='id043a0702c2d4'><button class='copyBtn' data-clipboard-target='#id043a0702c2d4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl -d UbuntuWsl2</pre> </div> <p> To set the default Ubuntu instance I typed: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf12f1bfd4e61'><button class='copyBtn' data-clipboard-target='#idf12f1bfd4e61' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --setdefault UbuntuWsl2</pre> </div> <p> Let's verify that worked: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id46895e4b6b5e'><button class='copyBtn' data-clipboard-target='#id46895e4b6b5e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --list <span class='unselectable'>Windows Subsystem for Linux Distributions: UbuntuWsl2 (Default) Ubuntu </span></pre> </div> <p> I then deleted the huge tar files that I had created in earlier steps. I never needed them because I used <code>LxRunOffline duplicate</code>. </p> <!-- endregion --> <!-- #region --> <h2 id="usb3">Ubuntu on 500GB USB3 SSD</h2> <div class='imgWrapper imgFlex right' style=' '> <a href='https://www.amazon.com/gp/product/B08GTXVG9P' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/wslBackup/sandiskExtreme.svg" type="image/svg"> <!---<source srcset="/blog/images/wslBackup/sandiskExtreme.avif" type="image/avif">--> <source srcset="/blog/images/wslBackup/sandiskExtreme.webp" type="image/webp"> <source srcset="/blog/images/wslBackup/sandiskExtreme.apng" type="image/apng"> <source srcset="/blog/images/wslBackup/sandiskExtreme.png" type="image/png"> <source srcset="/blog/images/wslBackup/sandiskExtreme.jpg" type="image/jpeg"> <source srcset="/blog/images/wslBackup/sandiskExtreme.jpeg" type="image/jpeg"> <source srcset="/blog/images/wslBackup/sandiskExtreme.jfif" type="image/jpeg"> <source srcset="/blog/images/wslBackup/sandiskExtreme.pjpeg" type="image/jpeg"> <source srcset="/blog/images/wslBackup/sandiskExtreme.pjp" type="image/jpeg"> <source srcset="/blog/images/wslBackup/sandiskExtreme.gif" type="image/gif"> <source srcset="/blog/images/wslBackup/sandiskExtreme.tif" type="image/tiff"> <source srcset="/blog/images/wslBackup/sandiskExtreme.tiff" type="image/tiff"> <source srcset="/blog/images/wslBackup/sandiskExtreme.bmp" type="image/bmp"> <source srcset="/blog/images/wslBackup/sandiskExtreme.ico" type="image/x-icon"> <source srcset="/blog/images/wslBackup/sandiskExtreme.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/wslBackup/sandiskExtreme.png" style='width: 100%; ' /> </picture> </a> </div> <p> I have a <a href='https://www.amazon.com/gp/product/B08GTXVG9P' target='_blank' rel="nofollow">Sandisk Extreme 500GB USB3 SSD drive</a>. This tiny, light, and very portable storage device should be perfect for holding my Ubuntu development system. How cool it would be to be able to drop it into a small pocket! </p> <p> Better yet, the <a href='https://www.quora.com/Can-a-USB-3-0-external-SSD-be-faster-than-an-internal-laptop%E2%80%99s-HDD' target='_blank' rel="nofollow">performance of portable USB3 SSD drives</a> blows away traditional hard drives. </p> <p> For example, assume that your SSD drive appears as drive <code>X</code>. Also assume that <code>LxRunOffline.exe</code> and <code>LxRunOffline.dll</code> have been copied to the root directory of the SSD drive. To register your portable Ubuntu you would simply type: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3d3b6d3c0014'><button class='copyBtn' data-clipboard-target='#id3d3b6d3c0014' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\> </span>X:LxRunOffline register X:\MyUbuntu -n UbuntuMSlinn</pre> </div> <div class="pullQuote"> You can walk up to any recently updated Windows 10 machine, plug your tiny USB3 SSD drive into a USB3 or USB3.1 port, run the registration command, and boom! you are productive with great storage performance. </div> <div style='font-size: 3em;;text-align: center'>&#x1F601;</div> <!-- endregion --> <!-- #region --> <h2 id="duplicate">Duplicating to SSD</h2> <p> Again, the <code>Ubuntu</code> VM must be terminated. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id65bf63927f81'><button class='copyBtn' data-clipboard-target='#id65bf63927f81' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl -t Ubuntu<br> <span class='unselectable'>C:\Users\Mike Slinn> </span>LxRunOffline duplicate -n Ubuntu -N UbuntuWsl2Extreme -d I:\UbuntuWsl2Extreme</pre> </div> <p> The above ran for a couple of hours and concluded without error. <code>LxRunOffline duplicate</code> automatically registers the new instance. </p> <p> Let's see the Ubuntu instances that are currently set: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idde42fe55286a'><button class='copyBtn' data-clipboard-target='#idde42fe55286a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>wsl --list <span class='unselectable'>Windows Subsystem for Linux Distributions: Ubuntu (Default) UbuntuWsl2 UbuntuWsl2Extreme </span></pre> </div> <div style='font-size: 3em;;text-align: center'>&#x1F601;</div> <!-- endregion --> <!-- #region --> <h2 id="LxRunOfflineUpdate"><span class="code">LxRunOffline Update 2023-04-25</span></h2> <p> <a href='https://github.com/armanexplorer' target='_blank' rel="nofollow">Arman Mazloumzadeh</a> posted the <a href='https://github.com/microsoft/WSL/issues/4735#issuecomment-1522509689' target='_blank' rel="nofollow">following</a>: </p> <blockquote> <p> I finally resolved this amiss during the following steps (Windows 10.0.19045 Build 19045): </p> <!-- #region --> <div class="jekyll_pre" > <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd691c0fbd3c2'><button class='copyBtn' data-clipboard-target='#idd691c0fbd3c2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>Running wsl.exe --import Ubuntu A:\wsl\Ubuntu A:\wsl\Ubuntu.tar = > Unspecified error Running LxRunOffline.exe install -n Ubuntu -d A:\wsl\Ubuntu -f A:\wsl\Ubuntu.tar => [ERROR] Couldn't get the value "DistributionName" of the registry key "Software\Microsoft\Windows\CurrentVersion\Lxss\AppxInstallerCache" Removing the Software\Microsoft\Windows\CurrentVersion\Lxss\AppxInstallerCache registry key Running LxRunOffline.exe install -n Ubuntu -d A:\wsl\Ubuntu -f A:\wsl\Ubuntu.tar => [ERROR] Couldn't get the value "DistributionName" of the registry key "Software\Microsoft\Windows\CurrentVersion\Lxss\TryStoreWSL" Removing the Software\Microsoft\Windows\CurrentVersion\Lxss\TryStoreWSL registry key Running LxRunOffline.exe install -n Ubuntu -d A:\wsl\Ubuntu -f A:\wsl\Ubuntu.tar => ✔ Running ubuntu.exe config --default-user arman => ✔ done 🎉</pre> </div> <!-- endregion --> </blockquote> <!-- endregion --> <!-- #region --> <h2 id="startStop">Starting and Stopping Portable VMs</h2> <p> You can start the WSL VM called <code>UbuntuMSlinn</code> by using <code>wsl -d</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id35ccc7e98ba0'><button class='copyBtn' data-clipboard-target='#id35ccc7e98ba0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\> </span>wsl -d UbuntuMSlinn</pre> </div> <p> You can stop it by using <code>wsl -t</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id388cece3c6da'><button class='copyBtn' data-clipboard-target='#id388cece3c6da' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\> </span>wsl -t UbuntuMSlinn</pre> </div> <!-- endregion --> <!-- #region --> <h2 id="issues">Microsoft WSL Issue Reporting</h2> <p> In the course of writing this article I found the <a href='https://github.com/microsoft/WSL/blob/master/CONTRIBUTING.md#8-collect-wsl-logs' target='_blank' rel="nofollow">web page</a> for reporting a WSL issue, so that Microsoft can look at it. <!-- endregion --> AI / ML System Behavior Reflects the Society That Produced It 2021-12-26T00:00:00-05:00 https://mslinn.github.io/blog/2021/12/26/ai-society <p> Artificial intelligence systems, including machine learning systems, are primarily software-driven. Like all software, AI and ML systems reflect the societies that produced them. <a href='https://www.thoughtworks.com/insights/articles/demystifying-conways-law' target='_blank' rel="nofollow">Conway&rsquo;s Law</a> explains how the internal organization of software reflects the organization that created it. </p> <p> I postulate that the behavior of AI & ML systems reflect the society that the systems are embedded in. Do you find the AI systems are <a href='https://www.bruegel.org/blog-post/dark-side-artificial-intelligence-manipulation-human-behaviour' target='_blank' rel="nofollow">instrusive or exploitive</a>? Look in the collective mirror for the society it was created by. </p> <p class="alert rounded shadow"> If the populace is viewed as merely something for corporations to exploit, and this view is accepted at enough levels in society, that becomes the status quo. In my opinion, the USA in 2022 has definitely reached that point. </p> <div class="quoteCite shadow rounded"> <h2 style="margin-top: 0">Conway&rsquo;s Law</h2> Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure. <br><br> &nbsp;&ndash; Melvin E. Conway, 1967 </div> <div class="quoteCite shadow rounded"> <h2 style="margin-top: 0">Allan Kelly’s Corollary</h2> Organisational design is system design. <br><br> &nbsp;&ndash; Allan Kelly, 2005 </div> <p> The <a href='https://www.hbs.edu/ris/Publication%20Files/16-124_7ae90679-0ce6-4d72-9e9d-828872c7af49.pdf' target='_blank' rel="nofollow">The Mirroring Hypothesis: Theory, Evidence and Exceptions</a> by Lyra J. Colfer and Carliss Y. Baldwin was published by the Harvard Business School in 2016. </p> <div class="quoteCite shadow rounded"> A large-scale system is one that stretches across time and space. A large organization similarly stretches across time and space. Conway’s Law ties the two. How we organize defines how we think collectively, and thus what we make collectively...<br><br> ... To build software, first build a community...<br><br> Software is essentially about people, and the larger-scale we make it, the more that truth becomes visible. One program reflects how one person thinks. A large-scale application reflects how many people think together. <br><br> &nbsp;&ndash; Pieter Hintjes, <a href='http://hintjens.com/blog:73' target='_blank' rel="nofollow">Sex in Title, and Other Stories</a>, Dec 16, 2013 </div> <p> Do you even care? If so, please be the change you want to see in the world. </p> <div class='quote'> <div class='imgWrapper imgFlex right quartersize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/gandhi.svg" type="image/svg"> <!---<source srcset="/blog/images/gandhi.avif" type="image/avif">--> <source srcset="/blog/images/gandhi.webp" type="image/webp"> <source srcset="/blog/images/gandhi.apng" type="image/apng"> <source srcset="/blog/images/gandhi.png" type="image/png"> <source srcset="/blog/images/gandhi.jpg" type="image/jpeg"> <source srcset="/blog/images/gandhi.jpeg" type="image/jpeg"> <source srcset="/blog/images/gandhi.jfif" type="image/jpeg"> <source srcset="/blog/images/gandhi.pjpeg" type="image/jpeg"> <source srcset="/blog/images/gandhi.pjp" type="image/jpeg"> <source srcset="/blog/images/gandhi.gif" type="image/gif"> <source srcset="/blog/images/gandhi.tif" type="image/tiff"> <source srcset="/blog/images/gandhi.tiff" type="image/tiff"> <source srcset="/blog/images/gandhi.bmp" type="image/bmp"> <source srcset="/blog/images/gandhi.ico" type="image/x-icon"> <source srcset="/blog/images/gandhi.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/gandhi.png" style='width: 100%; ' /> </picture> </div> We but mirror the world. All the tendencies present in the outer world are to be found in the world of our body. If we could change ourselves, the tendencies in the world would also change. As a man changes his own nature, so does the attitude of the world change towards him. This is the divine mystery supreme. A wonderful thing it is and the source of our happiness. We need not wait to see what others do. <span class='quoteAttribution'> &nbsp;&ndash; From <a href='https://commonground.ca/be-the-change-you-want-to-see-in-the-world/' rel='nofollow' target='_blank'>Mahatma Gandhi</a></span> </div> Spring-Breezifier: Solving COVID-19 With HVAC 2021-12-22T00:00:00-05:00 https://mslinn.github.io/blog/2021/12/22/covid-hvac <!-- #region intro --> <p> When the COVID-19 pandemic began, the world was unprepared and no one knew how the virus was transmitted. As the medical profession lurched awkwardly into action, it gave advice that later turned out to be incorrect. As time went by, important details about how the virus is transmitted were revealed, and the previous medical advice had become politicized. The new advice has not yet been generally adopted as policy and is currently unknown to most people. </p> <div class='quote'> <div class='quoteText clearfix'> <div class='imgWrapper imgFlex right quartersize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/covidHvac/mencken.svg" type="image/svg"> <!---<source srcset="/blog/images/covidHvac/mencken.avif" type="image/avif">--> <source srcset="/blog/images/covidHvac/mencken.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/mencken.apng" type="image/apng"> <source srcset="/blog/images/covidHvac/mencken.png" type="image/png"> <source srcset="/blog/images/covidHvac/mencken.jpg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/mencken.jpeg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/mencken.jfif" type="image/jpeg"> <source srcset="/blog/images/covidHvac/mencken.pjpeg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/mencken.pjp" type="image/jpeg"> <source srcset="/blog/images/covidHvac/mencken.gif" type="image/gif"> <source srcset="/blog/images/covidHvac/mencken.tif" type="image/tiff"> <source srcset="/blog/images/covidHvac/mencken.tiff" type="image/tiff"> <source srcset="/blog/images/covidHvac/mencken.bmp" type="image/bmp"> <source srcset="/blog/images/covidHvac/mencken.ico" type="image/x-icon"> <source srcset="/blog/images/covidHvac/mencken.cur" type="image/x-icon"> <img class="imgImg rounded" src="/blog/images/covidHvac/mencken.png" style='width: 100%; ' /> </picture> </div> For every complex problem there is an answer that is clear, simple, and wrong. <br><br><br><br><br><br> </div><div class='quoteAttribution'> &nbsp;&ndash; <a href='https://www.britannica.com/biography/H-L-Mencken' rel='nofollow' target='_blank'>H. L. Mencken 1880-1956.</a></div> </div> <!-- endregion --> <!-- #region about --> <h2 id="about">About This Article</h2> <p> This article's intent is to show how the COVID-19 pandemic could be dealt with simply, cheaply, and with a minimum of aggravation by raising awareness of the potential role of <a href='https://www.cdc.gov/infectioncontrol/guidelines/environmental/background/air.html#c3' target='_blank' rel="nofollow">HVAC</a> (heating, ventilation and air conditioning) equipment. </p> <p> The author is an electrical engineer with no health-related or HVAC-related qualifications. However, he can read and write, ideas often come to mind when presented with new information. </p> <div class="formalNotice rounded shadow"> After initially publishing this article, an old friend showed me a YouTube video made by a consulting civil engineer who specializes in this topic. <a href="#youtube2">You can skip to it if you are impatient</a> and want to know in-depth technical specifics. </div> <p> All the new information referenced in this article is generally available, from qualified and reputable sources, and this article just disseminates it. </p> <p> This article concludes with actionable suggestions. </p> <!-- endregion --> <!-- #region update --> <h2 id="update">Update 2023-06-08</h2> <div class='quote'> <p> [The] federal agency has set a target - five air changes per hour - for how much rooms and buildings should be ventilated...<br><br> it&rsquo;s easy to see the guidance only in the context of Covid-19, it will help with many other airborne hazards like wildfire smoke, allergens and other infectious diseases, such as the flu...<br><br> &ldquo;If they had broadcast and implemented these changes at the beginning, there never would have been a pandemic&rdquo;, said Kimberly Prather, an atmospheric chemist at the University of California at San Diego and the Scripps Institution of Oceanography. </p> <span class='quoteAttribution'> &nbsp;&ndash; From <a href='https://www.cnn.com/2023/05/12/health/cdc-new-ventilation-target/index.html' rel='nofollow' target='_blank'>CDC sets first target for indoor air ventilation to prevent spread of Covid-19</a></span> </div> <!-- endregion --> <!-- #region aerosols --> <h2 id="trans">COVID-19 Is Primarily Transmitted Via Aerosols</h2> <p> The most significant bit of information about COVID-19, which was not generally accepted at the beginning of the pandemic, is that it is <a href='https://www.nature.com/articles/d41586-021-00251-4' target='_blank' rel="nofollow">mostly transmitted via aerosols</a>; that is, via small airborne droplets only a few microns in diameter. (One micron is a millionth of a meter, or one twenty-five thousandth of an inch). Aerosol particles can hang in the air for hours and travel hundreds of feet. </p> <p> The Centers for Disease Control and Prevention (CDC) officially recently officially recognized that SARS-CoV-2 (the virus that causes COVID-19) is airborne, meaning it is highly transmissible through the air. </p> <div class="quoteCite shadow rounded"> <div class='imgWrapper imgFlex right' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/covidHvac/cdc.svg" type="image/svg"> <!---<source srcset="/blog/images/covidHvac/cdc.avif" type="image/avif">--> <source srcset="/blog/images/covidHvac/cdc.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/cdc.apng" type="image/apng"> <source srcset="/blog/images/covidHvac/cdc.png" type="image/png"> <source srcset="/blog/images/covidHvac/cdc.jpg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/cdc.jpeg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/cdc.jfif" type="image/jpeg"> <source srcset="/blog/images/covidHvac/cdc.pjpeg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/cdc.pjp" type="image/jpeg"> <source srcset="/blog/images/covidHvac/cdc.gif" type="image/gif"> <source srcset="/blog/images/covidHvac/cdc.tif" type="image/tiff"> <source srcset="/blog/images/covidHvac/cdc.tiff" type="image/tiff"> <source srcset="/blog/images/covidHvac/cdc.bmp" type="image/bmp"> <source srcset="/blog/images/covidHvac/cdc.ico" type="image/x-icon"> <source srcset="/blog/images/covidHvac/cdc.cur" type="image/x-icon"> <img class="imgImg " src="/blog/images/covidHvac/cdc.png" style='width: 100%; maxwidth; 25%; width: 130px; height: auto;' /> </picture> </div> Exposure occurs in three principal ways: <ol> <li>inhalation of very fine respiratory droplets and aerosol particles</li> <li> deposition of respiratory droplets and particles on exposed mucous membranes in the mouth, nose, or eye by direct splashes and sprays </li> <li> touching mucous membranes with hands that have been soiled either directly by virus-containing respiratory fluids or indirectly by touching surfaces with virus on them. </li> </ol> <p> People release respiratory fluids during exhalation (e.g., quiet breathing, speaking, singing, exercise, coughing, sneezing) in the form of droplets across a spectrum of sizes. These droplets carry virus and transmit infection. To stay healthy, avoid these droplets. </p> <p> The largest droplets settle out of the air rapidly, within seconds to minutes. The smallest very fine droplets, and aerosol particles formed when these fine droplets rapidly dry, are small enough that they can remain suspended in the air for minutes to hours. </p> &nbsp;&ndash;From the CDC <a href='https://www.cdc.gov/coronavirus/2019-ncov/science/science-briefs/sars-cov-2-transmission.html' target='_blank' rel="nofollow">Scientific Brief: SARS-CoV-2 Transmission</a>, Updated May 7, 2021. </div> <p> The idea that physical distancing of 6 feet (2 meters) might help keep people healthy is an example of bad science from a study <a href='https://www.businessinsider.com/6-foot-distancing-rule-is-outdated-oxford-mit-new-system-2020-8' target='_blank' rel="nofollow">80 years ago</a> that was not debunked until recently. </p> <div class="quoteCite shadow rounded"> <div class='imgWrapper imgFlex right quartersize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/covidHvac/epaLogo.svg" type="image/svg"> <!---<source srcset="/blog/images/covidHvac/epaLogo.avif" type="image/avif">--> <source srcset="/blog/images/covidHvac/epaLogo.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/epaLogo.apng" type="image/apng"> <source srcset="/blog/images/covidHvac/epaLogo.png" type="image/png"> <source srcset="/blog/images/covidHvac/epaLogo.jpg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/epaLogo.jpeg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/epaLogo.jfif" type="image/jpeg"> <source srcset="/blog/images/covidHvac/epaLogo.pjpeg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/epaLogo.pjp" type="image/jpeg"> <source srcset="/blog/images/covidHvac/epaLogo.gif" type="image/gif"> <source srcset="/blog/images/covidHvac/epaLogo.tif" type="image/tiff"> <source srcset="/blog/images/covidHvac/epaLogo.tiff" type="image/tiff"> <source srcset="/blog/images/covidHvac/epaLogo.bmp" type="image/bmp"> <source srcset="/blog/images/covidHvac/epaLogo.ico" type="image/x-icon"> <source srcset="/blog/images/covidHvac/epaLogo.cur" type="image/x-icon"> <img class="imgImg " src="/blog/images/covidHvac/epaLogo.png" style='width: 100%; ' /> </picture> </div> Transmission of COVID-19 from inhalation of virus in the air can occur at distances greater than six feet. Particles from an infected person can move throughout an entire room or indoor space. The particles can also linger in the air after a person has left the room – they can remain airborne for hours in some cases. <br><br> &nbsp;&ndash;From the US Environmental Protection Agency (EPA): <a href='https://www.epa.gov/coronavirus/indoor-air-and-coronavirus-covid-19' target='_blank' rel="nofollow">Indoor Air and Coronavirus (COVID-19)</a>, web page was updated December 15, 2021. </div> <p> Distance between people is not a significant transmissibility factor. Imagine two people, back to back, one (downwind) very sick with COVID-19, and the other (healthy) upwind, and the wind is blowing at 20 mph (32 km/h). The sick person will not infect the healthy person, unless they exchange positions. Just control the air that is breathed, and all airborne viruses such as COVID-19 will be controled. </p> <div class="quoteCite shadow rounded"> <div class='imgWrapper imgFlex right' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/covidHvac/ucsdMedicine.svg" type="image/svg"> <!---<source srcset="/blog/images/covidHvac/ucsdMedicine.avif" type="image/avif">--> <source srcset="/blog/images/covidHvac/ucsdMedicine.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/ucsdMedicine.apng" type="image/apng"> <source srcset="/blog/images/covidHvac/ucsdMedicine.png" type="image/png"> <source srcset="/blog/images/covidHvac/ucsdMedicine.jpg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/ucsdMedicine.jpeg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/ucsdMedicine.jfif" type="image/jpeg"> <source srcset="/blog/images/covidHvac/ucsdMedicine.pjpeg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/ucsdMedicine.pjp" type="image/jpeg"> <source srcset="/blog/images/covidHvac/ucsdMedicine.gif" type="image/gif"> <source srcset="/blog/images/covidHvac/ucsdMedicine.tif" type="image/tiff"> <source srcset="/blog/images/covidHvac/ucsdMedicine.tiff" type="image/tiff"> <source srcset="/blog/images/covidHvac/ucsdMedicine.bmp" type="image/bmp"> <source srcset="/blog/images/covidHvac/ucsdMedicine.ico" type="image/x-icon"> <source srcset="/blog/images/covidHvac/ucsdMedicine.cur" type="image/x-icon"> <img class="imgImg " src="/blog/images/covidHvac/ucsdMedicine.png" style='width: 100%; maxwidth; 25%; width: 130px; height: auto;' /> </picture> </div> What we learned during the pandemic is that aerosols were one of the main drivers in spreading the COVID-19 virus and that their importance in the transmission of many other respiratory pathogens has been systematically underappreciated. <br><br> &nbsp;&ndash; Dr. Robert Schooley, Professor in the Department of Medicine at the University of California San Diego (November 22, 2021), quoted from <a href='https://ucsdnews.ucsd.edu/pressrelease/covid-gets-airborne' target='_blank' rel="nofollow">COVID Gets Airborne &ndash; UC San Diego develops computer model to aid understanding of how viruses travel through the air</a> </div> <!-- endregion --> <!-- #region controlling virus infiltration --> <h2 id="virus">Controlling Virus Infiltration</h2> <p> We would do well to remember that the COVID-19 virus causes sickness. Without exposure to the virus people would not get sick. The risk that a person might catch the disease is directly related to the number of viruses inhaled per unit of time. For example, the more viruses someone inhales in an hour, the more likely they will get sick. </p> <p> <a href='https://www.statnews.com/2020/04/14/how-much-of-the-coronavirus-does-it-take-to-make-you-sick/' target='_blank' rel="nofollow">The amount of virus necessary to make a person sick is called the infectious dose.</a> All the countermeasures that have been used (hand washing, physical distancing, masks, curfews, etc.) are intended to reduce the number of viruses people are exposed to per unit time. Pandemic policy treats countermeasures as proxies for virus transmission vectors. By not updating pandemic policy as new information becomes available, we loose the ability to control the spread of the virus. </p> <!-- endregion --> <!-- #region hepa --> <h2 id="hepa">HEPA Filters</h2> <p> HEPA filters on HVAC equipment are designed to remove these aerosols. They come in a huge variety of sizes, shapes and air flow capacities. If you could just breathe the pure air emitted from a fan that had a HEPA filter, you would not get sick from any airborne virus. Any number of people could gather safely, if each of them received an adequate supply of pure air in their face at all times. </p> <div class="quoteCite shadow rounded"> <div class='imgWrapper imgFlex right quartersize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/covidHvac/epaLogo.svg" type="image/svg"> <!---<source srcset="/blog/images/covidHvac/epaLogo.avif" type="image/avif">--> <source srcset="/blog/images/covidHvac/epaLogo.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/epaLogo.apng" type="image/apng"> <source srcset="/blog/images/covidHvac/epaLogo.png" type="image/png"> <source srcset="/blog/images/covidHvac/epaLogo.jpg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/epaLogo.jpeg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/epaLogo.jfif" type="image/jpeg"> <source srcset="/blog/images/covidHvac/epaLogo.pjpeg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/epaLogo.pjp" type="image/jpeg"> <source srcset="/blog/images/covidHvac/epaLogo.gif" type="image/gif"> <source srcset="/blog/images/covidHvac/epaLogo.tif" type="image/tiff"> <source srcset="/blog/images/covidHvac/epaLogo.tiff" type="image/tiff"> <source srcset="/blog/images/covidHvac/epaLogo.bmp" type="image/bmp"> <source srcset="/blog/images/covidHvac/epaLogo.ico" type="image/x-icon"> <source srcset="/blog/images/covidHvac/epaLogo.cur" type="image/x-icon"> <img class="imgImg " src="/blog/images/covidHvac/epaLogo.png" style='width: 100%; ' /> </picture> </div> HEPA is a type of pleated mechanical air filter. It is an acronym for “high efficiency particulate air filter” (as officially defined by the U.S. Dept. of Energy). This type of air filter can theoretically remove at least 99.97% of dust, pollen, mold, bacteria, and any airborne particles with a size of 0.3 microns (µm). <br><br> &nbsp;&ndash;<a href='https://www.epa.gov/indoor-air-quality-iaq/what-hepa-filter-1' target='_blank' rel="nofollow">US Environment Protection Agency</a> </div> <p> HEPA filters are inexpensive and are readily available. They are not new. In fact, <a href='https://www.apcfilters.com/the-history-of-hepa-filters/' target='_blank' rel="nofollow">HEPA filter technology was created in the 1940s</a> by the US Army Chemical Corps and National Defense Research Committee as part of the Manhattan Project. HEPA technology was declassified after World War II and became available for commercial and personal use. HEPA filters play a key role in the research and development of modern pharmaceuticals, aerospace engineering, and computer chip manufacturing. </p> <div class="quoteCite shadow rounded"> HEPA air cleaners, which remain little-used in Canadian hospitals, are a cheap and easy way to reduce risk from airborne pathogens. <br><br> &nbsp;&ndash;From Nature Magazine, October 6, 2021: <a href='https://www.nature.com/articles/d41586-021-02669-2' target='_blank' rel="nofollow">Real-world data show that filters clean COVID-causing virus from air &ndash; An inexpensive type of portable filter efficiently screened SARS-CoV-2 and other disease-causing organisms from hospital air.</a> </div> <p> It is easy to attach a HEPA filter to a fan, or to replace an existing HVAC filter with a HEPA filter. Here is a very <a href='https://www.jefftk.com/p/ceiling-air-purifier' target='_blank' rel="nofollow">scientific approach to a home-made solution</a>. </p> <div class='imgWrapper imgBlock inline fullsize' style=' '> <figure> <a href='https://youtu.be/kH5APw_SLUU?t=106' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/covidHvac/hepaFan.svg" type="image/svg"> <!---<source srcset="/blog/images/covidHvac/hepaFan.avif" type="image/avif">--> <source srcset="/blog/images/covidHvac/hepaFan.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/hepaFan.apng" type="image/apng"> <source srcset="/blog/images/covidHvac/hepaFan.png" type="image/png"> <source srcset="/blog/images/covidHvac/hepaFan.jpg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/hepaFan.jpeg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/hepaFan.jfif" type="image/jpeg"> <source srcset="/blog/images/covidHvac/hepaFan.pjpeg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/hepaFan.pjp" type="image/jpeg"> <source srcset="/blog/images/covidHvac/hepaFan.gif" type="image/gif"> <source srcset="/blog/images/covidHvac/hepaFan.tif" type="image/tiff"> <source srcset="/blog/images/covidHvac/hepaFan.tiff" type="image/tiff"> <source srcset="/blog/images/covidHvac/hepaFan.bmp" type="image/bmp"> <source srcset="/blog/images/covidHvac/hepaFan.ico" type="image/x-icon"> <source srcset="/blog/images/covidHvac/hepaFan.cur" type="image/x-icon"> <img alt='Dr. Jeffrey E. Terrell, director of the Michigan Sinus Center, demonstrates how to build an air purifier with a HEPA filter for about $25 with parts from your local hardware store.' class="imgImg rounded shadow" src="/blog/images/covidHvac/hepaFan.png" style='width: 100%; ' title='Dr. Jeffrey E. Terrell, director of the Michigan Sinus Center, demonstrates how to build an air purifier with a HEPA filter for about $25 with parts from your local hardware store.' /> </picture> </a> <figcaption class='imgFigCaption fullsize'> <a href="https://youtu.be/kH5APw_SLUU?t=106" target='_blank' > Dr. Jeffrey E. Terrell, director of the Michigan Sinus Center, demonstrates how to build an air purifier with a HEPA filter for about $25 with parts from your local hardware store. </a> </figcaption> </figure> </div> <p> One of the comments on the above video, from Rick Rude in 2014, was: </p> <div class="quoteCite shadow rounded"> Better to pull air thru the filter, if you push air through the filter dust that builds up will get blown off and go back into the room. Also, if the fan is pulling air through, the filter will stick to the fan and won't need as much tape to hold it on. Always handle used filters with care because handling them rough will release concentrated dust back into your air. If possible, always take air cleaners or vacuum cleaners outside to change the filters or bags. </div> <!-- endregion --> <!-- #region merv filters --> <h2 id="merv">MERV Filters</h2> <p> MERV is a graduated standard; MERV 7 is the minimum standard for furnaces, while MERV 13 is the highest. </p> <p> MERV 13 filters are more efficient at removing large particles from the air, while HEPA filters are more efficient at removing small particles from the air. MERV 13 filters can remove up to 99.97% of particles from the air, while HEPA filters can remove up to 99.99% of particles from the air. </p> <p> <a href='https://engineering.ucdavis.edu/news/science-action-how-build-corsi-rosenthal-box' target='_blank' rel="nofollow">How to Build a Corsi-Rosenthal Box</a>. </p> <!-- endregion --> <!-- #region youtube2 --> <h2 id="youtube2">How to Identify and Rectify Poorly Ventilated Indoor Spaces Using Engineering Controls</h2> <p> <a href='https://www.linkedin.com/in/elfstrom/' target='_blank' rel="nofollow">David Elfstrom, P. Eng.</a>, is a independent civil engineer based in Simcoe, Ontario who consults on the efficient use of energy in buildings. During the pandemic he has been drawing attention to the importance of ventilation, filtration, and overall indoor environmental quality. Earlier this year Mr. Elfstrom completed a ventilation and air pathways assessment of an apartment building in outbreak for a public health unit. This presentation was part of Passive Buildings Canada's 2021 Annual General Meeting. </p> <style>.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }</style><div class='embed-container'> <iframe title="YouTube video player" width="640" height="390" src="//www.youtube.com/embed/9J96F32tv1s" frameborder="0" allowfullscreen></iframe></div> <p style="margin-top: 1em;"> Mr. Elfstrom also co-authored the <a href='https://docs.google.com/document/d/17tKk8Da8tnchtnp9ZRe7fPazGAmXtvoA-n4GZcY0_fQ/edit' target='_blank' rel="nofollow">Masks4Canada Room Ventilation/Filtration Guide and Tip Sheet</a>. </p> <!-- endregion --> <!-- #region N95 --> <h2 id="hvac">N95, KF94, FFP2, and 9152 Masks and Their Cousins</h2> <p> If you need to move about in a crowd, or enter a space that contained people a few hours ago, you need to wear a properly fitting face mask that filters out the tiny aerosol particles that contain the COVID-19 virus. <a href='https://www.travelawaits.com/2559161/n95-vs-kn95-vs-kf94-masks-for-travel/' target='_blank' rel="nofollow">N95 masks and their cousins</a> (KN95, KF94, etc.) do the job because they filter particles as small as 0.3 microns. This is similar to the particle size filtered by HEPA filters. Caution: KN95 is a self-reported test standard, and lacks strict government regulation by China, resulting in many underperforming and often flat-out fake masks. </p> <p> The US National Personal Protective Technology Laboratory (NPPTL), which is part of the US National Institute for Occupational Safety and Health (NIOSH), which itself is part of the CDC, evaluated various masks and published the results as <a href='https://www.cdc.gov/niosh/npptl/respirators/testing/NonNIOSHresults.html' target='_blank' rel="nofollow">NPPTL Respirator Assessments to Support the COVID-19 Response</a>. </p> <p> The following types of masks do <b>not</b> reliably filter tiny aerosols, so they do not adequately protect you from COVID-19 variants such as Omicron: </p> <ol> <li> <a href='https://www.theguardian.com/commentisfree/2021/dec/27/best-masks-covid-tests-cloth-surgical-respirators' target='_blank' rel="nofollow">Surgical masks</a> (what most people wear these days) </li> <li>Masks with activated charcoal</li> <li>Vented masks</li> <li>Cloth masks</li> <li>Gaiters</li> <li>Ill-fitting masks that do not cover and seal the mouth and nose</li> </ol> <iframe width="690" height="388" src="https://www.youtube.com/embed/WE5Uo3F2TdU" frameborder="0" allowfullscreen class="rounded shadow liImg"></iframe> <!-- endregion --> <!-- #region inoculations and pills --> <h2 id="other">Inoculations and Pills</h2> <div class='imgWrapper imgFlex right quartersize' style=' '> <a href='https://www.reuters.com/business/healthcare-pharmaceuticals/us-fda-set-authorize-pfizer-merck-covid-19-pills-this-week-bloomberg-news-2021-12-21/' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/covidHvac/covidPills.svg" type="image/svg"> <!---<source srcset="/blog/images/covidHvac/covidPills.avif" type="image/avif">--> <source srcset="/blog/images/covidHvac/covidPills.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/covidPills.apng" type="image/apng"> <source srcset="/blog/images/covidHvac/covidPills.png" type="image/png"> <source srcset="/blog/images/covidHvac/covidPills.jpg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/covidPills.jpeg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/covidPills.jfif" type="image/jpeg"> <source srcset="/blog/images/covidHvac/covidPills.pjpeg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/covidPills.pjp" type="image/jpeg"> <source srcset="/blog/images/covidHvac/covidPills.gif" type="image/gif"> <source srcset="/blog/images/covidHvac/covidPills.tif" type="image/tiff"> <source srcset="/blog/images/covidHvac/covidPills.tiff" type="image/tiff"> <source srcset="/blog/images/covidHvac/covidPills.bmp" type="image/bmp"> <source srcset="/blog/images/covidHvac/covidPills.ico" type="image/x-icon"> <source srcset="/blog/images/covidHvac/covidPills.cur" type="image/x-icon"> <img alt='COVID-19 pills' class="imgImg rounded shadow" src="/blog/images/covidHvac/covidPills.png" style='width: 100%; ' title='COVID-19 pills' /> </picture> </a> </div> <p> Inoculations, including boosters, have proven to be very helpful. Pills from Pfizer Inc. and Merck & Co. for people sick with COVID-19 will also be very helpful once they become available. However, pills will only help sick people get better, they will not prevent getting sick. </p> <p> Until everyone in the entire world has somehow acquired an effective level of antibodies, COVID-19 will continue to mutate. </p> <!-- endregion --> <!-- #region the only solution --> <h2 id="go">The Only Solution Available Today</h2> <p> Inoculations and masks are good, but they are not perfect preventative measures against COVID-19. Protection against infection is not 100%. At present, the only preventative measure against COVID-19 that would allow people to safely mingle in person would be to guarantee continuous streams of pure air directed individually at each person's face. </p> <p> This measure would also prevent the spread of <a href='https://bmcinfectdis.biomedcentral.com/articles/10.1186/s12879-019-3707-y' target='_blank' rel="nofollow">all other diseases and their variants that are primarily spread via aerosols</a>, including coronoviruses, colds, flus, tuberculosis, MERS-CoV, measles, ebola and chickenpox. Pollen, dust and other particulates would also be removed, so athsma sufffers would feel relief from extended periods breathing pure air. </p> <p> The solution to living with COVID-19 is simple, safe, inexpensive and is not disruptive: </p> <div class="formalNotice rounded shadow"> <h2 class="centered" style="margin: 0; padding: 0">Live your social life with a pure breeze in your face.</h2> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/covidHvac/breeze-dandelion.svg" type="image/svg"> <!---<source srcset="/blog/images/covidHvac/breeze-dandelion.avif" type="image/avif">--> <source srcset="/blog/images/covidHvac/breeze-dandelion.webp" type="image/webp"> <source srcset="/blog/images/covidHvac/breeze-dandelion.apng" type="image/apng"> <source srcset="/blog/images/covidHvac/breeze-dandelion.png" type="image/png"> <source srcset="/blog/images/covidHvac/breeze-dandelion.jpg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/breeze-dandelion.jpeg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/breeze-dandelion.jfif" type="image/jpeg"> <source srcset="/blog/images/covidHvac/breeze-dandelion.pjpeg" type="image/jpeg"> <source srcset="/blog/images/covidHvac/breeze-dandelion.pjp" type="image/jpeg"> <source srcset="/blog/images/covidHvac/breeze-dandelion.gif" type="image/gif"> <source srcset="/blog/images/covidHvac/breeze-dandelion.tif" type="image/tiff"> <source srcset="/blog/images/covidHvac/breeze-dandelion.tiff" type="image/tiff"> <source srcset="/blog/images/covidHvac/breeze-dandelion.bmp" type="image/bmp"> <source srcset="/blog/images/covidHvac/breeze-dandelion.ico" type="image/x-icon"> <source srcset="/blog/images/covidHvac/breeze-dandelion.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/covidHvac/breeze-dandelion.png" style='width: 100%; ' /> </picture> </div> <div style="clear:both; font-size: 0"></div> </div> <p> Life as we knew it before the pandemic started 2 years ago could resume, mostly. </p> <!-- endregion --> <!-- #region hvac upgrades --> <h2 id="hvac">HVAC Upgrades Are Key</h2> <p> The engineering group that encompasses HVAC is ASHRAE. ASHRAE was formed as the American Society of Heating, Refrigerating and Air-Conditioning Engineers by the merger in 1959 of American Society of Heating and Air-Conditioning Engineers (ASHAE), founded in 1894 and The American Society of Refrigerating Engineers (ASRE), founded in 1904. </p> <p> ASHRAE offers <a href='https://www.ashrae.org/file%20library/technical%20resources/covid-19/core-recommendations-for-reducing-airborne-infectious-aerosol-exposure.pdf' target='_blank' rel="nofollow">Core Recommendations for Reducing Airborne Infectious Aerosol Exposure</a>. <a href='https://www.iso-aire.com/blog/what-are-the-differences-between-a-merv-13-and-a-hepa-filter' target='_blank' rel="nofollow">MERV 13 or better</a> levels of performance are recommended, and HEPA filters surpass that specifation. In fact, all HEPA filters have a rating of a MERV 17 or higher. </p> <!-- endregion --> <!-- #region entrepreneurs --> <h2 id="entre">Entrepreneurs</h2> <p> This represents an opportunity for entrepreneurs. </p> <!-- #region restaurants --> <h3 id="restos">Restaurants</h2> <p> Imagine a restaurant that stays open with near-normal seating capacity because each table is equipped with a fan and ducting that blows a gentle breeze of pure air directly into the face of every patron. The air would recirculate within the restaurant, so there would be no need to upgrade the existing HVAC system because the HEPA filters located at each table would continuously clean the air within the restaurant. Those filters would need to be cleaned and disinfected daily. The wait staff would all wear N95 or similar masks, however each cashier could instead enjoy their own gentle breeeze of pure air. </p> <p> Cost to equip each seat in the restaurant could be less than $50. For do-it-yourself owners, the cost could approach $10 per seat. For example, a 100-seat restaurant might be able to retrofit 75 of those seats at a cost of less than $3750. They would never have to close again due to any pandemic caused by airborne aerosols... provided, of course, that local regulations recognized this approach. </p> <!-- endregion --> <!-- #region retail --> <h3 id="stores">Retail Stores and Office Buildings</h2> <p> Enclosing sales clerks in stores behind clear plastic walls is wrong because it decreases air circulation. Instead, each those staff members should have a dedicated ducted fan that blows a gentle breeze of pure air into their face at all times. </p> <p> Similarly, queues of people awaiting their turn at checking out should have fans blowing pure air at their heads. The air would recirculate within the store, there would be no need to upgrade the existing HVAC system because the HEPA filters would continuously clean the air within the building. </p> <!-- endregion --> <!-- #region spring-breezifier --> <h2 id="domain">Spring-Breezifier Is Offered Into the Public Domain</h2> <p> I offer this idea to the world; I am an idea machine so I cannot act on most of them. Just for fun, let's call this idea the <i>Spring-Breezifier</i>. </p> <p> Designing and building Spring-Breezifiers seems like it might be a good high school or youth group project. HVAC installers in particular should be able to make short work of this idea; go ahead and make lots of money providing pure airflows by building custom Spring-Breezifiers for your customers, we all thank you! </p> <p> If anyone would like to talk to me about their Spring-Breezifier project, I would be happy to speak with them. </p> <!-- endregion --> <!-- #region next steps --> <h2 id="next">Next Steps</h2> <ol> <li> Medical HVAC specialists could suggest the necessary cubic feet per minute of air required per person sufficient to guarantee a continuous stream of pure air, and provide guidelines for designing ductwork to direct that air effectively. </li> <li> Build and test prototypes. Most of the effort will be in designing and building the ducting and/or the scaffolding. Adding inner baffles would raise the cost and weight, lower the noise and make the airflow less turbulent. Perhaps Spring-Breezifiers could be built entirely from off-the-shelf parts for some or even most installations. Large-format 3D printers for special circumstances might be helpful. </li> <li>Propose the inclusion of pure air streams into public spaces as a matter of public health policy.</li> </ol> <!-- endregion --> <!-- endregion --> Disappointing Scala 3 Installation Experience 2021-05-19T00:00:00-04:00 https://mslinn.github.io/blog/2021/05/19/installing-scala-3.0 <p> I run <a href='https://scalacourses.com' target='_blank' rel="nofollow">ScalaCourses.com</a>, an online Scala training web site. After years of hype, Scala 3 is now available. </p> <h2 id="prime">Scala 3: Not Yet Ready for Production</h2> <p> <a href='https://www.infoq.com/news/2021/03/scala3/' target='_blank' rel="nofollow">Scala 3</a> was eight years in the making. You would never know that from the horrible installation process and the disappointing installation instructions. </p> <p> It is going to take quite a while before <a href='https://scalatimes.com/d374aea433' target='_blank' rel="nofollow">Scala 3</a>, which was known as Dotty before it was released, can be trusted in production. According to the <a href='https://github.com/lampepfl/dotty' target='_blank' rel="nofollow">Dotty GitHub project</a>, the only published future milestone is <a href='https://github.com/lampepfl/dotty/milestones' target='_blank' rel="nofollow">v3.1.0, which has no due date</a>. Given that Scala 3 uses an entirely new build process, and an entirely new (and nonstandard) installation process, and that the internals of the Scala compiler were almost completely replaced, I doubt that version will be stable enough for use on production projects. </p> <h2 id="choices">Installation Choices</h2> <p> Installation cholices include: </p> <ul> <li>Command-line Scala has limited use cases, but is nice to have around for occassional experimentation.</li> <li> <a href='https://www.scala-lang.org/blog/2021/04/08/scala-3-in-sbt.html' target='_blank' rel="nofollow">SBT</a> (which features an enhanced Scala REPL) is very helpful for interactively developing code, as well as for building and testing. </li> <li> <a href='https://www.jetbrains.com/help/idea/discover-intellij-idea-for-scala.html' target='_blank' rel="nofollow">IntelliJ</a> provides the best Scala coding productivity and code quality. </li> <li> <a href='https://shunsvineyard.info/2020/11/20/setting-up-vs-code-for-scala-development-on-wsl/' target='_blank' rel="nofollow">VSCode</a> has been playing catch-up but is not full-featured yet. </li> </ul> <h2 id="install">Installation Transcript</h2> <p> The following is the transcript of how I installed command-line Scala on Ubuntu 20.10 running under WSL2. </p> <h3 class="numbered" id="remove_scala2">Remove Scala 2</h3> <p> This step is not required. Scala 2 and Scala 3 can easily co-exist on the same system because their names are different. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idba5fd253764b'><button class='copyBtn' data-clipboard-target='#idba5fd253764b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt remove scala <span class='unselectable'>Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be REMOVED: scala 0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded. After this operation, 666 MB disk space will be freed. Do you want to continue? [Y/n] (Reading database ... 236024 files and directories currently installed.) Removing scala (2.13.4-400) ... Processing triggers for man-db (2.9.3-2) ... </span></pre> </div> <h3 class="numbered" id="cs">Install Coursier</h3> <p> Coursier is a multithreaded downloader for project dependencies, and it now also downloads Scala 3. SBT uses coursier internally. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf0a9c9a3db39'><button class='copyBtn' data-clipboard-target='#idf0a9c9a3db39' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -fLo cs https://git.io/coursier-cli-"$(uname | tr LD ld)" <span class='unselectable'>&nbsp; % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 144 100 144 0 0 285 0 --:--:-- --:--:-- --:--:-- 6000 100 57.1M 100 57.1M 0 0 3656k 0 0:00:15 0:00:15 --:--:-- 4092k </span> <span class='unselectable'>$ </span>mv cs ~/.local/bin/ <span class='unselectable'>$ </span>chmod a+x ~/.local/bin/cs <span class='unselectable'>$ </span>cs install cs <span class='unselectable'>https://repo1.maven.org/maven2/io/get-coursier/apps/maven-metadata.xml 100.0% [##########] 1.8 KiB (8.5 KiB / s) https://repo1.maven.org/maven2/io/get-coursier/coursier-cli_2.12/maven-metadata.xml No new update since 2021-03-23 14:35:16 Wrote cs Warning: /home/mslinn/.local/share/coursier/bin is not in your PATH To fix that, add the following line to ~/.bashrc export PATH="$PATH:/home/mslinn/.local/share/coursier/bin" </span> <span class='unselectable'>$ </span> export PATH="$PATH:/home/mslinn/.local/share/coursier/bin"</pre> </div> <p> Now let's test Coursier: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id04a4e1f6beb4'><button class='copyBtn' data-clipboard-target='#id04a4e1f6beb4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cs <span class='unselectable'>Coursier 2.0.16 Usage: cs [options] [command] [command-options] Available commands: bootstrap, channel, complete, fetch, get, install, java, java-home, launch, list, publish, resolve, setup, uninstall, update, search Type cs command --help for help on an individual command </span></pre> </div> <p> This is the Coursier help message for the <code>install</code> subcommand: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id46b3d3cca5e6'><button class='copyBtn' data-clipboard-target='#id46b3d3cca5e6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cs install --help <span class='unselectable'>Command: install Usage: cs install --cache &lt;string?&gt; Cache directory (defaults to environment variable COURSIER_CACHE, or ~/.cache/coursier/v1 on Linux and ~/Library/Caches/Coursier/v1 on Mac) --mode | -m &lt;offline|update-changing|update|missing|force&gt; Download mode (default: missing, that is fetch things missing from cache) --ttl | -l &lt;duration&gt; TTL duration (e.g. &quot;24 hours&quot;) --parallel | -n &lt;int&gt; Maximum number of parallel downloads (default: 6) --checksum &lt;checksum1,checksum2,...&gt; Checksum types to check - end with none to allow for no checksum validation if no checksum is available, example: SHA-256,SHA-1,none --retry-count &lt;int&gt; Retry limit for Checksum error when fetching a file --cache-file-artifacts | --cfa &lt;bool&gt; Flag that specifies if a local artifact should be cached. --follow-http-to-https-redirect &lt;bool&gt; Whether to follow http to https redirections --credentials &lt;host(realm) user:pass|host user:pass&gt; Credentials to be used when fetching metadata or artifacts. Specify multiple times to pass multiple credentials. Alternatively, use the COURSIER_CREDENTIALS environment variable --credential-file &lt;string*&gt; Path to credential files to read credentials from --use-env-credentials &lt;bool&gt; Whether to read credentials from COURSIER_CREDENTIALS (env) or coursier.credentials (Java property), along those passed with --credentials and --credential-file --quiet | -q &lt;counter&gt; Quiet output --verbose | -v &lt;counter&gt; Increase verbosity (specify several times to increase more) --progress | -P &lt;bool&gt; Force display of progress bars --log-changing &lt;bool&gt; Log changing artifacts --log-channel-version | --log-index-version | --log-jvm-index-version &lt;bool&gt; Log app channel or JVM index version --graalvm-home &lt;string?&gt; --graalvm-option &lt;string*&gt; --graalvm-default-version &lt;string?&gt; --install-dir | --dir &lt;string?&gt; --install-platform &lt;string?&gt; Platform for prebuilt binaries (e.g. &quot;x86_64-pc-linux&quot;, &quot;x86_64-apple-darwin&quot;, &quot;x86_64-pc-win32&quot;) --install-prefer-prebuilt &lt;bool&gt; --only-prebuilt &lt;bool&gt; Require prebuilt artifacts for native applications, don&#39;t try to build native executable ourselves --repository | -r &lt;maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|ivy:$pattern&gt; Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots) --default-repositories &lt;bool&gt; --proguarded &lt;bool?&gt; --channel &lt;org:name&gt; Channel for apps --default-channels &lt;bool&gt; Add default channels --contrib &lt;bool&gt; Add contrib channel --file-channels &lt;bool&gt; Add channels read from the configuration directory --jvm &lt;string?&gt; --jvm-dir &lt;string?&gt; --system-jvm &lt;bool?&gt; --local-only &lt;bool&gt; --update &lt;bool&gt; --jvm-index &lt;string?&gt; --repository | -r &lt;maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|scala-integration|scala-nightlies|ivy:$pattern|jitpack|clojars|jcenter|apache:$repo&gt; Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots) --no-default &lt;bool&gt; Do not add default repositories (~/.ivy2/local, and Central) --sbt-plugin-hack &lt;bool&gt; Modify names in Maven repository paths for sbt plugins --drop-info-attr &lt;bool&gt; Drop module attributes starting with &#39;info.&#39; - these are sometimes used by projects built with sbt --channel &lt;org:name&gt; Channel for apps --default-channels &lt;bool&gt; Add default channels --contrib &lt;bool&gt; Add contrib channel --file-channels &lt;bool&gt; Add channels read from the configuration directory --repository | -r &lt;maven|sonatype:$repo|ivy2local|bintray:$org/$repo|bintray-ivy:$org/$repo|typesafe:ivy-$repo|typesafe:$repo|sbt-plugin:$repo|scala-integration|scala-nightlies|ivy:$pattern|jitpack|clojars|jcenter|apache:$repo&gt; Repository - for multiple repositories, separate with comma and/or add this option multiple times (e.g. -r central,ivy2local -r sonatype:snapshots, or equivalently -r central,ivy2local,sonatype:snapshots) --no-default &lt;bool&gt; Do not add default repositories (~/.ivy2/local, and Central) --sbt-plugin-hack &lt;bool&gt; Modify names in Maven repository paths for sbt plugins --drop-info-attr &lt;bool&gt; Drop module attributes starting with &#39;info.&#39; - these are sometimes used by projects built with sbt --channel &lt;org:name&gt; Channel for apps --default-channels &lt;bool&gt; Add default channels --contrib &lt;bool&gt; Add contrib channel --file-channels &lt;bool&gt; Add channels read from the configuration directory --env &lt;bool&gt; --disable-env | --disable &lt;bool&gt; --setup &lt;bool&gt; --user-home &lt;string?&gt; --add-channel &lt;string*&gt; (deprecated) --force | -f &lt;bool&gt; </span></pre> </div> <h3 class="numbered" id="s3">Install Scala 3</h3> <p> The Scala 3 compiler and REPL are separate programs: <code>scala3-compiler</code> and <code>scala3-repl</code>. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id18b50b515936'><button class='copyBtn' data-clipboard-target='#id18b50b515936' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cs install scala3-compiler <span class='unselectable'>https://repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.0.0/scala3-compiler_3-3.0.0.pom 100.0% [##########] 4.8 KiB (79.7 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.0.0/tasty-core_3-3.0.0.pom 100.0% [##########] 3.5 KiB (69.5 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.0.0/scala3-library_3-3.0.0.pom 100.0% [##########] 3.6 KiB (53.9 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.0.0/scala3-interfaces-3.0.0.pom 100.0% [##########] 3.4 KiB (65.9 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.0.0/scala3-interfaces-3.0.0.jar 100.0% [##########] 3.4 KiB (113.9 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.0.0/tasty-core_3-3.0.0.jar 100.0% [##########] 71.9 KiB (192.7 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.0.0/scala3-library_3-3.0.0.jar 100.0% [##########] 1.1 MiB (1.8 MiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.0.0/scala3-compiler_3-3.0.0.jar 100.0% [##########] 14.7 MiB (3.7 MiB / s) Wrote scala3-compiler </span> <span class='unselectable'>$ </span>cs install scala3-repl <span class='unselectable'>https://repo1.maven.org/maven2/io/get-coursier/apps/maven-metadata.xml No new update since 2021-05-14 04:42:19 Wrote scala3-repl </span></pre> </div> <h3 class="numbered" id="repl3">Run Scala 3 REPL</h3> <p> The part is easy! </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3c31901452b2'><button class='copyBtn' data-clipboard-target='#id3c31901452b2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>scala3-repl --version <span class='unselectable'>Scala code runner version 3.0.0 -- Copyright 2002-2021, LAMP/EPFL </span> <span class='unselectable'>$ </span>scala3-repl <span class='unselectable'>scala&gt; </span></pre> </div> <h2 id="sbt">Easily Run Scala REPL With SBT</h2> <p> If you do not mind directories called <code>project/</code> and <code>target/</code> being created in your current directory, and you have already <a href='https://www.scala-sbt.org/download.html' target='_blank' rel="nofollow">installed sbt</a>, you can get a REPL powered by Scala 3 like this: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id156db411d016'><button class='copyBtn' data-clipboard-target='#id156db411d016' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sbt "-Dsbt.version=1.5.2" ++3.0.0! console <span class='unselectable'>[info] welcome to sbt 1.5.2 (Ubuntu Java 11.0.11) [info] loading global plugins from /home/mslinn/.sbt/1.0/plugins [info] loading project definition from /var/work/ancientWarmth/ancientWarmth/project [info] set current project to ancientwarmth (in build file:/var/work/ancientWarmth/ancientWarmth/) [info] Forcing Scala version to 3.0.0 on all projects. [info] Reapplying settings... [info] set current project to ancientwarmth (in build file:/var/work/ancientWarmth/ancientWarmth/) [info] Updating [info] Resolved dependencies [info] Updating https://repo1.maven.org/maven2/org/scala-lang/scaladoc_3/3.0.0/scaladoc_3-3.0.0.pom 100.0% [##########] 6.1 KiB (82.6 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala3-tasty-inspector_3/3.0.0/scala3-tasty-inspector_3-3.0.0.pom 100.0% [##########] 3.6 KiB (80.8 KiB / s) [info] Resolved dependencies [info] Fetching artifacts of [info] Fetched artifacts of [info] Fetching artifacts of https://repo1.maven.org/maven2/org/scala-lang/scala3-tasty-inspector_3/3.0.0/scala3-tasty-inspector_3-3.0.0.jar 100.0% [##########] 16.6 KiB (338.1 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scaladoc_3/3.0.0/scaladoc_3-3.0.0.jar 100.0% [##########] 1.5 MiB (3.1 MiB / s) [info] Fetched artifacts of scala&gt; </span></pre> </div> <p> Thanks to <a href='https://twitter.com/renghenKornel/status/1395684928440791040' target='_blank' rel="nofollow">@renghen</a> for this tip. </p> <h2 id="sc">ScalaCourses</h2> <p> If you want to learn how to work effectively with Scala for functional and object-oriented programming, <a href='https://scalacourses.com' target='_blank' rel="nofollow">ScalaCourses.com</a> is your best option. The course material is suitable for Scala 2 and Scala 3. Visit ScalaCourses.com to learn how to become a proficient Scala programmer. </p> OCI / Docker / AWS Lambda / Django / Buildah / podman 2021-04-29T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/29/buildah-podman-python-lambda <style> body { counter-reset: pcounter; } p.count:before { counter-increment: pcounter; content: counter(pcounter) ")\A0"; } </style> <!-- #region intro --> <p> This article is a work in progress. Some of it may be incorrect, and some thoughts might lead nowhere. I am publicly posting it in this state so I can discuss it with others. This article will be improved as information becomes available. </p> <!-- endregion --> <!-- #region Goal --> <h2 id="goal">Goal</h2> <div class='imgWrapper imgFlex right' style=' '> <a href='https://podman.io' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/buildahPodman/podman-logo-crop.svg" type="image/svg"> <!---<source srcset="/blog/images/buildahPodman/podman-logo-crop.avif" type="image/avif">--> <source srcset="/blog/images/buildahPodman/podman-logo-crop.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/podman-logo-crop.apng" type="image/apng"> <source srcset="/blog/images/buildahPodman/podman-logo-crop.png" type="image/png"> <source srcset="/blog/images/buildahPodman/podman-logo-crop.jpg" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/podman-logo-crop.jpeg" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/podman-logo-crop.jfif" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/podman-logo-crop.pjpeg" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/podman-logo-crop.pjp" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/podman-logo-crop.gif" type="image/gif"> <source srcset="/blog/images/buildahPodman/podman-logo-crop.tif" type="image/tiff"> <source srcset="/blog/images/buildahPodman/podman-logo-crop.tiff" type="image/tiff"> <source srcset="/blog/images/buildahPodman/podman-logo-crop.bmp" type="image/bmp"> <source srcset="/blog/images/buildahPodman/podman-logo-crop.ico" type="image/x-icon"> <source srcset="/blog/images/buildahPodman/podman-logo-crop.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/buildahPodman/podman-logo-crop.png" style='width: 100%; padding: 1em; height: 191px; width: auto;' /> </picture> </a> </div> <div class='imgWrapper imgFlex right' style=' '> <a href='https://buildah.io/' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.svg" type="image/svg"> <!---<source srcset="/blog/images/buildahPodman/buildah-logo-crop.avif" type="image/avif">--> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.apng" type="image/apng"> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.png" type="image/png"> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.jpg" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.jpeg" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.jfif" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.pjpeg" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.pjp" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.gif" type="image/gif"> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.tif" type="image/tiff"> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.tiff" type="image/tiff"> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.bmp" type="image/bmp"> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.ico" type="image/x-icon"> <source srcset="/blog/images/buildahPodman/buildah-logo-crop.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/buildahPodman/buildah-logo-crop.png" style='width: 100%; padding: 0.73em; height: 191px; width: auto;' /> </picture> </a> </div> <p> <a href='/blog/2021/04/28/buildah-podman.html'>As previously discussed</a>, Buildah is a drop-in replacement for using <code>docker build</code> and a <code>Dockerfile</code>. Buildah’s <code>build-using-dockerfile</code>, or <code>bud</code> argument makes it behave just like <code>docker build</code> does. </p> <p> The goal of this article is to use Buildah / <code>podman</code> to create an Open Container Initiative (OCI) container image with a Django app, including the Python 3.8 runtime installed. The Django app will start when the container is created. The code for the Django app will be stored on the local machine where its source code can be edited, and it will be mapped into the container from the host system. Changes made to the code from the host system will be immediately visible inside the container. </p> <!-- endregion --> <!-- #region TODO --> <h2 id="todo">TODO</h2> <p class="count"> Background: AWS publishes <a href='https://docs.aws.amazon.com/lambda/latest/dg/python-image.html' target='_blank' rel="nofollow">Deploying Python with an AWS base image</a>, but that does not discuss running or testing. <a href='https://docs.aws.amazon.com/lambda/latest/dg/getting-started-create-function.html' target='_blank' rel="nofollow">Create a Lambda function with the console</a> is a more complete article, but is focused on using the web browser console, using Docker, and Node.js. So many differences from the desired goal make the articles difficult to translate to AWS CLI, Buildah / <code>podman</code> and Python. </p> <p class="count"> Talk about the <a href='https://github.com/aws/aws-lambda-runtime-interface-emulator' target='_blank' rel="nofollow">AWS Lambda Runtime Interface Emulator</a>, compare and contrast with the <a href='https://pypi.org/project/awslambdaric/' target='_blank' rel="nofollow">AWS Lambda Python Runtime Interface Client</a>. </p> <p class="count"> Compare these <a href='https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html' target='_blank' rel="nofollow">AWS Lambda Runtimes</a> with other, equivalant runtimes. </p> <p class="count"> OCI images are swapped in when AWS Lambda is invoked. Do larger images cost more to use? If so, discuss. </p> <!-- endregion --> <!-- #region Deploy Python Lambda function with Container Image --> <h2 id="main">Deploy Python Lambda function with Container Image</h2> <p> Consider this <code>Dockerfile</code>, which launches a Python 3.8 command-line application in a manner compatible with AWS Lambda: </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,Dockerfile' download='Dockerfile' title='Click on the file name to download the file'>Dockerfile</a> </div> <pre data-lt-active="false" class="pre_tag maxOneScreenHigh copyContainer" id="id7134221661b6">FROM public.ecr.aws/lambda/python:3.8 COPY app.py ./ CMD ["app.handler"] </pre> <p> Following is a small Python app called <code>app.py</code>, which will be launched by the <code>Dockerfile</code>. The Python app can be run as an AWS Lambda program because it implements the <code>handler</code> entry point. </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,app.py' download='app.py' title='Click on the file name to download the file'>app.py</a> </div> <pre data-lt-active="false" class="pre_tag maxOneScreenHigh copyContainer" id="idd6010dda0796">import sys def handler(event, context): return f"Hello from AWS Lambda using Python &#123;sys.version&#125;!" </pre> <!-- endregion --> <!-- #region Build image --> <h2 id="build">Build Image</h2> <p> Buildah builds the image, just the same way that Docker would: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida4a3c5aec0d1'><button class='copyBtn' data-clipboard-target='#ida4a3c5aec0d1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>buildah bud -t hello . <span class='unselectable'>STEP 1: FROM public.ecr.aws/lambda/python:3.8 Getting image source signatures Copying blob 03ac043af787 skipped: already exists Copying blob 420e64b38334 done Copying blob ff259f25b075 done Copying blob 3ff716981d54 done Copying blob 6b6e623a48a8 done Copying blob 9aa8f1e66d54 done Copying config 67dc3a2a54 done Writing manifest to image destination Storing signatures STEP 2: COPY app.py ./ STEP 3: CMD ["app.handler"] STEP 4: COMMIT hello Getting image source signatures Copying blob 683073d39306 skipped: already exists Copying blob 658871a69e1f skipped: already exists Copying blob 6fa16f35d11e skipped: already exists Copying blob d6fa53d6caa6 skipped: already exists Copying blob 61c062506436 skipped: already exists Copying blob 1c1d66a5fd95 skipped: already exists Copying blob 33af9dc6463a done Copying config 98862dfd20 done Writing manifest to image destination Storing signatures --&gt; 98862dfd208 98862dfd2087152ee821553d6cb1c033e735af06e5f11c814bcc9300fb65584e </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Test Lambda function Locally --> <h2 id="deploy_local">Test Lambda function Locally</h2> <p> Before calling the Lambda API from a local container, first run the container. Containers default to running in the foreground, but the <code>-d</code> option causes a container to be run as a background process. This container is given the name <code>hello</code>, the external HTTP endpoint at 9000 is mapped to internal port 8080, and the latest version of the <code>hello</code> lambda function is run in the container. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida190b17ea2a9'><button class='copyBtn' data-clipboard-target='#ida190b17ea2a9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman run \ -d \ --name hello \ -p 9000:8080 \ hello:latest <span class='unselectable'>d4d296e4c91d01c98d312e3f79599dca53990d95218e94bbdfbbac6a43cde9e8 </span></pre> </div> <p> Call the local version of the Lambda API: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0a8f3701b387'><button class='copyBtn' data-clipboard-target='#id0a8f3701b387' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl \ -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" \ -d '{}' <span class='unselectable'>"Hello from AWS Lambda using Python 3.8.9 (default, Apr 20 2021, 13:58:54) \n[GCC 7.3.1 20180712 (Red Hat 7.3.1-12)]!" </span></pre> </div> <p> Stop the container called <code>hello</code>. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfaa00f1cc51d'><button class='copyBtn' data-clipboard-target='#idfaa00f1cc51d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman stop hello <span class='unselectable'>96cc1b1ed92368a1165d6a6ad0b1e5544d4ac751b64e94df33bf2322e6d7b30c </span></pre> </div> <!-- endregion --> <!-- #region Create AWS ECR Repository --> <h2 id="podman_tag">Create AWS ECR Repository</h2> <p> AWS provides a registry for OCI-compatible image repositories called the <a href='https://aws.amazon.com/ecr/' target='_blank' rel="nofollow">AWS Elastic Container Registry (ECR)</a>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3858df510e26'><button class='copyBtn' data-clipboard-target='#id3858df510e26' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ecr create-repository help CREATE-REPOSITORY() CREATE-REPOSITORY()<br/> NAME create-repository -<br/> DESCRIPTION Creates a repository. For more information, see Amazon ECR Repositories in the Amazon Elastic Container Registry User Guide .<br/> See also: AWS API Documentation<br/> See &#39;aws help&#39; for descriptions of global parameters.<br/> SYNOPSIS create-repository --repository-name &lt;value&gt; [--tags &lt;value&gt;] [--image-tag-mutability &lt;value&gt;] [--image-scanning-configuration &lt;value&gt;] [--cli-input-json &lt;value&gt;] [--generate-cli-skeleton &lt;value&gt;]<br/> OPTIONS --repository-name (string) The name to use for the repository. The repository name may be spec- ified on its own (such as nginx-web-app ) or it can be prepended with a namespace to group the repository into a category (such as project-a/nginx-web-app ).<br/> --tags (list) The metadata that you apply to the repository to help you categorize and organize them. Each tag consists of a key and an optional value, both of which you define. Tag keys can have a maximum character length of 128 characters, and tag values can have a maximum length of 256 characters.<br/> (structure) The metadata that you apply to a resource to help you categorize and organize them. Each tag consists of a key and an optional value, both of which you define. Tag keys can have a maximum character length of 128 characters, and tag values can have a maximum length of 256 characters.<br/> Key -&gt; (string) One part of a key-value pair that make up a tag. A key is a general label that acts like a category for more specific tag values.<br/> Value -&gt; (string) The optional part of a key-value pair that make up a tag. A value acts as a descriptor within a tag category (key).<br/> Shorthand Syntax:<br/> Key=string,Value=string ...<br/> JSON Syntax:<br/> [ { &quot;Key&quot;: &quot;string&quot;, &quot;Value&quot;: &quot;string&quot; } ... ]<br/> --image-tag-mutability (string) The tag mutability setting for the repository. If this parameter is omitted, the default setting of MUTABLE will be used which will al- low image tags to be overwritten. If IMMUTABLE is specified, all im- age tags within the repository will be immutable which will prevent them from being overwritten.<br/> Possible values:<br/> o MUTABLE<br/> o IMMUTABLE<br/> --image-scanning-configuration (structure) The image scanning configuration for the repository. This setting determines whether images are scanned for known vulnerabilities af- ter being pushed to the repository.<br/> scanOnPush -&gt; (boolean) The setting that determines whether images are scanned after be- ing pushed to a repository. If set to true , images will be scanned after being pushed. If this parameter is not specified, it will default to false and images will not be scanned unless a scan is manually started with the StartImageScan API.<br/> Shorthand Syntax:<br/> scanOnPush=boolean<br/> JSON Syntax:<br/> { &quot;scanOnPush&quot;: true|false }<br/> --cli-input-json (string) Performs service operation based on the JSON string provided. The JSON string follows the format provided by --gen- erate-cli-skeleton. If other arguments are provided on the command line, the CLI values will override the JSON-provided values. It is not possible to pass arbitrary binary values using a JSON-provided value as the string will be taken literally.<br/> --generate-cli-skeleton (string) Prints a JSON skeleton to standard output without sending an API request. If provided with no value or the value input, prints a sample input JSON that can be used as an argument for --cli-input-json. If provided with the value output, it validates the command inputs and returns a sample output JSON for that command.<br/> See &#39;aws help&#39; for descriptions of global parameters.<br/> EXAMPLES Example 1: To create a repository<br/> The following create-repository example creates a repository inside the specified namespace in the default registry for an account.<br/> aws ecr create-repository \ --repository-name project-a/nginx-web-app<br/> Output:<br/> { &quot;repository&quot;: { &quot;registryId&quot;: &quot;123456789012&quot;, &quot;repositoryName&quot;: &quot;sample-repo&quot;, &quot;repositoryArn&quot;: &quot;arn:aws:ecr:us-west-2:123456789012:repository/project-a/nginx-web-app&quot; } }<br/> For more information, see Creating a Repository in the Amazon ECR User Guide.<br/> Example 2: To create a repository configured with image tag immutabil- ity<br/> The following create-repository example creates a repository configured for tag immutability in the default registry for an account.<br/> aws ecr create-repository \ --repository-name sample-repo \ --image-tag-mutability IMMUTABLE<br/> Output:<br/> { &quot;repository&quot;: { &quot;registryId&quot;: &quot;123456789012&quot;, &quot;repositoryName&quot;: &quot;sample-repo&quot;, &quot;repositoryArn&quot;: &quot;arn:aws:ecr:us-west-2:123456789012:repository/sample-repo&quot;, &quot;imageTagMutability&quot;: &quot;IMMUTABLE&quot; } }<br/> For more information, see Image Tag Mutability in the Amazon ECR User Guide.<br/> Example 3: To create a repository configured with a scanning configura- tion<br/> The following create-repository example creates a repository configured to perform a vulnerability scan on image push in the default registry for an account.<br/> aws ecr create-repository \ --repository-name sample-repo \ --image-scanning-configuration scanOnPush=true<br/> Output:<br/> { &quot;repository&quot;: { &quot;registryId&quot;: &quot;123456789012&quot;, &quot;repositoryName&quot;: &quot;sample-repo&quot;, &quot;repositoryArn&quot;: &quot;arn:aws:ecr:us-west-2:123456789012:repository/sample-repo&quot;, &quot;imageScanningConfiguration&quot;: { &quot;scanOnPush&quot;: true } } }<br/> For more information, see Image Scanning in the Amazon ECR User Guide.<br/> OUTPUT repository -&gt; (structure) The repository that was created.<br/> repositoryArn -&gt; (string) The Amazon Resource Name (ARN) that identifies the repository. The ARN contains the arn:aws:ecr namespace, followed by the re- gion of the repository, AWS account ID of the repository owner, repository namespace, and repository name. For example, arn:aws:ecr:region:012345678910:repository/test .<br/> registryId -&gt; (string) The AWS account ID associated with the registry that contains the repository.<br/> repositoryName -&gt; (string) The name of the repository.<br/> repositoryUri -&gt; (string) The URI for the repository. You can use this URI for Docker push or pull operations.<br/> createdAt -&gt; (timestamp) The date and time, in JavaScript date format, when the reposi- tory was created.<br/> imageTagMutability -&gt; (string) The tag mutability setting for the repository.<br/> imageScanningConfiguration -&gt; (structure) The image scanning configuration for a repository.<br/> scanOnPush -&gt; (boolean) The setting that determines whether images are scanned after being pushed to a repository. If set to true , images will be scanned after being pushed. If this parameter is not speci- fied, it will default to false and images will not be scanned unless a scan is manually started with the StartImageScan API.<br/> <br/> CREATE-REPOSITORY()</pre> </div> <!-- endregion --> <p> The following creates an AWS ECR image repository in called <code>hello</code> within the <code>test</code> namespace. <a href='https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-scanning.html' target='_blank' rel="nofollow">Images are scanned</a> for known vulnerabilities after they are pushed to the repository. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb1852c23f1c6'><button class='copyBtn' data-clipboard-target='#idb1852c23f1c6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ecr create-repository \ --repository-name test/hello \ --image-scanning-configuration scanOnPush=true <span class='unselectable'>{ "repository": { "repositoryArn": "arn:aws:ecr:us-east-1:031372724784:repository/test/hello", "registryId": "031372724784", "repositoryName": "test/hello", "repositoryUri": "031372724784.dkr.ecr.us-east-1.amazonaws.com/test/hello", "createdAt": 1620232146.0, "imageTagMutability": "MUTABLE", "imageScanningConfiguration": { "scanOnPush": true } } } </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Tag Image --> <h2 id="podman_tag">Tag Image</h2> <p class="quote"> <b><code>podman tag</code></b> &ndash; Assigns a new image name to an existing image. A full name refers to the entire image name, including the optional tag after the <code>:</code>. If there is no tag provided, then podman will default to latest for both the image and the target-name. &nbsp; &ndash; From <code>man podman-tag</code>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbe76752c1858'><button class='copyBtn' data-clipboard-target='#idbe76752c1858' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>IMAGE_NAME=hello <span class='unselectable'>$ </span>IMAGE_VERSION=0.1 <span class='unselectable'>$ </span>podman tag $IMAGE_NAME:$IMAGE_VERSION \ $REGISTRY/$IMAGE_NAME:$IMAGE_VERSION <span class='unselectable'>$ </span>podman images <span class='unselectable'>REPOSITORY TAG IMAGE ID CREATED SIZE localhost/hello 0.1 98862dfd2087 39 minutes ago 622 MB 752246127823.dkr.ecr.us-east-1.amazonaws.com/hello latest 98862dfd2087 39 minutes ago 622 MB public.ecr.aws/lambda/python 3.8 67dc3a2a54fb 25 hours ago 622 MB 752246127823.dkr.ecr.us-east-1.amazonaws.com/ancientwarmth latest 5d18ea34fc30 28 hours ago 2.03 GB localhost/ancientwarmth latest 5d18ea34fc30 28 hours ago 2.03 GB &lt;none&gt; &lt;none&gt; 40ef32b39cf4 5 days ago 622 MB docker.io/library/amazonlinux latest 53ef897d731f 5 days ago 170 MB docker.io/amazon/aws-lambda-python 3.8 e12ea62c5582 9 days ago 622 MB docker.io/library/alpine latest 6dbb9cc54074 2 weeks ago 5.88 MB docker.io/lambci/lambda build-python3.8 714c659c9f6f 3 months ago 2.03 GB </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Push Image to ECR --> <h2 id="push">Push Image to ECR</h2> <p> <code>Podman</code> will use the IAM credentials for the <code>dev</code> <a href='https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html' target='_blank' rel="nofollow">profile</a> in <code>~/.aws/credentials</code> to log into that AWS account: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>~/.aws/credentials</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4dbba51c898e'><button class='copyBtn' data-clipboard-target='#id4dbba51c898e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[default] aws_access_key_id = ******************** aws_secret_access_key = **************************************** region = us-east-1<br> [dev] aws_access_key_id = ******************** aws_secret_access_key = **************************************** region = us-east-1</pre> </div> <!-- endregion --> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcd96e16bb3b0'><button class='copyBtn' data-clipboard-target='#idcd96e16bb3b0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>export AWS_PROFILE=dev<br> <span class='unselectable'>$ </span>AWS_ACCOUNT="$( aws sts get-caller-identity \ --query Account \ --output text )"<br> <span class='unselectable'>$ </span>AWS_REGION="$( aws configure get region )"<br> <span class='unselectable'>$ </span>REGISTRY="$AWS_ACCOUNT.dkr.ecr.$AWS_REGION.amazonaws.com"<br> <span class='unselectable'>$ </span>aws ecr get-login-password \ --region "$AWS_REGION" | \ podman login \ --password-stdin \ --username AWS \ "$REGISTRY" <span class='unselectable'>Login Succeeded! </span></pre> </div> <!-- endregion --> <p> Now that <code>podman</code> is logged into AWS, use <code>podman</code> push the image to AWS ECR: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6484bee5c454'><button class='copyBtn' data-clipboard-target='#id6484bee5c454' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman push test/$IMAGE_NAME \ $REGISTRY/$IMAGE_NAME:$IMAGE_VERSION <span class='unselectable'>Getting image source signatures Copying blob 692590faf2d1 [--------------------------------------] 8.0b / 8.2MiB Copying blob 397718cff58d [--------------------------------------] 8.0b / 206.2MiB Copying blob 9ca787b1c91c [--------------------------------------] 8.0b / 93.1MiB Copying blob ef26f5221b79 [--------------------------------------] 8.0b / 196.7MiB Copying blob 0a3f69c27a89 [--------------------------------------] 8.0b / 316.4MiB Copying blob 5b3cbb76df75 [--------------------------------------] 8.0b / 1.1GiB Copying blob e9cad39831b0 [--------------------------------------] 8.0b / 3.5KiB Error: Error copying image to the remote destination: Error writing blob: Error initiating layer upload to /v2/ancientwarmth/blobs/uploads/ in 752246127823.dkr.ecr.us-east-1.amazonaws.com: name unknown: The repository with name 'hello' does not exist in the registry with id '752246127823' </span></pre> </div> <!-- endregion --> <p> The results of an <a href='https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ecr/describe-image-scan-findings.html' target='_blank' rel="nofollow">image scan</a> for the new repository can be retrieved as follows: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id021800ae1d62'><button class='copyBtn' data-clipboard-target='#id021800ae1d62' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ecr describe-image-scan-findings \ --repository-name test/hello \ --image-id imageTag=tag_name</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Deploy Python Lambda function with Container Image --> <h2 id="buildah_python">Deploy Python Lambda function with Container Image</h2> <p> <code>Podman</code> can invoke the app using an OCI container with Amazon Linux 2 and Python 3.8: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcc6d033b0ff1'><button class='copyBtn' data-clipboard-target='#idcc6d033b0ff1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman container run -ti \ public.ecr.aws/lambda/python:3.8 \ blog/docker/podman/app.py <span class='unselectable'>Trying to pull public.ecr.aws/lambda/python:3.8... Getting image source signatures Copying blob 1de4740de1c2 done Copying blob 03ac043af787 done Copying blob 2e2bb77ae2dc done Copying blob 842c9dce67e8 done Copying blob df513d38f4d9 done Copying blob 031c6369fb2b done Copying config e12ea62c55 done Writing manifest to image destination Storing signatures time="2021-05-02T23:38:30.971" level=info msg="exec '/var/runtime/bootstrap' (cwd=/var/task, handler=)" </span></pre> </div> <!-- endregion --> <!-- endregion --> Docker, OCI Images, Buildah and podman 2021-04-28T00:00:00-04:00 https://mslinn.github.io/blog/2021/04/28/buildah-podman <!-- #region intro --> <p> There are many ways to create and run Docker-compatible images. Docker is probably the worst option, mostly because it runs as a daemon, and all *nix daemons run with <code>root</code> privileges. Also, the <code>docker-ce</code> package lists <code>iptables</code> as a dependency, which needs <code>systemd</code> to be running normally, and WSL2 only partially supports <code>systemd</code>. </p> <p> <a href='https://www.capitalone.com/tech/cloud/container-runtime/' target='_blank' rel="nofollow">A Comprehensive Container Runtime Comparison</a> provides helpful background information and an interesting historical viewpoint. </p> <!-- endregion --> <!-- #region Open Container Initiative (OCI) --> <h2 id="oci">Open Container Initiative (OCI)</h2> <div class='imgWrapper imgFlex inline fullsize' style=' '> <a href='https://opencontainers.org/' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/buildahPodman/oci_logo.svg" type="image/svg"> <!---<source srcset="/blog/images/buildahPodman/oci_logo.avif" type="image/avif">--> <source srcset="/blog/images/buildahPodman/oci_logo.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/oci_logo.apng" type="image/apng"> <source srcset="/blog/images/buildahPodman/oci_logo.png" type="image/png"> <source srcset="/blog/images/buildahPodman/oci_logo.jpg" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/oci_logo.jpeg" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/oci_logo.jfif" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/oci_logo.pjpeg" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/oci_logo.pjp" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/oci_logo.gif" type="image/gif"> <source srcset="/blog/images/buildahPodman/oci_logo.tif" type="image/tiff"> <source srcset="/blog/images/buildahPodman/oci_logo.tiff" type="image/tiff"> <source srcset="/blog/images/buildahPodman/oci_logo.bmp" type="image/bmp"> <source srcset="/blog/images/buildahPodman/oci_logo.ico" type="image/x-icon"> <source srcset="/blog/images/buildahPodman/oci_logo.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/buildahPodman/oci_logo.png" style='width: 100%; ' /> </picture> </a> </div> <p> The latest evolution of Docker-compatible images, <a href='https://github.com/opencontainers/image-spec' target='_blank' rel="nofollow">OCI image format</a> (not to be confused with <a href='https://www.oracle.com/ca-en/cloud/' target='_blank' rel="nofollow">Oracle Cloud Infrastructure</a>), is compatible with: </p> <ul style="column-count: 2;"> <li><a href='https://aws.amazon.com/lambda/' target='_blank' rel="nofollow">AWS Lambda</a></li> <li><a href='https://azure.microsoft.com/en-us/services/functions/' target='_blank' rel="nofollow">Azure Functions</a></li> <li><a href='https://azure.microsoft.com/en-us/services/kubernetes-service/' target='_blank' rel="nofollow">Azure Kubernetes Service</a></li> <li><a href='https://buildah.io/' target='_blank' rel="nofollow">Buildah</a></li> <li><a href='https://buildpacks.io/' target='_blank' rel="nofollow">Cloud Native Buildpacks</a></li> <li><a href='https://circleci.com/' target='_blank' rel="nofollow">CircleCI</a></li> <li><a href='https://www.docker.com/' target='_blank' rel="nofollow">Docker</a></li> <li><a href='https://dokku.com/' target='_blank' rel="nofollow">Dokku</a></li> <li><a href='https://gitlab.com' target='_blank' rel="nofollow">GitLab</a></li> <li><a href='https://cloud.google.com/container-registry/docs/image-formats' target='_blank' rel="nofollow">Google Cloud</a></li> <li><a href='https://heroku.com' target='_blank' rel="nofollow">Heroku</a></li> <li><a href='https://containerjournal.com/topics/container-management/what-is-knative-and-what-can-it-do-for-you/' target='_blank' rel="nofollow">Knative</a></li> <li><a href='https://kubernetes.io/' target='_blank' rel="nofollow">Kubernetes</a></li> <li><a href='https://podman.io/' target='_blank' rel="nofollow"><code>podman</code></a></li> <li><a href='https://github.com/containers/skopeo' target='_blank' rel="nofollow"><code>skopeo</code></a></li> <li><a href='https://spring.io/guides/topicals/spring-boot-docker/' target='_blank' rel="nofollow">Spring Boot</a></li> <li><a href='https://cloud.google.com/tekton' target='_blank' rel="nofollow">Tekton</a></li> </ul> <p> Supported OCI formats include: </p> <ul style="column-count: 2;"> <li>Docker containers schema 1</li> <li>Docker containers schema 2</li> <li>Pods (groups of containers)</li> <li>Images</li> <li>Volumes</li> </ul> <!-- endregion --> <!-- #region Buildah, podman and skopeo --> <h2 id="three">Buildah, podman and skopeo</h2> <!-- #region implicit --> <p> This article discusses 3 related open source projects from RedHat / IBM that provide an alternative to Docker: Buildah, <code>podman</code> and <code>skopeo</code>. These 3 projects share a common source code base, and are daemonless tools for managing Open Container Initiative (OCI) images. </p> <p> Paraphrasing the reasons expressed in <a href='https://developers.redhat.com/blog/2019/02/21/podman-and-buildah-for-docker-users/' target='_blank' rel="nofollow">Podman and Buildah for Docker Users</a> for using <code>podman</code> instead of Docker, wherever <code>podman</code> is mentioned, read &ldquo;<code>podman</code>, Buildah and <code>skopeo</code>&rdquo;: </p> <p class="quoteCite" cite="From &ldquo;Podman and Buildah for Docker Users&rdquo;"> The Podman approach is simply to directly interact with the image registry, with the container and image storage, and with the Linux kernel through the <code>runC</code> container runtime process (not a daemon).<br><br> Running Podman as a normal user means that Podman will, by default, store images and containers in the user’s home directory. Podman uses a repository in the user’s home directory: <code>~/.local/share/containers</code> (instead of <code>/var/lib/docker</code>).<br><br> Despite the new locations for the local repositories, the images created by Docker and Podman are compatible with the OCI standard. Podman can push to and pull from popular container registries like Quay.io and Docker hub, as well as private registries. </p> <!-- endregion --> <!-- #region Buildah vs. podman --> <h2 id="buildah_vs_podman">Buildah vs. podman</h2> <p> <code>Podman</code> can build OCI containers interactively or in batch mode. You can either build using a <code>Dockerfile</code> using <code>podman build</code> (batch mode), or you can interactively run a container, make changes to the running image, and then <code>podman commit</code> those changes to a new image tag. </p> <p> Buildah was written before <code>podman</code>. Some of Buildah's source code for creating and managing container images was ported to <code>podman</code>. The <code>podman build</code> command is a subset of Buildah&rsquo;s functionality. </p> <p> <p> However, apparently the differences between the two programs are important: </p> <p class="quote"> Buildah builds OCI images. Confusingly, <code>podman build</code> can also be used to build Docker images also, but it’s incredibly slow and used up a lot of disk space by using the <code>vfs</code> storage driver by default. <code>buildah bud</code> (‘build using Dockerfile’) was much faster for me, and uses the overlay storage driver. <br><br> &nbsp; &ndash; From <a href='https://zwischenzugs.com/page/3/' target='_blank' rel="nofollow">Goodbye Docker: Purging is Such Sweet Sorrow</a> by Ian Miell. </p> <!-- endregion --> <!-- endregion --> <!-- #region podman --> <h2 id="podman">podman</h2> <!-- #region implicit --> <div class='imgWrapper imgFlex inline' style=' '> <a href='https://podman.io' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/buildahPodman/podman-logo.svg" type="image/svg"> <!---<source srcset="/blog/images/buildahPodman/podman-logo.avif" type="image/avif">--> <source srcset="/blog/images/buildahPodman/podman-logo.webp" type="image/webp"> <source srcset="/blog/images/buildahPodman/podman-logo.apng" type="image/apng"> <source srcset="/blog/images/buildahPodman/podman-logo.png" type="image/png"> <source srcset="/blog/images/buildahPodman/podman-logo.jpg" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/podman-logo.jpeg" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/podman-logo.jfif" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/podman-logo.pjpeg" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/podman-logo.pjp" type="image/jpeg"> <source srcset="/blog/images/buildahPodman/podman-logo.gif" type="image/gif"> <source srcset="/blog/images/buildahPodman/podman-logo.tif" type="image/tiff"> <source srcset="/blog/images/buildahPodman/podman-logo.tiff" type="image/tiff"> <source srcset="/blog/images/buildahPodman/podman-logo.bmp" type="image/bmp"> <source srcset="/blog/images/buildahPodman/podman-logo.ico" type="image/x-icon"> <source srcset="/blog/images/buildahPodman/podman-logo.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/buildahPodman/podman-logo.png" style='width: 100%; padding: 1em' /> </picture> </a> </div> <p> <code>Podman</code> supports developing, managing, and running OCI Containers on Linux systems, including WSL, without requiring <code>root</code> privilege. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>shell Installation on Ubuntu</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id873ea4a617a0'><button class='copyBtn' data-clipboard-target='#id873ea4a617a0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install buildah podman skopeo</pre> </div> <div class="pullQuote"> Podman commands are very nearly the same as Docker’s. </div> <p> Because <code>podman</code> is a drop-in replacement for <code>docker</code>, the following alias enables the <code>docker</code> command to invoke <code>podman</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>~/.bash_aliases</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7baf32cd3471'><button class='copyBtn' data-clipboard-target='#id7baf32cd3471' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>alias docker=podman</pre> </div> <p> As described in <a href='https://www.vultr.com/docs/how-to-install-and-use-podman-on-ubuntu-20-04' target='_blank' rel="nofollow">How to Install and Use Podman on Ubuntu 20.04</a>, I added <code>'registry.access.redhat.com'</code> to the list of <code>registries</code> in <code>/etc/containers/registries.conf</code>. I also added <a href='https://gallery.ecr.aws/' target='_blank' rel="nofollow"><code>gallery.ecr.aws</code></a> and <a href='https://cloud.google.com/container-registry/docs/pushing-and-pulling#add-registry' target='_blank' rel="nofollow"><code>gcr.io</code></a>. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>/etc/containers/registries.conf</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id87c5bc245cb8'><button class='copyBtn' data-clipboard-target='#id87c5bc245cb8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[registries.search] registries = ['docker.io', 'gallery.ecr.aws', 'gcr.io', 'quay.io', 'registry.access.redhat.com']</pre> </div> <!-- endregion --> <!-- #region podman Help--> <h3 id="podmanHelp"><span class="code">podman</span> Help</h3> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id53553b0e7b17'><button class='copyBtn' data-clipboard-target='#id53553b0e7b17' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman --help <span class='unselectable'>Manage pods, containers and images<br/> Usage: podman [flags] podman [command]<br/> Available Commands: attach Attach to a running container auto-update Auto update containers according to their auto-update policy build Build an image using instructions from Containerfiles commit Create new image based on the changed container container Manage containers cp Copy files/folders between a container and the local filesystem create Create but do not start a container diff Display the changes to the object&#39;s file system events Show podman events exec Run a process in a running container export Export container&#39;s filesystem contents as a tar archive generate Generate structured data based on containers and pods. healthcheck Manage health checks on containers help Help about any command history Show history of a specified image image Manage images images List images in local storage import Import a tarball to create a filesystem image info Display podman system information init Initialize one or more containers inspect Display the configuration of object denoted by ID kill Kill one or more running containers with a specific signal load Load an image from container archive login Login to a container registry logout Logout of a container registry logs Fetch the logs of one or more containers manifest Manipulate manifest lists and image indexes mount Mount a working container&#39;s root filesystem network Manage networks pause Pause all the processes in one or more containers play Play a pod and its containers from a structured file. pod Manage pods port List port mappings or a specific mapping for the container ps List containers pull Pull an image from a registry push Push an image to a specified destination restart Restart one or more containers rm Remove one or more containers rmi Removes one or more images from local storage run Run a command in a new container save Save image to an archive search Search registry for image start Start one or more containers stats Display a live stream of container resource usage statistics stop Stop one or more containers system Manage podman tag Add an additional name to a local image top Display the running processes of a container unmount Unmounts working container&#39;s root filesystem unpause Unpause the processes in one or more containers unshare Run a command in a modified user namespace untag Remove a name from a local image version Display the Podman Version Information volume Manage volumes wait Block on one or more containers<br/> Flags: --cgroup-manager string Cgroup manager to use (&quot;cgroupfs&quot;|&quot;systemd&quot;) (default &quot;cgroupfs&quot;) --cni-config-dir string Path of the configuration directory for CNI networks --conmon string Path of the conmon binary -c, --connection string Connection to use for remote Podman service --events-backend string Events backend to use (&quot;file&quot;|&quot;journald&quot;|&quot;none&quot;) (default &quot;file&quot;) --help Help for podman --hooks-dir strings Set the OCI hooks directory path (may be set multiple times) (default [/usr/share/containers/oci/hooks.d]) --identity string path to SSH identity file, (CONTAINER_SSHKEY) --log-level string Log messages above specified level (debug, info, warn, error, fatal, panic) (default &quot;error&quot;) --namespace string Set the libpod namespace, used to create separate views of the containers and pods on the system --network-cmd-path string Path to the command for configuring the network -r, --remote Access remote Podman service (default false) --root string Path to the root directory in which data, including images, is stored --runroot string Path to the &#39;run directory&#39; where all state information is stored --runtime string Path to the OCI-compatible binary used to run containers, default is /usr/bin/runc --storage-driver string Select which storage driver is used to manage storage of images and containers (default is overlay) --storage-opt stringArray Used to pass an option to the storage driver --syslog Output logging information to syslog as well as the console (default false) --tmpdir string Path to the tmp directory for libpod state content.<br> Note: use the environment variable &#39;TMPDIR&#39; to change the temporary storage location for container images, &#39;/var/tmp&#39;.<br> --url string URL to access Podman service (CONTAINER_HOST) (default &quot;unix:/home/mslinn/.docker/run/podman/podman.sock&quot;) -v, --version Version of Podman<br/> Use &quot;podman [command] --help&quot; for more information about a command. </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region podman info --> <h3 id="padman_info"><span class="code">podman info</span></h3> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id29b232abf2f9'><button class='copyBtn' data-clipboard-target='#id29b232abf2f9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman info <span class='unselectable'>host: arch: amd64 buildahVersion: 1.15.2 cgroupVersion: v1 conmon: package: &#39;conmon: /usr/libexec/podman/conmon&#39; path: /usr/libexec/podman/conmon version: &#39;conmon version 2.0.20, commit: unknown&#39; cpus: 8 distribution: distribution: ubuntu version: &quot;20.10&quot; eventLogger: file hostname: Bear idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 kernel: 5.4.72-microsoft-standard-WSL2 linkmode: dynamic memFree: 897724416 memTotal: 6231638016 ociRuntime: name: runc package: &#39;containerd.io: /usr/bin/runc&#39; path: /usr/bin/runc version: |- runc version 1.0.0-rc93 commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec spec: 1.0.2-dev go: go1.13.15 libseccomp: 2.5.1 os: linux remoteSocket: path: /home/mslinn/.docker/run/podman/podman.sock rootless: true slirp4netns: executable: /bin/slirp4netns package: Unknown version: |- slirp4netns version 1.0.1 commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4 libslirp: 4.3.1 swapFree: 0 swapTotal: 0 uptime: 306h 23m 6.37s (Approximately 12.75 days) registries: search: - quay.io - docker.io - gallery.ecr.aws - registry.access.redhat.com store: configFile: /home/mslinn/.config/containers/storage.conf containerStore: number: 8 paused: 0 running: 0 stopped: 8 graphDriverName: overlay graphOptions: overlay.mount_program: Executable: /bin/fuse-overlayfs Package: Unknown Version: |- fusermount3 version: 3.9.3 fuse-overlayfs: version 1.0.0 FUSE library version 3.9.3 using FUSE kernel interface version 7.31 graphRoot: /home/mslinn/.local/share/containers/storage graphStatus: Backing Filesystem: extfs Native Overlay Diff: &quot;false&quot; Supports d_type: &quot;true&quot; Using metacopy: &quot;false&quot; imageStore: number: 4 runRoot: /home/mslinn/.docker/run/containers volumePath: /home/mslinn/.local/share/containers/storage/volumes version: APIVersion: 1 Built: 0 BuiltTime: Wed Dec 31 19:00:00 1969 GitCommit: &quot;&quot; GoVersion: go1.14.7 OsArch: linux/amd64 Version: 2.0.6 </span></pre> </div> <!-- endregion --> <!-- #region podman container help --> <h3 id="padman_container_help"><span class="code">podman container</span> Help</h3> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5f1fe1b3b774'><button class='copyBtn' data-clipboard-target='#id5f1fe1b3b774' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>man podman-container <span class='unselectable'>podman-container(1) General Commands Manual podman-container(1)<br/> NAME podman-container - Manage containers<br/> SYNOPSIS podman container subcommand<br/> DESCRIPTION The container command allows you to manage containers<br/> COMMANDS &#9484;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9516;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9516;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9488; &#9474;Command &#9474; Man Page &#9474; Description &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;attach &#9474; podman-attach(1) &#9474; Attach to a running container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;checkpoint &#9474; podman-container-checkpoint(1) &#9474; Checkpoints one or more running &#9474; &#9474; &#9474; &#9474; containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;cleanup &#9474; podman-container-cleanup(1) &#9474; Cleanup the container&#39;s network &#9474; &#9474; &#9474; &#9474; and mountpoints. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;commit &#9474; podman-commit(1) &#9474; Create new image based on the &#9474; &#9474; &#9474; &#9474; changed container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;cp &#9474; podman-cp(1) &#9474; Copy files/folders between a &#9474; &#9474; &#9474; &#9474; container and the local &#9474; &#9474; &#9474; &#9474; filesystem. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;create &#9474; podman-create(1) &#9474; Create a new container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;diff &#9474; podman-diff(1) &#9474; Inspect changes on a container or &#9474; &#9474; &#9474; &#9474; image&#39;s filesystem. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;exec &#9474; podman-exec(1) &#9474; Execute a command in a running &#9474; &#9474; &#9474; &#9474; container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;exists &#9474; podman-container-exists(1) &#9474; Check if a container exists in &#9474; &#9474; &#9474; &#9474; local storage &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;export &#9474; podman-export(1) &#9474; Export a container&#39;s filesystem &#9474; &#9474; &#9474; &#9474; contents as a tar archive. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;init &#9474; podman-init(1) &#9474; Initialize a container &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;inspect &#9474; podman-inspect(1) &#9474; Display a container or image&#39;s &#9474; &#9474; &#9474; &#9474; configuration. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;kill &#9474; podman-kill(1) &#9474; Kill the main process in one or &#9474; &#9474; &#9474; &#9474; more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;list &#9474; podman-ps(1) &#9474; List the containers on the &#9474; &#9474; &#9474; &#9474; system.(alias ls) &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;logs &#9474; podman-logs(1) &#9474; Display the logs of a container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;mount &#9474; podman-mount(1) &#9474; Mount a working container&#39;s root &#9474; &#9474; &#9474; &#9474; filesystem. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;pause &#9474; podman-pause(1) &#9474; Pause one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;port &#9474; podman-port(1) &#9474; List port mappings for the &#9474; &#9474; &#9474; &#9474; container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;prune &#9474; podman-container-prune(1) &#9474; Remove all stopped containers &#9474; &#9474; &#9474; &#9474; from local storage. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;restart &#9474; podman-restart(1) &#9474; Restart one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;restore &#9474; podman-container-restore(1) &#9474; Restores one or more containers &#9474; &#9474; &#9474; &#9474; from a checkpoint. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;rm &#9474; podman-rm(1) &#9474; Remove one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;run &#9474; podman-run(1) &#9474; Run a command in a container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;runlabel &#9474; podman-container-runlabel(1) &#9474; Executes a command as described &#9474; &#9474; &#9474; &#9474; by a container image label. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;start &#9474; podman-start(1) &#9474; Starts one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;stats &#9474; podman-stats(1) &#9474; Display a live stream of one or &#9474; &#9474; &#9474; &#9474; more container&#39;s resource usage &#9474; &#9474; &#9474; &#9474; statistics. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;stop &#9474; podman-stop(1) &#9474; Stop one or more running &#9474; &#9474; &#9474; &#9474; containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;top &#9474; podman-top(1) &#9474; Display the running processes of &#9474; &#9474; &#9474; &#9474; a container. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;unmount &#9474; podman-unmount(1) &#9474; Unmount a working container&#39;s &#9474; &#9474; &#9474; &#9474; root filesystem.(Alias unmount) &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;unpause &#9474; podman-unpause(1) &#9474; Unpause one or more containers. &#9474; &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9532;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9508; &#9474;wait &#9474; podman-wait(1) &#9474; Wait on one or more containers to &#9474; &#9474; &#9474; &#9474; stop and print their exit codes. &#9474; &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9524;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9524;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9496;<br/> SEE ALSO podman, podman-exec, podman-run<br/> podman-container(1) </span></pre> </div> <!-- endregion --> <!-- #region Podman Run Help --> <h3 id="podman_help">Podman Run Help</h3> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id62d21eed53d7'><button class='copyBtn' data-clipboard-target='#id62d21eed53d7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>podman container run --help <span class='unselectable'>Run a command in a new container<br/> Description: Runs a command in a new container from the given image<br/> Usage: podman container run [flags] IMAGE [COMMAND [ARG...]]<br/> Examples: podman container run imageID ls -alF /etc podman container run --network=host imageID dnf -y install java podman container run --volume /var/hostdir:/var/ctrdir -i -t fedora /bin/bash<br/> Flags: --add-host strings Add a custom host-to-IP mapping (host:ip) (default []) --annotation strings Add annotations to container (key:value) -a, --attach strings Attach to STDIN, STDOUT or STDERR --authfile string Path of the authentication file. Use REGISTRY_AUTH_FILE environment variable to override --blkio-weight string Block IO weight (relative weight) accepts a weight value between 10 and 1000. --blkio-weight-device DEVICE_NAME:WEIGHT Block IO weight (relative device weight, format: DEVICE_NAME:WEIGHT) --cap-add strings Add capabilities to the container --cap-drop strings Drop capabilities from the container --cgroup-parent string Optional parent cgroup for the container --cgroupns string cgroup namespace to use --cgroups string control container cgroup configuration (&quot;enabled&quot;|&quot;disabled&quot;|&quot;no-conmon&quot;) (default &quot;enabled&quot;) --cidfile string Write the container ID to the file --conmon-pidfile string Path to the file that will receive the PID of conmon --cpu-period uint Limit the CPU CFS (Completely Fair Scheduler) period --cpu-quota int Limit the CPU CFS (Completely Fair Scheduler) quota --cpu-rt-period uint Limit the CPU real-time period in microseconds --cpu-rt-runtime int Limit the CPU real-time runtime in microseconds --cpu-shares uint CPU shares (relative weight) --cpus float Number of CPUs. The default is 0.000 which means no limit --cpuset-cpus string CPUs in which to allow execution (0-3, 0,1) --cpuset-mems string Memory nodes (MEMs) in which to allow execution (0-3, 0,1). Only effective on NUMA systems. -d, --detach Run container in background and print container ID --detach-keys [a-Z] Override the key sequence for detaching a container. Format is a single character [a-Z] or a comma separated sequence of `ctrl-&lt;value&gt;`, where `&lt;value&gt;` is one of: `a-cf`, `@`, `^`, `[`, `\`, `]`, `^` or `_` (default &quot;ctrl-p,ctrl-q&quot;) --device strings Add a host device to the container --device-cgroup-rule strings Add a rule to the cgroup allowed devices list --device-read-bps strings Limit read rate (bytes per second) from a device (e.g. --device-read-bps=/dev/sda:1mb) --device-read-iops strings Limit read rate (IO per second) from a device (e.g. --device-read-iops=/dev/sda:1000) --device-write-bps strings Limit write rate (bytes per second) to a device (e.g. --device-write-bps=/dev/sda:1mb) --device-write-iops strings Limit write rate (IO per second) to a device (e.g. --device-write-iops=/dev/sda:1000) --disable-content-trust This is a Docker specific option and is a NOOP --dns strings Set custom DNS servers --dns-opt strings Set custom DNS options --dns-search strings Set custom DNS search domains --entrypoint string Overwrite the default ENTRYPOINT of the image -e, --env stringArray Set environment variables in container (default [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,TERM=xterm]) --env-file strings Read in a file of environment variables --env-host Use all current