Mike Slinn's Blog 2024-07-11T16:35:26-04:00 https://mslinn.github.io/blog Mike Slinn mslinn@gmail.com Thermal Control 2024-07-05T00:00:00-04:00 https://mslinn.github.io/blog/2024/07/05/cooling <!-- #region intro --> <p> This article discusses some of the physics and a few of the electrical components for controlling temperature in a consumer electronics product. </p> <h2 id="miniseries">Mini-Series</h2> <p> This article can be read standalone; however, is part of a mini-series that discusses how to increase the storage capacity and performance of the Ableton Push 3 Standalone (P3S). If you own a P3S, or are thinking of purchasing one, you might want to first <a href='/av_studio/570-ableton-push-standalone.html'>read my review</a>. </p> <div class='imgWrapper imgFlex inline' style=' '> <a href='/av_studio/570-ableton-push-standalone.html' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/p3s_top.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/p3s_top.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/p3s_top.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/p3s_top.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/p3s_top.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/p3s_top.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/p3s_top.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/p3s_top.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/p3s_top.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/p3s_top.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/p3s_top.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/p3s_top.png" style='width: 100%; ' /> </picture> </a> </div> <p> The articles in this mini-series are: </p> <ol> <li><a href='/av_studio/570-ableton-push-nvme.html'>Ableton Push 3 NVMe Replacement</a></li> <li><a href='/blog/2024/07/05/cooling.html'>Thermal Control</a></li> <li><a href='/blog/2024/06/11/virtualbox.html'>VirtualBox</a></li> <li><a href='/blog/2024/06/12/clonezilla.html'>Clonezilla</a></li> <li><a href='/blog/2024/06/12/supersize-fs.html'>Supersizing a Partition and Its File System</a></li> </ol> <!-- endregion --> <!-- #region Thermal Pads --> <h2 id="pads">Thermal Pads</h2> <!-- #region implicit --> <p> A thermal pad is made from <a href='https://en.wikipedia.org/wiki/Thermal_interface_material' target='_blank' rel="nofollow">thermal interface material</a>; its purpose in the article I wrote entitled <a href='/570-ableton-push-nvme.html'>Ableton Push 3 NVMe Replacement</a> is to thermally connect the chips on the NVMe drive to a metal heat sink. Thermal pads are often baby blue; however, as shown in the above photographs, the P3S uses a pink thermal pad between the CPU module and the bottom plate, and, as we saw at the beginning of this article, another pink thermal pad between the external heat sink and the bottom plate. </p> <div class='imgWrapper imgBlock center' style='width: 450px; '> <figure> <a href='https://www.amazon.com/gp/product/B07X82MNF2' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/thermal_pads.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/thermal_pads.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/thermal_pads.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/thermal_pads.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/thermal_pads.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/thermal_pads.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/thermal_pads.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/thermal_pads.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/thermal_pads.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/thermal_pads.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/thermal_pads.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/thermal_pads.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/thermal_pads.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/thermal_pads.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/thermal_pads.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/thermal_pads.cur" type="image/x-icon"> <img alt='Various thicknesses of thermal pads' class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/thermal_pads.png" style='width: 100%; padding: 1em;' title='Various thicknesses of thermal pads' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://www.amazon.com/gp/product/B07X82MNF2" target='_blank' > Various thicknesses of thermal pads </a> </figcaption> </figure> </div> <p> Depending on the quality and conditions of use, <a href='https://www.aok-technologies.com/news/did-the-thermal-pad-paste-degrade-to-a-level-that-is-less-efficient.html' target='_blank' rel="nofollow">a thermal pad can last anywhere from 5 to 10 years</a>. </p> <div class='imgWrapper imgFlex inline' style=' '> <figure> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/push_open_8.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/push_open_8.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/push_open_8.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/push_open_8.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/push_open_8.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/push_open_8.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/push_open_8.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/push_open_8.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/push_open_8.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/push_open_8.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/push_open_8.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/push_open_8.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/push_open_8.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/push_open_8.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/push_open_8.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/push_open_8.cur" type="image/x-icon"> <img alt='The pink thermal pad cools the P3S CPU module' class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/push_open_8.png" style='width: 100%; ' title='The pink thermal pad cools the P3S CPU module' /> </picture> <figcaption class='imgFigCaption '> The pink thermal pad cools the P3S CPU module </figcaption> </figure> </div> <!-- endregion --> <!-- #region Folklore vs. Provable Facts --> <h2 id="fvpf">Folklore vs. Provable Facts</h2> <p> Some manufacturers state that thermal pads should not ever be reused, citing the introduction of air gaps and bubbles. Apparently reusing a thermal pad would cause the thermal pads to act more as an insulator than as a conductor. I do not have data that would indicate how well-founded the statement is. Someone should test this; the information source obviously has a profit motive. </p> <p> Also from two manufacturers: When as thermal pad is removed, the surface should be cleaned with alcohol wipes and a microfiber cloth to ensure no traces are left behind. Again, I would like to see data on this. </p> <!-- endregion --> <!-- endregion --> <!-- #region Thermal Conductivity --> <h2 id="tc">Thermal Conductivity</h2> <!-- #region implicit --> <p> Heat transfer only occurs 3 ways: via conduction, convection, and radiation. Of the three, conduction is by far the most effective. </p> <div class='imgWrapper imgFlex inline' style=' '> <figure> <a href='https://www.noaa.gov/jetstream/atmosphere/transfer-of-heat-energy' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/cooling/conv_rad_cond.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/cooling/conv_rad_cond.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/cooling/conv_rad_cond.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/cooling/conv_rad_cond.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/cooling/conv_rad_cond.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/cooling/conv_rad_cond.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/conv_rad_cond.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/conv_rad_cond.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/conv_rad_cond.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/conv_rad_cond.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/conv_rad_cond.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/cooling/conv_rad_cond.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/cooling/conv_rad_cond.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/cooling/conv_rad_cond.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/cooling/conv_rad_cond.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/cooling/conv_rad_cond.cur" type="image/x-icon"> <img alt='NOAA: The Transfer of Heat Energy' class="imgImg rounded shadow" src="/av_studio/images/ableton/push/cooling/conv_rad_cond.png" style='width: 100%; ' title='NOAA: The Transfer of Heat Energy' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://www.noaa.gov/jetstream/atmosphere/transfer-of-heat-energy" target='_blank' > NOAA: The Transfer of Heat Energy </a> </figcaption> </figure> </div> <table class='noborder table'> <tr> <th>Mode</th> <th>Transfer Mechanism</th> <th>Material</th> </tr> <tr> <th>Conduction</th> <td>Diffusion of electrons and <a href='https://en.wikipedia.org/wiki/Phonon' target='_blank' rel="nofollow">phonon</a> vibrations</td> <td>Solids</td> </tr> <tr> <th>Convection</th> <td>Density differences cause currents</td> <td>Gases and liquids</td> </tr> <tr> <th>Radiation</th> <td>Electromagnetic radiation</td> <td>Non-opaque mediums</td> </tr> </table> <p> This article focuses on conduction, because it is the dominant cooling mode for the P3S. </p> <p> The <i>thermal conductivity</i> of a material characterizes its ability to transfer heat. For solids, thermal conductivity is due to the flow of free electrons, rather like how electricity is conducted. Materials with high thermal conductivity transfer heat faster than materials with low thermal conductivity. The thermal conductivity tells you the rate at which thermal energy is transported across a unit length of a material that has a unit temperature difference applied across that length. </p> <p> Thermal conductivity in an electrical device depends upon: </p> <ul> <li>Density of material</li> <li>Pressure</li> <li>Temperature</li> <li>Temperature differential</li> <li>Material structure</li> </ul> <p> In the <a href='https://en.wikipedia.org/wiki/International_System_of_Units' target='_blank' rel="nofollow">SI unit system</a>, thermal conductivity is measured in watts per meter per kelvin (W/mK). This defines the number of watts conducted per meter thickness of the material, per degree of temperature difference between the two surfaces. </p> <p> The following table is ordered by increasing thermal conductivity, with thermal insulators at the top and the best thermal conductors at the bottom. </p> <style> #table1 td:nth-child(3) { text-align: center; } </style> <table id='table1' class='table'> <tr> <th>Material</th> <th>Comment</th> <th>Conductivity</th> </tr> <tr> <th>Air</th> <td> Effectively a thermal insulator. </td> <td>0.026 W/mK</td> </tr> <tr> <th>Powder coating</th> <td> Used on computer chassis, and on the P3S bottom plate. Effectively a thermal insulator. </td> <td>0.174 W/mK</td> </tr> <tr> <th>Anodized aluminum coating</th> <td> Used to coat the P3S heat sink underneath the P3S bottom plate. Anodizing means adding a layer of <a href='https://www.britannica.com/science/alumina' target='_blank' rel="nofollow">alumina</a> to the aluminum surface. This thin layer of oxide is effectively a thermal insulator. Anodized aluminum has a higher surface area than unfinished aluminum and, therefore, radiates more infrared energy; however, radiation is orders of magnitude less effective than conduction for cooling. </td> <td>0.53-1.62 W/mK</td> </tr> <tr> <th>Marble</th> <td> Used for kitchen table tops. </td> <td>2.8 W/mK</td> </tr> <tr> <th>Thermal paste</th> <td> Commonly used between a CPU and its heatsink. Shelf life is 3 to 5 years, durability (working life) is 5 to 8 years. </td> <td>5 &mdash; 8 W/mK</td> </tr> <tr> <th>Thermal pads</th> <td>Easily replaced</td> <td> <p style="text-align: center; "> 3 &mdash; 16 W/mK </p> <p> The blue thermal silica gel pads that I purchased were rated at 6 W/mK. </p> <p> <a href='https://www.tglobaltechnology.com/product/tg-a1450-ultra-soft-thermal-pad/' target='_blank' rel="nofollow">The pink thermal pads of the P3S might be rated at 14.5 W/mK.</a> </p> </td> </tr> <tr> <th>Low-carbon steel</th> <td> Sheet metal, used for computer chassis, which might include the P3S bottom plate. Surface treatment, such as anodizing, generally reduces this value. </td> <td>34-54 W/mK</td> </tr> <tr> <th>Liquid metal paste (<a href='https://www.thermal-grizzly.com/en/conductonaut/s-tg-c-001-r' target='_blank' rel="nofollow">Conductonaut</a>)</th> <td>Can be <a href='https://www.reddit.com/r/overclocking/comments/100vcgm/liquid_metal_thermal_compound_2_years_after/' target='_blank' rel="nofollow">highly dangerous</a> for electronics and aluminum parts.</td> <td>73 W/mK</td> </tr> <tr> <th>Solid aluminum</th> <td> Often used in heat sinks, for example underneath the P3S bottom plate. However, the P3S exterior heat sink is anodized, which greatly reduces its thermal conductivity. </td> <td>150-205 W/mK</td> </tr> <tr> <th>Solid copper</th> <td>Used on motherboards</td> <td>357-413 W/mK</td> </tr> <tr style="border: red thick solid"> <th>Heat pipe</th> <td>Used in a wide variety of electronics, including laptops and mobile devices</td> <td><span class='bg_yellow'>4,000 to 100,000 W/mK</span></td> </tr> </table> <p class="alert rounded shadow"> Notice the huge disparity between the thermal conductivity of thermal pads, the metals and the coatings that the bottom of the P3S is constructed from. This tells us that although thermal pads are much better than simply relying on convection or radiation, thermal pads and the coatings greatly restrict the transfer of heat generated by the P3S electronics. </p> <p> If a heat pipe was used effectively, the overheating problems of the P3S would completely go away. Furthermore, the heat pipe would allow more powerful and faster electrical components could be used. </p> <!-- #region Heat Pipes --> <h3 id="pipes">Heat Pipes</h3> <div class='imgWrapper imgBlock inline' style='width: 100%; '> <figure> <a href='https://commons.wikimedia.org/w/index.php?curid=13144188' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.cur" type="image/x-icon"> <img alt='Heat pipes in a laptop<br>Photo by Kristoferb at English Wikipedia, CC BY-SA 3.0' class="imgImg rounded shadow" src="/av_studio/images/ableton/push/cooling/Laptop_Heat_Pipe.png" style='width: 100%; ' title='Heat pipes in a laptop<br>Photo by Kristoferb at English Wikipedia, CC BY-SA 3.0' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://commons.wikimedia.org/w/index.php?curid=13144188" target='_blank' > Heat pipes in a laptop<br>Photo by Kristoferb at English Wikipedia, CC BY-SA 3.0 </a> </figcaption> </figure> </div> <p> Thermal pads, fans and <a href='https://en.wikipedia.org/wiki/Heat_pipe' target='_blank' rel="nofollow">heat pipes</a> are commonly used in laptops for dissipating heat. Heat pipes do not require power, are very light, and provide extremely high <i>heat flux</i> &mdash; from 10x to 200x more than pure copper. <a href='https://en.wikipedia.org/wiki/Heat_flux' target='_blank' rel="nofollow">Heat flux</a> (also known as thermal flux, heat flux density, heat flow density and heat flow rate intensity) is the amount of heat energy passing through a surface, measured as the flow of energy per unit of area per unit of time. It is a vector quantity. </p> <div class='imgWrapper imgFlex inline' style=' '> <figure> <a href='https://www.arquimea.com/products/heat-pipes-satellite-space-applications/' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/cooling/heat-pipes.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/cooling/heat-pipes.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/cooling/heat-pipes.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/cooling/heat-pipes.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/cooling/heat-pipes.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/cooling/heat-pipes.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/heat-pipes.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/heat-pipes.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/heat-pipes.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/heat-pipes.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/heat-pipes.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/cooling/heat-pipes.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/cooling/heat-pipes.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/cooling/heat-pipes.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/cooling/heat-pipes.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/cooling/heat-pipes.cur" type="image/x-icon"> <img alt='Heat pipes for thermally controlling a satellite' class="imgImg rounded shadow" src="/av_studio/images/ableton/push/cooling/heat-pipes.png" style='width: 100%; ' title='Heat pipes for thermally controlling a satellite' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://www.arquimea.com/products/heat-pipes-satellite-space-applications/" target='_blank' > Heat pipes for thermally controlling a satellite </a> </figcaption> </figure> </div> <p> The following video by James Orgill of <a href='https://www.youtube.com/@TheActionLab' target='_blank' rel="nofollow">The Action Lab</a> shows that heat pipes transfer heat energy dramatically faster than solid materials. </p> <a href="https://www.youtube.com/shorts/ZEGazGMEhtw"> <video controls class="rounded shadow" src="/av_studio/images/ableton/push/cooling/The_Fastest_Heat_Conductor.mp4" style="display: block; margin: 0 auto 2em auto;"></video> </a> <p> This video explains how heat pipesmic work in detail: </p> <style>.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }</style><div class='embed-container'> <iframe title="YouTube video player" width="640" height="390" src="//www.youtube.com/embed/51bwzEO8XCw" frameborder="0" allowfullscreen></iframe></div> <p> Perhaps Ableton might look at using a heat pipe in a future version of the P3S that features a more powerful CPU and larger NVMe drive. </p> <!-- endregion --> <!-- endregion --> <!-- #region Heat Sinks --> <h2>Heat Sinks</h2> <p> Heat pipes can be combined with other components for heat transfer, such as <a href='https://www.chtechnology.com/heat-sinks/extruded-heat-sinks.html' target='_blank' rel="nofollow">extruded heat sinks</a>, <a href='https://www.famosheatsink.com/news/die-cast-heat-sinks-vs-extruded-heat-sinks/' target='_blank' rel="nofollow">die-cast heat sinks</a>, <a href='https://www.lorithermal.com/stacked-fin-heat-sink' target='_blank' rel="nofollow">fin-stack heat sinks</a>, and <a href='https://myheatsinks.com/custom/skived-heat-sinks/' target='_blank' rel="nofollow">skived heat sinks</a>. Heat transfer materials thermally bond the heat pipes to the heat sinks. This creates a thermal management system that can be individually tuned for each application. </p> <div class='imgWrapper imgFlex inline' style=' '> <figure> <a href='https://www.lorithermal.com/stacked-fin-heat-sink' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/cooling/stacked_fin.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/cooling/stacked_fin.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/cooling/stacked_fin.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/cooling/stacked_fin.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/cooling/stacked_fin.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/cooling/stacked_fin.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/stacked_fin.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/stacked_fin.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/stacked_fin.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/stacked_fin.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/stacked_fin.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/cooling/stacked_fin.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/cooling/stacked_fin.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/cooling/stacked_fin.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/cooling/stacked_fin.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/cooling/stacked_fin.cur" type="image/x-icon"> <img alt='Stacked fin heat sinks offer better performance than extrusion heat sinks, due to thinner fin profiles leading to higher surface area and lower pressure drop.' class="imgImg rounded shadow" src="/av_studio/images/ableton/push/cooling/stacked_fin.png" style='width: 100%; ' title='Stacked fin heat sinks offer better performance than extrusion heat sinks, due to thinner fin profiles leading to higher surface area and lower pressure drop.' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://www.lorithermal.com/stacked-fin-heat-sink" target='_blank' > Stacked fin heat sinks offer better performance than extrusion heat sinks, due to thinner fin profiles leading to higher surface area and lower pressure drop. </a> </figcaption> </figure> </div> <div class='imgWrapper imgFlex inline' style=' '> <figure> <a href='https://myheatsinks.com/custom/skived-heat-sinks/' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/cooling/skived_fin.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/cooling/skived_fin.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/cooling/skived_fin.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/cooling/skived_fin.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/cooling/skived_fin.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/cooling/skived_fin.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/skived_fin.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/skived_fin.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/skived_fin.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/skived_fin.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/skived_fin.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/cooling/skived_fin.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/cooling/skived_fin.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/cooling/skived_fin.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/cooling/skived_fin.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/cooling/skived_fin.cur" type="image/x-icon"> <img alt='Skived heat sinks provide the best possible thermal conductivity between the fins and the base' class="imgImg rounded shadow" src="/av_studio/images/ableton/push/cooling/skived_fin.png" style='width: 100%; ' title='Skived heat sinks provide the best possible thermal conductivity between the fins and the base' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://myheatsinks.com/custom/skived-heat-sinks/" target='_blank' > Skived heat sinks provide the best possible thermal conductivity between the fins and the base </a> </figcaption> </figure> </div> <!-- endregion --> <!-- #region Vapor Chambers --> <h2 id="vc">Vapor Chambers</h2> <p> <a href='https://en.wikipedia.org/wiki/Heat_pipe#Vapor_chamber' target='_blank' rel="nofollow">Vapor chambers</a>, also known as planar heat pipes, are relatively new. Tubular heat pipes act as heat dissipaters, while planar heat pipes act as heat spreaders. </p> <div class='imgWrapper imgBlock center' style='width: 50%; '> <figure> <a href='https://www.1-act.com/thermal-solutions/passive/heat-pipes/vapor-chambers/' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/cooling/vapor-chamber.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/cooling/vapor-chamber.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/cooling/vapor-chamber.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/cooling/vapor-chamber.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/cooling/vapor-chamber.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/cooling/vapor-chamber.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/vapor-chamber.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/vapor-chamber.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/vapor-chamber.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/vapor-chamber.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/cooling/vapor-chamber.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/cooling/vapor-chamber.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/cooling/vapor-chamber.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/cooling/vapor-chamber.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/cooling/vapor-chamber.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/cooling/vapor-chamber.cur" type="image/x-icon"> <img alt='Vapor chamber assembly' class="imgImg rounded shadow" src="/av_studio/images/ableton/push/cooling/vapor-chamber.png" style='width: 100%; padding: 1em;' title='Vapor chamber assembly' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://www.1-act.com/thermal-solutions/passive/heat-pipes/vapor-chambers/" target='_blank' > Vapor chamber assembly </a> </figcaption> </figure> </div> <!-- endregion --> <!-- #region Aftermarket Upgrade --> <h3 id="au">Aftermarket Upgrade</h3> <p> I can imagine an aftermarket upgrade that would increase the height of a PS3 slightly, uses a heat pipe or vapor chamber for cooling, and has a variable-speed fan. This would support the biggest, fastest and hottest NVMe drive you can afford. I wonder if it might be possible to overclock the CPU if it was properly cooled? </p> <!-- endregion --> <!-- endregion --> <!-- #region Sources --> <h3>Sources</h3> <ul> <li><a href='https://www.engineeringtoolbox.com/thermal-conductivity-metals-d_858.html' target='_blank' rel="nofollow">Metals, Metallic Elements and Alloys - Thermal Conductivities</a></li> <li><a href='https://www.techpowerup.com/forums/threads/kryonaut-12-w-mk-vs-conductonaut-liquid-metal-73-w-mk.229167/' target='_blank' rel="nofollow">Kryonaut vs Conductonaut (Liquid Metal) </a></li> <li><a href='https://www.powerelectronicsnews.com/thermal-paste-is-it-really-necessary' target='_blank' rel="nofollow">Is Thermal Paste Necessary?</a></li> <li><a href='https://www.sciencedirect.com/science/article/abs/pii/S0254058413004409' target='_blank' rel="nofollow">Thermal conductivity of anodized aluminum oxide layer: The effect of electrolyte and temperature</a></li> </ul> <p> The University of Cambridge has an online course which goes into the physics of conductivity: <a href='https://www.doitpoms.ac.uk/tlplib/thermal_electrical/printall.ph' target='_blank' rel="nofollow">Introduction to thermal and electrical conductivity</a>. </p> <!-- endregion --> <!-- endregion --> Upgrading Modular PSUs 2024-06-28T00:00:00-04:00 https://mslinn.github.io/blog/2024/06/28/upgrade_psu <!-- #region intro --> <p> This article discusses modular PSUs conforming to the family of standards for power supplies designed for <a href='https://en.wikipedia.org/wiki/ATX#ATX12V_2.x' target='_blank' rel="nofollow">ATX form factor motherboards</a>, specifically, the ATX 2.x and ATX 3.x PSU specifications. These two major versions of the ATX PSU specification are collectively called the ATX12V specifications. ATX12V PSUs are the most common types of PSU today, having evolved from the original ATX PSU specification. </p> <p> Some of the information in this article applies generally to all modular PSUs, in particular the <a href='#summary'>summary</a>. </p> <div class='imgWrapper imgFlex center' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/psu/psu.svg" type="image/svg"> <!---<source srcset="/blog/images/psu/psu.avif" type="image/avif">--> <source srcset="/blog/images/psu/psu.webp" type="image/webp"> <source srcset="/blog/images/psu/psu.apng" type="image/apng"> <source srcset="/blog/images/psu/psu.png" type="image/png"> <source srcset="/blog/images/psu/psu.jpg" type="image/jpeg"> <source srcset="/blog/images/psu/psu.jpeg" type="image/jpeg"> <source srcset="/blog/images/psu/psu.jfif" type="image/jpeg"> <source srcset="/blog/images/psu/psu.pjpeg" type="image/jpeg"> <source srcset="/blog/images/psu/psu.pjp" type="image/jpeg"> <source srcset="/blog/images/psu/psu.gif" type="image/gif"> <source srcset="/blog/images/psu/psu.tif" type="image/tiff"> <source srcset="/blog/images/psu/psu.tiff" type="image/tiff"> <source srcset="/blog/images/psu/psu.bmp" type="image/bmp"> <source srcset="/blog/images/psu/psu.ico" type="image/x-icon"> <source srcset="/blog/images/psu/psu.cur" type="image/x-icon"> <img class="imgImg " src="/blog/images/psu/psu.png" style='width: 100%; ' /> </picture> </div> <!-- endregion --> <!-- #region ATX12V Standard --> <h2 id="atx12v">ATX12V Standard</h2> <p> The ATX12V standard is upwards compatible. This means you can install a PSU that conforms to a more recent version of the standard in an older motherboard that only supports an older version of the standard. For more information, please see the <a href='https://xdevs.com/doc/Standards/ATX/ATX12V_Power_Supply_Design_Guide_Rev1.1.pdf' target='_blank' rel="nofollow">ATX / ATX12V Power Supply Design Guide</a>. </p> <!-- #region ATX 2.x PSUs --> <h3 id="atx3">ATX 2.x PSUs</h3> <p> The current version of the ATX12V standard, v2.53, was released in December 2021. The most recent version of the standard that I could find a PDF of is <a href='https://cdn.instructables.com/ORIG/FS8/5ILB/GU59Z1AT/FS85ILBGU59Z1AT.pdf' target='_blank' rel="nofollow">ATX Specificaiton Version 2.2</a>, published in 2004. </p> <p> This is the <a href='https://www.lifewire.com/atx-24-pin-12v-power-supply-pinout-2624578' target='_blank' rel="nofollow">ATX 2.2</a> 24-pin motherboard power connector pinout: </p> <div class='imgWrapper imgBlock center' style='width: 65%; '> <figure> <picture class='imgPicture'> <source srcset="/blog/images/psu/atx_2_pinout.svg" type="image/svg"> <!---<source srcset="/blog/images/psu/atx_2_pinout.avif" type="image/avif">--> <source srcset="/blog/images/psu/atx_2_pinout.webp" type="image/webp"> <source srcset="/blog/images/psu/atx_2_pinout.apng" type="image/apng"> <source srcset="/blog/images/psu/atx_2_pinout.png" type="image/png"> <source srcset="/blog/images/psu/atx_2_pinout.jpg" type="image/jpeg"> <source srcset="/blog/images/psu/atx_2_pinout.jpeg" type="image/jpeg"> <source srcset="/blog/images/psu/atx_2_pinout.jfif" type="image/jpeg"> <source srcset="/blog/images/psu/atx_2_pinout.pjpeg" type="image/jpeg"> <source srcset="/blog/images/psu/atx_2_pinout.pjp" type="image/jpeg"> <source srcset="/blog/images/psu/atx_2_pinout.gif" type="image/gif"> <source srcset="/blog/images/psu/atx_2_pinout.tif" type="image/tiff"> <source srcset="/blog/images/psu/atx_2_pinout.tiff" type="image/tiff"> <source srcset="/blog/images/psu/atx_2_pinout.bmp" type="image/bmp"> <source srcset="/blog/images/psu/atx_2_pinout.ico" type="image/x-icon"> <source srcset="/blog/images/psu/atx_2_pinout.cur" type="image/x-icon"> <img alt='ATX 2.2 24-Pin Connector Pinout Diagram<br>Pin numbering is clearly shown' class="imgImg rounded shadow" src="/blog/images/psu/atx_2_pinout.png" style='width: 100%; padding: 1em;' title='ATX 2.2 24-Pin Connector Pinout Diagram<br>Pin numbering is clearly shown' /> </picture> <figcaption class='imgFigCaption '> ATX 2.2 24-Pin Connector Pinout Diagram<br>Pin numbering is clearly shown </figcaption> </figure> </div> <p> The above diagram shows that pins are numbered in columns from left to right. pin 1 is at the top left, and pin 24 (the highest-numbered pin) is at the bottom right. </p> <div class='imgWrapper imgBlock inline fullsize' style=' '> <figure> <a href='https://cdn.instructables.com/ORIG/FS8/5ILB/GU59Z1AT/FS85ILBGU59Z1AT.pdf' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/psu/atx_2_2_pinout.svg" type="image/svg"> <!---<source srcset="/blog/images/psu/atx_2_2_pinout.avif" type="image/avif">--> <source srcset="/blog/images/psu/atx_2_2_pinout.webp" type="image/webp"> <source srcset="/blog/images/psu/atx_2_2_pinout.apng" type="image/apng"> <source srcset="/blog/images/psu/atx_2_2_pinout.png" type="image/png"> <source srcset="/blog/images/psu/atx_2_2_pinout.jpg" type="image/jpeg"> <source srcset="/blog/images/psu/atx_2_2_pinout.jpeg" type="image/jpeg"> <source srcset="/blog/images/psu/atx_2_2_pinout.jfif" type="image/jpeg"> <source srcset="/blog/images/psu/atx_2_2_pinout.pjpeg" type="image/jpeg"> <source srcset="/blog/images/psu/atx_2_2_pinout.pjp" type="image/jpeg"> <source srcset="/blog/images/psu/atx_2_2_pinout.gif" type="image/gif"> <source srcset="/blog/images/psu/atx_2_2_pinout.tif" type="image/tiff"> <source srcset="/blog/images/psu/atx_2_2_pinout.tiff" type="image/tiff"> <source srcset="/blog/images/psu/atx_2_2_pinout.bmp" type="image/bmp"> <source srcset="/blog/images/psu/atx_2_2_pinout.ico" type="image/x-icon"> <source srcset="/blog/images/psu/atx_2_2_pinout.cur" type="image/x-icon"> <img alt='All ATX 2.2 Cable Pinout Diagrams<br>See previous diagram for full pinout numbering' class="imgImg rounded shadow" src="/blog/images/psu/atx_2_2_pinout.png" style='width: 100%; ' title='All ATX 2.2 Cable Pinout Diagrams<br>See previous diagram for full pinout numbering' /> </picture> </a> <figcaption class='imgFigCaption fullsize'> <a href="https://cdn.instructables.com/ORIG/FS8/5ILB/GU59Z1AT/FS85ILBGU59Z1AT.pdf" target='_blank' > All ATX 2.2 Cable Pinout Diagrams<br>See previous diagram for full pinout numbering </a> </figcaption> </figure> </div> <!-- endregion --> <!-- #region ATX 3.x PSUs --> <h3 id="atx3">ATX 3.x PSUs</h3> <p> ATX 3.x is upwards compatible with ATX 2.x, which means that an ATX 2.x tester can determine if an ATX 3.x PSU works, except for being able to test either of the new high-power PCIe 5.0 connectors for top-tier GPUs. </p> <p> ATX 3.0 introduced the 12VHPWR connector, which was problematic and was replaced by the new 12V-2×6 connector for ATX 3.1. To learn more about ATX 3.0, see <a href='https://www.xda-developers.com/what-is-atx-30-and-why-you-need-to-know-before-your-next-pc-upgrade' target='_blank' rel="nofollow">What is ATX 3.0 and why you need to know before your next PC upgrade.</a> </p> <p> Anandtech has an excellent <a href='https://www.anandtech.com/show/21367/the-xpg-core-reactor-ii-ve-850w-atx-31-psu-review/2' target='_blank' rel="nofollow">description of ATX 3.1</a> that explains why the 12VHPWR connector introduced by ATX 3.0 was a failure and discusses the 12V-2×6 connector introduced by ATX 3.1 to replace the ATX 3.0 12VHPWR connector. </p> <p class="alert rounded shadow"> I would not install an ATX 3.x PSU unless the computer had an NVIDIA 4090 because this PSU standard is still bleeding edge. </p> <!-- endregion --> <!-- endregion --> <!-- #region Modular PSUs --> <h2 id="mpsu">Modular PSUs</h2> <p> I recently encountered a bad batch of computer power supplies (PSUs). This article specifically discusses replacing / upgrading modular PSUs. Most of the issues raised do not arise unless the PSU is at least partially modular. </p> <p class="alert rounded shadow"> Non-modular PSUs lack one significant source of failure: only the cables provided with the PSU can be used because they are permanently attached. </p> <!-- endregion --> <!-- #region Cables --> <h2 id="cables">Cables</h2> <p> Modular power supplies have removable cables; you only connect the cables that you need. This makes building computers easier and improves airflow within a computer. </p> <div class='imgWrapper imgFlex center' style='width: 75%; '> <picture class='imgPicture'> <source srcset="/blog/images/psu/cabling.svg" type="image/svg"> <!---<source srcset="/blog/images/psu/cabling.avif" type="image/avif">--> <source srcset="/blog/images/psu/cabling.webp" type="image/webp"> <source srcset="/blog/images/psu/cabling.apng" type="image/apng"> <source srcset="/blog/images/psu/cabling.png" type="image/png"> <source srcset="/blog/images/psu/cabling.jpg" type="image/jpeg"> <source srcset="/blog/images/psu/cabling.jpeg" type="image/jpeg"> <source srcset="/blog/images/psu/cabling.jfif" type="image/jpeg"> <source srcset="/blog/images/psu/cabling.pjpeg" type="image/jpeg"> <source srcset="/blog/images/psu/cabling.pjp" type="image/jpeg"> <source srcset="/blog/images/psu/cabling.gif" type="image/gif"> <source srcset="/blog/images/psu/cabling.tif" type="image/tiff"> <source srcset="/blog/images/psu/cabling.tiff" type="image/tiff"> <source srcset="/blog/images/psu/cabling.bmp" type="image/bmp"> <source srcset="/blog/images/psu/cabling.ico" type="image/x-icon"> <source srcset="/blog/images/psu/cabling.cur" type="image/x-icon"> <img class="imgImg " src="/blog/images/psu/cabling.png" style='width: 100%; ' /> </picture> </div> <p> If you attach a cable from a modular PSU to a modular PSU of a different model, then some or all of the electronics in your computer may be fried. The actual results depend on many variables. </p> <div class="pullQuote"> Replace old PSU cables with the cables that come with the replacement PSU </div> <p> You might want to get in the habit of marking each end of every cable with nail polish, and also paint a dab on a small portion of the PSU label as well. Using a unique color of nail polish for every different model of PSU that you own will make it easy to recognize the cables that came with each model of PSU that you have. Nail polish will not stick to the plastic used in some connectors, so if that is the case, place the dabs of nail polish on the wire sleeves, if present, or on the wires themselves. </p> <!-- endregion --> <!-- #region Four Special Wires --> <h3 id="four">Four Special Wires</h3> <p> Four of the wires in 24-pin ATX v2 and v3 cables have special functions. It is important to test new PSUs before you install them, to verify that the signals in these wires are functioning properly. This section describes the signals to be tested. The next section describes how to test them. </p> <ul> <li> <b>PS_ON</b> (power on) on pin 16 is a signal originating from the motherboard to the power supply. This pin&rsquo;s voltage is internally pulled up to +5 V inside the power supply. The PSU will attempt turn on after the voltage on this pin is pulled low by being connected to ground. </li> <li> <b>PWR_OK</b> (power good) on pin 8 is an signal from the power supply that indicates that the +5VDC and +3.3VDC voltages provided by the PSU&rsquo;s 24-pin connector have stabilized. The motherboard will not attempt to power up unless this signal is provided. Normally the singal remains low for a brief time (100 to 500 milliseconds) after the <b>PS_ON</b> signal is pulled low; however, the signal will only go high after the PSU ascertains that its +5VDC and +3.3VDC outputs are stable and within normal operating parameters. </li> <li> <b>+5 V<sub>SB</sub></b> (+5 V standby) on pin 9 supplies power even when the rest of the supply wire lines are off. This can be used to power the circuitry that controls the power-on signal. </li> <li> <b>+3.3 V sense</b> on pin 13 allows the PSU to sense the voltage drop in the PSU wiring. </li> </ul> <!-- endregion --> <!-- #region Use a PSU Tester --> <h2 id="utpt">Use a PSU Tester</h2> <div class='imgWrapper imgBlock center' style='width: 75%; '> <figure> <a href='https://www.amazon.com/Computer-PC-Tester-Connectors-Enclosure/dp/B076CLNPPK' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/psu/psu_tester.svg" type="image/svg"> <!---<source srcset="/blog/images/psu/psu_tester.avif" type="image/avif">--> <source srcset="/blog/images/psu/psu_tester.webp" type="image/webp"> <source srcset="/blog/images/psu/psu_tester.apng" type="image/apng"> <source srcset="/blog/images/psu/psu_tester.png" type="image/png"> <source srcset="/blog/images/psu/psu_tester.jpg" type="image/jpeg"> <source srcset="/blog/images/psu/psu_tester.jpeg" type="image/jpeg"> <source srcset="/blog/images/psu/psu_tester.jfif" type="image/jpeg"> <source srcset="/blog/images/psu/psu_tester.pjpeg" type="image/jpeg"> <source srcset="/blog/images/psu/psu_tester.pjp" type="image/jpeg"> <source srcset="/blog/images/psu/psu_tester.gif" type="image/gif"> <source srcset="/blog/images/psu/psu_tester.tif" type="image/tiff"> <source srcset="/blog/images/psu/psu_tester.tiff" type="image/tiff"> <source srcset="/blog/images/psu/psu_tester.bmp" type="image/bmp"> <source srcset="/blog/images/psu/psu_tester.ico" type="image/x-icon"> <source srcset="/blog/images/psu/psu_tester.cur" type="image/x-icon"> <img alt='Buy a tester for your next PSU installation' class="imgImg " src="/blog/images/psu/psu_tester.png" style='width: 100%; ' title='Buy a tester for your next PSU installation' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://www.amazon.com/Computer-PC-Tester-Connectors-Enclosure/dp/B076CLNPPK" target='_blank' > Buy a tester for your next PSU installation </a> </figcaption> </figure> </div> <p class="alert center rounded shadow"> Always use a <a href='https://www.amazon.com/Computer-PC-Tester-Connectors-Enclosure/dp/B076CLNPPK' target='_blank' rel="nofollow">PSU tester</a> to test a new power supply before you install it. </p> <p> Most PSU testers have no battery and are completely powered by the PSU's 20-pin or 24-pin interface. If no power is provided by the 20-pin or 24-pin interface, the tester will not function. </p> <p> To use the PSU tester shown above, plug the 24-pin cable between the PSU and the right-hand-side connector on the tester. Now turn on the PSU. You should observe: </p> <ul> <li> The tester should NOT start beeping. </li> <li> <b>PG</b> stands for <i>Power Good</i>, also known as &ldquo;Power OK&rdquo;. As previously mentioned, the motherboard will not attempt to power up unless this signal is provided. The LCD indicates how many milliseconds it takes to receive the <b>Power Good</b> signal from the PSU. If it reads 0, your PSU is non-functional. </li> <li> The text under the <b>+12V2</b> heading will flash <b>LL</b> when no PCI-E device is connected. Its value is only relevant when a PCI-E connector (4-pin, 6-pin or 8-pin) is used by an IDE (HDD) or a SATA drive, or a floppy disk. </li> </ul> <!-- endregion --> <!-- #region Replace The Cabling When Replacing a Power Supply --> <h2 id="rp">Replace The Cabling When Replacing a Power Supply</h2> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/psu/warning-auto.svg" type="image/svg"> <!---<source srcset="/blog/images/psu/warning-auto.avif" type="image/avif">--> <source srcset="/blog/images/psu/warning-auto.webp" type="image/webp"> <source srcset="/blog/images/psu/warning-auto.apng" type="image/apng"> <source srcset="/blog/images/psu/warning-auto.png" type="image/png"> <source srcset="/blog/images/psu/warning-auto.jpg" type="image/jpeg"> <source srcset="/blog/images/psu/warning-auto.jpeg" type="image/jpeg"> <source srcset="/blog/images/psu/warning-auto.jfif" type="image/jpeg"> <source srcset="/blog/images/psu/warning-auto.pjpeg" type="image/jpeg"> <source srcset="/blog/images/psu/warning-auto.pjp" type="image/jpeg"> <source srcset="/blog/images/psu/warning-auto.gif" type="image/gif"> <source srcset="/blog/images/psu/warning-auto.tif" type="image/tiff"> <source srcset="/blog/images/psu/warning-auto.tiff" type="image/tiff"> <source srcset="/blog/images/psu/warning-auto.bmp" type="image/bmp"> <source srcset="/blog/images/psu/warning-auto.ico" type="image/x-icon"> <source srcset="/blog/images/psu/warning-auto.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/psu/warning-auto.png" style='width: 100%; ' /> </picture> </div> <p class="alert rounded shadow"> When replacing a power supply with another model or brand, <i>even if it seems almost identical</i>, <b>also replace the power cabling</b>. Seemingly trivial differences between PSU models can cause the new power supply to fail if an old power supply's cabling is used, and the components it is electrically connected to might be damaged. </p> <span style='font-size: 3em; float: left; margin-right: 5px;;'>&#x1F4A5;</span> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F4A5;</span> <p> In other words, unless you are really, totally, absolutely 100% sure that the cables are in every way identical, replace them when you replace the PSU. Leaving the previous cabling in place and just swapping the power supply can fry the power supply, the motherboard, and more. </p><!-- endregion --> <!-- #region Corsair --> <h2 id="corsair">Corsair</h2> <!-- #region implicit --> <p> I have been purchasing PSUs for the computers that I build since the early 1980s. Some of those PSUs were made by Corsair. </p> <p> Following the lead of other vendors, <a href='https://www.corsair.com/c/psu' target='_blank' rel="nofollow">Corsair</a> introduced its first fully modular PSU in 2012 for computer enthusiasts to build their own computers. </p> <p> Technical support is not provided, citing &lsquo;legal liability issues&rsquo;. I call bullshit. </p> <div class='imgWrapper imgBlock center' style='width: 75%; '> <figure> <picture class='imgPicture'> <source srcset="/blog/images/psu/elephant_shit.svg" type="image/svg"> <!---<source srcset="/blog/images/psu/elephant_shit.avif" type="image/avif">--> <source srcset="/blog/images/psu/elephant_shit.webp" type="image/webp"> <source srcset="/blog/images/psu/elephant_shit.apng" type="image/apng"> <source srcset="/blog/images/psu/elephant_shit.png" type="image/png"> <source srcset="/blog/images/psu/elephant_shit.jpg" type="image/jpeg"> <source srcset="/blog/images/psu/elephant_shit.jpeg" type="image/jpeg"> <source srcset="/blog/images/psu/elephant_shit.jfif" type="image/jpeg"> <source srcset="/blog/images/psu/elephant_shit.pjpeg" type="image/jpeg"> <source srcset="/blog/images/psu/elephant_shit.pjp" type="image/jpeg"> <source srcset="/blog/images/psu/elephant_shit.gif" type="image/gif"> <source srcset="/blog/images/psu/elephant_shit.tif" type="image/tiff"> <source srcset="/blog/images/psu/elephant_shit.tiff" type="image/tiff"> <source srcset="/blog/images/psu/elephant_shit.bmp" type="image/bmp"> <source srcset="/blog/images/psu/elephant_shit.ico" type="image/x-icon"> <source srcset="/blog/images/psu/elephant_shit.cur" type="image/x-icon"> <img alt='&lsquo;Legal liability issues&rsquo;' class="imgImg rounded shadow" src="/blog/images/psu/elephant_shit.png" style='width: 100%; ' title='&lsquo;Legal liability issues&rsquo;' /> </picture> <figcaption class='imgFigCaption '> &lsquo;Legal liability issues&rsquo; </figcaption> </figure> </div> <p> Technical information for specific products is not organized properly and contains inaccurate compatibility statements. Providing incomplete information seems to be a goal, perhaps again due to paranoia about legal liability. I attribute the inaccuracies to sloppiness. </p> <p> Corsair's PSU products are described as mysterious black boxes, with stern warnings of fatal hazards if opened. </p> <!-- endregion --> <!-- #region Connector Labels --> <h3 id="labels">Connector Labels</h3> <p> Corsair has been inconsistent about how it has labeled the modular connectors over the years. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/psu/connectors.svg" type="image/svg"> <!---<source srcset="/blog/images/psu/connectors.avif" type="image/avif">--> <source srcset="/blog/images/psu/connectors.webp" type="image/webp"> <source srcset="/blog/images/psu/connectors.apng" type="image/apng"> <source srcset="/blog/images/psu/connectors.png" type="image/png"> <source srcset="/blog/images/psu/connectors.jpg" type="image/jpeg"> <source srcset="/blog/images/psu/connectors.jpeg" type="image/jpeg"> <source srcset="/blog/images/psu/connectors.jfif" type="image/jpeg"> <source srcset="/blog/images/psu/connectors.pjpeg" type="image/jpeg"> <source srcset="/blog/images/psu/connectors.pjp" type="image/jpeg"> <source srcset="/blog/images/psu/connectors.gif" type="image/gif"> <source srcset="/blog/images/psu/connectors.tif" type="image/tiff"> <source srcset="/blog/images/psu/connectors.tiff" type="image/tiff"> <source srcset="/blog/images/psu/connectors.bmp" type="image/bmp"> <source srcset="/blog/images/psu/connectors.ico" type="image/x-icon"> <source srcset="/blog/images/psu/connectors.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/psu/connectors.png" style='width: 100%; ' /> </picture> </div> <p> In the above photograph, we see: </p> <ul> <li> Three connectors were clearly labeled with the PSUs that they were compatible with (&ldquo;TXM/HX Series only&rdquo;, &ldquo;760/860AX ONLY&rdquo;, &ldquo;Type4&rdquo;) </li> <li>One connector had a label that only had meaning if you knew the PSU it was for (&ldquo;CD08&rdquo;)</li> <li> One connector labels its purpose but does not provide any clue about the PSU that it was designed for (&ldquo;PCI-E&rdquo;). As we shall see, not all PCI-E cables are created equal. </li> </ul> <p> Not shown are connectors without labels. </p> <p> The inconsistent and unhelpful labeling continues today. </p> <!-- endregion --> <!-- #region Inaccurate Compatibility Statements About Cabling Variants --> <h3 id="cv">Inaccurate Compatibility Statements About Cabling Variants</h3> <p> The main Corsair product page for PSUs says: </p> <div class='quote'> <div class='quoteText clearfix'> The only difference between Type 3 and Type 4 cables is the pinout of the 24-pin ATX cable; all other cables (SATA, PCIe, etc) are the same. </div><div class='quoteAttribution'> &nbsp;&ndash; <a href='https://www.corsair.com/ca/en/s/psu-cable-compatibility' rel='nofollow' target='_blank'>PSU Cable Compatibility Statement</a></div> </div> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/psu/compat_statement.svg" type="image/svg"> <!---<source srcset="/blog/images/psu/compat_statement.avif" type="image/avif">--> <source srcset="/blog/images/psu/compat_statement.webp" type="image/webp"> <source srcset="/blog/images/psu/compat_statement.apng" type="image/apng"> <source srcset="/blog/images/psu/compat_statement.png" type="image/png"> <source srcset="/blog/images/psu/compat_statement.jpg" type="image/jpeg"> <source srcset="/blog/images/psu/compat_statement.jpeg" type="image/jpeg"> <source srcset="/blog/images/psu/compat_statement.jfif" type="image/jpeg"> <source srcset="/blog/images/psu/compat_statement.pjpeg" type="image/jpeg"> <source srcset="/blog/images/psu/compat_statement.pjp" type="image/jpeg"> <source srcset="/blog/images/psu/compat_statement.gif" type="image/gif"> <source srcset="/blog/images/psu/compat_statement.tif" type="image/tiff"> <source srcset="/blog/images/psu/compat_statement.tiff" type="image/tiff"> <source srcset="/blog/images/psu/compat_statement.bmp" type="image/bmp"> <source srcset="/blog/images/psu/compat_statement.ico" type="image/x-icon"> <source srcset="/blog/images/psu/compat_statement.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/psu/compat_statement.png" style='width: 100%; ' /> </picture> </div> <p> However, that is incorrect. Corsair&rsquo;s blog states that the 24-pin cables have the same pin-outs, and that the PCIe and EPS12V cables are not the same. This is also incorrect, because the 24-pin cables have different pin-outs. </p> <div class='quote'> <div class='quoteText clearfix'> The Type 4 cables have the same pin-out as Type 3 cables, but include small, solid capacitors on the +12V, +5V and +3.3V leads on the 24-pin, PCIe and EPS12V cables. </div><div class='quoteAttribution'> &nbsp;&ndash; <a href='https://forum.corsair.com/blog/power-supply-units/explanation-of-rmi-new-type-4-cables/' rel='nofollow' target='_blank'>Explanation of RMi's New Type 4 Cables</a></div> </div> <p> In summary, I understand all of the above to mean that, except for the 24-pin ATX cable, type 4 cables are compatible with type 3 cables, but they provide cleaner power. </p> <!-- endregion --> <!-- #region Diagnostics --> <h3 id="diagnostics">Diagnostics</h3> <p> Little <a href='https://help.corsair.com/hc/en-us/articles/360025085372-PSU-How-to-test-a-PSU-Power-Supply-Unit' target='_blank' rel="nofollow">diagnostic information</a> is provided. "Use a paperclip," they say, and then they also mention that might not be very helpful. In general, paperclips have limited value as a diagnostic tool for debugging PSUs. Next, Corsair discusses how to use a PSU tester for an ATX 24-pin cable but does not talk about how to test the other cables. </p> <p> Corsair provides a table on the same page showing pin signal levels for an ATX 2 24-pin cable but neglects to indicate how pins are numbered. </p> <!-- endregion --> <!-- endregion --> <!-- #region Alternative Manufacturers --> <h2 id="alt">Alternative Manufacturers</h2> <p> A brief search yielded the following PSU manufacturers, all of whom provide product support: </p> <ul> <li><a href='https://www.coolermaster.com/en-global/catalog/power-supplies/' target='_blank' rel="nofollow">CoolerMaster</a></li> <li><a href='https://www.evga.com/products/productlist.aspx?type=10' target='_blank' rel="nofollow">EVGA</a></li> <li><a href='https://www.gigabyte.com/Power-Supply' target='_blank' rel="nofollow">Gigabyte</a></li> <li><a href='https://www.rosewill.com/collection/power-supplies' target='_blank' rel="nofollow">Rosewill</a></li> <li><a href='https://seasonic.com/' target='_blank' rel="nofollow">Seasonic</a></li> <li><a href='https://www.super-flower.com.tw/en' target='_blank' rel="nofollow">Super Flower</a></li> <li><a href='https://www.thermaltake.com/products/power-supply.html' target='_blank' rel="nofollow">ThermalTake</a></li> </ul> <!-- endregion --> <!-- #region EVGA SuperNOVA 1000 GT --> <h2 id="EVGA_SuperNOVA">EVGA SuperNOVA 1000 GT</h2> <!-- #region implicit --> <p> I moved away from the bad batch of Corsair PSUs and purchased an <a href='https://www.evga.com/products/product.aspx?pn=220-GT-1000-X1' target='_blank' rel="nofollow">EVGA SuperNOVA 1000 GT</a>. </p> <p> NewEgg now has a new AI-generated <a href='https://www.newegg.ca/evga-supernova-1000-gt-220-gt-1000-x1-1000w/p/N82E16817438221#IsFeedbackTab' target='_blank' rel="nofollow">customer recommendation summary</a> for their products, and those comments helped me make a purchasing decision. I recommend the NewEgg purchasing experience. </p> <div class='imgWrapper imgFlex inline' style=' '> <figure> <a href='https://www.newegg.ca/evga-supernova-1000-gt-220-gt-1000-x1-1000w/p/N82E16817438221#IsFeedbackTab' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/psu/evga_supernova_1000_gt_review.svg" type="image/svg"> <!---<source srcset="/blog/images/psu/evga_supernova_1000_gt_review.avif" type="image/avif">--> <source srcset="/blog/images/psu/evga_supernova_1000_gt_review.webp" type="image/webp"> <source srcset="/blog/images/psu/evga_supernova_1000_gt_review.apng" type="image/apng"> <source srcset="/blog/images/psu/evga_supernova_1000_gt_review.png" type="image/png"> <source srcset="/blog/images/psu/evga_supernova_1000_gt_review.jpg" type="image/jpeg"> <source srcset="/blog/images/psu/evga_supernova_1000_gt_review.jpeg" type="image/jpeg"> <source srcset="/blog/images/psu/evga_supernova_1000_gt_review.jfif" type="image/jpeg"> <source srcset="/blog/images/psu/evga_supernova_1000_gt_review.pjpeg" type="image/jpeg"> <source srcset="/blog/images/psu/evga_supernova_1000_gt_review.pjp" type="image/jpeg"> <source srcset="/blog/images/psu/evga_supernova_1000_gt_review.gif" type="image/gif"> <source srcset="/blog/images/psu/evga_supernova_1000_gt_review.tif" type="image/tiff"> <source srcset="/blog/images/psu/evga_supernova_1000_gt_review.tiff" type="image/tiff"> <source srcset="/blog/images/psu/evga_supernova_1000_gt_review.bmp" type="image/bmp"> <source srcset="/blog/images/psu/evga_supernova_1000_gt_review.ico" type="image/x-icon"> <source srcset="/blog/images/psu/evga_supernova_1000_gt_review.cur" type="image/x-icon"> <img alt='NewEgg Customer Review Summary' class="imgImg rounded shadow" src="/blog/images/psu/evga_supernova_1000_gt_review.png" style='width: 100%; ' title='NewEgg Customer Review Summary' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://www.newegg.ca/evga-supernova-1000-gt-220-gt-1000-x1-1000w/p/N82E16817438221#IsFeedbackTab" target='_blank' > NewEgg Customer Review Summary </a> </figcaption> </figure> </div> <!-- endregion --> <!-- #region Setup --> <h3 id="es_setup">Setup</h3> <p> I opened the EVGA SuperNOVA 1000 GT box and found a proper printed manual, in color. The manual showed a US phone number for support (888-881-3842) and an email address, <a href='https://support@evga.com'><code>support@evga.com</code></a>. </p> <p> The included 'tester' was just a jumper that could perform the same function as a paperclip. It was almost completely useless. I used my tester, <a href='#utpt'>shown above</a>. The test failed with PG=0. </p> <p> I called the phone number on a Thursday at 4 PM ET, but was told it was closed until Monday at 9 AM PT. That was odd. I also called their other customer support phone number, (714) 528-4500, but got the same message. </p> <p> I emailed EVGA&rsquo;s support address, but the email bounced back with the following error. </p> <pre style="margin-bottom: 1em;">&lt;support@evga.com>: host d290004a.ess.barracudanetworks.com[209.222.82.255] said: 550 permanent failure for one or more recipients (support@evga.com:blocked) (in reply to end of DATA command)</pre> <p> I emailed their European support address, <a href='https://supportEU@evga'><code>supportEU@evga</code></a>, with the same error. I then enrolled in the <a href='https://secure.evga.com/us/signup.asp' target='_blank' rel="nofollow">EVGA Member</a>. However, I could not log in due to several bizarre errors. </p> <p> What is going on? Is EVGA bankrupt? </p> <!-- endregion --> <!-- endregion --> <!-- #region Summary --> <h2 id="summary">Summary</h2> <p> Replace all cables attached to a PSU when upgrading it. </p> <p> I forgive Corsair's bad batch of product, it happens. I do not forgive the policy of not providing technical support. </p> <p> I will update this article after I receive and install the EVGA SuperNOVA 1000 GT. </p> <!-- endregion --> Supersizing a Partition and Its File System 2024-06-12T00:00:00-04:00 https://mslinn.github.io/blog/2024/06/12/supersize-fs <!-- #region intro --> <p> This article discusses how to clone a Linux drive containing partitions with <code>ext4</code> file systems. If the original drive has the same storage capacity as the clone, your task is complete, and you do not need to read this article. </p> <h2 id="miniseries">Mini-Series</h2> <p> This article can be read standalone; however, is part of a mini-series that discusses how to increase the storage capacity and performance of the Ableton Push 3 Standalone (P3S). If you own a P3S, or are thinking of purchasing one, you might want to first <a href='/av_studio/570-ableton-push-standalone.html'>read my review</a>. </p> <div class='imgWrapper imgFlex inline' style=' '> <a href='/av_studio/570-ableton-push-standalone.html' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/p3s_top.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/p3s_top.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/p3s_top.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/p3s_top.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/p3s_top.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/p3s_top.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/p3s_top.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/p3s_top.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/p3s_top.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/p3s_top.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/p3s_top.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/p3s_top.png" style='width: 100%; ' /> </picture> </a> </div> <p> The articles in this mini-series are: </p> <ol> <li><a href='/av_studio/570-ableton-push-nvme.html'>Ableton Push 3 NVMe Replacement</a></li> <li><a href='/blog/2024/07/05/cooling.html'>Thermal Control</a></li> <li><a href='/blog/2024/06/11/virtualbox.html'>VirtualBox</a></li> <li><a href='/blog/2024/06/12/clonezilla.html'>Clonezilla</a></li> <li><a href='/blog/2024/06/12/supersize-fs.html'>Supersizing a Partition and Its File System</a></li> </ol> <p> Because a Linux file system is being modified, Linux must be used to perform the modification. It does not matter if the Linux OS runs on a physical machine or in a virtualized machine, so long as the file system to be modified is accessible from Linux. I wrote this article using an Ubuntu instance running in VirtualBox, as described in <a href='/blog/2024/06/11/virtualbox.html'>VirtualBox Setup</a>. It does not matter which OS you use to run VirtualBox; Windows, Mac and Linux all work equally well in this regard. </p> <!-- endregion --> <!-- #region Two Scenarios --> <h2 id="ts">Two Scenarios</h2> <p> This article works through the following scenarios when working with a cloned Linux drive that has more storage capacity than the original drive: </p> <ol> <li> <a href='#modestly_enlarge'>Modestly enlarge a partition</a> containing an <code>ext4</code> file system. </li> <li> <a href='#supersize'>Supersize a partition</a> containing a <code>ext4</code> file system by recreating it such that the resulting file system is much larger than the original. </li> </ol> <p> But first, some important technical details. </p> <!-- endregion --> <!-- #region Enlarging a File System --> <h2 id="eafs">Enlarging a File System</h2> <p> After testing to verify that a drive has been successfully cloned to a larger storage device, at least one of the partitions on the newly cloned NVMe drive must be enlarged, or the extra storage capacity of the larger drive will not be used. The file system within the partition also needs to be enlarged. However, there are limits to how much a file system should be enlarged instead of recreating it in the new and larger partition. </p> <p> Tiny file systems like the default file system in a <a href='/av_studio/570-ableton-push-nvme.html'>Push 3 Standalone</a>, which uses 1K blocks, should not be enlarged much, if at all. Instead, tiny file systems should be recreated at the size they will be used. See the next section for an explanation of the technical details of the Linux file system internals behind this statement. </p> <!-- endregion --> <!-- #region Linux File System Internals --> <h3 id="lfsi">Linux File System Internals</h3> <p> <a href='https://en.wikipedia.org/wiki/Theodore_Ts%27o' target='_blank' rel="nofollow">Theodore Tso</a> is a maintainer for the <code>ext4</code> file system. He is also one of the two authors of <a href='https://en.wikipedia.org/wiki/E2fsprogs' target='_blank' rel="nofollow"><code>e2fsprogs</code></a>, a collection of 31 commands for manipulating Linux <code>ext3</code> and <code>ext4</code> file systems. Theodore is one of the best authorities that one might wish for regarding how to work with <code>ext4</code> file systems. </p> <div class='quote'> <div class='quoteText clearfix'> For a small file system, we use a 1k block size, to reduce space wasted by internal fragmentation. We also reduce the size of the <code>ext4</code> journal to reduce file system overhead. But both of these changes have significant performance impacts&mdash;which might not matter if the file system is meant for a small, slow USB thumb drive, but if you are trying to use this for a higher-performance system, you'll be sorry. <br><br> We also don't enable 64-bit block numbers, since this increases metadata overhead. But like the 1K block size, this limits the maximum size that is supported by the base file system, and it is not at all simple to change the fundamental aspects of the file system so that the file system is scalable and performant. So this idea of "drop a small file system on a device" and then blowing it up to a huge size is a Really, REALLY, REALLY bad idea. <br><br> There are some ways that this can be mitigated (you can give options to <code>mke2fs</code> to use a 4k block size, enable 64k blocksize, and force a larger journal size), but if the file system has some files placed in the small file system, the block placement of the small file system will not be optimal after the file system is blown up to a large size. </div><div class='quoteAttribution'> &nbsp;&ndash; <a href='https://github.com/tytso/e2fsprogs/issues/60#issuecomment-783465199' rel='nofollow' target='_blank'><code>e2fsprogs</code> author Theodore Tso</a></div> </div> <p> Theodore also explains why a tiny file system like the one shipped with the Push 3 Standalone should not be resized past 1 TB: </p> <div class='quote'> <div class='quoteText clearfix'> The problem is that when you create a file system that is tiny (e.g., 200MiB), we use a blocksize of 1K. With a block size of 1K, the backup superblock is located at block 8193 (e.g., the first block of the second block group in the file system). But when using a block size of 1K, the number of block group descriptors is so large that it completely fills the first block group and spills into the second block group: <br><br> Group 0: (Blocks 1-8192) csum 0x0e73 Primary superblock at 1, Group descriptors at 2-8193 <br><br> and so when the backup superblock is written, it trashes the block group descriptors: <br><br> Group 1: (Blocks 8193-16384) csum 0x98cc [INODE_UNINIT] Backup superblock at 8193, Group descriptors at 8194-16385 </div> </div> <p> With the above in mind, let&rsquo;s work through resizing the file system on a partition. We will consider both of the scenarios introduced at the beginning of this article. </p> <!-- endregion --> <!-- #region Installing gparted --> <h2 id="gparted">Installing Gparted</h2> <p> The Linux <code>gparted</code> GUI program can change the size of a drive partition, and will attempt to enlarge or shrink the file system as required. </p> <p> I was able to run <code>gparted</code> on a WSL instance; however, I did not know how to map USB drives to the WSL VM, so <code>gparted</code> could work its magic. Maybe one day I will fight through the issues involved, but there is no reason for me to work that hard for no benefit. </p> <p> Instead, I installed Ubuntu Desktop 24.04 in a new virtual machine. Please see the article I wrote on <a href='/blog/2024/06/12/virtualbox.html'>VirtualBox</a> for details. Once Ubuntu was running in a VM, I started a terminal session and installed <code>gparted</code> on the new Ubuntu VM with this incantation: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfcd03d94d867'><button class='copyBtn' data-clipboard-target='#idfcd03d94d867' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install gparted</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region man gparted --> <h2 id="mg">Gparted Help</h2> <p> The help message for <code>gparted</code> is: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id40fed2f515d6'><button class='copyBtn' data-clipboard-target='#id40fed2f515d6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>man gparted <span class='unselectable'>GPARTED(8) GParted Manual GPARTED(8)<br/> NAME gparted - GNOME Partition Editor for manipulating disk parti&#8208; tions.<br/> SYNOPSIS gparted [device]...<br/> DESCRIPTION The gparted application is the GNOME partition editor for cre&#8208; ating, reorganizing, and deleting disk partitions.<br/> A disk device can be subdivided into one or more partitions. The gparted application enables you to change the partition or&#8208; ganization on a disk device while preserving the contents of the partition.<br/> With gparted you can accomplish the following tasks: - Create a partition table on a disk device. - Enable and disable partition flags such as boot and hidden. - Perform actions with partitions such as create, delete, re&#8208; size, move, check, label, copy, and paste.<br/> More documentation can be found in the application help manual, and online at: https://gparted.org<br/> EXAMPLES You can run gparted from a command line and specify one or more disk devices.<br/> For example, to start gparted with the devices /dev/sda and /dev/sdc you would use the following command:<br/> gparted /dev/sda /dev/sdc<br/> NOTES Editing partitions has the potential to cause LOSS of DATA.<br/> The gparted application is designed to enable you to edit par&#8208; titions while reducing the risk of data loss. The application is carefully tested and is used by the GParted project team. However, loss of data might occur due to software bugs, hard&#8208; ware problems, or power failure.<br/> You can help to reduce the risk of data loss by not mounting or unmounting partitions outside of the gparted application while gparted is running.<br/> You are advised to BACKUP your DATA before using the gparted application.<br/> REPORTING BUGS Report bugs at: https://gparted.org/bugs.php<br/> AUTHOR Manual page written by Curtis Gedak &lt;gedakc@users.sf.net&gt;<br/> SEE ALSO parted(8), fdisk(8), mkfs(8), ntfsprogs(8)<br/> gparted Jan 16th, 2011 GPARTED(8) </span></pre> </div> <!-- endregion --> <p> To expand on what the <code>gparted</code> help information says, <code>gparted</code> does not just resize partitions, it also runs <code>e2fsk</code> before doing the resize to verify that the partition was successfully cloned, then if that succeeds, it tries to run <code>resize2fs</code> to resize the file system in the partition and thereby complete the job. </p> <p> This means that if one of the steps in the above sequence fails, the remaining steps do not get executed. So the partition might resize, and the file system could successfully verify its integrity, but the file system resize could fail. If you clone a 256 GB NVMe drive to a 4 TB NVMe drive, then you are likely to encouter this problem. The <a href='#recreation'>Recreating a File System</a> section later in this article demonstrates the solution to that problem. </p> <!-- endregion --> <!-- #region Practicalities --> <h3 id="pgp">Practicalities</h3> <p> <code>Gparted</code> has no problem enlarging a 256 GB partition with a 1K block size to 500 GB. However, resizing the partition/file system to a much larger partition, for example, 3 TB, will fail. </p> <p> However, as Theodore Tso has <a href='#lfsi'>already informed us</a>, it would be unwise to expand a tiny file system at all, and also unwise to supersize a medium-sized file system to a much larger file system. </p> <p> When enlarging a file system is not possible or advisable, the file system must be recreated and then the contents of the old file system must be copied file-by-file to the new file system. </p> <!-- endregion --> <!-- #region Running Gparted --> <h3 id="rg">Running Gparted</h3> <p> In <a href='/blog/2024/06/12/clonezilla.html'>Clonezilla</a>, I showed how the 256 GB NVMe from an Ableton Push 3 Standalone could be cloned to a 4 TB NVMe. After cloning, the <code>data </code> partition should be enlarged to use all available space. This is a job for <code>gparted</code>! If the <code>data</code> partition is not enlarged, then the extra space on the drive would never be used. </p> <p> Running <code>gparted</code> showed the VM drive used by Ubuntu as <code>/dev/sda</code>, and the 4 TB NVMe drive as <code>/dev/sdb</code>: </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/gparted_1.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/gparted_1.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/gparted_1.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/gparted_1.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/gparted_1.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/gparted_1.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_1.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_1.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_1.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_1.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_1.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/gparted_1.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gparted_1.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gparted_1.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/gparted_1.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/gparted_1.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/gparted_1.png" style='width: 100%; ' /> </picture> </div> <p> Selecting <code>/dev/sdb</code> showed 5 partitions: </p> <ol> <li> The <code>msdos</code> partition has two flags: <code>legacy_boot</code> and <code>msftdata</code>. This partition&rsquo;s file system is <a href='https://en.wikipedia.org/wiki/File_Allocation_Table' target='_blank' rel="nofollow">FAT16</a>. UEFI boot partitions (Windows calls these <i>EFI system partition</i>s) are always formatted with <code>FAT16</code> or <code>FAT32</code> file systems. There should be no reason to ever modify this partition. </li> <li> The <code>swap1</code> partition has one flag: <a href='https://phoenixnap.com/kb/swap-partition' target='_blank' rel="nofollow"><code>swap</code></a>. This swap partition is equal to the amount of RAM in the P3S. There should no reason to ever modify this partition. </li> <li>I do not know what the <code>primary</code> partition is for or why its file system type is unknown.</li> <li> The <code>secondary</code> partition contains the Linux operating system for the P3S. This partition is 47.7% full; most users will probably not need to modify it. </li> <li> The <code>data</code> partition is where all user data is stored. This partition should be expanded to utilize all available space. </li> <li> 3.41 TB of unused space follows. If you clone a P3S NVMe drive to an NVMe with different capacity, say 1 TB, the amount of unused space will differ. </li> </ol> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/gparted_2.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/gparted_2.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/gparted_2.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/gparted_2.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/gparted_2.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/gparted_2.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_2.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_2.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_2.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_2.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_2.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/gparted_2.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gparted_2.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gparted_2.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/gparted_2.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/gparted_2.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/gparted_2.png" style='width: 100%; ' /> </picture> </div> <p> Right-clicking on the <code>data</code> partition displayed the <b>Resize/Move</b> menu item, which I selected: </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/gparted_3.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/gparted_3.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/gparted_3.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/gparted_3.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/gparted_3.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/gparted_3.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_3.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_3.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_3.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_3.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_3.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/gparted_3.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gparted_3.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gparted_3.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/gparted_3.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/gparted_3.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/gparted_3.png" style='width: 100%; ' /> </picture> </div> <p> I dragged the right-most handle to the far right. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/gparted_4.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/gparted_4.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/gparted_4.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/gparted_4.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/gparted_4.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/gparted_4.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_4.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_4.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_4.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_4.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_4.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/gparted_4.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gparted_4.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gparted_4.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/gparted_4.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/gparted_4.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/gparted_4.png" style='width: 100%; ' /> </picture> </div> <p> The <code>data</code> partition was now set to include all available free space on the drive: </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/gparted_5.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/gparted_5.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/gparted_5.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/gparted_5.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/gparted_5.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/gparted_5.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_5.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_5.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_5.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_5.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_5.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/gparted_5.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gparted_5.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gparted_5.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/gparted_5.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/gparted_5.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/gparted_5.png" style='width: 100%; ' /> </picture> </div> <p> I pressed the <kbd>Resize/Move</kbd> button, then I pressed the green checkbox icon to actually resize the <code>data</code> partition. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/gparted_6.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/gparted_6.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/gparted_6.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/gparted_6.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/gparted_6.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/gparted_6.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_6.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_6.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_6.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_6.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_6.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/gparted_6.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gparted_6.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gparted_6.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/gparted_6.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/gparted_6.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/gparted_6.png" style='width: 100%; ' /> </picture> </div> <p> Finally, I pressed the <kbd>Apply</kbd> button: </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/gparted_7.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/gparted_7.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/gparted_7.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/gparted_7.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/gparted_7.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/gparted_7.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_7.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_7.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_7.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_7.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_7.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/gparted_7.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gparted_7.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gparted_7.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/gparted_7.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/gparted_7.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/gparted_7.png" style='width: 100%; ' /> </picture> </div> <p> The resize operation first verified the integrity of the contents of the <code>data</code> partition, which took about 5 minutes. Then it aborted with the following error before resizing the file system: </p> <p class="error rounded shadow code" style="padding-left: 1em;"> resize2fs: New size results in too many block group descriptors. </p> <p> The <code>data</code> partition had been resized, but its file system had not. <code>Gparted</code> shows the unused portion of <code>/dev/sdb5</code> in gray: </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/gparted_8.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/gparted_8.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/gparted_8.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/gparted_8.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/gparted_8.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/gparted_8.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_8.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_8.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_8.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_8.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gparted_8.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/gparted_8.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gparted_8.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gparted_8.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/gparted_8.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/gparted_8.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/gparted_8.png" style='width: 100%; ' /> </picture> </div> <p> Not to worry, the partition is fine. The error message tells us that we must recreate the file system inside the partition, and that will necessitate copying all the files and directories into the file system once it has been recreated. </p> <!-- endregion --> <!-- #region Modestly Enlarging the Filesystem --> <h2 id="modestly_enlarge">Modestly Enlarging the Filesystem</h2> <p> A file system can be resized to utilize all the space in the newly expanded partition, subject to the limitations that have already been discussed. </p> <p> This section shows how I enlarged a partition and its file system after cloning a 256 GB NVMe drive to a 500 GB NVMe drive. As previously mentioned, I suggest that you not enlarge tiny file systems, so if you are reading this because you have a P3S and want to clone the 256 GB NVMe drive that it came with to a larger drive, you should skip this section and read the <a href='#findmnt'>Recreating a File System</a> section. </p> <div class="alert rounded shadow clearfix" style="padding-bottom: 0"> <div class='imgWrapper imgFlex right' style='width: 225px; margin-right: 1em;'> <picture class='imgPicture'> <source srcset="/assets/images/control/noRunningWithScissors.svg" type="image/svg"> <!---<source srcset="/assets/images/control/noRunningWithScissors.avif" type="image/avif">--> <source srcset="/assets/images/control/noRunningWithScissors.webp" type="image/webp"> <source srcset="/assets/images/control/noRunningWithScissors.apng" type="image/apng"> <source srcset="/assets/images/control/noRunningWithScissors.png" type="image/png"> <source srcset="/assets/images/control/noRunningWithScissors.jpg" type="image/jpeg"> <source srcset="/assets/images/control/noRunningWithScissors.jpeg" type="image/jpeg"> <source srcset="/assets/images/control/noRunningWithScissors.jfif" type="image/jpeg"> <source srcset="/assets/images/control/noRunningWithScissors.pjpeg" type="image/jpeg"> <source srcset="/assets/images/control/noRunningWithScissors.pjp" type="image/jpeg"> <source srcset="/assets/images/control/noRunningWithScissors.gif" type="image/gif"> <source srcset="/assets/images/control/noRunningWithScissors.tif" type="image/tiff"> <source srcset="/assets/images/control/noRunningWithScissors.tiff" type="image/tiff"> <source srcset="/assets/images/control/noRunningWithScissors.bmp" type="image/bmp"> <source srcset="/assets/images/control/noRunningWithScissors.ico" type="image/x-icon"> <source srcset="/assets/images/control/noRunningWithScissors.cur" type="image/x-icon"> <img class="imgImg " src="/assets/images/control/noRunningWithScissors.png" style='width: 100%; ' /> </picture> </div> <p style="margin-top: 1em;"> We can use the standard Linux command <code>resize2fs</code> to resize the filesystem. As the documentation says, <code>resize2fs</code> can expand the size of an <code>ext4</code> mounted file system. </p> <p> Since I discourage people from running with scissors, I suggest you first <a href='#unmount'>unmount</a> any drives whose file systems will be resized. </p> </div> <p> Before running <code>resize2fs</code> it is a good idea to verify the integrity of the partition to be resized. The <code>e2fsck</code> is the command normally used to do that. </p> <!-- #region pre --> <div class="jekyll_pre clear" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida178157a368e'><button class='copyBtn' data-clipboard-target='#ida178157a368e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>man e2fsck <span class='unselectable'>E2FSCK(8) System Manager&#39;s Manual E2FSCK(8)<br/> NAME e2fsck - check a Linux ext2/ext3/ext4 file system<br/> SYNOPSIS e2fsck [ -pacnyrdfkvtDFV ] [ -b superblock ] [ -B blocksize ] [ -l|-L bad_blocks_file ] [ -C fd ] [ -j external-journal ] [ -E extended_options ] [ -z undo_file ] device<br/> DESCRIPTION e2fsck is used to check the ext2/ext3/ext4 family of file sys&#8208; tems. For ext3 and ext4 file systems that use a journal, if the system has been shut down uncleanly without any errors, normally, after replaying the committed transactions in the journal, the file system should be marked as clean. Hence, for file systems that use journaling, e2fsck will normally re&#8208; play the journal and exit, unless its superblock indicates that further checking is required.<br/> device is a block device (e.g., /dev/sdc1) or file containing the file system.<br/> Note that in general it is not safe to run e2fsck on mounted file systems. The only exception is if the -n option is speci&#8208; fied, and -c, -l, or -L options are not specified. However, even if it is safe to do so, the results printed by e2fsck are not valid if the file system is mounted. If e2fsck asks whether or not you should check a file system which is mounted, the only correct answer is ``no&#39;&#39;. Only experts who really know what they are doing should consider answering this ques&#8208; tion in any other way.<br/> If e2fsck is run in interactive mode (meaning that none of -y, -n, or -p are specified), the program will ask the user to fix each problem found in the file system. A response of &#39;y&#39; will fix the error; &#39;n&#39; will leave the error unfixed; and &#39;a&#39; will fix the problem and all subsequent problems; pressing Enter will proceed with the default response, which is printed before the question mark. Pressing Control-C terminates e2fsck imme&#8208; diately.<br/> OPTIONS -a This option does the same thing as the -p option. It is provided for backwards compatibility only; it is sug&#8208; gested that people use -p option whenever possible.<br/> -b superblock Instead of using the normal superblock, use an alterna&#8208; tive superblock specified by superblock. This option is normally used when the primary superblock has been cor&#8208; rupted. The location of backup superblocks is dependent on the file system&#39;s blocksize, the number of blocks per group, and features such as sparse_super.<br/> Additional backup superblocks can be determined by using the mke2fs program using the -n option to print out where the superblocks exist, supposing mke2fs is sup&#8208; plied with arguments that are consistent with the file system&#39;s layout (e.g. blocksize, blocks per group, sparse_super, etc.).<br/> If an alternative superblock is specified and the file system is not opened read-only, e2fsck will make sure that the primary superblock is updated appropriately upon completion of the file system check.<br/> -B blocksize Normally, e2fsck will search for the superblock at vari&#8208; ous different block sizes in an attempt to find the ap&#8208; propriate block size. This search can be fooled in some cases. This option forces e2fsck to only try locating the superblock at a particular blocksize. If the su&#8208; perblock is not found, e2fsck will terminate with a fa&#8208; tal error.<br/> -c This option causes e2fsck to use badblocks(8) program to do a read-only scan of the device in order to find any bad blocks. If any bad blocks are found, they are added to the bad block inode to prevent them from being allo&#8208; cated to a file or directory. If this option is speci&#8208; fied twice, then the bad block scan will be done using a non-destructive read-write test.<br/> -C fd This option causes e2fsck to write completion informa&#8208; tion to the specified file descriptor so that the progress of the file system check can be monitored. This option is typically used by programs which are run&#8208; ning e2fsck. If the file descriptor number is negative, then absolute value of the file descriptor will be used, and the progress information will be suppressed ini&#8208; tially. It can later be enabled by sending the e2fsck process a SIGUSR1 signal. If the file descriptor speci&#8208; fied is 0, e2fsck will print a completion bar as it goes about its business. This requires that e2fsck is run&#8208; ning on a video console or terminal.<br/> -d Print debugging output (useless unless you are debugging e2fsck).<br/> -D Optimize directories in file system. This option causes e2fsck to try to optimize all directories, either by re- indexing them if the file system supports directory in&#8208; dexing, or by sorting and compressing directories for smaller directories, or for file systems using tradi&#8208; tional linear directories.<br/> Even without the -D option, e2fsck may sometimes opti&#8208; mize a few directories --- for example, if directory in&#8208; dexing is enabled and a directory is not indexed and would benefit from being indexed, or if the index struc&#8208; tures are corrupted and need to be rebuilt. The -D op&#8208; tion forces all directories in the file system to be op&#8208; timized. This can sometimes make them a little smaller and slightly faster to search, but in practice, you should rarely need to use this option.<br/> The -D option will detect directory entries with dupli&#8208; cate names in a single directory, which e2fsck normally does not enforce for performance reasons.<br/> -E extended_options Set e2fsck extended options. Extended options are comma separated, and may take an argument using the equals (&#39;=&#39;) sign. The following options are supported:<br/> ea_ver=extended_attribute_version Set the version of the extended attribute blocks which e2fsck will require while checking the file system. The version num&#8208; ber may be 1 or 2. The default extended at&#8208; tribute version format is 2.<br/> journal_only Only replay the journal if required, but do not perform any further checks or repairs.<br/> fragcheck During pass 1, print a detailed report of any discontiguous blocks for files in the file system.<br/> discard Attempt to discard free blocks and unused inode blocks after the full file system check (discarding blocks is useful on solid state devices and sparse / thin-provisioned storage). Note that discard is done in pass 5 AFTER the file system has been fully checked and only if it does not contain rec&#8208; ognizable errors. However there might be cases where e2fsck does not fully recognize a problem and hence in this case this option may prevent you from further manual data re&#8208; covery.<br/> nodiscard Do not attempt to discard free blocks and unused inode blocks. This option is exactly the opposite of discard option. This is set as default.<br/> no_optimize_extents Do not offer to optimize the extent tree by eliminating unnecessary width or depth. This can also be enabled in the options sec&#8208; tion of /etc/e2fsck.conf.<br/> optimize_extents Offer to optimize the extent tree by elimi&#8208; nating unnecessary width or depth. This is the default unless otherwise specified in /etc/e2fsck.conf.<br/> inode_count_fullmap Trade off using memory for speed when check&#8208; ing a file system with a large number of hard-linked files. The amount of memory re&#8208; quired is proportional to the number of in&#8208; odes in the file system. For large file systems, this can be gigabytes of memory. (For example, a 40TB file system with 2.8 billion inodes will consume an additional 5.7 GB memory if this optimization is en&#8208; abled.) This optimization can also be en&#8208; abled in the options section of /etc/e2fsck.conf.<br/> no_inode_count_fullmap Disable the inode_count_fullmap optimiza&#8208; tion. This is the default unless otherwise specified in /etc/e2fsck.conf.<br/> readahead_kb Use this many KiB of memory to pre-fetch metadata in the hopes of reducing e2fsck runtime. By default, this is set to the size of two block groups&#39; inode tables (typ&#8208; ically 4MiB on a regular ext4 file system); if this amount is more than 1/50th of total physical memory, readahead is disabled. Set this to zero to disable readahead entirely.<br/> bmap2extent Convert block-mapped files to extent-mapped files.<br/> fixes_only Only fix damaged metadata; do not optimize htree directories or compress extent trees. This option is incompatible with the -D and -E bmap2extent options.<br/> check_encoding Force verification of encoded filenames in case-insensitive directories. This is the default mode if the file system has the strict flag enabled.<br/> unshare_blocks If the file system has shared blocks, with the shared blocks read-only feature enabled, then this will unshare all shared blocks and unset the read-only feature bit. If there is not enough free space then the operation will fail. If the file system does not have the read-only feature bit, but has shared blocks anyway, then this option will have no effect. Note when using this option, if there is no free space to clone blocks, there is no prompt to delete files and in&#8208; stead the operation will fail.<br/> Note that unshare_blocks implies the &quot;-f&quot; option to ensure that all passes are run. Additionally, if &quot;-n&quot; is also specified, e2fsck will simulate trying to allocate enough space to deduplicate. If this fails, the exit code will be non-zero.<br/> -f Force checking even if the file system seems clean.<br/> -F Flush the file system device&#39;s buffer caches before be&#8208; ginning. Only really useful for doing e2fsck time tri&#8208; als.<br/> -j external-journal Set the pathname where the external-journal for this file system can be found.<br/> -k When combined with the -c option, any existing bad blocks in the bad blocks list are preserved, and any new bad blocks found by running badblocks(8) will be added to the existing bad blocks list.<br/> -l filename Add the block numbers listed in the file specified by filename to the list of bad blocks. The format of this file is the same as the one generated by the bad&#8208; blocks(8) program. Note that the block numbers are based on the blocksize of the file system. Hence, bad&#8208; blocks(8) must be given the blocksize of the file system in order to obtain correct results. As a result, it is much simpler and safer to use the -c option to e2fsck, since it will assure that the correct parameters are passed to the badblocks program.<br/> -L filename Set the bad blocks list to be the list of blocks speci&#8208; fied by filename. (This option is the same as the -l option, except the bad blocks list is cleared before the blocks listed in the file are added to the bad blocks list.)<br/> -n Open the file system read-only, and assume an answer of `no&#39; to all questions. Allows e2fsck to be used non-in&#8208; teractively. This option may not be specified at the same time as the -p or -y options.<br/> -p Automatically repair (&quot;preen&quot;) the file system. This option will cause e2fsck to automatically fix any file system problems that can be safely fixed without human intervention. If e2fsck discovers a problem which may require the system administrator to take additional cor&#8208; rective action, e2fsck will print a description of the problem and then exit with the value 4 logically or&#39;ed into the exit code. (See the EXIT CODE section.) This option is normally used by the system&#39;s boot scripts. It may not be specified at the same time as the -n or -y options.<br/> -r This option does nothing at all; it is provided only for backwards compatibility.<br/> -t Print timing statistics for e2fsck. If this option is used twice, additional timing statistics are printed on a pass by pass basis.<br/> -v Verbose mode.<br/> -V Print version information and exit.<br/> -y Assume an answer of `yes&#39; to all questions; allows e2fsck to be used non-interactively. This option may not be specified at the same time as the -n or -p op&#8208; tions.<br/> -z undo_file Before overwriting a file system block, write the old contents of the block to an undo file. This undo file can be used with e2undo(8) to restore the old contents of the file system should something go wrong. If the empty string is passed as the undo_file argument, the undo file will be written to a file named e2fsck-de&#8208; vice.e2undo in the directory specified via the E2FSPROGS_UNDO_DIR environment variable.<br/> WARNING: The undo file cannot be used to recover from a power or system crash.<br/> EXIT CODE The exit code returned by e2fsck is the sum of the following conditions: 0 - No errors 1 - File system errors corrected 2 - File system errors corrected, system should be rebooted 4 - File system errors left uncorrected 8 - Operational error 16 - Usage or syntax error 32 - E2fsck canceled by user request 128 - Shared library error<br/> SIGNALS The following signals have the following effect when sent to e2fsck.<br/> SIGUSR1 This signal causes e2fsck to start displaying a comple&#8208; tion bar or emitting progress information. (See discus&#8208; sion of the -C option.)<br/> SIGUSR2 This signal causes e2fsck to stop displaying a comple&#8208; tion bar or emitting progress information.<br/> REPORTING BUGS Almost any piece of software will have bugs. If you manage to find a file system which causes e2fsck to crash, or which e2fsck is unable to repair, please report it to the author.<br/> Please include as much information as possible in your bug re&#8208; port. Ideally, include a complete transcript of the e2fsck run, so I can see exactly what error messages are displayed. (Make sure the messages printed by e2fsck are in English; if your system has been configured so that e2fsck&#39;s messages have been translated into another language, please set the the LC_ALL environment variable to C so that the transcript of e2fsck&#39;s output will be useful to me.) If you have a writable file system where the transcript can be stored, the script(1) program is a handy way to save the output of e2fsck to a file.<br/> It is also useful to send the output of dumpe2fs(8). If a spe&#8208; cific inode or inodes seems to be giving e2fsck trouble, try running the debugfs(8) command and send the output of the stat(1u) command run on the relevant inode(s). If the inode is a directory, the debugfs dump command will allow you to extract the contents of the directory inode, which can sent to me after being first run through uuencode(1). The most useful data you can send to help reproduce the bug is a compressed raw image dump of the file system, generated using e2image(8). See the e2image(8) man page for more details.<br/> Always include the full version string which e2fsck displays when it is run, so I know which version you are running.<br/> ENVIRONMENT E2FSCK_CONFIG Determines the location of the configuration file (see e2fsck.conf(5)).<br/> AUTHOR This version of e2fsck was written by Theodore Ts&#39;o &lt;tytso@mit.edu&gt;.<br/> SEE ALSO e2fsck.conf(5), badblocks(8), dumpe2fs(8), debugfs(8), e2im&#8208; age(8), mke2fs(8), tune2fs(8)<br/> E2fsprogs version 1.47.0 February 2023 E2FSCK(8) </span></pre> </div> <!-- endregion --> <p> Here is the documentation for <code>resize2fs</code>: </p> <!-- #region pre man resize2fs --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idccf049961490'><button class='copyBtn' data-clipboard-target='#idccf049961490' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>man resize2fs <span class='unselectable'>RESIZE2FS(8) System Manager&#39;s Manual RESIZE2FS(8)<br/> NAME resize2fs - ext2/ext3/ext4 file system resizer<br/> SYNOPSIS resize2fs [ -fFpPMbs ] [ -d debug-flags ] [ -S RAID-stride ] [ -z undo_file ] device [ size ]<br/> DESCRIPTION The resize2fs program will resize ext2, ext3, or ext4 file sys&#8208; tems. It can be used to enlarge or shrink an unmounted file system located on device. If the file system is mounted, it can be used to expand the size of the mounted file system, as&#8208; suming the kernel and the file system supports on-line resiz&#8208; ing. (Modern Linux 2.6 kernels will support on-line resize for file systems mounted using ext3 and ext4; ext3 file systems will require the use of file systems with the resize_inode fea&#8208; ture enabled.)<br/> The size parameter specifies the requested new size of the file system. If no units are specified, the units of the size para&#8208; meter shall be the file system blocksize of the file system. Optionally, the size parameter may be suffixed by one of the following units designators: &#39;K&#39;, &#39;M&#39;, &#39;G&#39;, &#39;T&#39; (either upper- case or lower-case) or &#39;s&#39; for power-of-two kilobytes, megabytes, gigabytes, terabytes or 512 byte sectors respec&#8208; tively. The size of the file system may never be larger than the size of the partition. If size parameter is not specified, it will default to the size of the partition.<br/> The resize2fs program does not manipulate the size of parti&#8208; tions. If you wish to enlarge a file system, you must make sure you can expand the size of the underlying partition first. This can be done using fdisk(8) by deleting the partition and recreating it with a larger size or using lvextend(8), if you&#39;re using the logical volume manager lvm(8). When recreat&#8208; ing the partition, make sure you create it with the same start&#8208; ing disk cylinder as before! Otherwise, the resize operation will certainly not work, and you may lose your entire file sys&#8208; tem. After running fdisk(8), run resize2fs to resize the ext2 file system to use all of the space in the newly enlarged par&#8208; tition.<br/> If you wish to shrink an ext2 partition, first use resize2fs to shrink the size of file system. Then you may use fdisk(8) to shrink the size of the partition. When shrinking the size of the partition, make sure you do not make it smaller than the new size of the ext2 file system!<br/> The -b and -s options enable and disable the 64bit feature, re&#8208; spectively. The resize2fs program will, of course, take care of resizing the block group descriptors and moving other data blocks out of the way, as needed. It is not possible to resize the file system concurrent with changing the 64bit status.<br/> OPTIONS -b Turns on the 64bit feature, resizes the group descrip&#8208; tors as necessary, and moves other metadata out of the way.<br/> -d debug-flags Turns on various resize2fs debugging features, if they have been compiled into the binary. debug-flags should be computed by adding the numbers of the desired fea&#8208; tures from the following list: 2 - Debug block relocations 4 - Debug inode relocations 8 - Debug moving the inode table 16 - Print timing information 32 - Debug minimum file system size (-M) calcula&#8208; tion<br/> -f Forces resize2fs to proceed with the file system resize operation, overriding some safety checks which resize2fs normally enforces.<br/> -F Flush the file system device&#39;s buffer caches before be&#8208; ginning. Only really useful for doing resize2fs time trials.<br/> -M Shrink the file system to minimize its size as much as possible, given the files stored in the file system.<br/> -p Print out percentage completion bars for each resize2fs phase during an offline (non-trivial) resize operation, so that the user can keep track of what the program is doing. (For very fast resize operations, no progress bars may be displayed.)<br/> -P Print an estimate of the number of file system blocks in the file system if it is shrunk using resize2fs&#39;s -M op&#8208; tion and then exit.<br/> -s Turns off the 64bit feature and frees blocks that are no longer in use.<br/> -S RAID-stride The resize2fs program will heuristically determine the RAID stride that was specified when the file system was created. This option allows the user to explicitly specify a RAID stride setting to be used by resize2fs instead.<br/> -z undo_file Before overwriting a file system block, write the old contents of the block to an undo file. This undo file can be used with e2undo(8) to restore the old contents of the file system should something go wrong. If the empty string is passed as the undo_file argument, the undo file will be written to a file named resize2fs-de&#8208; vice.e2undo in the directory specified via the E2FSPROGS_UNDO_DIR environment variable.<br/> WARNING: The undo file cannot be used to recover from a power or system crash.<br/> KNOWN BUGS The minimum size of the file system as estimated by resize2fs may be incorrect, especially for file systems with 1k and 2k blocksizes.<br/> AUTHOR resize2fs was written by Theodore Ts&#39;o &lt;tytso@mit.edu&gt;.<br/> COPYRIGHT Resize2fs is Copyright 1998 by Theodore Ts&#39;o and PowerQuest, Inc. All rights reserved. As of April, 2000 Resize2fs may be redistributed under the terms of the GPL.<br/> SEE ALSO fdisk(8), e2fsck(8), mke2fs(8), lvm(8), lvextend(8)<br/> E2fsprogs version 1.47.0 February 2023 RESIZE2FS(8) </span></pre> </div> <!-- endregion --> <p> Now let&rsquo;s finally get the job done. First, we verify and optimize the filesystem on the <code>data</code> partition: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide1e5d21609b4'><button class='copyBtn' data-clipboard-target='#ide1e5d21609b4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo e2fsck -Dfp /dev/sdb5 <span class='unselectable'>data: 48424/27583488 files (2.1% non-contiguous), 98269049/220661060 blocks </span></pre> </div> <!-- endregion --> <p> The file system is good, and very little of it is non-continuous, so let&rsquo;s go ahead and resize it. </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7105ec5643c5'><button class='copyBtn' data-clipboard-target='#id7105ec5643c5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo resize2fs /dev/sdb5 <span class='unselectable'>resize2fs 1.47.0 (5-Feb-2023) The filesystem is already 458988544 (1k) blocks long. Nothing to do! </span></pre> </div> <!-- endregion --> <p> You are done, <a href='#done'>time to clean up</a>. </p> <!-- endregion --> <!-- #region Recreating a File System --> <h2 id="recreation">Recreating a File System</h2> <!-- #region implicit --> <p> When resizing an existing file system is either impossible or inadvisable, a new file system should be created using <code>mke2fs</code>, and the contents of the old file system copied to the new file system. </p> <p> This is the help information for <code>mke2fs</code>: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>man mke2fs</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1966d2095d08'><button class='copyBtn' data-clipboard-target='#id1966d2095d08' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>MKE2FS(8) System Manager&#39;s Manual MKE2FS(8)<br/> NAME mke2fs - create an ext2/ext3/ext4 file system<br/> SYNOPSIS mke2fs [ -c | -l filename ] [ -b block-size ] [ -C cluster-size ] [ -d root-directory ] [ -D ] [ -g blocks-per-group ] [ -G number-of-groups ] [ -i bytes-per-inode ] [ -I inode-size ] [ -j ] [ -J journal-options ] [ -N number-of-inodes ] [ -n ] [ -m reserved-blocks-percentage ] [ -o creator-os ] [ -O [^]fea&#8208; ture[,...] ] [ -q ] [ -r fs-revision-level ] [ -E extended-op&#8208; tions ] [ -v ] [ -F ] [ -L volume-label ] [ -M last-mounted-di&#8208; rectory ] [ -S ] [ -t fs-type ] [ -T usage-type ] [ -U UUID ] [ -V ] [ -e errors-behavior ] [ -z undo_file ] device [ fs-size ]<br/> mke2fs -O journal_dev [ -b block-size ] [ -L volume-label ] [ -n ] [ -q ] [ -v ] external-journal [ fs-size ]<br/> DESCRIPTION mke2fs is used to create an ext2, ext3, or ext4 file system, usually in a disk partition (or file) named by device.<br/> The file system size is specified by fs-size. If fs-size does not have a suffix, it is interpreted as power-of-two kilobytes, unless the -b blocksize option is specified, in which case fs- size is interpreted as the number of blocksize blocks. If the fs-size is suffixed by &#39;k&#39;, &#39;m&#39;, &#39;g&#39;, &#39;t&#39; (either upper-case or lower-case), then it is interpreted in power-of-two kilobytes, megabytes, gigabytes, terabytes, etc. If fs-size is omitted, mke2fs will create the file system based on the device size.<br/> If mke2fs is run as mkfs.XXX (i.e., mkfs.ext2, mkfs.ext3, or mkfs.ext4) the option -t XXX is implied; so mkfs.ext3 will cre&#8208; ate a file system for use with ext3, mkfs.ext4 will create a file system for use with ext4, and so on.<br/> The defaults of the parameters for the newly created file sys&#8208; tem, if not overridden by the options listed below, are con&#8208; trolled by the /etc/mke2fs.conf configuration file. See the mke2fs.conf(5) manual page for more details.<br/> OPTIONS -b block-size Specify the size of blocks in bytes. Valid block-size values are powers of two from 1024 up to 65536 (however note that the kernel is able to mount only file systems with block-size smaller or equal to the system page size - 4k on x86 systems, up to 64k on ppc64 or aarch64 de&#8208; pending on kernel configuration). If omitted, block- size is heuristically determined by the file system size and the expected usage of the file system (see the -T option). In most common cases, the default block size is 4k. If block-size is preceded by a negative sign (&#39;-&#39;), then mke2fs will use heuristics to determine the appropriate block size, with the constraint that the block size will be at least block-size bytes. This is useful for certain hardware devices which require that the blocksize be a multiple of 2k.<br/> -c Check the device for bad blocks before creating the file system. If this option is specified twice, then a slower read-write test is used instead of a fast read- only test.<br/> -C cluster-size Specify the size of cluster in bytes for file systems using the bigalloc feature. Valid cluster-size values are from 2048 to 256M bytes per cluster. This can only be specified if the bigalloc feature is enabled. (See the ext4 (5) man page for more details about bigalloc.) The default cluster size if bigalloc is enabled is 16 times the block size.<br/> -d root-directory Copy the contents of the given directory into the root directory of the file system.<br/> -D Use direct I/O when writing to the disk. This avoids mke2fs dirtying a lot of buffer cache memory, which may impact other applications running on a busy server. This option will cause mke2fs to run much more slowly, however, so there is a tradeoff to using direct I/O.<br/> -e error-behavior Change the behavior of the kernel code when errors are detected. In all cases, a file system error will cause e2fsck(8) to check the file system on the next boot. error-behavior can be one of the following:<br/> continue Continue normal execution.<br/> remount-ro Remount file system read-only.<br/> panic Cause a kernel panic.<br/> -E extended-options Set extended options for the file system. Extended op&#8208; tions are comma separated, and may take an argument us&#8208; ing the equals (&#39;=&#39;) sign. The -E option used to be -R in earlier versions of mke2fs. The -R option is still accepted for backwards compatibility, but is deprecated. The following extended options are supported:<br/> encoding=encoding-name Enable the casefold feature in the super block and set encoding-name as the encoding to be used. If encoding-name is not speci&#8208; fied, the encoding defined in mke2fs.conf(5) is used.<br/> encoding_flags=encoding-flags Define parameters for file name character encoding operations. If a flag is not changed using this parameter, its default value is used. encoding-flags should be a comma-separated lists of flags to be en&#8208; abled. To disable a flag, add it to the list with the prefix &quot;no&quot;.<br/> The only flag that can be set right now is strict which means that invalid strings should be rejected by the file system. In the default configuration, the strict flag is disabled.<br/> mmp_update_interval=interval Adjust the initial MMP update interval to interval seconds. Specifying an interval of 0 means to use the default interval. The specified interval must be less than 300 seconds. Requires that the mmp feature be enabled.<br/> stride=stride-size Configure the file system for a RAID array with stride-size file system blocks. This is the number of blocks read or written to disk before moving to the next disk, which is sometimes referred to as the chunk size. This mostly affects placement of file system metadata like bitmaps at mke2fs time to avoid placing them on a single disk, which can hurt performance. It may also be used by the block allocator.<br/> stripe_width=stripe-width Configure the file system for a RAID array with stripe-width file system blocks per stripe. This is typically stride-size * N, where N is the number of data-bearing disks in the RAID (e.g. for RAID 5 there is one parity disk, so N will be the number of disks in the array minus 1). This allows the block allocator to prevent read-modify- write of the parity in a RAID stripe if pos&#8208; sible when the data is written.<br/> offset=offset Create the file system at an offset from the beginning of the device or file. This can be useful when creating disk images for vir&#8208; tual machines.<br/> resize=max-online-resize Reserve enough space so that the block group descriptor table can grow to support a file system that has max-online-resize blocks.<br/> lazy_itable_init[= &lt;0 to disable, 1 to enable&gt;] If enabled and the uninit_bg feature is en&#8208; abled, the inode table will not be fully initialized by mke2fs. This speeds up file system initialization noticeably, but it re&#8208; quires the kernel to finish initializing the file system in the background when the file system is first mounted. If the option value is omitted, it defaults to 1 to enable lazy inode table zeroing.<br/> lazy_journal_init[= &lt;0 to disable, 1 to enable&gt;] If enabled, the journal inode will not be fully zeroed out by mke2fs. This speeds up file system initialization noticeably, but carries some small risk if the system crashes before the journal has been over&#8208; written entirely one time. If the option value is omitted, it defaults to 1 to enable lazy journal inode zeroing.<br/> assume_storage_prezeroed[= &lt;0 to disable, 1 to en&#8208; able&gt;] If enabled, mke2fs assumes that the storage device has been prezeroed, skips zeroing the journal and inode tables, and annotates the block group flags to signal that the inode table has been zeroed.<br/> no_copy_xattrs Normally mke2fs will copy the extended at&#8208; tributes of the files in the directory hier&#8208; archy specified via the (optional) -d op&#8208; tion. This will disable the copy and leaves the files in the newly created file system without any extended attributes.<br/> num_backup_sb=&lt;0|1|2&gt; If the sparse_super2 file system feature is enabled this option controls whether there will be 0, 1, or 2 backup superblocks cre&#8208; ated in the file system.<br/> packed_meta_blocks[= &lt;0 to disable, 1 to enable&gt;] Place the allocation bitmaps and the inode table at the beginning of the disk. This option requires that the flex_bg file system feature to be enabled in order for it to have effect, and will also create the jour&#8208; nal at the beginning of the file system. This option is useful for flash devices that use SLC flash at the beginning of the disk. It also maximizes the range of contiguous data blocks, which can be useful for certain specialized use cases, such as supported Shingled Drives.<br/> root_owner[=uid:gid] Specify the numeric user and group ID of the root directory. If no UID:GID is specified, use the user and group ID of the user run&#8208; ning mke2fs. In mke2fs 1.42 and earlier the UID and GID of the root directory were set by default to the UID and GID of the user running the mke2fs command. The root_owner= option allows explicitly specifying these values, and avoid side-effects for users that do not expect the contents of the file system to change based on the user running mke2fs.<br/> test_fs Set a flag in the file system superblock in&#8208; dicating that it may be mounted using exper&#8208; imental kernel code, such as the ext4dev file system.<br/> orphan_file_size=size Set size of the file for tracking unlinked but still open inodes and inodes with trun&#8208; cate in progress. Larger file allows for better scalability, reserving a few blocks per cpu is ideal.<br/> discard Attempt to discard blocks at mkfs time (dis&#8208; carding blocks initially is useful on solid state devices and sparse / thin-provisioned storage). When the device advertises that discard also zeroes data (any subsequent read after the discard and before write re&#8208; turns zero), then mark all not-yet-zeroed inode tables as zeroed. This significantly speeds up file system initialization. This is set as default.<br/> nodiscard Do not attempt to discard blocks at mkfs time.<br/> quotatype Specify the which quota types (usrquota, grpquota, prjquota) which should be enabled in the created file system. The argument of this extended option should be a colon sepa&#8208; rated list. This option has effect only if the quota feature is set. The default quota types to be initialized if this option is not specified is both user and group quo&#8208; tas. If the project feature is enabled that project quotas will be initialized as well.<br/> -F Force mke2fs to create a file system, even if the speci&#8208; fied device is not a partition on a block special de&#8208; vice, or if other parameters do not make sense. In or&#8208; der to force mke2fs to create a file system even if the file system appears to be in use or is mounted (a truly dangerous thing to do), this option must be specified twice.<br/> -g blocks-per-group Specify the number of blocks in a block group. There is generally no reason for the user to ever set this para&#8208; meter, as the default is optimal for the file system. (For administrators who are creating file systems on RAID arrays, it is preferable to use the stride RAID pa&#8208; rameter as part of the -E option rather than manipulat&#8208; ing the number of blocks per group.) This option is generally used by developers who are developing test cases.<br/> If the bigalloc feature is enabled, the -g option will specify the number of clusters in a block group.<br/> -G number-of-groups Specify the number of block groups that will be packed together to create a larger virtual block group (or &quot;flex_bg group&quot;) in an ext4 file system. This improves meta-data locality and performance on meta-data heavy workloads. The number of groups must be a power of 2 and may only be specified if the flex_bg file system feature is enabled.<br/> -i bytes-per-inode Specify the bytes/inode ratio. mke2fs creates an inode for every bytes-per-inode bytes of space on the disk. The larger the bytes-per-inode ratio, the fewer inodes will be created. This value generally shouldn&#39;t be smaller than the blocksize of the file system, since in that case more inodes would be made than can ever be used. Be warned that it is not possible to change this ratio on a file system after it is created, so be care&#8208; ful deciding the correct value for this parameter. Note that resizing a file system changes the number of inodes to maintain this ratio.<br/> -I inode-size Specify the size of each inode in bytes. The inode-size value must be a power of 2 larger or equal to 128. The larger the inode-size the more space the inode table will consume, and this reduces the usable space in the file system and can also negatively impact performance. It is not possible to change this value after the file system is created.<br/> File systems with an inode size of 128 bytes do not sup&#8208; port timestamps beyond January 19, 2038. Inodes which are 256 bytes or larger will support extended time&#8208; stamps, project id&#39;s, and the ability to store some ex&#8208; tended attributes in the inode table for improved per&#8208; formance.<br/> The default inode size is controlled by the mke2fs.conf(5) file. In the mke2fs.conf file shipped with e2fsprogs, the default inode size is 256 bytes for most file systems, except for small file systems where the inode size will be 128 bytes.<br/> -j Create the file system with an ext3 journal. If the -J option is not specified, the default journal parameters will be used to create an appropriately sized journal (given the size of the file system) stored within the file system. Note that you must be using a kernel which has ext3 support in order to actually make use of the journal.<br/> -J journal-options Create the ext3 journal using options specified on the command-line. Journal options are comma separated, and may take an argument using the equals (&#39;=&#39;) sign. The following journal options are supported:<br/> size=journal-size Create an internal journal (i.e., stored in&#8208; side the file system) of size journal-size megabytes. The size of the journal must be at least 1024 file system blocks (i.e., 1MB if using 1k blocks, 4MB if using 4k blocks, etc.) and may be no more than 10,240,000 file system blocks or half the total file system size (whichever is smaller)<br/> fast_commit_size=fast-commit-size Create an additional fast commit journal area of size fast-commit-size kilobytes. This option is only valid if fast_commit feature is enabled on the file system. If this option is not specified and if fast_commit feature is turned on, fast com&#8208; mit area size defaults to journal-size / 64 megabytes. The total size of the journal with fast_commit feature set is journal-size + ( fast-commit-size * 1024) megabytes. The total journal size may be no more than 10,240,000 file system blocks or half the total file system size (whichever is smaller).<br/> location=journal-location Specify the location of the journal. The argument journal-location can either be specified as a block number, or if the num&#8208; ber has a units suffix (e.g., &#39;M&#39;, &#39;G&#39;, etc.) interpret it as the offset from the beginning of the file system.<br/> device=external-journal Attach the file system to the journal block device located on external-journal. The ex&#8208; ternal journal must already have been cre&#8208; ated using the command<br/> mke2fs -O journal_dev external-journal<br/> Note that external-journal must have been created with the same block size as the new file system. In addition, while there is support for attaching multiple file systems to a single external journal, the Linux ker&#8208; nel and e2fsck(8) do not currently support shared external journals yet.<br/> Instead of specifying a device name di&#8208; rectly, external-journal can also be speci&#8208; fied by either LABEL=label or UUID=UUID to locate the external journal by either the volume label or UUID stored in the ext2 su&#8208; perblock at the start of the journal. Use dumpe2fs(8) to display a journal device&#39;s volume label and UUID. See also the -L op&#8208; tion of tune2fs(8).<br/> Only one of the size or device options can be given for a file system.<br/> -l filename Read the bad blocks list from filename. Note that the block numbers in the bad block list must be generated using the same block size as used by mke2fs. As a re&#8208; sult, the -c option to mke2fs is a much simpler and less error-prone method of checking a disk for bad blocks be&#8208; fore formatting it, as mke2fs will automatically pass the correct parameters to the badblocks program.<br/> -L new-volume-label Set the volume label for the file system to new-volume- label. The maximum length of the volume label is 16 bytes.<br/> -m reserved-blocks-percentage Specify the percentage of the file system blocks re&#8208; served for the super-user. This avoids fragmentation, and allows root-owned daemons, such as syslogd(8), to continue to function correctly after non-privileged processes are prevented from writing to the file system. The default percentage is 5%.<br/> -M last-mounted-directory Set the last mounted directory for the file system. This might be useful for the sake of utilities that key off of the last mounted directory to determine where the file system should be mounted.<br/> -n Causes mke2fs to not actually create a file system, but display what it would do if it were to create a file system. This can be used to determine the location of the backup superblocks for a particular file system, so long as the mke2fs parameters that were passed when the file system was originally created are used again. (With the -n option added, of course!)<br/> -N number-of-inodes Overrides the default calculation of the number of in&#8208; odes that should be reserved for the file system (which is based on the number of blocks and the bytes-per-inode ratio). This allows the user to specify the number of desired inodes directly.<br/> -o creator-os Overrides the default value of the &quot;creator operating system&quot; field of the file system. The creator field is set by default to the name of the OS the mke2fs exe&#8208; cutable was compiled for.<br/> -O [^]feature[,...] Create a file system with the given features (file sys&#8208; tem options), overriding the default file system op&#8208; tions. The features that are enabled by default are specified by the base_features relation, either in the [defaults] section in the /etc/mke2fs.conf configuration file, or in the [fs_types] subsections for the usage types as specified by the -T option, further modified by the features relation found in the [fs_types] subsec&#8208; tions for the file system and usage types. See the mke2fs.conf(5) manual page for more details. The file system type-specific configuration setting found in the [fs_types] section will override the global default found in [defaults].<br/> The file system feature set will be further edited using either the feature set specified by this option, or if this option is not given, by the default_features rela&#8208; tion for the file system type being created, or in the [defaults] section of the configuration file.<br/> The file system feature set is comprised of a list of features, separated by commas, that are to be enabled. To disable a feature, simply prefix the feature name with a caret (&#39;^&#39;) character. Features with dependen&#8208; cies will not be removed successfully. The pseudo-file system feature &quot;none&quot; will clear all file system fea&#8208; tures.<br/> For more information about the features which can be set, please see the manual page ext4(5).<br/> -q Quiet execution. Useful if mke2fs is run in a script.<br/> -r revision Set the file system revision for the new file system. Note that 1.2 kernels only support revision 0 file sys&#8208; tems. The default is to create revision 1 file systems.<br/> -S Write superblock and group descriptors only. This is an extreme measure to be taken only in the very unlikely case that all of the superblock and backup superblocks are corrupted, and a last-ditch recovery method is de&#8208; sired by experienced users. It causes mke2fs to reini&#8208; tialize the superblock and group descriptors, while not touching the inode table and the block and inode bitmaps. The e2fsck program should be run immediately after this option is used, and there is no guarantee that any data will be salvageable. Due to the wide va&#8208; riety of possible options to mke2fs that affect the on- disk layout, it is critical to specify exactly the same format options, such as blocksize, fs-type, feature flags, and other tunables when using this option, or the file system will be further corrupted. In some cases, such as file systems that have been resized, or have had features enabled after format time, it is impossible to overwrite all of the superblocks correctly, and at least some file system corruption will occur. It is best to run this on a full copy of the file system so other op&#8208; tions can be tried if this doesn&#39;t work.<br/> -t fs-type Specify the file system type (i.e., ext2, ext3, ext4, etc.) that is to be created. If this option is not specified, mke2fs will pick a default either via how the command was run (for example, using a name of the form mkfs.ext2, mkfs.ext3, etc.) or via a default as defined by the /etc/mke2fs.conf file. This option controls which file system options are used by default, based on the fstypes configuration stanza in /etc/mke2fs.conf.<br/> If the -O option is used to explicitly add or remove file system options that should be set in the newly cre&#8208; ated file system, the resulting file system may not be supported by the requested fs-type. (e.g., &quot;mke2fs -t ext3 -O extent /dev/sdXX&quot; will create a file system that is not supported by the ext3 implementation as found in the Linux kernel; and &quot;mke2fs -t ext3 -O ^has_journal /dev/hdXX&quot; will create a file system that does not have a journal and hence will not be supported by the ext3 file system code in the Linux kernel.)<br/> -T usage-type[,...] Specify how the file system is going to be used, so that mke2fs can choose optimal file system parameters for that use. The usage types that are supported are de&#8208; fined in the configuration file /etc/mke2fs.conf. The user may specify one or more usage types using a comma separated list.<br/> If this option is is not specified, mke2fs will pick a single default usage type based on the size of the file system to be created. If the file system size is less than 3 megabytes, mke2fs will use the file system type floppy. If the file system size is greater than or equal to 3 but less than 512 megabytes, mke2fs(8) will use the file system type small. If the file system size is greater than or equal to 4 terabytes but less than 16 terabytes, mke2fs(8) will use the file system type big. If the file system size is greater than or equal to 16 terabytes, mke2fs(8) will use the file system type huge. Otherwise, mke2fs(8) will use the default file system type default.<br/> -U UUID Set the universally unique identifier (UUID) of the file system to UUID. The format of the UUID is a series of hex digits separated by hyphens, like this: &quot;c1b9d5a2-f162-11cf-9ece-0020afc76f16&quot;. The UUID para&#8208; meter may also be one of the following:<br/> clear clear the file system UUID<br/> random generate a new randomly-generated UUID<br/> time generate a new time-based UUID<br/> -v Verbose execution.<br/> -V Print the version number of mke2fs and exit.<br/> -z undo_file Before overwriting a file system block, write the old contents of the block to an undo file. This undo file can be used with e2undo(8) to restore the old contents of the file system should something go wrong. If the empty string is passed as the undo_file argument, the undo file will be written to a file named mke2fs-de&#8208; vice.e2undo in the directory specified via the E2FSPROGS_UNDO_DIR environment variable or the undo_dir directive in the configuration file.<br/> WARNING: The undo file cannot be used to recover from a power or system crash.<br/> ENVIRONMENT MKE2FS_SYNC If set to non-zero integer value, its value is used to determine how often sync(2) is called during inode table initialization.<br/> MKE2FS_CONFIG Determines the location of the configuration file (see mke2fs.conf(5)).<br/> MKE2FS_FIRST_META_BG If set to non-zero integer value, its value is used to determine first meta block group. This is mostly for de&#8208; bugging purposes.<br/> MKE2FS_DEVICE_SECTSIZE If set to non-zero integer value, its value is used to determine logical sector size of the device.<br/> MKE2FS_DEVICE_PHYS_SECTSIZE If set to non-zero integer value, its value is used to determine physical sector size of the device.<br/> MKE2FS_SKIP_CHECK_MSG If set, do not show the message of file system automatic check caused by mount count or check interval.<br/> AUTHOR This version of mke2fs has been written by Theodore Ts&#39;o &lt;tytso@mit.edu&gt;.<br/> AVAILABILITY mke2fs is part of the e2fsprogs package and is available from http://e2fsprogs.sourceforge.net.<br/> SEE ALSO mke2fs.conf(5), badblocks(8), dumpe2fs(8), e2fsck(8), tune2fs(8), ext4(5)<br/> E2fsprogs version 1.47.0 February 2023 MKE2FS(8)</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Journaling --> <h3 id="journal">Journaling</h3> <p> The <code>mke2fs</code> command will create an <code>ext4</code> <a href='https://www.kernel.org/doc/html/latest/filesystems/ext4/journal.html' target='_blank' rel="nofollow">journal</a> when creating an <code>ext4</code> file system. </p> <p class="alert rounded shadow"> There is no <code>mke2fs</code> option to specify that an <code>ext4</code> journal should be created; instead, an <code>ext4</code> journal is created by default. <br><br> Unfortunately, lots of the &lsquo;advice&rsquo; you can find on the interwebs advise people to issue options for the <code>mke2fs</code> command that create a journal &hellip; however, those commands create <code>ext3</code> journals, not <code>ext4</code> journals. <br><br> This is undesirable because <code>ext4</code> journals are better than <code>ext3</code> journals. </p> <!-- endregion --> <!-- #region UUID --> <h3 id="uuid">UUID</h3> <p> File systems are best mounted by specifying their <a href='https://www.linux.com/news/what-uuids-can-do-you/' target='_blank' rel="nofollow">UUID</a>. In contrast, specifying the drive to be mounted by their device name is problematic when device assignments shift as hardware is modified. </p> <p> When you clone a drive, you should also replicate the original UUIDs of the file systems that should be mounted. That allows you to replace the original drive with the clone without modifying the system. </p> <p> You can obtain the original file system UUID for <code>/dev/sdb5</code> whether or not it is mounted with the following incantation: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd1501ebf5634'><button class='copyBtn' data-clipboard-target='#idd1501ebf5634' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>lsblk -no UUID /dev/sdb5 <span class='unselectable'>64f5365e-edfc-4781-a285-9a053e6937fa </span></pre> </div> <!-- endregion --> <p> The above shows that the UUID for <code>/dev/sdb5</code> on my virtualized Ubuntu system was <code>64f5365e-edfc-4781-a285-9a053e6937fa</code>. </p> <p> Let&rsquo;s store the file system UUID in an environment variable called <code>UUID</code> for convenience in the next step: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id33a1f8131b57'><button class='copyBtn' data-clipboard-target='#id33a1f8131b57' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>UUID="$( lsblk -no UUID /dev/sdb5 )"<br> <span class='unselectable'>$ </span>echo $UUID <span class='unselectable'>64f5365e-edfc-4781-a285-9a053e6937fa </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Example --> <h3 id="ex">Example</h3> <p> As an example, let&rsquo;s consider the case of cloning the NVMe drive of an Ableton P3S to a larger NVMe drive. The <code>data</code> partition is the only partition that should be enlarged, and it should be enlarged to utilize all available storage capacity. </p> <p> Before we throw away the file system on <code>/dev/sdb5</code> we need to unmount it: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2a3fd6f1bd2f'><button class='copyBtn' data-clipboard-target='#id2a3fd6f1bd2f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo umount /dev/sdb5</pre> </div> <!-- endregion --> <p> Let&rsquo;s use <code>mke2fs</code> to create an optimal file system in the <code>data</code> partition, mapped as <code>/dev/sdb5</code>. This file system will automatically be optimized to use all available space, with the same <code>UUID</code> as the original file system. </p> <p class="alert rounded shadow"> The <code>mke2fs</code> command is destructive. It places a new, formatted filesystem onto the specified partition. Typing the wrong device name will wipe all existing data in that device! </p> <p> We can use the <code>-n</code> option to do a <a href='https://www.merriam-webster.com/dictionary/dry%20run' target='_blank' rel="nofollow">dry run</a> to see what would happen: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide3ca774e87a1'><button class='copyBtn' data-clipboard-target='#ide3ca774e87a1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo mke2fs \ -n \ -c -L data -O filetype,extents -t ext4 -U "$UUID" \ /dev/sdb5 <span class='unselectable'>mke2fs 1.47.0 (5-Feb-2023) /dev/sdb5 contains a ext4 file system labelled 'data' last mounted on /media/mslinn/data1 on Thu Jun 20 06:18:21 2024 Proceed anyway? (y,N) </span>y <span class='unselectable'>Creating filesystem with 969404928 4k blocks and 242352128 inodes Filesystem UUID: 64f5365e-edfc-4781-a285-9a053e6937fa Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 102400000, 214990848, 512000000, 550731776, 644972544 </span></pre> </div> <!-- endregion --> <p> This is the actual invocation: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id86bbb652c204'><button class='copyBtn' data-clipboard-target='#id86bbb652c204' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo mke2fs \ -c -L data -O filetype,extents -t ext4 -U "$UUID" \ /dev/sdb5 <span class='unselectable'>mke2fs 1.47.0 (5-Feb-2023) /dev/sdb5 contains a ext4 file system labelled 'data' last mounted on /media/mslinn/data1 on Thu Jun 20 06:18:21 2024 Proceed anyway? (y,N) </span>y <span class='unselectable'>Creating filesystem with 969404928 4k blocks and 242352128 inodes Filesystem UUID: 64f5365e-edfc-4781-a285-9a053e6937fa Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 102400000, 214990848, 512000000, 550731776, 644972544 </span></pre> </div> <!-- endregion --> <p> Now the <code>data</code> partition contains an optimized file system that is completely empty. We need to copy the files and directories from the <code>data</code> partition of the original drive to the empty <code>data</code> partition on the clone. </p> <!-- endregion --> <!-- #region Copying Files and Directories --> <h3 id="copy">Copying Files and Directories</h3> <!-- #region implicit --> <p> Partitions need to be mounted before files and directories can be copied to them, or from them. I inserted the original P3S NVMe drive into my second Sabrent NVMe / USB enclosure, and connected the enclosure to the computer. The VirtualBox VM for the Ubuntu instance I was working with already knew to take control of both Sabrent USB devices. </p> <p> Once plugged in, the virtualized Ubuntu instance running in VirtualBox presented me with all the file systems on both of the NVMe drives in Sabrent enclosures, including the file systems called <code>data</code> found on each of the NVMe drives. </p> <p> Clicking on each of the <code>data</code> partitions in the Ubuntu file manager mounted them. </p> <p> The <code>data</code> partition on the original drive (now mounted as <code>/dev/sdc5</code>) contained all the files and directories to be copied, and the <code>data</code> partition on the clone (still mounted as <code>/dev/sdb5</code>) was empty because we had just recreated its file system. </p> <!-- endregion --> <!-- #region Cut and Paste --> <h4 id="rsync">Cut and Paste</h4> <p> One way to copy all the files would be to just copy and paste. This is likely to require superuser privilege to copy some files or directories. Unfortunately, Ubuntu 20.04 removed the ability for a file manager such as <code>nautilus</code> or <code>thunar</code> to run with elevated permissions. Thus, you are likely to find that some or even most files copy fine, but the process terminates with an error. </p> <p> If you were able to perform the copy without incident, skip to the <a href='#done'>All Done</a> section. </p> <!-- endregion --> <h4 id="findmnt"><span class="code">Findmnt</span></h4> <p> If you prefer to use the command line instead of drag and drop, or if you used drag and drop, but an error occurred after copying some files, you should use the <code>rsync</code> command to complete the copy. </p> <p> Before you can use <code>rsync</code>, however, you need to know the mount points for the file systems in both <code>data</code> partitions. </p> <p> The <code>findmnt</code> command shows information about mounted file systems. Here is the <code>man</code> page: </p> <!-- #region pre man findmnt --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>man findmnt</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5317a5b7420d'><button class='copyBtn' data-clipboard-target='#id5317a5b7420d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>FINDMNT(8) System Administration FINDMNT(8)<br/> NAME findmnt - find a filesystem<br/> SYNOPSIS findmnt [options]<br/> findmnt [options] device|mountpoint<br/> findmnt [options] [--source] device [--target path|--mountpoint mountpoint]<br/> DESCRIPTION findmnt will list all mounted filesystems or search for a filesystem. The findmnt command is able to search in /etc/fstab, /etc/mtab or /proc/self/mountinfo. If device or mountpoint is not given, all filesystems are shown.<br/> The device may be specified by device name, major:minor numbers, filesystem label or UUID, or partition label or UUID. Note that findmnt follows mount(8) behavior where a device name may be interpreted as a mountpoint (and vice versa) if the --target, --mountpoint or --source options are not specified.<br/> The command-line option --target accepts any file or directory and then findmnt displays the filesystem for the given path.<br/> The command prints all mounted filesystems in the tree-like format by default. The default output, is subject to change. So whenever possible, you should avoid using default output in your scripts. Always explicitly define expected columns by using --output columns-list in environments where a stable output is required.<br/> The relationship between block devices and filesystems is not always one-to-one. The filesystem may use more block devices. This is why findmnt provides SOURCE and SOURCES (pl.) columns. The column SOURCES displays all devices where it is possible to find the same filesystem UUID (or another tag specified in fstab when executed with --fstab and --evaluate).<br/> OPTIONS -A, --all Disable all built-in filters and print all filesystems.<br/> -a, --ascii Use ascii characters for tree formatting.<br/> -b, --bytes Print the sizes in bytes rather than in a human-readable format.<br/> By default, the unit, sizes are expressed in, is byte, and unit prefixes are in power of 2^10 (1024). Abbreviations of symbols are exhibited truncated in order to reach a better readability, by exhibiting alone the first letter of them; examples: &quot;1 KiB&quot; and &quot;1 MiB&quot; are respectively exhibited as &quot;1 K&quot; and &quot;1 M&quot;, then omitting on purpose the mention &quot;iB&quot;, which is part of these abbreviations.<br/> -C, --nocanonicalize Do not canonicalize paths at all. This option affects the comparing of paths and the evaluation of tags (LABEL, UUID, etc.).<br/> -c, --canonicalize Canonicalize all printed paths.<br/> --deleted Print filesystems where target (mountpoint) is marked as deleted by kernel.<br/> -D, --df Imitate the output of df(1). This option is equivalent to -o SOURCE,FSTYPE,SIZE,USED,AVAIL,USE%,TARGET but excludes all pseudo filesystems. Use --all to print all filesystems.<br/> -d, --direction word The search direction, either forward or backward.<br/> -e, --evaluate Convert all tags (LABEL, UUID, PARTUUID, or PARTLABEL) to the corresponding device names for the SOURCE column. It&rsquo;s an unusual situation, but the same tag may be duplicated (used for more devices). For this purpose, there is SOURCES (pl.) column. This column displays by multi-line cell all devices where the tag is detected by libblkid. This option makes sense for fstab only.<br/> -F, --tab-file path Search in an alternative file. If used with --fstab, --mtab or --kernel, then it overrides the default paths. If specified more than once, then tree-like output is disabled (see the --list option).<br/> -f, --first-only Print the first matching filesystem only.<br/> -i, --invert Invert the sense of matching.<br/> -J, --json Use JSON output format.<br/> -k, --kernel Search in /proc/self/mountinfo. The output is in the tree-like format. This is the default. The output contains only mount options maintained by kernel (see also --mtab).<br/> -l, --list Use the list output format. This output format is automatically enabled if the output is restricted by the -t, -O, -S or -T option and the option --submounts is not used or if more that one source file (the option -F) is specified.<br/> -M, --mountpoint path Explicitly define the mountpoint file or directory. See also --target.<br/> -m, --mtab Search in /etc/mtab. The output is in the list format by default (see --tree). The output may include user space mount options.<br/> -N, --task tid Use alternative namespace /proc/&lt;tid&gt;/mountinfo rather than the default /proc/self/mountinfo. If the option is specified more than once, then tree-like output is disabled (see the --list option). See also the unshare(1) command.<br/> -n, --noheadings Do not print a header line.<br/> -O, --options list Limit the set of printed filesystems. More than one option may be specified in a comma-separated list. The -t and -O options are cumulative in effect. It is different from -t in that each option is matched exactly; a leading no at the beginning does not have global meaning. The &quot;no&quot; can used for individual items in the list. The &quot;no&quot; prefix interpretation can be disabled by &quot;+&quot; prefix.<br/> -o, --output list Define output columns. See the --help output to get a list of the currently supported columns. The TARGET column contains tree formatting if the --list or --raw options are not specified.<br/> The default list of columns may be extended if list is specified in the format +list (e.g., findmnt -o +PROPAGATION).<br/> --output-all Output almost all available columns. The columns that require --poll are not included.<br/> -P, --pairs Produce output in the form of key=&quot;value&quot; pairs. All potentially unsafe value characters are hex-escaped (\x&lt;code&gt;). See also option --shell.<br/> -p, --poll[=list] Monitor changes in the /proc/self/mountinfo file. Supported actions are: mount, umount, remount and move. More than one action may be specified in a comma-separated list. All actions are monitored by default.<br/> The time for which --poll will block can be restricted with the --timeout or --first-only options.<br/> The standard columns always use the new version of the information from the mountinfo file, except the umount action which is based on the original information cached by findmnt. The poll mode allows using extra columns:<br/> ACTION mount, umount, move or remount action name; this column is enabled by default<br/> OLD-TARGET available for umount and move actions<br/> OLD-OPTIONS available for umount and remount actions<br/> --pseudo Print only pseudo filesystems.<br/> --shadow Print only filesystems over-mounted by another filesystem.<br/> -R, --submounts Print recursively all submounts for the selected filesystems. The restrictions defined by options -t, -O, -S, -T and --direction are not applied to submounts. All submounts are always printed in tree-like order. The option enables the tree-like output format by default. This option has no effect for --mtab or --fstab.<br/> -r, --raw Use raw output format. All potentially unsafe characters are hex-escaped (\x&lt;code&gt;).<br/> --real Print only real filesystems.<br/> -S, --source spec Explicitly define the mount source. Supported specifications are device, maj:min, LABEL=label, UUID=uuid, PARTLABEL=label and PARTUUID=uuid.<br/> -s, --fstab Search in /etc/fstab. The output is in the list format (see --list).<br/> -T, --target path Define the mount target. If path is not a mountpoint file or directory, then findmnt checks the path elements in reverse order to get the mountpoint (this feature is supported only when searching in kernel files and unsupported for --fstab). It&rsquo;s recommended to use the option --mountpoint when checks of path elements are unwanted and path is a strictly specified mountpoint.<br/> -t, --types list Limit the set of printed filesystems. More than one type may be specified in a comma-separated list. The list of filesystem types can be prefixed with no to specify the filesystem types on which no action should be taken. For more details see mount(8).<br/> --tree Enable tree-like output if possible. The options is silently ignored for tables where is missing child-parent relation (e.g., fstab).<br/> --shadowed Print only filesystems over-mounted by another filesystem.<br/> -U, --uniq Ignore filesystems with duplicate mount targets, thus effectively skipping over-mounted mount points.<br/> -u, --notruncate Do not truncate text in columns. The default is to not truncate the TARGET, SOURCE, UUID, LABEL, PARTUUID, PARTLABEL columns. This option disables text truncation also in all other columns.<br/> -v, --nofsroot Do not print a [/dir] in the SOURCE column for bind mounts or btrfs subvolumes.<br/> -w, --timeout milliseconds Specify an upper limit on the time for which --poll will block, in milliseconds.<br/> -x, --verify Check mount table content. The default is to verify /etc/fstab parsability and usability. It&rsquo;s possible to use this option also with --tab-file. It&rsquo;s possible to specify source (device) or target (mountpoint) to filter mount table. The option --verbose forces findmnt to print more details.<br/> --verbose Force findmnt to print more information (--verify only for now).<br/> --vfs-all When used with VFS-OPTIONS column, print all VFS (fs-independent) flags. This option is designed for auditing purposes to list also default VFS kernel mount options which are normally not listed.<br/> -y, --shell The column name will be modified to contain only characters allowed for shell variable identifiers. This is usable, for example, with --pairs. Note that this feature has been automatically enabled for --pairs in version 2.37, but due to compatibility issues, now it&rsquo;s necessary to request this behavior by --shell.<br/> -h, --help Display help text and exit.<br/> -V, --version Print version and exit.<br/> EXIT STATUS The exit value is 0 if there is something to display, or 1 on any error (for example if no filesystem is found based on the user&rsquo;s filter specification, or the device path or mountpoint does not exist).<br/> ENVIRONMENT LIBMOUNT_FSTAB=&lt;path&gt; overrides the default location of the fstab file<br/> LIBMOUNT_MTAB=&lt;path&gt; overrides the default location of the mtab file<br/> LIBMOUNT_DEBUG=all enables libmount debug output<br/> LIBSMARTCOLS_DEBUG=all enables libsmartcols debug output<br/> LIBSMARTCOLS_DEBUG_PADDING=on use visible padding characters.<br/> EXAMPLES findmnt --fstab -t nfs Prints all NFS filesystems defined in /etc/fstab.<br/> findmnt --fstab /mnt/foo Prints all /etc/fstab filesystems where the mountpoint directory is /mnt/foo. It also prints bind mounts where /mnt/foo is a source.<br/> findmnt --fstab --target /mnt/foo Prints all /etc/fstab filesystems where the mountpoint directory is /mnt/foo.<br/> findmnt --fstab --evaluate Prints all /etc/fstab filesystems and converts LABEL= and UUID= tags to the real device names.<br/> findmnt -n --raw --evaluate --output=target LABEL=/boot Prints only the mountpoint where the filesystem with label &quot;/boot&quot; is mounted.<br/> findmnt --poll --mountpoint /mnt/foo Monitors mount, unmount, remount and move on /mnt/foo.<br/> findmnt --poll=umount --first-only --mountpoint /mnt/foo Waits for /mnt/foo unmount.<br/> findmnt --poll=remount -t ext3 -O ro Monitors remounts to read-only mode on all ext3 filesystems.<br/> AUTHORS Karel Zak &lt;kzak@redhat.com&gt;<br/> SEE ALSO fstab(5), mount(8)<br/> REPORTING BUGS For bug reports, use the issue tracker at https://github.com/util-linux/util-linux/issues.<br/> AVAILABILITY The findmnt command is part of the util-linux package which can be downloaded from Linux Kernel Archive &lt;https://www.kernel.org/pub/linux/utils/util-linux/&gt;.<br/> util-linux 2.39.3 2023-12-01 FINDMNT(8)</pre> </div> <!-- endregion --> <p> The following incantation shows the mount points for the <code>data</code> partitions: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3b68e40f0697'><button class='copyBtn' data-clipboard-target='#id3b68e40f0697' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>findmnt -lo source,target | grep data <span class='unselectable'>/dev/sdb5 /media/mslinn/data /dev/sdc5 /media/mslinn/data1 </span></pre> </div> <!-- endregion --> <p> We now know that <b>for my Ubuntu instance</b>, the files and directories mounted at <code>/media/mslinn/data1/</code> need to be copied to <code>/media/mslinn/data/</code>. </p> <!-- endregion --> <!-- #region Rsync --> <h4 id="rsync"><span class="code">Rsync</span></h4> <p> <code>Rsync</code> can resume a large copy that failed earlier, without recopying files that were successfully copied. You can rerun <code>rsync</code> as many times as you like without fear of duplicating information. </p> <p> This is the general form of the <code>rsync</code> command that you should use. Instead of <code>source_directory</code> and <code>destination_directory</code>, specify the mount points for the two <code>data</code> directories. </p> <p class="alert rounded shadow"> Be sure to append a trailing slash to both directories when using <code>rsync</code>. </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idde4d494f34f4'><button class='copyBtn' data-clipboard-target='#idde4d494f34f4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>rsync -axHAXS --numeric-ids --info=progress2 \ source_directory<span style='background:yellow;'>/</span> destination_directory<span style='background:yellow;'>/</span></pre> </div> <!-- endregion --> <p> The <code>rsync</code> options I used are: </p> <table class='table'> <tr> <td><code>-a</code></td> <td>Copy all files, with permissions, etc.</td> </tr> <tr> <td><code>-x</code></td> <td>Stay on one file system</td> </tr> <tr> <td><code>-H</code> </td> <td>Preserve hard links</td> </tr> <tr> <td><code>-A</code></td> <td>Preserve ACLs/permissions</td> </tr> <tr> <td><code>-X</code></td> <td>Preserve extended attributes</td> </tr> <tr> <td><code>-S</code></td> <td>Handle sparse files</td> </tr> <tr> <td nowrap><code>--numeric-ids</code></td> <td>Transfer numeric group and user IDs, rather than using user and group names and mapping them at both ends</td> </tr> <tr> <td nowrap><code>--info=progress2</code></td> <td>shows statistics based on the whole transfer, rather than individual files</td> </tr> </table> <p> We can now write the proper incantation to run <code>rsync</code> to copy from <code>/media/mslinn/data1/</code> to <code>/media/mslinn/data/</code>. </p> <p class="alert rounded shadow"> Note that <code>sudo</code> is used to ensure that all files and directories are copied, without problems from insufficient permissions. </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id085d326f371c'><button class='copyBtn' data-clipboard-target='#id085d326f371c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span><span style='background:yellow;'>sudo</span> rsync -axHAXS --numeric-ids --info=progress2 \ /media/mslinn/data1<span style='background:yellow;'>/</span> /media/mslinn/data<span style='background:yellow;'>/</span></pre> </div> <!-- endregion --> <p> The copy took 25 seconds on my system. </p> <!-- endregion --> <!-- endregion --> <!-- #region All Done --> <h2 id="done">All Done</h2> <p> Unmount both <code>data</code> partitions, unplug the Sabrent enclosures, and remove the NVMe drives. Store the original NVMe drive somewhere safe, and install the clone into the computer that it belongs to; in my case, this was the P3S. </p> <div class='imgWrapper imgFlex inline' style=' '> <figure> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.cur" type="image/x-icon"> <img alt='Keep your NVMe drive safe' class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/p3s_medicine_crop.png" style='width: 100%; ' title='Keep your NVMe drive safe' /> </picture> <figcaption class='imgFigCaption '> Keep your NVMe drive safe </figcaption> </figure> </div> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <p> We did what we set out to do: resized the partition and its file system. </p> <!-- endregion --> Clonezilla 2024-06-12T00:00:00-04:00 https://mslinn.github.io/blog/2024/06/12/clonezilla <!-- #region intro --> <p> Although other OSes are mentioned, this article focuses on using Clonezilla to clone drives used by Linux systems. </p> <h2 id="miniseries">Mini-Series</h2> <p> This article can be read standalone; however, is part of a mini-series that discusses how to increase the storage capacity and performance of the Ableton Push 3 Standalone (P3S). If you own a P3S, or are thinking of purchasing one, you might want to first <a href='/av_studio/570-ableton-push-standalone.html'>read my review</a>. </p> <div class='imgWrapper imgFlex inline' style=' '> <a href='/av_studio/570-ableton-push-standalone.html' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/p3s_top.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/p3s_top.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/p3s_top.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/p3s_top.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/p3s_top.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/p3s_top.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/p3s_top.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/p3s_top.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/p3s_top.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/p3s_top.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/p3s_top.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/p3s_top.png" style='width: 100%; ' /> </picture> </a> </div> <p> The articles in this mini-series are: </p> <ol> <li><a href='/av_studio/570-ableton-push-nvme.html'>Ableton Push 3 NVMe Replacement</a></li> <li><a href='/blog/2024/07/05/cooling.html'>Thermal Control</a></li> <li><a href='/blog/2024/06/11/virtualbox.html'>VirtualBox</a></li> <li><a href='/blog/2024/06/12/clonezilla.html'>Clonezilla</a></li> <li><a href='/blog/2024/06/12/supersize-fs.html'>Supersizing a Partition and Its File System</a></li> </ol> <!-- endregion --> <!-- #region Nomenclature --> <h2 id="nomenclature">Nomenclature</h2> <p> An <i>original</i> is the drive that needs to be cloned, and a <i>clone</i> is a drive containing an identical copy of the original drive&rsquo;s contents. </p> <div class='imgWrapper imgFlex right' style='width: 35%; '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/snapshot.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/snapshot.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/snapshot.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/snapshot.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/snapshot.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/snapshot.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/snapshot.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/snapshot.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/snapshot.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/snapshot.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/snapshot.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/snapshot.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/snapshot.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/snapshot.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/snapshot.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/snapshot.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/snapshot.png" style='width: 100%; ' /> </picture> </div> <p> A <i>snapshot</i> is a read-only copy of an entire file system and all the files contained in the file system. The contents of each snapshot reflect the state of the file system at the time the snapshot was created. It is easy to navigate through each snapshot as if it were still active. The directories, folders and files will appear as they were at the time that the snapshot was created. </p> <p> By default, Linux systems are not configured to use snapshots; for example, Ableton Push 3 Standalone devices do not use snapshots. It is not difficult to set up snapshots using <a href='https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)' target='_blank' rel="nofollow"><code>LVM</code></a>; perhaps I will write an article about how to do that one day. </p><!-- endregion --> <!-- #region About Clonezilla --> <h2 id="about">About Clonezilla</h2> <p> <a href='https://clonezilla.org/' target='_blank' rel="nofollow">Clonezilla</a> can clone just about any type of Linux hard drive. </p> <div class='quote'> <div class='quoteText clearfix'> <p>Many file systems are supported:</p> <ol> <li><code>ext2</code>, <code>ext3</code>, <code>ext4</code>, <code>reiserfs</code>, <code>reiser4</code>, <code>xfs</code>, <code>jfs</code>, <code>btrfs</code>, <code>f2fs</code> and <code>nilfs2</code> of GNU/Linux</li> <li><code>FAT12</code>, <code>FAT16</code>, <code>FAT32</code>, <code>exFAT</code> and <code>NTFS</code> of MS Windows</li> <li><code>HFS+</code> and <code>APFS</code> of macOS</li> <li><code>UFS</code> of FreeBSD, NetBSD, and OpenBSD</li> <li>Minix</li> <li><code>VMFS3</code> and <code>VMFS5</code> of VMWare ESX</li> </ol> <p> Therefore, you can clone GNU/Linux, Microsoft Windows, Intel-based macOS, FreeBSD, NetBSD, OpenBSD, Minix, VMWare ESXi and Chrome OS/Chromium OS, no matter if it's 32-bit (x86) or 64-bit (x86-64) OS. For these file systems, only used blocks in partitions are saved and restored by <a href='http://partclone.org/' target='_blank' rel="nofollow">Partclone</a>. For unsupported file systems, sector-to-sector copy is done by <code>dd</code> in Clonezilla. </p> </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://clonezilla.org/' rel='nofollow' target='_blank'>What is Clonezilla?</a></div> </div> <p> Clonezilla works with MBR- and GPT-formatted hard drives. However, I am only going to discuss GPT-formatted drives. There is no point in discussing obsolete technology unless a <a href='/expertArticles/'>legal firm is paying me for my time</a>. </p> <!-- endregion --> <!-- #region GPT Drives --> <h2 id="gpt">GPT Drives</h2> <p> The <a href='https://en.wikipedia.org/wiki/GUID_Partition_Table' target='_blank' rel="nofollow">GUID Partition Table (GPT)</a> is a standard for the layout of the partition tables on a physical hard disk. It forms a part of the <a href='https://en.wikipedia.org/wiki/UEFI' target='_blank' rel="nofollow">Unified Extensible Firmware Interface (UEFI) standard</a>, which Microsoft calls EFI. GPT allows for a maximum disk and partition size of 9.4 ZB (a zetabyte is a billion terabytes). </p> <p> The bootloader for GPT drives is stored in the EFI system partition, formatted as <code>FAT16</code> or <code>FAT32</code>, and resides at the beginning of the disk &mdash; <b>Partition 1</b> in the following diagram. </p> <div class='imgWrapper imgFlex inline' style=' '> <figure> <a href='https://www.easeus.com/partition-master/remove-gpt-disk-partition.html' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/gpt-disk-layout.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/gpt-disk-layout.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/gpt-disk-layout.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/gpt-disk-layout.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/gpt-disk-layout.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/gpt-disk-layout.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gpt-disk-layout.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gpt-disk-layout.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gpt-disk-layout.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gpt-disk-layout.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/gpt-disk-layout.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/gpt-disk-layout.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gpt-disk-layout.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/gpt-disk-layout.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/gpt-disk-layout.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/gpt-disk-layout.cur" type="image/x-icon"> <img alt='From 'How to Remove, Delete or Format GPT Disk Partition'' class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/gpt-disk-layout.png" style='width: 100%; ' title='From 'How to Remove, Delete or Format GPT Disk Partition'' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://www.easeus.com/partition-master/remove-gpt-disk-partition.html" target='_blank' > From 'How to Remove, Delete or Format GPT Disk Partition' </a> </figcaption> </figure> </div> <p> As illustrated by the above diagram, drives contain a primary partition table, at least one partition, and a backup partition table. You could think of the two partition tables as bookends and the partitions as books between them, with the introductory book (the partition dedicated to containing the bootloader) in first place. </p> <div class='imgWrapper imgFlex center halfsize' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/bookends.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/bookends.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/bookends.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/bookends.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/bookends.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/bookends.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/bookends.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/bookends.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/bookends.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/bookends.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/bookends.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/bookends.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/bookends.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/bookends.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/bookends.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/bookends.cur" type="image/x-icon"> <img class="imgImg " src="/av_studio/images/ableton/push/clone/bookends.png" style='width: 100%; ' /> </picture> </div> <!-- endregion --> <!-- #region Overview --> <h2 id="overview">Clonezilla Overview</h2> <p> Clonezilla can clone entire drives, including the two GPT partition tables that bookend all the partitions in between, or it can just clone specific partitions. </p> <p> The Clonezilla procedure is the same, no matter what size file system you start with. The process after using Clonezilla changes, depending on the scenario. To summarize: </p> <ol> <li> Once Clonezilla finishes cloning a drive, all of the partitions in the clone are identical to the partitions in the original drive; any extra storage capacity is not used. </li> <li> If the cloned drive is the same size as the original, you are done. </li> <li> <div class='imgWrapper imgFlex right' style='width: 25%; '> <a href='/blog/2024/06/12/supersize-fs.html' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/supersize_me.png" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/supersize_me.png" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/supersize_me.png" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/supersize_me.png" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/supersize_me.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/supersize_me.png" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/supersize_me.png" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/supersize_me.png" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/supersize_me.png" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/supersize_me.png" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/supersize_me.png" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/supersize_me.png" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/supersize_me.png" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/supersize_me.png" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/supersize_me.png" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/supersize_me.png" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/supersize_me.png" style='width: 100%; ' /> </picture> </a> </div> As I discuss in <a href='/blog/2024/06/12/supersize-fs.html'>Supersizing a Partition and Its File System</a>, tiny Linux file systems (those with 1K blocks) should be recreated larger, not enlarged, and the same is true for normal-sized file systems that are being migrated to much larger partitions. Normally, only one of the partitions on a drive should be enlarged, so this article makes that assumption. <br><br> To recreate a file system in a partition: <ol> <li> The file system in the partition being enlarged should be deleted. </li> <li> The partition should be enlarged. </li> <li> A new file system should be created in the enlarged partition. The creation process heavily optimizes the file system for the partition it is placed in. </li> <li> The files from the partition on the original drive should be copied to the new file system on the cloned drive. </li> </ol> </li> </ol> <!-- endregion --> <!-- #region Usage --> <h2 id="usage">Usage</h2> <p> The Clonezilla installation process depends on how you want to use it. I describe the installation processes separately for each scenario discussed in this article. </p> <div class='imgWrapper imgFlex right' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/virtualbox_logo.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/virtualbox_logo.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/virtualbox_logo.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/virtualbox_logo.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/virtualbox_logo.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/virtualbox_logo.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/virtualbox_logo.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/virtualbox_logo.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/virtualbox_logo.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/virtualbox_logo.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/virtualbox_logo.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/virtualbox_logo.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/virtualbox_logo.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/virtualbox_logo.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/virtualbox_logo.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/virtualbox_logo.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/virtualbox_logo.png" style='width: 100%; ' /> </picture> </div> <p> I would like to encourage you to get in the habit of running Clonezilla in a VirtualBox virtual machine, as I did when I wrote this article. Unless you want to clone the system drive in the machine that you prefer to run VirtualBox on, this is generally your best option for running Clonezilla. </p> <p> You have many other options as well. You can run Clonezilla from a normal Ubuntu system if you just want to clone a drive that is not mounted or if your file system supports snapshots. To clone a system disk that does not support snapshots, the drives on the disk will first have to be attached to the OS as a <code>/dev</code> device but not mounted, then the virtualized system should be booted, or an ISO system image on a CD / DVD / SSD should be booted. Once Ubuntu is running, Clonezilla can be run as a console text program in a terminal window. </p> <p> I was unable to use Clonezilla from WSL because I did not know how to map USB drives to the WSL VM so Clonezilla could work its magic. Maybe one day I will fight through the issues involved, but there is no reason for me to work that hard for no benefit. </p> <!-- endregion --> <!-- #region Cloning an Unmounted Drive --> <h2 id="unm">Cloning an Unmounted Drive</h2> <!-- #region Installation --> <h3 id="i2">Installation</h3> <p> If you want to clone unmounted drives, install Clonezilla on an Ubuntu system like this: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf180d8398032'><button class='copyBtn' data-clipboard-target='#idf180d8398032' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install clonezilla</pre> </div> <!-- endregion --> <p> Once installed, read the <code>clonezilla</code> help message: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2f5c9b07829c'><button class='copyBtn' data-clipboard-target='#id2f5c9b07829c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo clonezilla -h <span class='unselectable'>/usr/sbin/clonezilla: -h: invalid option Usage: Run clonezilla: /usr/sbin/clonezilla [OPTION] -l, --language INDEX Set the language to be shown by index number: [0|en_US.UTF-8]: English, [2|zh_TW.UTF-8]: Traditional Chinese (UTF-8, Unicode) - Taiwan [a|ask]: Prompt to ask the language index This option is for backward compatibility. It&#39;s recommended to use locales to assign that. -d0, --dialog Use dialog -d1, --Xdialog Use Xdialog -d2, --whiptail Use whiptail -d3, --gdialog Use gdialog -d4, --kdialog Use kdialog -k, --skip-ocs-prep-repo Skip preparing the clonezilla image home directory (assume it&#39;s ready), this is specially for device &lt;-&gt; image clone. -p, --postaction [choose|poweroff|reboot|command|CMD] When save/restoration finishs, choose action in the client, poweroff, reboot (default), in command prompt or run CMD -s, --skip-lite-menu Do not show live-server and lite-client in the dialog menu. Ex. /usr/sbin/clonezilla -l en </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Launch Clonezilla --> <h3 id="lc">Launch Clonezilla</h3> <p> Now launch <code>clonezilla</code>: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ideae578f65f5a'><button class='copyBtn' data-clipboard-target='#ideae578f65f5a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo clonezilla</pre> </div> <!-- endregion --> <p> If you are cloning an unmounted drive, you can skip the next section and continue to <a href='#rc'>Running Clonezilla</a>. </p> <!-- endregion --> <!-- endregion --> <!-- #region Cloning a System Drive --> <h2 id="sysclone">Cloning a System Drive</h2> <!-- #region implicit --> <p> <i>This section actually describes how to clone any Linux-compatible drive, not just system drives.</i> </p> <p> Generally speaking, a system cannot clone itself unless it has the ability to snapshot the drives that you want to back up. Without a snapshot, cloning a system drive requires Clonezilla to be booted, such that: </p> <ol> <li>The drive to be cloned is connected as a <code>/dev/</code> Linux device</li> <li>None of the file systems in any of the drive's partitions may be mounted</li> </ol> <!-- endregion --> <!-- #region Installation in a VirtualBox VM --> <h3 id="i2">Installation in a VirtualBox VM</h3> <p> Clonezilla is available as a package that includes a minimal Linux operating system. After the dedicated Linux OS boots, it runs Clonezilla. </p> <p> You can install Clonezilla on a CD, DVD, or SSD drive. It was more convenient for me to run Clonezilla in a VirtualBox virtual machine (VM) instead of running on bare metal using a CD or SSD drive. </p> <p> I downloaded the <a href='https://clonezilla.org/downloads/download.php?branch=stable' target='_blank' rel="nofollow">ISO image for standalone Clonezilla</a> for the <code>amd64</code> architecture. This will run on physical Windows desktops, Mac and Linux desktop computers that can boot from ISO images, CDs, DVDs and SSD drives. It will also run on a virtual 64-bit x86 computer. </p> <p> Then I started VirtualBox as described in my <a href='/blog/2024/06/11/virtualbox.html'>VirtualBox Setup</a> article and created a new virtual machine using the Clonezilla ISO image. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_1.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_1.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_1.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_1.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_1.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_1.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_1.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_1.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_1.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_1.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_1.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_1.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_1.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_1.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_1.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_1.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_1.png" style='width: 100%; ' /> </picture> </div> <p> Notice that Clonezilla does not require a virtual hard drive; it strictly runs from the downloaded ISO image. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_2.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_2.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_2.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_2.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_2.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_2.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_2.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_2.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_2.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_2.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_2.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_2.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_2.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_2.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_2.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_2.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_2.png" style='width: 100%; ' /> </picture> </div> <p> I launched the Clonezilla VM, and it instantly booted the Linux setup program. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_3.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_3.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_3.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_3.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_3.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_3.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_3.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_3.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_3.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_3.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_3.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_3.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_3.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_3.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_3.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_3.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_3.png" style='width: 100%; ' /> </picture> </div> <p> The Linux setup program asked me to select the keyboard layout I was using, and I pressed <kbd>Enter</kbd> to accept the default <code>en_US</code> layout. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_4.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_4.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_4.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_4.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_4.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_4.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_4.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_4.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_4.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_4.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_4.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_4.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_4.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_4.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_4.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_4.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_4.png" style='width: 100%; ' /> </picture> </div> <p> Yes, I am sure. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_5.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_5.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_5.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_5.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_5.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_5.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_5.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_5.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_5.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_5.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_5.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_5.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_5.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_5.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_5.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_5.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_5.png" style='width: 100%; ' /> </picture> </div> <p> Once Linux is configured, it is time to run the Clonezilla program. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_6.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_6.png" style='width: 100%; ' /> </picture> </div> <!-- endregion --> <!-- endregion --> <!-- #region Running Clonezilla --> <h2 id="rc">Running Clonezilla</h2> <p> Clonezilla uses different background colors when running from a command line, and when running from an ISO image. Otherwise, the program is exactly the same. </p> <p> The next two screenshots show the difference between Clonezilla running from a command-line, and when running from an ISO image. Both of the screens accomplish the same thing: defining the mode of operation. </p> <div class="rounded shadow" style="border: thin solid grey; padding: 1em; margin-bottom: 1em;"> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.cur" type="image/x-icon"> <img class="imgImg " src="/av_studio/images/ableton/push/clone/clonezilla_7_standalone.png" style='width: 100%; ' /> </picture> </div> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6b.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_6b.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6b.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6b.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6b.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6b.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6b.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6b.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6b.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6b.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6b.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6b.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6b.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6b.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6b.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_6b.cur" type="image/x-icon"> <img class="imgImg " src="/av_studio/images/ableton/push/clone/clonezilla_6b.png" style='width: 100%; ' /> </picture> </div> </div> <p> I wanted to copy from one physical disk to another, so I selected the second choice, <code>device-device</code>, using the up- and down-arrow keys, then pressing <kbd>Enter</kbd>. </p> <p> In order for the VM to 'see' the NVMe drives, I right-clicked on the little rocket icon at the bottom of the VirtualBox window. This displayed all the USB devices attached to the Windows computer. I then enabled the two <b>Sabrent [2001]</b> USB devices, which made them exclusively available to the VM running Clonezilla. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_7.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_7.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_7.png" style='width: 100%; ' /> </picture> </div> <p> Now I selected <code>disk_to_local_disk</code> and pressed the <kbd>Enter</kbd> key. </p> <p> After several long minutes, I was presented with the two USB drives and was asked to select the source drive to clone, which was of course the 256 GB NVMe. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_8.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_8.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_8.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_8.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_8.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_8.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_8.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_8.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_8.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_8.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_8.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_8.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_8.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_8.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_8.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_8.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_8.png" style='width: 100%; ' /> </picture> </div> <P> After pressing the <kbd>Enter</kbd> key, I had to wait a while once again while Clonezilla laboriously enumerated all available drives. Eventually, it displayed the 4 TB drive as the only possible candidate for the destination, which was correct. I pressed the <kbd>Enter</kbd> key once more. </P> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_9.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_9.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_9.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_9.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_9.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_9.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_9.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_9.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_9.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_9.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_9.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_9.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_9.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_9.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_9.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_9.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_9.png" style='width: 100%; ' /> </picture> </div> <p> Another long pause ensued, and then I was presented with this screen: </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_10.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_10.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_10.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_10.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_10.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_10.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_10.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_10.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_10.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_10.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_10.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_10.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_10.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_10.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_10.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_10.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_10.png" style='width: 100%; ' /> </picture> </div> <p> I pressed <kbd>Enter</kbd> and soon saw: </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_11.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_11.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_11.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_11.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_11.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_11.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_11.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_11.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_11.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_11.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_11.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_11.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_11.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_11.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_11.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_11.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_11.png" style='width: 100%; ' /> </picture> </div> <p> I just want to resize the <code>data</code> partition, so I selected the first option, <code>-k0</code>. I will resize the partition later using <code>gparted</code>. </p> <p> I will decide what to do later, so I picked the first option, <code>-p choose</code>. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_12.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_12.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_12.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_12.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_12.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_12.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_12.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_12.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_12.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_12.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_12.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_12.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_12.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_12.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_12.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_12.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_12.png" style='width: 100%; ' /> </picture> </div> <p> A helpful hit was displayed in case I need to run this command again. The information was presented in green text over a black background at the bottom of the window. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_13.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_13.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_13.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_13.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_13.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_13.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_13.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_13.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_13.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_13.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_13.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_13.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_13.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_13.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_13.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_13.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_13.png" style='width: 100%; ' /> </picture> </div> <p> The small blue LEDs on both Sabrent enclosures flashed for several minutes. Finally, I was asked twice if I wanted the contents of <code>sda</code> (the new 4 TB drive) to be overwritten. Yes! </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_14.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_14.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_14.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_14.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_14.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_14.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_14.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_14.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_14.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_14.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_14.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_14.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_14.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_14.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_14.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_14.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_14.png" style='width: 100%; ' /> </picture> </div> <p> <code>/dev/sda3</code> had a file with errors. I will run <code>fsck</code> on it shortly. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_15.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_15.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_15.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_15.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_15.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_15.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_15.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_15.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_15.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_15.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_15.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_15.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_15.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_15.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_15.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_15.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_15.png" style='width: 100%; ' /> </picture> </div> <p> I left Clonezilla running overnight. When I awoke, it had completed: </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_16.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_16.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_16.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_16.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_16.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_16.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_16.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_16.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_16.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_16.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_16.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_16.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_16.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_16.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_16.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_16.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_16.png" style='width: 100%; ' /> </picture> </div> <p> I turned off the Clonezilla VM. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_17.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/clonezilla_17.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_17.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_17.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_17.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_17.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_17.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_17.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_17.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_17.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_17.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_17.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_17.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_17.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_17.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/clonezilla_17.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/clonezilla_17.png" style='width: 100%; ' /> </picture> </div> <!-- endregion --> <!-- #region Store the Original NVMe --> <h2 id="store">Store the Original NVMe</h2> <p> I unplugged the original 256 GB NVMe drive and placed it into an empty prescription bottle, then stored the bottle with my extra drives. Small spice bottles also work well for storing NVMe drives. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/clone/p3s_medicine_crop.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/clone/p3s_medicine_crop.png" style='width: 100%; ' /> </picture> </div> <!-- endregion --> <!-- #region Clone Verification --> <h2 id="verify">Clone Verification</h2> <p> Now we can verify that the cloned drive works by putting it into the system that the original drive came from and verifying that the system still works. </p> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <p> If the drive containing the clone has the same storage capacity as the original drive, you are done. </p> <!-- endregion --> <!-- #region Next: Resizing --> <h2 id="next">Next: Resizing</h2> <p> If the drive containing the clone is larger or smaller than the original drive, at least one of the partitions on the clone will need to be resized. Remove the cloned drive from the system, place it back into the Sabrent NVMe / USB enclosure, and reattach the enclosure to the computer for further work. Please continue on to read <a href='/blog/2024/06/13/supersize-fs.html'>Supersizing a Partition and Its File System</a> for more information. </p> <!-- endregion --> <!-- endregion --> VirtualBox 2024-06-11T00:00:00-04:00 https://mslinn.github.io/blog/2024/06/11/virtualbox <!-- #region intro --> <p> This article describes how I set up Ubuntu Desktop 24.04 in a VirtualBox VM. </p> <h2 id="miniseries">Mini-Series</h2> <p> This article can be read standalone; however, is part of a mini-series that discusses how to increase the storage capacity and performance of the Ableton Push 3 Standalone (P3S). If you own a P3S, or are thinking of purchasing one, you might want to first <a href='/av_studio/570-ableton-push-standalone.html'>read my review</a>. </p> <div class='imgWrapper imgFlex inline' style=' '> <a href='/av_studio/570-ableton-push-standalone.html' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/av_studio/images/ableton/push/p3s_top.svg" type="image/svg"> <!---<source srcset="/av_studio/images/ableton/push/p3s_top.avif" type="image/avif">--> <source srcset="/av_studio/images/ableton/push/p3s_top.webp" type="image/webp"> <source srcset="/av_studio/images/ableton/push/p3s_top.apng" type="image/apng"> <source srcset="/av_studio/images/ableton/push/p3s_top.png" type="image/png"> <source srcset="/av_studio/images/ableton/push/p3s_top.jpg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.jpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.jfif" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.pjpeg" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.pjp" type="image/jpeg"> <source srcset="/av_studio/images/ableton/push/p3s_top.gif" type="image/gif"> <source srcset="/av_studio/images/ableton/push/p3s_top.tif" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/p3s_top.tiff" type="image/tiff"> <source srcset="/av_studio/images/ableton/push/p3s_top.bmp" type="image/bmp"> <source srcset="/av_studio/images/ableton/push/p3s_top.ico" type="image/x-icon"> <source srcset="/av_studio/images/ableton/push/p3s_top.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/av_studio/images/ableton/push/p3s_top.png" style='width: 100%; ' /> </picture> </a> </div> <p> The articles in this mini-series are: </p> <ol> <li><a href='/av_studio/570-ableton-push-nvme.html'>Ableton Push 3 NVMe Replacement</a></li> <li><a href='/blog/2024/07/05/cooling.html'>Thermal Control</a></li> <li><a href='/blog/2024/06/11/virtualbox.html'>VirtualBox</a></li> <li><a href='/blog/2024/06/12/clonezilla.html'>Clonezilla</a></li> <li><a href='/blog/2024/06/12/supersize-fs.html'>Supersizing a Partition and Its File System</a></li> </ol> <!-- endregion --> <!-- #region VirtualBox --> <h2 id="vb">VirtualBox</h2> <p> VirtualBox allows you to run another operating system within your computer&rsquo;s operating system. VirtualBox is available for Windows, Macintosh, Linux and other OSes. It can virtualize Windows, Linux, <a href='https://forums.virtualbox.org/viewtopic.php?t=92649&sid=b459106f6da250c0d79e573e3ee823b7' target='_blank' rel="nofollow">macOS</a>, Solaris, and the BSD UNIX work-alikes. </p> <p> I often use an Ubuntu Desktop instance in a <a href='https://www.virtualbox.org/' target='_blank' rel="nofollow">VirtualBox</a> VM. It is my default Ubuntu instance for general-purpose work. </p> <!-- endregion --> <!-- #region Terminology --> <h2 id="terminology">Terminology</h2> <dl> <dt>Host</dt> <dd>This is the OS of the physical computer on which Oracle VM VirtualBox was installed.</dd> <dt>Guest</dt> <dd>This is the OS of the physical computer on which Oracle VM VirtualBox was installed.</dd> <dt>Virtual machine (VM)</dt> <dd> This is the special environment that VirtualBox creates for your guest OS while it is running. In other words, you run your guest OS in a VM. </dd> </dl> <!-- endregion --> <!-- #region Installation --> <h2 id="install">Installation</h2> <p> I installed VirtualBox and the <a href='https://blogs.oracle.com/virtualization/post/friday-spotlight-virtualbox-extension-pack-guest-additions' target='_blank' rel="nofollow">VirtualBox Extension Pack</a> according to <a href='https://www.virtualbox.org/manual/ch04.html' target='_blank' rel="nofollow">these in&shy;struc&shy;tions</a>. </p> <p> I then installed Ubuntu Desktop 24.04 as a guest in a new VirtualBox VM. The steps are: </p> <ol> <li>Download the <a href='https://ubuntu.com/download/desktop' target='_blank' rel="nofollow">Ubuntu Desktop 24.04</a> ISO file.</li> <li>Open VirtualBox and click on the <kbd>New</kbd> button to create a new virtual machine.</li> <li>Name your virtual machine and select <b>Linux</b> as the type and <b>Ubuntu (64-bit)</b> as the version.</li> <li>Allocate at least 2 GB of RAM to the virtual machine; 4 GB is better.</li> <li>Choose <b>Create a virtual hard disk now</b> and click <kbd>Create</kbd>.</li> <li>Select <b>VDI (VirtualBox Disk Image)</b> as the hard disk file type.</li> </ol> 7 . Choose " Dynamically allocated " for the storage type . 8 . Allocate at least 20 GB of storage space for the virtual machine . 9 . Click on the " Start " button to launch the virtual machine . 10 . In the " Select start - up disk " window , click on the folder icon and browse for the Ubuntu 24.04 ISO file you downloaded . 11 . Click " Start " to begin the installation process . 12 . Follow the on - screen instructions to complete the installation . <!-- endregion --> <!-- #region VirtualBox Guest Additions --> <h2 id="vbga">VirtualBox Guest Additions</h2> <p> A third item needs to be installed in the guest OS, the VirtualBox Guest Additions. It is included in the VirtualBox download and provides: </p> <ul> <li>Seamless mouse support</li> <li>Shared folders</li> <li>Better video support</li> <li>Seamless windows</li> <li>Generic host/guest communication channels</li> <li>Time synchronization</li> <li>Shared clipboard</li> <li>Automated logins</li> </ul> <p> For Windows, the VirtualBox Guest Additions are installed at <code>%ProgramFiles%\Oracle\VirtualBox\VBoxGuestAdditions.iso</code>. For Mac OS X hosts, this file is provided in the VirtualBox application bundle. </p> <!-- endregion --> <!-- #region Installation --> <h3 id="vgai">Installation</h3> <p> Before installing the VirtualBox Guest Additions in a guest Linux instance, you should install <code>gcc</code>, <code>make</code> and <code>perl</code>. Launch the virtual machine containing the guest Ubuntu image and enter the following at a shell prompt: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2ff0d0051ebb'><button class='copyBtn' data-clipboard-target='#id2ff0d0051ebb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install build-essential gcc make perl dkms</pre> </div> <!-- endregion --> <p> Now reboot the guest Ubuntu instance. </p> <p> Each time a VirtualBox Ubuntu instance boots, the VirtualBox Guest Additions CD-ROM image automounts at <code>/media/$USER/VBox_GAs_7.0.18/</code>. </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbceaab3d126d'><button class='copyBtn' data-clipboard-target='#idbceaab3d126d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls /media/$USER/VBox_GAs_7.0.18/ <span class='unselectable'>AUTORUN.INF runasroot.sh VBoxSolarisAdditions.pkg autorun.sh TRANS.TBL VBoxWindowsAdditions-amd64.exe cert VBoxDarwinAdditions.pkg VBoxWindowsAdditions.exe NT3x VBoxDarwinAdditionsUninstall.tool VBoxWindowsAdditions-x86.exe OS2 VBoxLinuxAdditions.run windows11-bypass.reg </span><br></pre> </div> <!-- endregion --> <p> To install the <a href='https://www.virtualbox.org/manual/ch04.html' target='_blank' rel="nofollow">VirtualBox Guest Additions</a> in the guest Ubuntu instance, run <code>VBoxLinuxAdditions.run</code>, and then reboot, as follows: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2e6c86415e9b'><button class='copyBtn' data-clipboard-target='#id2e6c86415e9b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo /media/$USER/VBox_GAs_7.0.18/VBoxLinuxAdditions.run <span class='unselectable'>Verifying archive integrity... 100% MD5 checksums are OK. All good. Uncompressing VirtualBox 7.0.18 Guest Additions for Linux 100% VirtualBox Guest Additions installer Removing installed version 7.0.18 of VirtualBox Guest Additions... update-initramfs: Generating /boot/initrd.img-6.8.0-35-generic Copying additional installer modules ... Installing additional modules ... VirtualBox Guest Additions: Starting. VirtualBox Guest Additions: Setting up modules VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel modules. This may take a while. VirtualBox Guest Additions: To build modules for other installed kernels, run VirtualBox Guest Additions: /sbin/rcvboxadd quicksetup <version> VirtualBox Guest Additions: or VirtualBox Guest Additions: /sbin/rcvboxadd quicksetup all VirtualBox Guest Additions: Building the modules for kernel 6.8.0-35-generic. update-initramfs: Generating /boot/initrd.img-6.8.0-35-generic </span><br> <span class='unselectable'>$ </span>sudo reboot</pre> </div> <!-- endregion --> <!-- endregion --> Rust 2024-05-30T00:00:00-04:00 https://mslinn.github.io/blog/2024/05/30/rust <!-- #region intro --><p> Rust is a low-level programming language done right, using modern tools with good documentation and extensive support. </p> <p> I have a lot of <a href='/resume/bio.html'>experience</a> with the Scala and Ruby languages, as well as dozens of other computer languages. I believe that Rust has adopted the best parts of the Scala and Ruby philosophies and melded the syntaxes together nicely. For example, all of these languages provide <a href='https://rust-book.cs.brown.edu/ch03-03-how-functions-work.html#statements-and-expressions' target='_blank' rel="nofollow">expressions</a>, <a href='https://doc.rust-lang.org/book/ch13-01-closures.html' target='_blank' rel="nofollow">closures</a>, <a href='https://doc.rust-lang.org/book/ch05-00-structs.html' target='_blank' rel="nofollow">structs</a> (value classes) and <a href='https://doc.rust-lang.org/book/ch10-00-generics.html' target='_blank' rel="nofollow">generics</a>. </p> <p> The package management is a bit like Ruby&rsquo;s <code>gem</code>s. <a href='https://crates.io/' target='_blank' rel="nofollow">Cargo crates</a> and <a href='https://rubygems.org/' target='_blank' rel="nofollow">Ruby gems</a> are provided as source code and compiled on demand. </p> <p> Rust has many of the Scala features I enjoy the most: </p> <ul> <li> <a href='https://doc.rust-lang.org/book/ch17-00-oop.html' target='_blank' rel="nofollow">Object-oriented</a> and <a href='https://doc.rust-lang.org/book/ch13-00-functional-features.html' target='_blank' rel="nofollow">functional programming</a> styles are both supported. </li> <li> <a href='https://doc.rust-lang.org/book/ch18-00-patterns.html' target='_blank' rel="nofollow">Pattern matching</a>. </li> <li><a href='https://doc.rust-lang.org/book/ch19-03-advanced-traits.html' target='_blank' rel="nofollow">Traits.</a></li> <li><a href='https://en.wikipedia.org/wiki/Comparison_of_programming_languages_(algebraic_data_type)#Rust' target='_blank' rel="nofollow">Algebraic type system.</a></li> </ul> <p> What sets Rust apart, however, is: </p> <ol> <li><a href='https://doc.rust-lang.org/book/ch09-00-error-handling.html' target='_blank' rel="nofollow">Error handling</a>.</li> <li> <a href='https://doc.rust-lang.org/book/ch04-00-understanding-ownership.html' target='_blank' rel="nofollow">Data ownership</a> (borrow checker) and <a href='https://doc.rust-lang.org/book/ch15-00-smart-pointers.html' target='_blank' rel="nofollow">smart pointers</a>. </li> <li>Data has <a href='https://doc.rust-lang.org/book/ch10-00-generics.html' target='_blank' rel="nofollow">lifetimes</a>.</li> <li>Stable, yet advanced compile-time metaprogramming (<a href='https://doc.rust-lang.org/book/ch19-06-macros.html' target='_blank' rel="nofollow">macros</a>).</li> <li>No runtime types.</li> <li><a href='https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html' target='_blank' rel="nofollow">Unsafe system.</a></li> <li> Memory safety without a garbage collector; eal-time behavior is deterministic. Real-time audio/video applications are therefore possible. </li> </ol> <p> <a href='https://rust-lang.org'><code>rust-lang.org</code></a> is the home of the Rust language. </p> <!-- endregion --> <!-- #region Background --> <h2 id="background">Background</h2> <p> Rust was initially released in May 2015 by <a href='https://www.crunchbase.com/person/graydon-hoare' target='_blank' rel="nofollow">Graydon Hoare</a> (no relation to <a href='https://en.wikipedia.org/wiki/Tony_Hoare' target='_blank' rel="nofollow">Tony Hoare</a>). Graydon had created the <a href='https://www.swift.org/' target='_blank' rel="nofollow">Swift language</a> used by Apple the previous year. <a href='https://v4.chriskrycho.com/rust-and-swift.html' target='_blank' rel="nofollow">Chris Krycho</a> has written a detailed comparison of the Swift and Rust languages. </p> <p> <a href='https://www.thurrott.com/windows/282471/microsoft-is-rewriting-parts-of-the-windows-kernel-in-rust' target='_blank' rel="nofollow">Microsoft vice president David Weston said</a> on Apr 29, 2023 that Microsoft had rewritten 36,000 lines of code in the Windows kernel in Rust, in addition to another 152,000 lines of code it had written for a proof of concept involving the DirectWrite Core library. The performance is purportedly excellent, with no regressions compared to the old C++ code. </p> <p> Linus Torvalds accepts contributions for Linux written in Rust, and the Rust infrastructure now present in most Linux installations is mature. Linux 6.8, released on March 10, 2024, was the first version of Linux to include a <a href='https://www.phoronix.com/news/Linux-6.8-Rust-PHY-Driver' target='_blank' rel="nofollow">device driver written in Rust</a>, replacing an older network device driver written in C. </p> <p> The <a href='https://www.reddit.com/r/rust/comments/1bxppdr/todays_xkcd_comic_is_a_physics_game_built_with/' target='_blank' rel="nofollow">XKCD Rust API</a> looks like a fun way to blow off an afternoon. </p> <p> If you need help, see the <code>#beginners</code> <a href='https://discord.gg/rust-lang' target='_blank' rel="nofollow">Rust Discord channel</a> and the <a href='https://users.rust-lang.org/' target='_blank' rel="nofollow">Rust Users Forum</a>. </p> <!-- endregion --> <!-- #region Bindings To Other Languages --> <h2 id="bind">Bindings To Other Languages</h2><ul> <li> <b>Ruby / Rust bindings</b> &mdash; <a href='https://crates.io/crates/magnus' target='_blank' rel="nofollow">Magnus</a> allows you to write Ruby gems in Rust, and call Ruby code from Rust code. </li> <li> <b>Python / Rust bindings.</b> &mdash; <a href='https://www.infoworld.com/article/3664124/how-to-use-rust-with-python-and-python-with-rust.html' target='_blank' rel="nofollow">Py03</a> allows you to mix Rust with Python. Enjoy Python&rsquo;s convenience with Rust&rsquo;s speed. </li> <li> <a href='https://docs.rust-embedded.org/book/interoperability/c-with-rust.html' target='_blank' rel="nofollow">Calling C from Rust</a>. </li> <li> <a href='https://developers.redhat.com/articles/2022/09/05/how-create-c-binding-rust-library' target='_blank' rel="nofollow">How to create C binding for a Rust library</a>: Rust can generate C dynamic libraries (<code>.so</code> files) as well as static libraries (<code>.a</code> files), which can be easily wrapped in Go bindings and Python bindings and used in code written in those languages. </li> <li> <a href='https://github.com/rust-lang/rust-bindgen' target='_blank' rel="nofollow"><code>bindgen</code></a> automatically generates Rust FFI bindings to C (and some C++) libraries. </li> <li> <a href='https://github.com/giovanniberti/robusta' target='_blank' rel="nofollow"><code>Robusta</code></a> — easy interoperability between Rust and Java. This library provides a procedural macro to make it easier to write JNI-compatible code in Rust and automatically convert Rust input and output types. </li> <li> <a href='https://stackoverflow.com/questions/31666936/execute-a-shell-command' target='_blank' rel="nofollow">Execute a shell command from Rust.</a> </li> </ul> <!-- endregion --> <!-- #region rustup --> <h2 id="rustup" class="clear">rustup</h2> <!-- #region implicit --> <p> <code>Rustup</code> is an installer for the Rust systems programming language. </p> <!-- endregion --> <h3 id="wslup">WSL / Ubuntu</h3> <p> For WSL / Ubuntu you can easily install <code>rustup</code>, then install the Rust toolchain with the following commands: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id65140b9e9a4a'><button class='copyBtn' data-clipboard-target='#id65140b9e9a4a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install rustup<br> <span class='unselectable'>$ </span>rustup default stable <span class='unselectable'>info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu' info: latest update on 2024-05-02, rust version 1.78.0 (9b00956e5 2024-04-29) info: downloading component 'cargo' info: downloading component 'clippy' info: downloading component 'rust-docs' info: downloading component 'rust-std' info: downloading component 'rustc' 63.7 MiB / 63.7 MiB (100 %) 38.2 MiB/s in 1s ETA: 0s info: downloading component 'rustfmt' info: installing component 'cargo' info: installing component 'clippy' info: installing component 'rust-docs' info: installing component 'rust-std' 24.3 MiB / 24.3 MiB (100 %) 23.8 MiB/s in 1s ETA: 0s info: installing component 'rustc' 63.7 MiB / 63.7 MiB (100 %) 20.5 MiB/s in 2s ETA: 0s info: installing component 'rustfmt' info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'<br> <span style='color: green;'>stable-x86_64-unknown-linux-gnu installed</span> - rustc 1.78.0 (9b00956e5 2024-04-29) </span></pre> </div> <!-- endregion --> <!-- #region Other OSes --> <h3 id="ruother">Other OSes</h3> <p> Installation instructions for your computer's OS appear when you visit <a href='https://rustup.rs'><code>rustup.rs</code></a>. </p> <p> If you want to customize your setup, you need to know the host triple for your system. A &lsquo;host triple&rsquo; identifies the computer architecture and OS to install to. In truth, the value is no longer a triple but a <a href='https://docs.rs/target-lexicon/latest/target_lexicon/struct.Triple.html' target='_blank' rel="nofollow">quintic value</a>. </p> <p> For WSL / Ubuntu, the host triple is <code>x86_64-unknown-linux-gnu</code>. For Mac, the host triple is <code>x86_64-apple-darwin</code>. The <code>toolchain</code> string consists of the <a href='https://rust-lang.github.io/rustup/concepts/channels.html' target='_blank' rel="nofollow">channel prefix</a>, followed by the host triple; the channels are <code>stable</code>, <code>beta</code> and <code>nightly</code>. Thus, the complete toolchain strings for the <code>stable</code> channel are: </p> <dl> <dt>WSL / Ubuntu</dt> <dd><code>stable-x86_64-unknown-linux-gnu</code></dd> <dt>Mac</dt> <dd><code>stable-x86_64-apple-darwin</code></dd> </ul> <p> This is how I ran the <code>rustup</code> installer. Notice that I specified the default, option 1 - standard installation because the default values are optimal. </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0da4b6aa4175'><button class='copyBtn' data-clipboard-target='#id0da4b6aa4175' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh <span class='unselectable'>info: downloading installer<br/> Welcome to Rust!<br/> This will download and install the official compiler for the Rust programming language, and its package manager, Cargo.<br/> Rustup metadata and toolchains will be installed into the Rustup home directory, located at:<br/> /home/mslinn/.rustup<br/> This can be modified with the RUSTUP_HOME environment variable.<br/> The Cargo home directory is located at:<br/> /home/mslinn/.cargo<br/> This can be modified with the CARGO_HOME environment variable.<br/> The cargo, rustc, rustup and other commands will be added to Cargo&#39;s bin directory, located at:<br/> /home/mslinn/.cargo/bin<br/> This path will then be added to your PATH environment variable by modifying the profile files located at:<br/> /home/mslinn/.profile /home/mslinn/.bashrc<br/> You can uninstall at any time with rustup self uninstall and these changes will be reverted.<br/> Current installation options:<br/> default host triple: x86_64-unknown-linux-gnu default toolchain: stable (default) profile: default modify PATH variable: yes<br/> 1) Proceed with standard installation (default - just press enter) 2) Customize installation 3) Cancel installation &gt; </span><kbd>Enter</kbd><br/> <span class='unselectable'>info: profile set to &#39;default&#39; info: setting default host triple to x86_64-unknown-linux-gnu info: syncing channel updates for &#39;stable-x86_64-unknown-linux-gnu&#39; info: latest update on 2024-05-02, rust version 1.78.0 (9b00956e5 2024-04-29) info: downloading component &#39;cargo&#39; info: downloading component &#39;clippy&#39; info: downloading component &#39;rust-docs&#39; 15.1 MiB / 15.1 MiB (100 %) 9.6 MiB/s in 2s ETA: 0s info: downloading component &#39;rust-mingw&#39; info: downloading component &#39;rust-std&#39; 24.6 MiB / 24.6 MiB (100 %) 18.3 MiB/s in 1s ETA: 0s info: downloading component &#39;rustc&#39; 73.8 MiB / 73.8 MiB (100 %) 18.1 MiB/s in 4s ETA: 0s info: downloading component &#39;rustfmt&#39; info: installing component &#39;cargo&#39; info: installing component &#39;clippy&#39; info: installing component &#39;rust-docs&#39; 15.1 MiB / 15.1 MiB (100 %) 3.6 MiB/s in 3s ETA: 0s info: installing component &#39;rust-mingw&#39; info: installing component &#39;rust-std&#39; 24.6 MiB / 24.6 MiB (100 %) 10.9 MiB/s in 2s ETA: 0s info: installing component &#39;rustc&#39; 73.8 MiB / 73.8 MiB (100 %) 10.8 MiB/s in 7s ETA: 0s info: installing component &#39;rustfmt&#39; info: default toolchain set to &#39;stable-x86_64-unknown-linux-gnu&#39;<br/> stable-x86_64-unknown-linux-gnu installed - (rustc does not exist)<br/> Rust is installed now. Great!<br/> To get started you may need to restart your current shell. This would reload your PATH environment variable to include Cargo&#39;s bin directory ($HOME/.cargo/bin).<br/> To configure your current shell, you need to source the corresponding env file under $HOME/.cargo.<br/> This is usually done by running one of the following (note the leading DOT): . &quot;$HOME/.cargo/env&quot; # For sh/bash/zsh/ash/dash/pdksh source &quot;$HOME/.cargo/env.fish&quot; # For fish</span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region All OSes --> <h3>All OSes</h3> <p> The <code>rustup</code> installation program modifies both <code>~/.bashrc</code> and <code>~/.profile</code>, which is redundant. The following line was appended to both files. I deleted it from <code>~/.profile</code>. </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Extra line appended to ~/.profile and ~/.bashrc</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7829276d738f'><button class='copyBtn' data-clipboard-target='#id7829276d738f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>. "$HOME/.cargo/env"</pre> </div> <!-- endregion --> <p> Now I reloaded the shell: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9a20184fbd10'><button class='copyBtn' data-clipboard-target='#id9a20184fbd10' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>source ~/.bashrc</pre> </div> <!-- endregion --> <p> When <code>rustup</code> runs, it creates a directory called <code>~/.rustup/</code>: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id74e2662cbafc'><button class='copyBtn' data-clipboard-target='#id74e2662cbafc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>tree -d -L 3 ~/.rustup/ <span class='unselectable'>/home/mslinn/.rustup/ ├── downloads ├── tmp ├── toolchains │   └── stable-x86_64-unknown-linux-gnu │   ├── bin │   ├── etc │   ├── lib │   ├── libexec │   └── share └── update-hashes<br> 11 directories </span></pre> </div> <!-- endregion --> <p> <code>~/.rustup/settings.toml</code> is an important configuration file for Rust: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>~/.rustup/settings.toml</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4ade5afa2687'><button class='copyBtn' data-clipboard-target='#id4ade5afa2687' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>default_toolchain = "stable-x86_64-unknown-linux-gnu" profile = "default" version = "12"<br> [overrides]</pre> </div> <!-- endregion --> <!-- endregion --> <!-- endregion --> <!-- #region Hello, World! --> <h2 id="hw" class="clear">Hello, World!</h2> <div class='imgWrapper imgFlex right' style='width: 220px; '> <a href='/blog/2023/05/12/c.html' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/c/k&r.svg" type="image/svg"> <!---<source srcset="/blog/c/k&r.avif" type="image/avif">--> <source srcset="/blog/c/k&r.webp" type="image/webp"> <source srcset="/blog/c/k&r.apng" type="image/apng"> <source srcset="/blog/c/k&r.png" type="image/png"> <source srcset="/blog/c/k&r.jpg" type="image/jpeg"> <source srcset="/blog/c/k&r.jpeg" type="image/jpeg"> <source srcset="/blog/c/k&r.jfif" type="image/jpeg"> <source srcset="/blog/c/k&r.pjpeg" type="image/jpeg"> <source srcset="/blog/c/k&r.pjp" type="image/jpeg"> <source srcset="/blog/c/k&r.gif" type="image/gif"> <source srcset="/blog/c/k&r.tif" type="image/tiff"> <source srcset="/blog/c/k&r.tiff" type="image/tiff"> <source srcset="/blog/c/k&r.bmp" type="image/bmp"> <source srcset="/blog/c/k&r.ico" type="image/x-icon"> <source srcset="/blog/c/k&r.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/c/k&r.png" style='width: 100%; ' /> </picture> </a> </div> <p> <a href='https://en.wikipedia.org/wiki/%22Hello,_World!%22_program' target='_blank' rel="nofollow">&ldquo;Hello, world!&rdquo;</a> was the output of the first C program in the seminal book <a href='https://en.wikipedia.org/wiki/The_C_Programming_Language' target='_blank' rel="nofollow">The C Programming Language</a>, written by Brian Kernighan and Dennis Ritchie and published in 1983. This book is often called &ldquo;the K&amp;R book&rdquo;. </p> <p> Here is the program, in all its glory: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>hello_world.c</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id388fd534a8d4'><button class='copyBtn' data-clipboard-target='#id388fd534a8d4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>main() { printf("Hello, world!"); }</pre> </div> <!-- endregion --> <p> <a href='/blog/2023/05/12/c.html'>I learned C from the K&R book</a>; the language and its libraries were small and simple, so it could be mastered in only a few weeks. </p> <p> As my first Rust program, I decided to write a <code>hello_world</code> equivalent program. As you can see, the Rust version looks almost identical to the C version: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>hello_world.rs</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id200a0ff52a33'><button class='copyBtn' data-clipboard-target='#id200a0ff52a33' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>fn main() { println!("Hello, World!"); }</pre> </div> <!-- endregion --> <p> The <a href='https://doc.rust-lang.org/book' target='_blank' rel="nofollow">Rust Programming Language</a> online book is a great place to start learning Rust. </p> <p class="alert rounded shadow"> I did not need to write the above <b>Hello, World!</b> Rust program&ndash;<a href='https://doc.rust-lang.org/stable/cargo/' target='_blank' rel="nofollow">Cargo always writes this short program</a> when creating a new Rust project! This is what I actually typed to write the program: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6ef5e2e1f42b'><button class='copyBtn' data-clipboard-target='#id6ef5e2e1f42b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cargo new hello_world <span class='unselectable'><span style='color: green;'>Creating</span> binary (application) `hello_world` package <span style='color: green;'>note</span>: see more `Cargo.toml` keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html </span><br> <span class='unselectable'>$ </span>cd hello_world<br></pre> </div> <!-- endregion --> <aside> <label>Aside - Saving Some Typing</label> <p> The above commands have some redundancy; the project name, <code>hello_world</code>, is typed twice. We can use a Bash history word designator to remove the redundancy. The token <code class='bg_yellow'>!:2</code> <a href='https://www.baeldung.com/linux/bash-recall-previous-command#1-understanding-word-designators-in-bashs-history-expansion' target='_blank' rel="nofollow">expands to the second argument of the previous command</a>. Here is how we can rewrite the above commands without redundancy: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idef62f5068697'><button class='copyBtn' data-clipboard-target='#idef62f5068697' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cargo new hello_world<br> <span class='unselectable'>$ </span>cd <span class='bg_yellow'>!:2</span></pre> </div> <!-- endregion --> <p> We can combine these two lines into a one-line command list using the <a href='https://www.gnu.org/software/bash/manual/html_node/Lists.html' target='_blank' rel="nofollow"><code>&amp;&amp;</code> list operator</a>: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id26ac5e18f585'><button class='copyBtn' data-clipboard-target='#id26ac5e18f585' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cargo new hello_world && cd !:2</pre> </div> <!-- endregion --> </aside> <p> Now let&rsquo;s see what cargo generated in the new <code>hello_world</code> directory structure that it created: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8fd3df2c0890'><button class='copyBtn' data-clipboard-target='#id8fd3df2c0890' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>tree -a -L 1 <span class='unselectable'>. ├── .git ├── .gitignore ├── Cargo.toml └── src </span><br> <span class='unselectable'>$ </span>cat Cargo.toml <span class='unselectable'>[package] name = "hello_world" version = "0.1.0" edition = "2021"<br> [dependencies] </span></pre> </div> <!-- endregion --> <p> Every Rust project generated by Cargo starts as the <b>Hello, World!</b> program. </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbabd20920d1f'><button class='copyBtn' data-clipboard-target='#idbabd20920d1f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cat src/main.rs <span class='unselectable'>fn main() { println!("Hello, world!"); } </span></pre> </div> <!-- endregion --> <p> Now let&rsquo;s run the program: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id02879a86ad5c'><button class='copyBtn' data-clipboard-target='#id02879a86ad5c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cargo run <span class='unselectable'>Compiling hello_world v0.1.0 (/mnt/c/work/rust/hello_world) Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.64s Running `target/debug/hello_world` Hello, world! </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Visual Studio Code --> <h2 id="vscode">Visual Studio Code</h2> <p> IDEs had not been invented yet when I first learned C. I am very happy that they were, and today my IDE of choice is the F/OSS <a href='/blog/2021/03/22/vscode-notes.html'>Visual Studio Code</a> from Microsoft. JetBrains also makes excellent IDEs, and their Rust IDE, <a href='https://www.jetbrains.com/rust/' target='_blank' rel="nofollow">RustRover</a>, is free for non-commercial use. I find that JetBrains IDEs generally offer more capable refactoring than Visual Studio Code; however, JetBrains IDEs do not handle <a href='/blog/2018/08/20/wsl-revisited.html'>WSL</a> as well as Visual Studio Code does. </p> <div class='imgWrapper imgFlex right' style='width: 110px; '> <a href='https://marketplace.visualstudio.com/items?itemName=Primer4128.rust-analyzer-fork' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/rust/rust_analyzer.svg" type="image/svg"> <!---<source srcset="/blog/images/rust/rust_analyzer.avif" type="image/avif">--> <source srcset="/blog/images/rust/rust_analyzer.webp" type="image/webp"> <source srcset="/blog/images/rust/rust_analyzer.apng" type="image/apng"> <source srcset="/blog/images/rust/rust_analyzer.png" type="image/png"> <source srcset="/blog/images/rust/rust_analyzer.jpg" type="image/jpeg"> <source srcset="/blog/images/rust/rust_analyzer.jpeg" type="image/jpeg"> <source srcset="/blog/images/rust/rust_analyzer.jfif" type="image/jpeg"> <source srcset="/blog/images/rust/rust_analyzer.pjpeg" type="image/jpeg"> <source srcset="/blog/images/rust/rust_analyzer.pjp" type="image/jpeg"> <source srcset="/blog/images/rust/rust_analyzer.gif" type="image/gif"> <source srcset="/blog/images/rust/rust_analyzer.tif" type="image/tiff"> <source srcset="/blog/images/rust/rust_analyzer.tiff" type="image/tiff"> <source srcset="/blog/images/rust/rust_analyzer.bmp" type="image/bmp"> <source srcset="/blog/images/rust/rust_analyzer.ico" type="image/x-icon"> <source srcset="/blog/images/rust/rust_analyzer.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/rust/rust_analyzer.png" style='width: 100%; ' /> </picture> </a> </div> <p> The Visual Studio Code documentation has a section entitled <a href='https://code.visualstudio.com/docs/languages/rust' target='_blank' rel="nofollow">Rust in Visual Studio Code</a>. It discusses the <a href='https://marketplace.visualstudio.com/items?itemName=Primer4128.rust-analyzer-fork' target='_blank' rel="nofollow">Rust Analyzer plugin</a>. <a href='https://rust-analyzer.github.io/manual.html#vs-code-2' target='_blank' rel="nofollow">Read the fine manual.</a> <a href='https://rust-analyzer.github.io/' target='_blank' rel="nofollow">More documentation.</a> </p> <p> There is not much to say about Visual Studio Code and the Rust Analyzer plugin; they just work as they should. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/rust/vscode_rust.svg" type="image/svg"> <!---<source srcset="/blog/images/rust/vscode_rust.avif" type="image/avif">--> <source srcset="/blog/images/rust/vscode_rust.webp" type="image/webp"> <source srcset="/blog/images/rust/vscode_rust.apng" type="image/apng"> <source srcset="/blog/images/rust/vscode_rust.png" type="image/png"> <source srcset="/blog/images/rust/vscode_rust.jpg" type="image/jpeg"> <source srcset="/blog/images/rust/vscode_rust.jpeg" type="image/jpeg"> <source srcset="/blog/images/rust/vscode_rust.jfif" type="image/jpeg"> <source srcset="/blog/images/rust/vscode_rust.pjpeg" type="image/jpeg"> <source srcset="/blog/images/rust/vscode_rust.pjp" type="image/jpeg"> <source srcset="/blog/images/rust/vscode_rust.gif" type="image/gif"> <source srcset="/blog/images/rust/vscode_rust.tif" type="image/tiff"> <source srcset="/blog/images/rust/vscode_rust.tiff" type="image/tiff"> <source srcset="/blog/images/rust/vscode_rust.bmp" type="image/bmp"> <source srcset="/blog/images/rust/vscode_rust.ico" type="image/x-icon"> <source srcset="/blog/images/rust/vscode_rust.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/rust/vscode_rust.png" style='width: 100%; ' /> </picture> </div> <!-- endregion --><!-- #region Videos --> <h2 id="videos">Videos</h2> <style>.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }</style><div class='embed-container'> <iframe title="YouTube video player" width="640" height="390" src="//www.youtube.com/embed/u0VotuGzD_w" frameborder="0" allowfullscreen></iframe></div> <style>.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }</style><div class='embed-container'> <iframe title="YouTube video player" width="640" height="390" src="//www.youtube.com/embed/CJtvnepMVAU" frameborder="0" allowfullscreen></iframe></div> <!-- endregion --> Product Development Collaboration 2024-02-18T00:00:00-05:00 https://mslinn.github.io/blog/2024/02/18/collaboration <!-- #region intro --> <p> For product development, the goal of effective collaboration is to fully engage all parties. Good product-oriented collaborators maintain a flow of information, ideas and commensurate action. Hesitation is minimized. Feedback is rapid. Reactions are prompt, yet measured through constant practice. </p> <!-- endregion --> <!-- #region Product Collaborator Checklist --> <h2 id="check">Product Collaborator Checklist</h2> <p> The following criteria are a guide to effective collaboration candidates for product development. </p> <!-- #region Desirable Traits --> <h3 id="destr">Desirable Traits</h3> <ul> <li> Strong interest in optimizing the architecture and engineering processes. </li> <li> A focus on the contributing to the desired results for the team, which are: reliably providing an easily understood and valuable benefit to a well-defined customer base. </li> <li> Manage the difference between desired outcomes and actual outcomes. </li> <li> Strive for process and product consistency. </li> <li> Willing to expend the overhead of time and energy required for effective communication: <ul> <li>Driving their activities from a Kanban-style status board</li> <li>Planning interactions with others, minimizing impulsive chatter</li> <li>Preparing and distributing notes before meetings</li> <li>Describing alternatives for decisions</li> <li>Documenting the rationale for design decisions</li> <li>Summarizing discussions in project documentation</li> <li>Contributing to an up-to-date product roadmap</li> </ul> </li> </ul> <!-- endregion --> <!-- #region Undesirable Traits --> <h3 id="undestr">Undesirable Traits</h3> <p> People with the following traits disqualify themselves from being a member of the core team because they are unable to consistently contribute effectively to the ever-evolving vision for the project. </p> <ul> <li> "Guns for hire" just want to get paid for time spent; they have no vested interest in the long-term outcome. They are tools to be exploited, not valuable resources to be cultivated. The best use of a mercenary is to give them orders, supervise them, learn from them, and then dispose of them as soon as possible. </li> <li> "Putting in time" demonstrates a lack of personal engagement. Those people require constant supervision, and the cost of the supervision must be subtracted from any value they might provide when computing their net contribution. </li> </ul> <!-- endregion --> <!-- #region Software Architecture and Engineering Processes --> <h2 id="defs">Software Architecture and Engineering Processes</h2> <p> ChatGPT gave me the following response when I asked, &ldquo;Compare and contrast the processes of architecture and engineering for software&rdquo;. I like the quality of this response, and provide it verbatim because it exactly matches my understanding of those terms. </p> <p> When it comes to software development, there are parallels between the processes of architecture and engineering, but there are also significant differences due to the intangible and rapidly evolving nature of software. Let's compare and contrast the processes of software architecture and engineering: </p> <h3 id="pf">Purpose and Focus</h3> <p> Software Architecture: Similar to architectural design in physical structures, software architecture focuses on designing the overall structure and organization of a software system. It involves making high-level decisions about the system's components, their interactions, and the principles that guide their design. </p> <p> Software Engineering: Software engineering encompasses the systematic approach to developing software solutions. It involves various disciplines such as requirements analysis, design, coding, testing, deployment, and maintenance. The focus is on creating reliable, efficient, and maintainable software systems. </p> <h3 id="dp">Design Process</h3> <p> Software Architecture: The architectural design process involves defining the system's key components, interfaces, and their interactions. Architectural patterns, such as layered architecture, microservices, or event-driven architecture, are often used to guide the design. Considerations include scalability, performance, security, and maintainability. </p> <p> Software Engineering: Software engineering encompasses the entire software development lifecycle (SDLC). It begins with requirements gathering and analysis, followed by design, implementation (coding), testing, deployment, and maintenance. Various methodologies like Waterfall, Agile, and DevOps guide this process. </p> <h3 id="ct"> Creativity vs. Technicality</h3> <p> Software Architecture: Software architects must balance creative design decisions with technical constraints. They need to envision the overall structure of the system and choose appropriate technologies and patterns to achieve the desired functionality and quality attributes. </p> <p> Software Engineering: Software engineers focus on implementing the design created by architects. While creativity is still important, engineers primarily deal with the technical aspects of coding, testing, and debugging. They translate the architectural design into working software components. </p> <h3 id="coll">Collaboration</h3> <p> Software Architecture: Collaboration between architects, stakeholders, and development teams is crucial to ensure that architectural decisions align with business goals and technical requirements. Architects often communicate their designs through diagrams, documentation, and discussions. </p> <p> Software Engineering: Software engineers collaborate closely with other team members, including architects, designers, testers, and project managers. They work together to implement features, resolve technical challenges, and deliver software increments according to project timelines. </p> <h3>Regulatory and Environmental Considerations</h3> <p> Software Architecture: Architectural decisions may need to comply with regulatory standards, industry best practices, and security guidelines. Architects must consider factors like data privacy, compliance requirements (such as GDPR or HIPAA), and interoperability with other systems. </p> <p> Software Engineering: Engineers implement features and functionalities while adhering to coding standards, security protocols, and testing practices. They ensure that the software meets quality standards, performance targets, and regulatory requirements specified by the architecture and project stakeholders. </p> <p> In summary, while both software architecture and engineering involve designing and building software systems, they have distinct focuses and processes. Architecture concentrates on high-level design decisions and system structure, while engineering encompasses the entire software development lifecycle, from requirements to deployment. Collaboration and communication are essential in both disciplines to deliver successful software solutions.</p> </p> <!-- endregion --> <!-- #region Collaboration --> <h2 id="collabl">Collaboration</h2> <p> ChatGPT defined collaboration as the following. Again, I very much like the response, so I provide it verbatim. </p> <p class="quote"> Collaboration refers to the process of individuals or groups working together to achieve a common goal or objective. It involves the sharing of ideas, resources, responsibilities, and tasks in a coordinated manner to accomplish tasks more effectively and efficiently than if done individually. <br><br> Collaboration often emphasizes communication, cooperation, and mutual respect among participants. It can occur within various contexts, including businesses, organizations, teams, communities, and even across different disciplines or sectors. <br><br> Successful collaboration typically results in synergy, where the combined efforts of the participants produce outcomes that exceed what could be achieved by any individual or subgroup working alone. </p> <p> Following is the response when I asked "what is required for effective collaboration in a new software project?" I <span class="bg_yellow">highlighted</span> a passage that is important, and <s>struck out</s> portions that I disagree with. </p> <p> Effective collaboration in a new software project requires several key elements: </p> <ol> <li> <b>Clear Goals and Objectives</b>: Establishing clear, well-defined goals and objectives for the project ensures that all collaborators understand what they are working towards and why. </li> <li> <b>Communication Channels</b>: Implementing reliable communication channels is essential for keeping all team members informed and facilitating the exchange of ideas, updates, and feedback. This can include tools like instant messaging, email, project management software, or regular team meetings. </li> <li> <b>Defined Roles and Responsibilities</b>: Clearly defining roles and responsibilities for each team member helps avoid confusion and ensures that tasks are completed efficiently. This includes designating a project manager, developers, testers, designers, and other relevant roles. </li> <li> <b>Collaboration Tools</b>: Utilizing collaboration tools such as version control systems (e.g., Git), issue tracking systems <s>(e.g., Jira)</s>, document sharing platforms (e.g., Google Drive), and collaborative coding environments (e.g., GitHub) can streamline the development process and enhance teamwork. </li> <li class="bg_yellow"> <b>Regular Feedback and Reviews</b>: Encouraging open and constructive feedback throughout the project lifecycle allows team members to address issues promptly, make necessary adjustments, and continuously improve the software. </li> <li class="bg_yellow"> <b>Flexibility and Adaptability</b>: Remaining flexible and adaptable to changes in requirements, timelines, and priorities is crucial for navigating unforeseen challenges and ensuring the project stays on track. </li> <li> <b>Respect and Trust</b>: Cultivating an environment of respect and trust among team members fosters collaboration and encourages open communication, creativity, and innovation. </li> <li class="bg_yellow"> <b>Documentation</b>: Maintaining thorough documentation of project requirements, design decisions, codebase, and processes helps ensure continuity and facilitates knowledge transfer among team members. </li> <li> <b>Regular Progress Tracking</b>: Implementing mechanisms to track progress, <s>such as sprint planning, daily stand-up meetings, and</s> regular status reports, enables the team to monitor project milestones, identify potential bottlenecks, and make necessary adjustments to stay on schedule. </li> <li> <b>Quality Assurance and Testing</b>: Integrating quality assurance processes and testing procedures into the development workflow ensures that the software meets the specified requirements, functions as intended, and delivers value to end-users. </li> </ol> <p> By incorporating these elements into the collaborative process, teams can increase the likelihood of success in their new software projects. </p> <!-- endregion --> <!-- #region Design Decisions --> <h2 id="dd">Design Decisions</h2> <p> I asked ChatGPT "how would someone document design decisions in a software project?" The highlighted portion is covered in more detail later in this article. </p> <p> Documenting design decisions in a software project is essential for ensuring clarity, facilitating collaboration, and providing a reference for future development. Here's a structured approach to documenting design decisions: </p> <ol> <li> <b>Establish a Design Document</b>: Create a designated document or section within project documentation specifically for recording design decisions. This could be a separate document, a section in the project wiki, or part of the version control repository. </li> <li> <b>Provide Context</b>: Start by providing context for the decision. Explain the problem or requirement that the design decision aims to address. Include any relevant background information, constraints, or considerations that influenced the decision-making process. </li> <li class="bg_yellow"> <b>Describe Alternatives</b>: Outline the various alternatives considered during the decision-making process. This could include different approaches, technologies, architectures, or design patterns. Briefly discuss the pros and cons of each alternative. </li> <li> <b>Justify the Chosen Solution</b>: Explain why the chosen solution was selected over the alternatives. Highlight the reasons, benefits, and trade-offs associated with the chosen approach. Consider factors such as performance, scalability, maintainability, ease of implementation, and alignment with project goals. </li> <li> <b>Detail the Design</b>: Provide a detailed description of the chosen design solution. Break down the design into its constituent components, modules, or layers. Use diagrams, flowcharts, UML diagrams, or other visual aids to illustrate the architecture, structure, and relationships between components. </li> <li> <b>Document Dependencies and Interactions</b>: Document dependencies between different components or modules within the design. Describe how these components interact with each other and with external systems or services. Identify any interfaces, APIs, or protocols used for communication. </li> <li> <b>Address Risks and Mitigations</b>: Identify potential risks or challenges associated with the chosen design. Discuss strategies for mitigating these risks and ensuring the robustness and resilience of the design solution. </li> <li> <b>Include Implementation Details</b>: If applicable, include implementation details or guidelines for developers to follow when implementing the design. This could include coding standards, best practices, design patterns, or frameworks to be used. </li> <li> <b>Update and Maintain</b>: Keep the design document up to date as the project evolves. Update it to reflect any changes, refinements, or new decisions made during the development process. Ensure that the document remains a reliable reference for developers and stakeholders. </li> <li> <b>Review and Iterate</b>: Encourage team members to review the design document periodically and provide feedback. Iterate on the design as necessary based on feedback, changes in requirements, or lessons learned from implementation. </li> </ol> <p> By following these steps, you can effectively document design decisions in a software project, providing a comprehensive and structured reference for developers, stakeholders, and future contributors. </p> <!-- endregion --> <!-- #region Comparative Product Matrix --> <h2 id="mat">Comparative Product Matrix</h2> <p> Comparative product matrices are useful for guiding and documenting design decisions. Search for them when you need to identify pre-existing solutions to a problem, and generate them when considering the implementation of various approaches. </p> <div class='imgWrapper imgFlex inline' style=' '> <figure> <a href='https://github.com/snowfort-ai/awesome-llm-webapps?tab=readme-ov-file#project-table' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/comprative_product_matrix.svg" type="image/svg"> <!---<source srcset="/blog/images/comprative_product_matrix.avif" type="image/avif">--> <source srcset="/blog/images/comprative_product_matrix.webp" type="image/webp"> <source srcset="/blog/images/comprative_product_matrix.apng" type="image/apng"> <source srcset="/blog/images/comprative_product_matrix.png" type="image/png"> <source srcset="/blog/images/comprative_product_matrix.jpg" type="image/jpeg"> <source srcset="/blog/images/comprative_product_matrix.jpeg" type="image/jpeg"> <source srcset="/blog/images/comprative_product_matrix.jfif" type="image/jpeg"> <source srcset="/blog/images/comprative_product_matrix.pjpeg" type="image/jpeg"> <source srcset="/blog/images/comprative_product_matrix.pjp" type="image/jpeg"> <source srcset="/blog/images/comprative_product_matrix.gif" type="image/gif"> <source srcset="/blog/images/comprative_product_matrix.tif" type="image/tiff"> <source srcset="/blog/images/comprative_product_matrix.tiff" type="image/tiff"> <source srcset="/blog/images/comprative_product_matrix.bmp" type="image/bmp"> <source srcset="/blog/images/comprative_product_matrix.ico" type="image/x-icon"> <source srcset="/blog/images/comprative_product_matrix.cur" type="image/x-icon"> <img alt='Comparative product matrix' class="imgImg " src="/blog/images/comprative_product_matrix.png" style='width: 100%; ' title='Comparative product matrix' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://github.com/snowfort-ai/awesome-llm-webapps?tab=readme-ov-file#project-table" target='_blank' > Comparative product matrix </a> </figcaption> </figure> </div> <p> Comparative product matrices take time to read and to prepare, but they help guide the team to the best technical decisions. Without them, you are just hoping to get lucky. </p><!-- endregion --> Expanding the WSL virtual hard drive 2024-02-12T00:00:00-05:00 https://mslinn.github.io/blog/2024/02/12/wsl-expand <!-- #region intro --> <p> I ran out of room on my WSL2 virtual hard drive. <a href='https://learn.microsoft.com/en-us/windows/wsl/disk-space#how-to-expand-the-size-of-your-wsl-2-virtual-hard-disk' target='_blank' rel="nofollow">Microsoft&rsquo;s instructions</a> worked perfectly to increase the virtual drive size. </p> <!-- endregion --> <!-- #region Shut Down WSL --> <h2 id="stop">Shut Down WSL</h2> <p> The following command work from WSL bash, and cmd, and PowerShell: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5609180e6ac7'><button class='copyBtn' data-clipboard-target='#id5609180e6ac7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>wsl.exe --shutdown</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Expand VHD With Diskpart --> <h2 id="Diskpart">Expand VHD With Diskpart</h2> <p style="text-align: left;"> The virtual disk was in <code>%LocalAppData%\<wbr>Packages\<wbr>CanonicalGroupLimited.<wbr>UbuntuonWindows_<wbr>79rhkp1fndgsc\<wbr>LocalState\<wbr>ext4.vhdx</code> </p> <p> Open a <code>cmd.exe</code> window with admininstrator privileges and run <code>diskpart</code>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>diskpart</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd3382847fefb'><button class='copyBtn' data-clipboard-target='#idd3382847fefb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>Select vdisk file="%LocalAppData%\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState\ext4.vhdx"<br> <span class='unselectable'>DISKPART> </span>detail vdisk <span class='unselectable'>Device type ID: 0 (Unknown) Vendor ID: {00000000-0000-0000-0000-000000000000} (Unknown) State: Added Virtual size: 256 GB Physical size: 247 GB Filename: C:\Users\Mike Slinn\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState\ext4.vhdx Is Child: No Parent Filename: Associated disk#: Not found. </span></pre> </div> <!-- endregion --> <p> The virtual disk was 256 GB (256,000 MB). I want it to be twice as big: 512,000 MB. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>diskpart (continued)</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8ad6b295205e'><button class='copyBtn' data-clipboard-target='#id8ad6b295205e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>DISKPART> </span>expand vdisk maximum=512000<br> <span class='unselectable'>100 percent completed<br> DiskPart successfully expanded the virtual disk file. </span> <span class='unselectable'>DISKPART> </span>exit<br> <span class='unselectable'>Leaving DiskPart... </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Restart WSL and resize2fs --> <h2 id="restart">Restart WSL and Run <span class="code">resize2fs</span></h2> <p> Now I restarted WSL from a <code>cmd</code> window. The following command work from WSL bash, and cmd, and PowerShell: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>cmd.exe</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id86b59591b893'><button class='copyBtn' data-clipboard-target='#id86b59591b893' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>wsl.exe</pre> </div> <!-- endregion --> <p> Now tell Linux to use all available space: </p> <!-- #region --> <div class="jekyll_pre" style='margin-bottom: 1em;'> <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0083c9c3a8d3'><button class='copyBtn' data-clipboard-target='#id0083c9c3a8d3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo resize2fs /dev/sdc 512000M <span class='unselectable'>resize2fs 1.47.0 (5-Feb-2023) Filesystem at /dev/sdc is mounted on /; on-line resizing required old_desc_blocks = 32, new_desc_blocks = 63 The filesystem on /dev/sdc is now 131072000 (4k) blocks long. </span></pre> </div> <!-- endregion --> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <p> Done! </p> <!-- endregion --> Installing JDK 17 on Ubuntu 2023-09-28T00:00:00-04:00 https://mslinn.github.io/blog/2023/09/28/jdk <!-- #region install --> <p> The Java Development Kit (JDK) version 17 is compatible with v11 and v8. Installing the JDK also installs the JRE, as you can see from the highlighted dependency shown below: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6aa79ab9c2a4'><button class='copyBtn' data-clipboard-target='#id6aa79ab9c2a4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>apt show -a openjdk-17-jdk <span class='unselectable'>Package: openjdk-17-jdk Version: 17.0.8.1+1~us1-0ubuntu1~23.04 Priority: optional Section: java Source: openjdk-17 Origin: Ubuntu Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com> Original-Maintainer: OpenJDK Team <openjdk-17@packages.debian.org> Bugs: https://bugs.launchpad.net/ubuntu/+filebug Installed-Size: 1544 kB Provides: java-compiler, java-sdk (= 17), java10-sdk, java11-sdk, java12-sdk, java13-sdk, java14-sdk, java15-sdk, java16-sdk, java17-sdk, java2-sdk, java5-sdk, java6-sdk, java7-sdk, java8-sdk, java9-sdk Depends: <span class="bg_yellow">openjdk-17-jre (= 17.0.8.1+1~us1-0ubuntu1~23.04)</span>, openjdk-17-jdk-headless (= 17.0.8.1+1~us1-0ubuntu1~23.04), libc6 (>= 2.34), zlib1g (>= 1:1.1.4) Recommends: libxt-dev Suggests: openjdk-17-demo, openjdk-17-source, visualvm Homepage: https://openjdk.java.net/ Download-Size: 1486 kB APT-Sources: http://archive.ubuntu.com/ubuntu lunar-updates/main amd64 Packages Description: OpenJDK Development Kit (JDK) OpenJDK is a development environment for building applications, applets, and components using the Java programming language.<br> Package: openjdk-17-jdk Version: 17.0.6+10-1ubuntu2 Priority: optional Section: java Source: openjdk-17 Origin: Ubuntu Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com> Original-Maintainer: OpenJDK Team <openjdk-17@packages.debian.org> Bugs: https://bugs.launchpad.net/ubuntu/+filebug Installed-Size: 4732 kB Provides: java-compiler, java-sdk, java10-sdk, java11-sdk, java12-sdk, java13-sdk, java14-sdk, java15-sdk, java16-sdk, java17-sdk, java2-sdk, java5-sdk, java6-sdk, java7-sdk, java8-sdk, java9-sdk Depends: openjdk-17-jre (= 17.0.6+10-1ubuntu2), openjdk-17-jdk-headless (= 17.0.6+10-1ubuntu2), libc6 (>= 2.34) Recommends: libxt-dev Suggests: openjdk-17-demo, openjdk-17-source, visualvm Homepage: https://openjdk.java.net/ Download-Size: 4585 kB APT-Sources: http://archive.ubuntu.com/ubuntu lunar/main amd64 Packages Description: OpenJDK Development Kit (JDK) OpenJDK is a development environment for building applications, applets, and components using the Java programming language. </span></pre> </div> <!-- endregion --> <p> Install OpenJDK 17 and its dependencies: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id356f7ef15a33'><button class='copyBtn' data-clipboard-target='#id356f7ef15a33' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install openjdk-17-jdk</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region set default java --> <h2 id="def">Setting the Default Java Version</h2> <p> The man page for <code>update-java-alternatives</code> is: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id69857af40d0e'><button class='copyBtn' data-clipboard-target='#id69857af40d0e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>man update-java-alternatives <span class='unselectable'>UPDATE-JAVA-ALTERNATIVESystem Manager&#39;s MUPDATE-JAVA-ALTERNATIVES(8)<br/> NAME update-java-alternatives - update alternatives for jre/sdk installations<br/> SYNOPSIS update-java-alternatives [--jre] [--plugin] [-v|--verbose] -l|--list [&lt;jname&gt;] -s|--set &lt;jname&gt; -a|--auto -h|-?|--help<br/> DESCRIPTION update-java-alternatives updates all alternatives belonging to one runtime or development kit for the Java language. A package does provide these information of it&#39;s alternatives in /usr/lib/jvm/.&lt;jname&gt;.jinfo.<br/> OPTIONS -l|--list [&lt;jname&gt;] List all installed packages (or just &lt;jname&gt;) provid&#8208; ing information to set a bunch of java alternatives. Verbose output shows each alternative provided by the packages.<br/> -a|--auto Switch all alternatives of registered jre/sdk instal&#8208; lations to automatic mode.<br/> -s|--set &lt;jname&gt; Set all alternatives of the registered jre/sdk instal&#8208; lation to the program path provided by the &lt;jname&gt; in&#8208; stallation.<br/> --jre Limit the actions to alternatives belong to a runtime environment, not a development kit.<br/> --jre-headless Limit the actions to alternatives belong to the head&#8208; less part of a runtime environment.<br/> --plugin Limit the actions to alternatives providing browser plugins.<br/> -h|--help Display a help message.<br/> -v|--verbose Verbose output.<br/> FILES /usr/lib/jvm/.*.jinfo A text file describing a jre/sdk installation. Con&#8208; sists of some variables of the form &lt;var&gt;=&lt;value&gt; and a list of alternatives of the form jre|jdk &lt;name&gt; &lt;path&gt;.<br/> AUTHOR update-java-alternatives and this manual page was written by Matthias Klose &lt;doko@ubuntu.com&gt;.<br/> May 2006 UPDATE-JAVA-ALTERNATIVES(8) </span></pre> </div> <!-- endregion --> <p> The above man page neglects to say that a package version&rsquo;s priority is set from its version number. Thus a newer version would normally have a higher priority. To get a list of the installed Java versions, type: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8f63082e3cb7'><button class='copyBtn' data-clipboard-target='#id8f63082e3cb7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo update-java-alternatives --list <span class='unselectable'>java-1.11.0-openjdk-amd64 1111 /usr/lib/jvm/java-1.11.0-openjdk-amd64 java-1.17.0-openjdk-amd64 1711 /usr/lib/jvm/java-1.17.0-openjdk-amd64 </span></pre> </div> <!-- endregion --> <p> As you can see, my machine had Java 11 and 17 installed. To set the newest version to be the default, type: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6bb0edbd463f'><button class='copyBtn' data-clipboard-target='#id6bb0edbd463f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo update-java-alternatives -a <span class='unselectable'>$ </span>java -version <span class='unselectable'>openjdk version "17.0.8.1" 2023-08-24 OpenJDK Runtime Environment (build 17.0.8.1+1-Ubuntu-0ubuntu123.04) OpenJDK 64-Bit Server VM (build 17.0.8.1+1-Ubuntu-0ubuntu123.04, mixed mode, sharing) </span></pre> </div> <!-- endregion --> <!-- endregion --> C++ Boost library 2023-09-14T00:00:00-04:00 https://mslinn.github.io/blog/2023/09/14/boost <!-- #region intro --> <h2 id="about">About Boost</h2> <p> <a href='www.boost.org'>Boost</a> is a general-purpose open source library of utility functions for C++. The <a href='https://www.boost.org/doc/libs/?view=condensed' target='_blank' rel="nofollow">list of categories of fuctionality</a> is formidable. </p> <div class='imgWrapper imgFlex inline' style=' '> <a href='https://www.boost.org/doc/libs/?view=condensed' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/c/boostCategories.svg" type="image/svg"> <!---<source srcset="/blog/c/boostCategories.avif" type="image/avif">--> <source srcset="/blog/c/boostCategories.webp" type="image/webp"> <source srcset="/blog/c/boostCategories.apng" type="image/apng"> <source srcset="/blog/c/boostCategories.png" type="image/png"> <source srcset="/blog/c/boostCategories.jpg" type="image/jpeg"> <source srcset="/blog/c/boostCategories.jpeg" type="image/jpeg"> <source srcset="/blog/c/boostCategories.jfif" type="image/jpeg"> <source srcset="/blog/c/boostCategories.pjpeg" type="image/jpeg"> <source srcset="/blog/c/boostCategories.pjp" type="image/jpeg"> <source srcset="/blog/c/boostCategories.gif" type="image/gif"> <source srcset="/blog/c/boostCategories.tif" type="image/tiff"> <source srcset="/blog/c/boostCategories.tiff" type="image/tiff"> <source srcset="/blog/c/boostCategories.bmp" type="image/bmp"> <source srcset="/blog/c/boostCategories.ico" type="image/x-icon"> <source srcset="/blog/c/boostCategories.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/c/boostCategories.png" style='width: 100%; ' /> </picture> </a> </div> <p> Boost is used in many <a href='https://www.boost.org/users/uses_shrink.html' target='_blank' rel="nofollow">commercial products</a>, <a href='https://www.boost.org/users/uses_inhouse.html' target='_blank' rel="nofollow">in-house projects</a>, and <a href='https://www.boost.org/users/uses_open.html' target='_blank' rel="nofollow">open-source projects</a>. </p> <p> Ostensibly a C++ library for wide usage, it has become an techical underpinning for Python; that is why Boost documentation also contains information about Python. </p> <h2 id="start">Getting Started</h2> <p> I started by reading the online <a href='https://www.boost.org/doc/libs/1_83_0/more/getting_started/index.html' target='_blank' rel="nofollow">Getting Started</a> guide. </p> <div class='imgWrapper imgFlex inline' style=' '> <a href='https://www.boost.org/doc/libs/1_83_0/more/getting_started/index.html' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/c/boost_getting_started.svg" type="image/svg"> <!---<source srcset="/blog/c/boost_getting_started.avif" type="image/avif">--> <source srcset="/blog/c/boost_getting_started.webp" type="image/webp"> <source srcset="/blog/c/boost_getting_started.apng" type="image/apng"> <source srcset="/blog/c/boost_getting_started.png" type="image/png"> <source srcset="/blog/c/boost_getting_started.jpg" type="image/jpeg"> <source srcset="/blog/c/boost_getting_started.jpeg" type="image/jpeg"> <source srcset="/blog/c/boost_getting_started.jfif" type="image/jpeg"> <source srcset="/blog/c/boost_getting_started.pjpeg" type="image/jpeg"> <source srcset="/blog/c/boost_getting_started.pjp" type="image/jpeg"> <source srcset="/blog/c/boost_getting_started.gif" type="image/gif"> <source srcset="/blog/c/boost_getting_started.tif" type="image/tiff"> <source srcset="/blog/c/boost_getting_started.tiff" type="image/tiff"> <source srcset="/blog/c/boost_getting_started.bmp" type="image/bmp"> <source srcset="/blog/c/boost_getting_started.ico" type="image/x-icon"> <source srcset="/blog/c/boost_getting_started.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/c/boost_getting_started.png" style='width: 100%; ' /> </picture> </a> </div> <p> For most Boost functionality, only headers are required, and libraries need not be built. The exceptions are described <a href='https://www.boost.org/doc/libs/1_83_0/more/getting_started/unix-variants.html#header-only-libraries' target='_blank' rel="nofollow">here</a>. </p> <div class='imgWrapper imgFlex inline' style=' '> <a href='https://www.boost.org/doc/libs/1_83_0/more/getting_started/unix-variants.html#header-only-libraries' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/c/boost_header_only.svg" type="image/svg"> <!---<source srcset="/blog/c/boost_header_only.avif" type="image/avif">--> <source srcset="/blog/c/boost_header_only.webp" type="image/webp"> <source srcset="/blog/c/boost_header_only.apng" type="image/apng"> <source srcset="/blog/c/boost_header_only.png" type="image/png"> <source srcset="/blog/c/boost_header_only.jpg" type="image/jpeg"> <source srcset="/blog/c/boost_header_only.jpeg" type="image/jpeg"> <source srcset="/blog/c/boost_header_only.jfif" type="image/jpeg"> <source srcset="/blog/c/boost_header_only.pjpeg" type="image/jpeg"> <source srcset="/blog/c/boost_header_only.pjp" type="image/jpeg"> <source srcset="/blog/c/boost_header_only.gif" type="image/gif"> <source srcset="/blog/c/boost_header_only.tif" type="image/tiff"> <source srcset="/blog/c/boost_header_only.tiff" type="image/tiff"> <source srcset="/blog/c/boost_header_only.bmp" type="image/bmp"> <source srcset="/blog/c/boost_header_only.ico" type="image/x-icon"> <source srcset="/blog/c/boost_header_only.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/c/boost_header_only.png" style='width: 100%; ' /> </picture> </a> </div> <p> <a href='https://theboostcpplibraries.com/' target='_blank' rel="nofollow">The Boost C++ Libraries</a> is a good free online book, and you can order non-free printed copies. </p> <div class='imgWrapper imgFlex inline' style=' '> <a href='https://theBoostCppLibraries.com' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/c/the_boost_cpp_libraries.svg" type="image/svg"> <!---<source srcset="/blog/c/the_boost_cpp_libraries.avif" type="image/avif">--> <source srcset="/blog/c/the_boost_cpp_libraries.webp" type="image/webp"> <source srcset="/blog/c/the_boost_cpp_libraries.apng" type="image/apng"> <source srcset="/blog/c/the_boost_cpp_libraries.png" type="image/png"> <source srcset="/blog/c/the_boost_cpp_libraries.jpg" type="image/jpeg"> <source srcset="/blog/c/the_boost_cpp_libraries.jpeg" type="image/jpeg"> <source srcset="/blog/c/the_boost_cpp_libraries.jfif" type="image/jpeg"> <source srcset="/blog/c/the_boost_cpp_libraries.pjpeg" type="image/jpeg"> <source srcset="/blog/c/the_boost_cpp_libraries.pjp" type="image/jpeg"> <source srcset="/blog/c/the_boost_cpp_libraries.gif" type="image/gif"> <source srcset="/blog/c/the_boost_cpp_libraries.tif" type="image/tiff"> <source srcset="/blog/c/the_boost_cpp_libraries.tiff" type="image/tiff"> <source srcset="/blog/c/the_boost_cpp_libraries.bmp" type="image/bmp"> <source srcset="/blog/c/the_boost_cpp_libraries.ico" type="image/x-icon"> <source srcset="/blog/c/the_boost_cpp_libraries.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/c/the_boost_cpp_libraries.png" style='width: 100%; ' /> </picture> </a> </div> <!-- endregion --> <!-- #region --> <h2 id="install">Ubuntu Installation</h2> <h3 id="boost_ver">Discover the Latest Boost Version Number</h3> <p> The latest version of Boost is shown on <a href='https://https://www.boost.org/users/history/?ref=learnubuntu.com' target='_blank' rel="nofollow"><code>https://www.boost.org/users/history/?ref=learnubuntu.com</code></a>: </p> <p> Let&rsquo;s scrape the lastest version number from the above web page: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4da4d4ee45eb'><button class='copyBtn' data-clipboard-target='#id4da4d4ee45eb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install libxml2-utils <span class='unselectable'>$ </span>VER=$( wget -q -O - https://www.boost.org/users/history/?ref=learnubuntu.com | \ xmllint --html --xpath '//*[@id="intro"]/div/h2[1]/a[2]' - 2>/dev/null | \ grep -oEi 'Version ([0-9].)*' | \ cut -d' ' -f 2 ) <span class='unselectable'>$ </span>echo $VER <span class='unselectable'>1.83 </span></pre> </div> <!-- endregion --> <h3 id="dl">Download Source &amp; Dependencies</h3> <p> Download and unpack the library source to a new directory within your home directory as follows: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idda0e303ab9dc'><button class='copyBtn' data-clipboard-target='#idda0e303ab9dc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cd <span class='unselectable'>$ </span>V_R=`echo $VER | tr . _` <span class='unselectable'>$ </span>echo $V_R <span class='unselectable'>1_83 </span> <span class='unselectable'>$ </span>NAME=boost_${V_R}_0 <span class='unselectable'>$ </span>echo $NAME <span class='unselectable'>boost_1_83_0 </span> <span class='unselectable'>$ </span>wget -O $NAME.tar.gz \ https://sourceforge.net/projects/boost/files/boost/$VER.0/$NAME.tar.gz/download <span class='unselectable'>$ </span>tar xzvf $NAME.tar.gz <span class='unselectable'>$ </span>cd $NAME/</pre> </div> <!-- endregion --> <p> You should now have the following files and directories: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id39222b1905f4'><button class='copyBtn' data-clipboard-target='#id39222b1905f4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls -w72 <span class='unselectable'>INSTALL boost/ boostcpp.jam index.htm rst.css Jamroot boost-build.jam bootstrap.bat index.html status/ LICENSE_1_0.txt boost.css bootstrap.sh* libs/ tools/ README.md boost.png doc/ more/ </span></pre> </div> <!-- endregion --> <p> Install the required dependencies for building Boost. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2e3f71df5c70'><button class='copyBtn' data-clipboard-target='#id2e3f71df5c70' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt-get update <span class='unselectable'>$ </span>yes | sudo apt install autotools-dev build-essential g++ \ libbz2-dev libicu-dev python3-dev</pre> </div> <!-- endregion --> <h3 id="bootstrap">Build Installer Using <span class="code">Bootstrap</span></h3> <p> Two scripts for building the installation program Boost are provided: <code>bootstrap.sh</code> (for Bash) and <code>bootstrap.bat</code> (for Windows/DOS). By default the scripts build all possible static/shared debug/release Boost libraries. Another default is that the script uses all the CPU cores your computer has for the build process. Even using all the cores on a fast laptop, the build might take 10 minutes. </p> <p> Here is the <code>bootstrap.sh</code> help message: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7185b01ea20b'><button class='copyBtn' data-clipboard-target='#id7185b01ea20b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>./bootstrap.sh -h <span class='unselectable'>`./bootstrap.sh\&#39; builds the Boost build system B2 and prepares Boost for building. This includes setting defaults in the project-config.jam which you can adjust prior to invoking B2.<br/> Usage: ./bootstrap.sh [OPTION]...<br/> Defaults for the options are specified in brackets.<br/> Configuration: -h, --help display this help and exit --with-bjam=BJAM use existing Boost.Jam executable (bjam) [automatically built] --with-toolset=TOOLSET use specific TOOLSET to build B2 and as default for building Boost [automatically detected] --show-libraries show the set of libraries that require build and installation steps (i.e., those libraries that can be used with --with-libraries or --without-libraries), then exit --with-libraries=list build only a particular set of libraries, describing using either a comma-separated list of library names or &quot;all&quot; [all] --without-libraries=list build all libraries except the ones listed [] --with-icu enable Unicode/ICU support in Regex [automatically detected] --without-icu disable Unicode/ICU support in Regex --with-icu=DIR specify the root of the ICU library installation and enable Unicode/ICU support in Regex [automatically detected] --with-python=PYTHON specify the Python executable [python] --with-python-root=DIR specify the root of the Python installation [automatically detected] --with-python-version=X.Y specify the Python version as X.Y [automatically detected]<br/> Installation directories: --prefix=PREFIX install Boost into the given PREFIX [/usr/local] --exec-prefix=EPREFIX install Boost binaries into the given EPREFIX [PREFIX]<br/> More precise control over installation directories: --libdir=DIR install libraries here [EPREFIX/lib] --includedir=DIR install headers here [PREFIX/include] </span></pre> </div> <p> <code>Bootstrap.sh</code> compiles and links a build program called <code>b2</code>, which by default installs all of Boost into <code>/usr/<wbr>local/</code>. To be more specific, the default location for Boost include files to be installed into is <code>/usr/<wbr>local/<wbr>include/<wbr>boost/</code>, and Boost libraries are installed by default into <code>/usr/<wbr>local/<wbr>lib/</code>. </p> <!-- endregion --> <p> Create (and recreate) the <code>b2</code> installation program for Boost for <i>each</i> <a href='/blog/2021/04/09/python-venvs.html'>Python virtual environment</a>. By default, <code>bootstrap.sh</code> uses the currently active Python virtual environment, like this: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1820b1df5432'><button class='copyBtn' data-clipboard-target='#id1820b1df5432' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>./bootstrap.sh</pre> </div> <p> Following is how to build <code>b2</code> for the virtual environment at <code>~/venv/blah/</code>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc1e56d80d430'><button class='copyBtn' data-clipboard-target='#idc1e56d80d430' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>./bootstrap.sh --with-python-root=~/venv/blah <span class='unselectable'>Building B2 engine..<br/> ### ### ### Using &#39;gcc&#39; toolset. ### ###<br/> g++ (Ubuntu 12.3.0-1ubuntu1~23.04) 12.3.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.<br/><br/> ### ###<br/> &gt; g++ -x c++ -std=c++11 -O2 -s -DNDEBUG builtins.cpp class.cpp command.cpp compile.cpp constants.cpp cwd.cpp debug.cpp debugger.cpp execcmd.cpp execnt.cpp execunix.cpp filesys.cpp filent.cpp fileunix.cpp frames.cpp function.cpp glob.cpp hash.cpp hcache.cpp hdrmacro.cpp headers.cpp jam_strings.cpp jam.cpp jamgram.cpp lists.cpp make.cpp make1.cpp md5.cpp mem.cpp modules.cpp native.cpp object.cpp option.cpp output.cpp parse.cpp pathnt.cpp pathsys.cpp pathunix.cpp regexp.cpp rules.cpp scan.cpp search.cpp startup.cpp subst.cpp sysinfo.cpp timestamp.cpp variable.cpp w32_getreg.cpp modules/order.cpp modules/path.cpp modules/property-set.cpp modules/regex.cpp modules/sequence.cpp modules/set.cpp -o b2 tools/build/src/engine/b2 Detecting Python version... 3.11 Unicode/ICU support for Boost.Regex?... /usr Backing up existing B2 configuration in project-config.jam.2 Generating B2 configuration in project-config.jam for gcc...<br/> Bootstrapping is done. To build, run:<br/> ./b2<br/> To generate header files, run:<br/> ./b2 headers<br/> The configuration generated uses gcc to build by default. If that is unintended either use the --with-toolset option or adjust configuration, by editing &#39;project-config.jam&#39;.<br/> Further information:<br/> - Command line help: ./b2 --help<br/> - Getting started guide: http://www.boost.org/more/getting_started/unix-variants.html<br/> - B2 documentation: http://www.boost.org/build/ </span></pre> </div> <!-- endregion --> <h3 id="b2">Running <span class="code">B2</span>, the Boost Installer</h3> <p> Following is the help message for the newly created <code>b2</code> Boost installation program. As you can see, the defaults specified when creating the <code>b2</code> script can be overidden. Note that not all of the available options are described in this help message; for example, the <code>-j</code> option is only described in the <a href='https://www.boost.org/doc/libs/1_83_0/tools/build/doc/html/index.html' target='_blank' rel="nofollow">online help for <code>b2</code></a>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id46f195284b36'><button class='copyBtn' data-clipboard-target='#id46f195284b36' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>./b2 --help <span class='unselectable'>B2 4.10-git<br/> Project-specific help:<br/> Project has jamfile at Jamroot<br/> Usage:<br/> b2 [options] [properties] [install|stage]<br/> Builds and installs Boost.<br/> Targets and Related Options:<br/> install Install headers and compiled library files to the ======= configured locations (below).<br/> --prefix=&lt;PREFIX&gt; Install architecture independent files here. Default: C:\Boost on Windows Default: /usr/local on Unix, Linux, etc.<br/> --exec-prefix=&lt;EPREFIX&gt; Install architecture dependent files here. Default: &lt;PREFIX&gt;<br/> --libdir=&lt;LIBDIR&gt; Install library files here. Default: &lt;EPREFIX&gt;/lib<br/> --includedir=&lt;HDRDIR&gt; Install header files here. Default: &lt;PREFIX&gt;/include<br/> --cmakedir=&lt;CMAKEDIR&gt; Install CMake configuration files here. Default: &lt;LIBDIR&gt;/cmake<br/> --no-cmake-config Do not install CMake configuration files.<br/> stage Build and install only compiled library files to the ===== stage directory.<br/> --stagedir=&lt;STAGEDIR&gt; Install library files here Default: ./stage<br/> Other Options:<br/> --build-type=&lt;type&gt; Build the specified pre-defined set of variations of the libraries. Note, that which variants get built depends on what each library supports.<br/> -- minimal -- (default) Builds a minimal set of variants. On Windows, these are static multithreaded libraries in debug and release modes, using shared runtime. On Linux, these are static and shared multithreaded libraries in release mode.<br/> -- complete -- Build all possible variations.<br/> --build-dir=DIR Build in this location instead of building within the distribution tree. Recommended!<br/> --show-libraries Display the list of Boost libraries that require build and installation steps, and then exit.<br/> --layout=&lt;layout&gt; Determine whether to choose library names and header locations such that multiple versions of Boost or multiple compilers can be used on the same system.<br/> -- versioned -- Names of boost binaries include the Boost version number, name and version of the compiler and encoded build properties. Boost headers are installed in a subdirectory of &lt;HDRDIR&gt; whose name contains the Boost version number.<br/> -- tagged -- Names of boost binaries include the encoded build properties such as variant and threading, but do not including compiler name and version, or Boost version. This option is useful if you build several variants of Boost, using the same compiler.<br/> -- system -- Binaries names do not include the Boost version number or the name and version number of the compiler. Boost headers are installed directly into &lt;HDRDIR&gt;. This option is intended for system integrators building distribution packages.<br/> The default value is &#39;versioned&#39; on Windows, and &#39;system&#39; on Unix.<br/> --buildid=ID Add the specified ID to the name of built libraries. The default is to not add anything.<br/> --python-buildid=ID Add the specified ID to the name of built libraries that depend on Python. The default is to not add anything. This ID is added in addition to --buildid.<br/> --help This message.<br/> --with-&lt;library&gt; Build and install the specified &lt;library&gt;. If this option is used, only libraries specified using this option will be built.<br/> --without-&lt;library&gt; Do not build, stage, or install the specified &lt;library&gt;. By default, all libraries are built.<br/> Properties:<br/> toolset=toolset Indicate the toolset to build with.<br/> variant=debug|release Select the build variant<br/> link=static|shared Whether to build static or shared libraries<br/> threading=single|multi Whether to build single or multithreaded binaries<br/> runtime-link=static|shared Whether to link to static or shared C and C++ runtime.<br/><br/> General command line usage:<br/> b2 [options] [properties] [targets]<br/> Options, properties and targets can be specified in any order.<br/> Important Options:<br/> * --clean Remove targets instead of building * -a Rebuild everything * -n Don&#39;t execute the commands, only print them * -d+2 Show commands as they are executed * -d0 Suppress all informational messages * -q Stop at first error * --reconfigure Rerun all configuration checks * --durations[=N] Report top N targets by execution time * --debug-configuration Diagnose configuration * --debug-building Report which targets are built with what properties * --debug-generator Diagnose generator search/execution<br/> Further Help:<br/> The following options can be used to obtain additional documentation.<br/> * --help-options Print more obscure command line options. * --help-internal B2 implementation details. * --help-doc-options Implementation details doc formatting.<br/> ...found 1 target... </span></pre> </div> <!-- endregion --> <p> The following runs the newly created <code>b2</code> program, which builds and installs Boost using all CPU cores. </p> <ol> <li> The Python version is stored into an environment variable called <code>PVER</code>. </li> <li> The <code>CPLUS_INCLUDE_PATH</code> environment variable is pointed at the location of the Python 3 include files. <code>PVER</code> was used to set the proper value for <code>CPLUS_INCLUDE_PATH</code>. </li> <li> <code>B2</code> generates screen upon screen of compiler warnings. The <code>-d 1</code> option suppresses them. </li> </ol> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idebe6abe163e3'><button class='copyBtn' data-clipboard-target='#idebe6abe163e3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>PVER="$(python --version | cut -d' ' -f 2 | grep -oEi '(3.[0-9]*)')" <span class='unselectable'>$ </span>echo $PVER <span class='unselectable'>3.11 </span> <span class='unselectable'>$ </span>sudo CPLUS_INCLUDE_PATH=/usr/include/python$PVER ./b2 -d 1 install <span class='unselectable'>Performing configuration checks<br/> - default address-model : 64-bit (cached) [1] - default architecture : x86 (cached) [1] - compiler supports SSE2 : yes (cached) [2] - compiler supports SSE4.1 : yes (cached) [2] - has std::atomic_ref : no (cached) [2] - has -Wl,--no-undefined : yes (cached) [2] - has statx : yes (cached) [2] - has init_priority attribute : yes (cached) [2] - has stat::st_blksize : yes (cached) [2] - has stat::st_mtim : yes (cached) [2] - has stat::st_mtimensec : no (cached) [2] - has stat::st_mtimespec : no (cached) [2] - has stat::st_birthtim : no (cached) [2] - has stat::st_birthtimensec : no (cached) [2] - has stat::st_birthtimespec : no (cached) [2] - has fdopendir(O_NOFOLLOW) : yes (cached) [2] - has dirent::d_type : yes (cached) [2] - has POSIX *at APIs : yes (cached) [2] - cxx11_auto_declarations : yes (cached) [2] - cxx11_constexpr : yes (cached) [2] - cxx11_defaulted_functions : yes (cached) [2] - cxx11_final : yes (cached) [2] - cxx11_hdr_mutex : yes (cached) [2] - cxx11_hdr_tuple : yes (cached) [2] - cxx11_lambdas : yes (cached) [2] - cxx11_noexcept : yes (cached) [2] - cxx11_nullptr : yes (cached) [2] - cxx11_rvalue_references : yes (cached) [2] - cxx11_template_aliases : yes (cached) [2] - cxx11_thread_local : yes (cached) [2] - cxx11_variadic_templates : yes (cached) [2] - has_icu builds : yes (cached) [2] warning: Graph library does not contain MPI-based parallel components. note: to enable them, add &quot;using mpi ;&quot; to your user-config.jam. note: to suppress this message, pass &quot;--without-graph_parallel&quot; to bjam. - zlib : yes (cached) - bzip2 : yes (cached) - lzma : yes (cached) - zstd : yes (cached) - has_lzma_cputhreads builds : yes (cached) [2] - cxx11_decltype : yes (cached) [2] - cxx11_basic_alignas : yes (cached) [2] - iconv (libc) : yes (cached) [2] - icu : yes (cached) [2] - cxx11_defaulted_moves : yes (cached) [2] - cxx11_hdr_functional : yes (cached) [2] - cxx11_hdr_type_traits : yes (cached) [2] - cxx11_override : yes (cached) [2] - cxx11_range_based_for : yes (cached) [2] - cxx11_scoped_enums : yes (cached) [2] - cxx11_smart_ptr : yes (cached) [2] - cxx11_static_assert : yes (cached) [2] - lockfree boost::atomic_flag : yes (cached) [2] - native atomic int32 supported : yes (cached) [2] - native syslog supported : yes (cached) [2] - pthread supports robust mutexes : yes (cached) [2] - compiler supports SSSE3 : yes (cached) [2] - compiler supports AVX2 : yes (cached) [2] - gcc visibility : yes (cached) [2] - sfinae_expr : yes (cached) [2] - cxx11_unified_initialization_syntax : yes (cached) [2] - cxx11_hdr_initializer_list : yes (cached) [2] - cxx11_hdr_chrono : yes (cached) [2] - cxx11_numeric_limits : yes (cached) [2] - cxx11_hdr_array : yes (cached) [2] - cxx11_hdr_atomic : yes (cached) [2] - cxx11_allocator : yes (cached) [2] - cxx11_explicit_conversion_operators : yes (cached) [2] - long double support : yes (cached) [2] warning: skipping optional Message Passing Interface (MPI) library. note: to enable MPI support, add &quot;using mpi ;&quot; to user-config.jam. note: to suppress this message, pass &quot;--without-mpi&quot; to bjam. note: otherwise, you can safely ignore this message. - cxx11_char16_t : yes (cached) [2] - cxx11_char32_t : yes (cached) [2] - Has Large File Support : yes (cached) [2] - Has attribute init_priority : yes (cached) [2] - libbacktrace builds : yes (cached) [2] - addr2line builds : yes (cached) [2] - WinDbg builds : no (cached) [2] - WinDbg builds : no (cached) [3] - WinDbgCached builds : no (cached) [2] - WinDbgCached builds : no (cached) [3] - BOOST_COMP_GNUC &gt;= 4.3.0 : yes (cached) [2] - BOOST_COMP_GNUC &gt;= 4.3.0 : yes (cached) [4] - cxx11_hdr_thread : yes (cached) [2] - cxx11_hdr_regex : yes (cached) [2] - compiler supports SSE2 : yes (cached) [4] - compiler supports SSE4.1 : yes (cached) [4] - has std::atomic_ref : no (cached) [4] - has statx : yes (cached) [4] - has init_priority attribute : yes (cached) [4] - has stat::st_blksize : yes (cached) [4] - has stat::st_mtim : yes (cached) [4] - has stat::st_mtimensec : no (cached) [4] - has stat::st_mtimespec : no (cached) [4] - has stat::st_birthtim : no (cached) [4] - has stat::st_birthtimensec : no (cached) [4] - has stat::st_birthtimespec : no (cached) [4] - has fdopendir(O_NOFOLLOW) : yes (cached) [4] - has dirent::d_type : yes (cached) [4] - has POSIX *at APIs : yes (cached) [4] - cxx11_auto_declarations : yes (cached) [4] - cxx11_constexpr : yes (cached) [4] - cxx11_defaulted_functions : yes (cached) [4] - cxx11_final : yes (cached) [4] - cxx11_hdr_mutex : yes (cached) [4] - cxx11_hdr_tuple : yes (cached) [4] - cxx11_lambdas : yes (cached) [4] - cxx11_noexcept : yes (cached) [4] - cxx11_nullptr : yes (cached) [4] - cxx11_rvalue_references : yes (cached) [4] - cxx11_template_aliases : yes (cached) [4] - cxx11_thread_local : yes (cached) [4] - cxx11_variadic_templates : yes (cached) [4] - has_icu builds : yes (cached) [4] - zlib : yes (cached) [5] - bzip2 : yes (cached) [5] - lzma : yes (cached) [5] - zstd : yes (cached) [5] - has_lzma_cputhreads builds : yes (cached) [4] - cxx11_decltype : yes (cached) [4] - cxx11_basic_alignas : yes (cached) [4] - iconv (libc) : yes (cached) [4] - icu : yes (cached) [4] - cxx11_defaulted_moves : yes (cached) [4] - cxx11_hdr_functional : yes (cached) [4] - cxx11_hdr_type_traits : yes (cached) [4] - cxx11_override : yes (cached) [4] - cxx11_range_based_for : yes (cached) [4] - cxx11_scoped_enums : yes (cached) [4] - cxx11_smart_ptr : yes (cached) [4] - cxx11_static_assert : yes (cached) [4] - lockfree boost::atomic_flag : yes (cached) [4] - native atomic int32 supported : yes (cached) [4] - native syslog supported : yes (cached) [4] - pthread supports robust mutexes : yes (cached) [4] - compiler supports SSSE3 : yes (cached) [4] - compiler supports AVX2 : yes (cached) [4] - gcc visibility : yes (cached) [4] - sfinae_expr : yes (cached) [4] - cxx11_unified_initialization_syntax : yes (cached) [4] - cxx11_hdr_initializer_list : yes (cached) [4] - cxx11_hdr_chrono : yes (cached) [4] - cxx11_numeric_limits : yes (cached) [4] - cxx11_hdr_array : yes (cached) [4] - cxx11_hdr_atomic : yes (cached) [4] - cxx11_allocator : yes (cached) [4] - cxx11_explicit_conversion_operators : yes (cached) [4] - long double support : yes (cached) [4] - cxx11_char16_t : yes (cached) [4] - cxx11_char32_t : yes (cached) [4] - Has Large File Support : yes (cached) [4] - Has attribute init_priority : yes (cached) [4] - libbacktrace builds : yes (cached) [4] - addr2line builds : yes (cached) [4] - WinDbg builds : no (cached) [4] - WinDbg builds : no (cached) [6] - WinDbgCached builds : no (cached) [4] - WinDbgCached builds : no (cached) [6] - cxx11_hdr_thread : yes (cached) [4] - cxx11_hdr_regex : yes (cached) [4]<br/> [1] gcc-12 [2] gcc-12/release/python-3.11/threading-multi/visibility-hidden [3] gcc-12/release/build-no/python-3.11/threading-multi/visibility-hidden [4] gcc-12/release/link-static/python-3.11/threading-multi/visibility-hidden [5] link-static [6] gcc-12/release/build-no/link-static/python-3.11/threading-multi/visibility-hidden<br/> Component configuration:<br/> - atomic : building - chrono : building - container : building - context : building - contract : building - coroutine : building - date_time : building - exception : building - fiber : building - filesystem : building - graph : building - graph_parallel : building - headers : building - iostreams : building - json : building - locale : building - log : building - math : building - mpi : building - nowide : building - program_options : building - python : building - random : building - regex : building - serialization : building - stacktrace : building - system : building - test : building - thread : building - timer : building - type_erasure : building - url : building - wave : building<br/> ...patience... ...patience... ...patience... ...patience... ...patience... ...patience... ...patience... ...found 52031 targets... </span></pre> </div> <!-- endregion --> <p> The <code>b2</code> installation program stored the following include files in <code>/usr/<wbr>local/<wbr>include/<wbr>boost/</code> </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id99270c7f5896'><button class='copyBtn' data-clipboard-target='#id99270c7f5896' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls -w72 /usr/local/include/boost/ <span class='unselectable'>accumulators/ make_unique.hpp algorithm/ math/ align/ math_fwd.hpp align.hpp mem_fn.hpp aligned_storage.hpp memory_order.hpp any/ metaparse/ any.hpp metaparse.hpp archive/ move/ array.hpp mp11/ asio/ mp11.hpp asio.hpp mpi/ assert/ mpi.hpp assert.hpp mpl/ assign/ msm/ assign.hpp multi_array/ atomic/ multi_array.hpp atomic.hpp multi_index/ beast/ multi_index_container.hpp beast.hpp multi_index_container_fwd.hpp bimap/ multiprecision/ bimap.hpp mysql/ bind/ mysql.hpp bind.hpp next_prior.hpp blank.hpp non_type.hpp blank_fwd.hpp noncopyable.hpp call_traits.hpp nondet_random.hpp callable_traits/ none.hpp callable_traits.hpp none_t.hpp cast.hpp nowide/ cerrno.hpp numeric/ checked_delete.hpp operators.hpp chrono/ operators_v1.hpp chrono.hpp optional/ circular_buffer/ optional.hpp circular_buffer.hpp outcome/ circular_buffer_fwd.hpp outcome.hpp compat/ parameter/ compatibility/ parameter.hpp compressed_pair.hpp pending/ compute/ pfr/ compute.hpp pfr.hpp concept/ phoenix/ concept_archetype.hpp phoenix.hpp concept_check/ pointee.hpp concept_check.hpp pointer_cast.hpp config/ pointer_to_other.hpp config.hpp poly_collection/ container/ polygon/ container_hash/ polymorphic_cast.hpp context/ polymorphic_pointer_cast.hpp contract/ pool/ contract.hpp predef/ contract_macro.hpp predef.h convert/ preprocessor/ convert.hpp preprocessor.hpp core/ process/ coroutine/ process.hpp coroutine2/ program_options/ crc.hpp program_options.hpp cregex.hpp progress.hpp cstdfloat.hpp property_map/ cstdint.hpp property_tree/ cstdlib.hpp proto/ current_function.hpp ptr_container/ cxx11_char_types.hpp python/ date_time/ python.hpp date_time.hpp qvm/ describe/ qvm.hpp describe.hpp qvm_lite.hpp detail/ random/ dll/ random.hpp dll.hpp range/ dynamic_bitset/ range.hpp dynamic_bitset.hpp ratio/ dynamic_bitset_fwd.hpp ratio.hpp enable_shared_from_this.hpp rational.hpp endian/ ref.hpp endian.hpp regex/ exception/ regex.h exception_ptr.hpp regex.hpp fiber/ regex_fwd.hpp filesystem/ safe_numerics/ filesystem.hpp scope_exit.hpp flyweight/ scoped_array.hpp flyweight.hpp scoped_ptr.hpp foreach.hpp serialization/ foreach_fwd.hpp shared_array.hpp format/ shared_container_iterator.hpp format.hpp shared_ptr.hpp function/ signals2/ function.hpp signals2.hpp function_equal.hpp smart_ptr/ function_output_iterator.hpp smart_ptr.hpp function_types/ sort/ functional/ spirit/ functional.hpp spirit.hpp fusion/ stacktrace/ generator_iterator.hpp stacktrace.hpp geometry/ statechart/ geometry.hpp static_assert.hpp get_pointer.hpp static_string/ gil/ static_string.hpp gil.hpp stl_interfaces/ graph/ swap.hpp hana/ system/ hana.hpp system.hpp heap/ test/ histogram/ thread/ histogram.hpp thread.hpp hof/ throw_exception.hpp hof.hpp timer/ icl/ timer.hpp implicit_cast.hpp token_functions.hpp indirect_reference.hpp token_iterator.hpp integer/ tokenizer.hpp integer.hpp tti/ integer_fwd.hpp tuple/ integer_traits.hpp type.hpp interprocess/ type_erasure/ intrusive/ type_index/ intrusive_ptr.hpp type_index.hpp io/ type_traits/ io_fwd.hpp type_traits.hpp iostreams/ typeof/ is_placeholder.hpp units/ iterator/ unordered/ iterator.hpp unordered_map.hpp iterator_adaptors.hpp unordered_set.hpp json/ url/ json.hpp url.hpp lambda/ utility/ lambda2/ utility.hpp lambda2.hpp uuid/ leaf/ variant/ leaf.hpp variant.hpp lexical_cast/ variant2/ lexical_cast.hpp variant2.hpp limits.hpp version.hpp local_function/ visit_each.hpp local_function.hpp vmd/ locale/ wave/ locale.hpp wave.hpp lockfree/ weak_ptr.hpp log/ winapi/ logic/ xpressive/ make_default.hpp yap/ make_shared.hpp </span></pre> </div> <!-- endregion --> <p> The <code>b2</code> installation program stored the following libraries in <code>/usr/<wbr>local/<wbr>lib/</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide3c506780be8'><button class='copyBtn' data-clipboard-target='#ide3c506780be8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls -w72 /usr/local/lib <span class='unselectable'>cmake/ libboost_atomic.a libboost_atomic.so@ libboost_atomic.so.1.83.0* libboost_chrono.a libboost_chrono.so@ libboost_chrono.so.1.83.0* libboost_container.a libboost_container.so@ libboost_container.so.1.83.0* libboost_context.a libboost_context.so@ libboost_context.so.1.83.0* libboost_contract.a libboost_contract.so@ libboost_contract.so.1.83.0* libboost_coroutine.a libboost_coroutine.so@ libboost_coroutine.so.1.83.0* libboost_date_time.a libboost_date_time.so@ libboost_date_time.so.1.83.0* libboost_exception.a libboost_fiber.a libboost_fiber.so@ libboost_fiber.so.1.83.0* libboost_filesystem.a libboost_filesystem.so@ libboost_filesystem.so.1.83.0* libboost_graph.a libboost_graph.so@ libboost_graph.so.1.83.0* libboost_iostreams.a libboost_iostreams.so@ libboost_iostreams.so.1.83.0* libboost_json.a libboost_json.so@ libboost_json.so.1.83.0* libboost_locale.a libboost_locale.so@ libboost_locale.so.1.83.0* libboost_log.a libboost_log.so@ libboost_log.so.1.83.0* libboost_log_setup.a libboost_log_setup.so@ libboost_log_setup.so.1.83.0* libboost_math_c99.a libboost_math_c99.so@ libboost_math_c99.so.1.83.0* libboost_math_c99f.a libboost_math_c99f.so@ libboost_math_c99f.so.1.83.0* libboost_math_c99l.a libboost_math_c99l.so@ libboost_math_c99l.so.1.83.0* libboost_math_tr1.a libboost_math_tr1.so@ libboost_math_tr1.so.1.83.0* libboost_math_tr1f.a libboost_math_tr1f.so@ libboost_math_tr1f.so.1.83.0* libboost_math_tr1l.a libboost_math_tr1l.so@ libboost_math_tr1l.so.1.83.0* libboost_nowide.a libboost_nowide.so@ libboost_nowide.so.1.83.0* libboost_prg_exec_monitor.a libboost_prg_exec_monitor.so@ libboost_prg_exec_monitor.so.1.83.0* libboost_program_options.a libboost_program_options.so@ libboost_program_options.so.1.83.0* libboost_random.a libboost_random.so@ libboost_random.so.1.83.0* libboost_regex.a libboost_regex.so@ libboost_regex.so.1.83.0* libboost_serialization.a libboost_serialization.so@ libboost_serialization.so.1.83.0* libboost_stacktrace_addr2line.a libboost_stacktrace_addr2line.so@ libboost_stacktrace_addr2line.so.1.83.0* libboost_stacktrace_backtrace.a libboost_stacktrace_backtrace.so@ libboost_stacktrace_backtrace.so.1.83.0* libboost_stacktrace_basic.a libboost_stacktrace_basic.so@ libboost_stacktrace_basic.so.1.83.0* libboost_stacktrace_noop.a libboost_stacktrace_noop.so@ libboost_stacktrace_noop.so.1.83.0* libboost_system.a libboost_system.so@ libboost_system.so.1.83.0* libboost_test_exec_monitor.a libboost_thread.a libboost_thread.so@ libboost_thread.so.1.83.0* libboost_timer.a libboost_timer.so@ libboost_timer.so.1.83.0* libboost_type_erasure.a libboost_type_erasure.so@ libboost_type_erasure.so.1.83.0* libboost_unit_test_framework.a libboost_unit_test_framework.so@ libboost_unit_test_framework.so.1.83.0* libboost_url.a libboost_url.so@ libboost_url.so.1.83.0* libboost_wave.a libboost_wave.so@ libboost_wave.so.1.83.0* libboost_wserialization.a libboost_wserialization.so@ libboost_wserialization.so.1.83.0* node_modules/ python3.11/ </span></pre> </div> <!-- endregion --> <h3 id="load_path">Verify Boost System Libraries</h3> <p> The <code>ldconfig</code> command manages Linux system libraries. Here is the help message: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id089dd10f0c27'><button class='copyBtn' data-clipboard-target='#id089dd10f0c27' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>man ldconfig <span class='unselectable'>ldconfig(8) System Manager&#39;s Manual ldconfig(8)<br/> NAME ldconfig - configure dynamic linker run-time bindings<br/> SYNOPSIS /sbin/ldconfig [-nNvVX] [-C cache] [-f conf] [-r root] directory ...<br/> /sbin/ldconfig -l [-v] library ...<br/> /sbin/ldconfig -p<br/> DESCRIPTION ldconfig creates the necessary links and cache to the most re&#8208; cent shared libraries found in the directories specified on the command line, in the file /etc/ld.so.conf, and in the trusted directories, /lib and /usr/lib. On some 64-bit architectures such as x86-64, /lib and /usr/lib are the trusted directories for 32-bit libraries, while /lib64 and /usr/lib64 are used for 64-bit libraries.<br/> The cache is used by the run-time linker, ld.so or ld-linux.so. ldconfig checks the header and filenames of the libraries it encounters when determining which versions should have their links updated. ldconfig should normally be run by the supe&#8208; ruser as it may require write permission on some root owned di&#8208; rectories and files.<br/> ldconfig will look only at files that are named lib*.so* (for regular shared objects) or ld-*.so* (for the dynamic loader it&#8208; self). Other files will be ignored. Also, ldconfig expects a certain pattern to how the symbolic links are set up, like this example, where the middle file (libfoo.so.1 here) is the SONAME for the library:<br/> libfoo.so -&gt; libfoo.so.1 -&gt; libfoo.so.1.12<br/> Failure to follow this pattern may result in compatibility is&#8208; sues after an upgrade.<br/> OPTIONS -c fmt --format=fmt (Since glibc 2.2) Use cache format fmt, which is one of old, new, or compat. Since glibc 2.32, the default is new. Before that, it was compat.<br/> -C cache Use cache instead of /etc/ld.so.cache.<br/> -f conf Use conf instead of /etc/ld.so.conf.<br/> -i --ignore-aux-cache (Since glibc 2.7) Ignore auxiliary cache file.<br/> -l (Since glibc 2.2) Interpret each operand as a libary name and configure its links. Intended for use only by experts.<br/> -n Process only the directories specified on the command line; don&#39;t process the trusted directories, nor those specified in /etc/ld.so.conf. Implies -N.<br/> -N Don&#39;t rebuild the cache. Unless -X is also specified, links are still updated.<br/> -p --print-cache Print the lists of directories and candidate libraries stored in the current cache.<br/> -r root Change to and use root as the root directory.<br/> -v --verbose Verbose mode. Print current version number, the name of each directory as it is scanned, and any links that are created. Overrides quiet mode.<br/> -V --version Print program version.<br/> -X Don&#39;t update links. Unless -N is also specified, the cache is still rebuilt.<br/> FILES /lib/ld.so is the run-time linker/loader. /etc/ld.so.conf contains a list of directories, one per line, in which to search for libraries. /etc/ld.so.cache contains an ordered list of libraries found in the di&#8208; rectories specified in /etc/ld.so.conf, as well as those found in the trusted directories.<br/> SEE ALSO ldd(1), ld.so(8)<br/> Linux man-pages 6.03 2023-01-07 ldconfig(8) </span></pre> </div> <!-- endregion --> <p> The Boost installation procedure automatically causes the newly built Boost libraries to be added to the configured system libraries when <code>bootstrap.sh</code> is not invoked with a target path, for example by specifying the <code>--with-python-root</code> option. </p> <p> To manually add the newly built Boost libraries to the system load path, use <code>ldconfig</code>. The following adds the newly compiled and installed Boost libraries in <code>/usr/local/lib/</code> (and any other libraries that might happen to be in that directory) to the <code>ld.so</code> cache. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2624ab5e5094'><button class='copyBtn' data-clipboard-target='#id2624ab5e5094' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo ldconfig -n /usr/local/lib</pre> </div> <!-- endregion --> <p> We can verify that the library cache now contains the newly built Boost libraries by using <code>ldconfig</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2f5463d49dcb'><button class='copyBtn' data-clipboard-target='#id2f5463d49dcb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ldconfig -p | grep boost <span class='unselectable'>libboost_wserialization.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_wserialization.so.1.83.0 libboost_wserialization.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_wserialization.so libboost_wave.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_wave.so.1.83.0 libboost_wave.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_wave.so libboost_url.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_url.so.1.83.0 libboost_url.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_url.so libboost_unit_test_framework.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_unit_test_framework.so.1.83.0 libboost_unit_test_framework.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_unit_test_framework.so libboost_type_erasure.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_type_erasure.so.1.83.0 libboost_type_erasure.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_type_erasure.so libboost_timer.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_timer.so.1.83.0 libboost_timer.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_timer.so libboost_thread.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_thread.so.1.83.0 libboost_thread.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_thread.so libboost_system.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_system.so.1.83.0 libboost_system.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_system.so libboost_stacktrace_noop.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_stacktrace_noop.so.1.83.0 libboost_stacktrace_noop.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_stacktrace_noop.so libboost_stacktrace_basic.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_stacktrace_basic.so.1.83.0 libboost_stacktrace_basic.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_stacktrace_basic.so libboost_stacktrace_backtrace.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_stacktrace_backtrace.so.1.83.0 libboost_stacktrace_backtrace.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_stacktrace_backtrace.so libboost_stacktrace_addr2line.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_stacktrace_addr2line.so.1.83.0 libboost_stacktrace_addr2line.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_stacktrace_addr2line.so libboost_serialization.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_serialization.so.1.83.0 libboost_serialization.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_serialization.so libboost_regex.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_regex.so.1.83.0 libboost_regex.so.1.74.0 (libc6,x86-64) =&gt; /lib/x86_64-linux-gnu/libboost_regex.so.1.74.0 libboost_regex.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_regex.so libboost_random.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_random.so.1.83.0 libboost_random.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_random.so libboost_python311.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_python311.so.1.83.0 libboost_python311.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_python311.so libboost_program_options.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_program_options.so.1.83.0 libboost_program_options.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_program_options.so libboost_prg_exec_monitor.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_prg_exec_monitor.so.1.83.0 libboost_prg_exec_monitor.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_prg_exec_monitor.so libboost_nowide.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_nowide.so.1.83.0 libboost_nowide.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_nowide.so libboost_math_tr1l.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_tr1l.so.1.83.0 libboost_math_tr1l.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_tr1l.so libboost_math_tr1f.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_tr1f.so.1.83.0 libboost_math_tr1f.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_tr1f.so libboost_math_tr1.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_tr1.so.1.83.0 libboost_math_tr1.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_tr1.so libboost_math_c99l.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_c99l.so.1.83.0 libboost_math_c99l.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_c99l.so libboost_math_c99f.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_c99f.so.1.83.0 libboost_math_c99f.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_c99f.so libboost_math_c99.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_c99.so.1.83.0 libboost_math_c99.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_math_c99.so libboost_log_setup.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_log_setup.so.1.83.0 libboost_log_setup.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_log_setup.so libboost_log.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_log.so.1.83.0 libboost_log.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_log.so libboost_locale.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_locale.so.1.83.0 libboost_locale.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_locale.so libboost_json.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_json.so.1.83.0 libboost_json.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_json.so libboost_iostreams.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_iostreams.so.1.83.0 libboost_iostreams.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_iostreams.so libboost_graph.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_graph.so.1.83.0 libboost_graph.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_graph.so libboost_filesystem.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_filesystem.so.1.83.0 libboost_filesystem.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_filesystem.so libboost_fiber.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_fiber.so.1.83.0 libboost_fiber.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_fiber.so libboost_date_time.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_date_time.so.1.83.0 libboost_date_time.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_date_time.so libboost_coroutine.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_coroutine.so.1.83.0 libboost_coroutine.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_coroutine.so libboost_contract.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_contract.so.1.83.0 libboost_contract.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_contract.so libboost_context.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_context.so.1.83.0 libboost_context.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_context.so libboost_container.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_container.so.1.83.0 libboost_container.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_container.so libboost_chrono.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_chrono.so.1.83.0 libboost_chrono.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_chrono.so libboost_atomic.so.1.83.0 (libc6,x86-64) =&gt; /usr/local/lib/libboost_atomic.so.1.83.0 libboost_atomic.so (libc6,x86-64) =&gt; /usr/local/lib/libboost_atomic.so </span></pre> </div> <!-- endregion --> <p> As a further check, examine the full path of an arbitrary include file (<code>src.hpp</code>) and list the Boost libraries: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf942491d4748'><button class='copyBtn' data-clipboard-target='#idf942491d4748' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>locate src.hpp <span class='unselectable'>/usr/include/boost/asio/impl/src.hpp /usr/include/boost/asio/ssl/impl/src.hpp /usr/include/boost/beast/src.hpp </span> <span class='unselectable'>$ </span>find /usr/local/lib/ -iname *boost* <span class='unselectable'>/usr/local/lib/libboost_math_tr1f.a /usr/local/lib/libboost_log_setup.so /usr/local/lib/libboost_system.so.1.83.0 /usr/local/lib/libboost_system.so /usr/local/lib/libboost_serialization.a /usr/local/lib/libboost_context.a /usr/local/lib/libboost_nowide.so /usr/local/lib/libboost_contract.so.1.83.0 /usr/local/lib/libboost_prg_exec_monitor.a /usr/local/lib/libboost_stacktrace_backtrace.so /usr/local/lib/libboost_iostreams.so /usr/local/lib/libboost_math_c99.so.1.83.0 /usr/local/lib/libboost_stacktrace_addr2line.so /usr/local/lib/libboost_locale.so.1.83.0 /usr/local/lib/libboost_math_c99l.so.1.83.0 /usr/local/lib/libboost_timer.a /usr/local/lib/libboost_atomic.so.1.83.0 /usr/local/lib/libboost_url.so.1.83.0 /usr/local/lib/libboost_math_c99.a /usr/local/lib/libboost_date_time.so /usr/local/lib/libboost_fiber.a /usr/local/lib/libboost_filesystem.so.1.83.0 /usr/local/lib/libboost_date_time.a /usr/local/lib/libboost_log_setup.a /usr/local/lib/libboost_type_erasure.so /usr/local/lib/libboost_json.so.1.83.0 /usr/local/lib/libboost_wave.so.1.83.0 /usr/local/lib/libboost_unit_test_framework.so /usr/local/lib/libboost_type_erasure.so.1.83.0 /usr/local/lib/libboost_random.so.1.83.0 /usr/local/lib/libboost_filesystem.so /usr/local/lib/libboost_stacktrace_basic.a /usr/local/lib/libboost_math_c99.so /usr/local/lib/libboost_regex.a /usr/local/lib/libboost_coroutine.so.1.83.0 /usr/local/lib/libboost_filesystem.a /usr/local/lib/libboost_date_time.so.1.83.0 /usr/local/lib/libboost_wave.a /usr/local/lib/libboost_regex.so /usr/local/lib/libboost_container.a /usr/local/lib/libboost_locale.a /usr/local/lib/libboost_graph.so /usr/local/lib/libboost_graph.so.1.83.0 /usr/local/lib/libboost_math_tr1f.so.1.83.0 /usr/local/lib/libboost_math_tr1.so.1.83.0 /usr/local/lib/libboost_program_options.so.1.83.0 /usr/local/lib/libboost_iostreams.so.1.83.0 /usr/local/lib/libboost_math_c99f.so.1.83.0 /usr/local/lib/libboost_nowide.so.1.83.0 /usr/local/lib/libboost_thread.so.1.83.0 /usr/local/lib/libboost_unit_test_framework.a /usr/local/lib/cmake/boost_math_c99f-1.83.0 /usr/local/lib/cmake/boost_math_c99f-1.83.0/libboost_math_c99f-variant-shared.cmake /usr/local/lib/cmake/boost_math_c99f-1.83.0/boost_math_c99f-config.cmake /usr/local/lib/cmake/boost_math_c99f-1.83.0/libboost_math_c99f-variant-static.cmake /usr/local/lib/cmake/boost_math_c99f-1.83.0/boost_math_c99f-config-version.cmake /usr/local/lib/cmake/boost_test_exec_monitor-1.83.0 /usr/local/lib/cmake/boost_test_exec_monitor-1.83.0/libboost_test_exec_monitor-variant-static.cmake /usr/local/lib/cmake/boost_test_exec_monitor-1.83.0/libboost_test_exec_monitor-variant-shared.cmake /usr/local/lib/cmake/boost_test_exec_monitor-1.83.0/boost_test_exec_monitor-config.cmake /usr/local/lib/cmake/boost_test_exec_monitor-1.83.0/boost_test_exec_monitor-config-version.cmake /usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0 /usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0/libboost_stacktrace_backtrace-variant-shared.cmake /usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0/boost_stacktrace_backtrace-config-version.cmake /usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0/libboost_stacktrace_backtrace-variant-static.cmake /usr/local/lib/cmake/boost_stacktrace_backtrace-1.83.0/boost_stacktrace_backtrace-config.cmake /usr/local/lib/cmake/boost_wave-1.83.0 /usr/local/lib/cmake/boost_wave-1.83.0/libboost_wave-variant-shared.cmake /usr/local/lib/cmake/boost_wave-1.83.0/boost_wave-config-version.cmake /usr/local/lib/cmake/boost_wave-1.83.0/libboost_wave-variant-static.cmake /usr/local/lib/cmake/boost_wave-1.83.0/boost_wave-config.cmake /usr/local/lib/cmake/boost_log-1.83.0 /usr/local/lib/cmake/boost_log-1.83.0/libboost_log-variant-shared.cmake /usr/local/lib/cmake/boost_log-1.83.0/boost_log-config.cmake /usr/local/lib/cmake/boost_log-1.83.0/libboost_log-variant-static.cmake /usr/local/lib/cmake/boost_log-1.83.0/boost_log-config-version.cmake /usr/local/lib/cmake/boost_stacktrace_basic-1.83.0 /usr/local/lib/cmake/boost_stacktrace_basic-1.83.0/boost_stacktrace_basic-config.cmake /usr/local/lib/cmake/boost_stacktrace_basic-1.83.0/boost_stacktrace_basic-config-version.cmake /usr/local/lib/cmake/boost_stacktrace_basic-1.83.0/libboost_stacktrace_basic-variant-static.cmake /usr/local/lib/cmake/boost_stacktrace_basic-1.83.0/libboost_stacktrace_basic-variant-shared.cmake /usr/local/lib/cmake/boost_coroutine-1.83.0 /usr/local/lib/cmake/boost_coroutine-1.83.0/boost_coroutine-config-version.cmake /usr/local/lib/cmake/boost_coroutine-1.83.0/boost_coroutine-config.cmake /usr/local/lib/cmake/boost_coroutine-1.83.0/libboost_coroutine-variant-shared.cmake /usr/local/lib/cmake/boost_coroutine-1.83.0/libboost_coroutine-variant-static.cmake /usr/local/lib/cmake/boost_timer-1.83.0 /usr/local/lib/cmake/boost_timer-1.83.0/boost_timer-config-version.cmake /usr/local/lib/cmake/boost_timer-1.83.0/boost_timer-config.cmake /usr/local/lib/cmake/boost_timer-1.83.0/libboost_timer-variant-shared.cmake /usr/local/lib/cmake/boost_timer-1.83.0/libboost_timer-variant-static.cmake /usr/local/lib/cmake/boost_program_options-1.83.0 /usr/local/lib/cmake/boost_program_options-1.83.0/boost_program_options-config.cmake /usr/local/lib/cmake/boost_program_options-1.83.0/libboost_program_options-variant-static.cmake /usr/local/lib/cmake/boost_program_options-1.83.0/libboost_program_options-variant-shared.cmake /usr/local/lib/cmake/boost_program_options-1.83.0/boost_program_options-config-version.cmake /usr/local/lib/cmake/boost_math_tr1f-1.83.0 /usr/local/lib/cmake/boost_math_tr1f-1.83.0/libboost_math_tr1f-variant-static.cmake /usr/local/lib/cmake/boost_math_tr1f-1.83.0/boost_math_tr1f-config.cmake /usr/local/lib/cmake/boost_math_tr1f-1.83.0/boost_math_tr1f-config-version.cmake /usr/local/lib/cmake/boost_math_tr1f-1.83.0/libboost_math_tr1f-variant-shared.cmake /usr/local/lib/cmake/boost_date_time-1.83.0 /usr/local/lib/cmake/boost_date_time-1.83.0/libboost_date_time-variant-static.cmake /usr/local/lib/cmake/boost_date_time-1.83.0/libboost_date_time-variant-shared.cmake /usr/local/lib/cmake/boost_date_time-1.83.0/boost_date_time-config.cmake /usr/local/lib/cmake/boost_date_time-1.83.0/boost_date_time-config-version.cmake /usr/local/lib/cmake/boost_math_c99l-1.83.0 /usr/local/lib/cmake/boost_math_c99l-1.83.0/libboost_math_c99l-variant-static.cmake /usr/local/lib/cmake/boost_math_c99l-1.83.0/libboost_math_c99l-variant-shared.cmake /usr/local/lib/cmake/boost_math_c99l-1.83.0/boost_math_c99l-config.cmake /usr/local/lib/cmake/boost_math_c99l-1.83.0/boost_math_c99l-config-version.cmake /usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0 /usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0/libboost_prg_exec_monitor-variant-shared.cmake /usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0/libboost_prg_exec_monitor-variant-static.cmake /usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0/boost_prg_exec_monitor-config.cmake /usr/local/lib/cmake/boost_prg_exec_monitor-1.83.0/boost_prg_exec_monitor-config-version.cmake /usr/local/lib/cmake/boost_wserialization-1.83.0 /usr/local/lib/cmake/boost_wserialization-1.83.0/libboost_wserialization-variant-shared.cmake /usr/local/lib/cmake/boost_wserialization-1.83.0/boost_wserialization-config-version.cmake /usr/local/lib/cmake/boost_wserialization-1.83.0/libboost_wserialization-variant-static.cmake /usr/local/lib/cmake/boost_wserialization-1.83.0/boost_wserialization-config.cmake /usr/local/lib/cmake/boost_serialization-1.83.0 /usr/local/lib/cmake/boost_serialization-1.83.0/boost_serialization-config.cmake /usr/local/lib/cmake/boost_serialization-1.83.0/libboost_serialization-variant-static.cmake /usr/local/lib/cmake/boost_serialization-1.83.0/boost_serialization-config-version.cmake /usr/local/lib/cmake/boost_serialization-1.83.0/libboost_serialization-variant-shared.cmake /usr/local/lib/cmake/boost_random-1.83.0 /usr/local/lib/cmake/boost_random-1.83.0/boost_random-config.cmake /usr/local/lib/cmake/boost_random-1.83.0/libboost_random-variant-static.cmake /usr/local/lib/cmake/boost_random-1.83.0/libboost_random-variant-shared.cmake /usr/local/lib/cmake/boost_random-1.83.0/boost_random-config-version.cmake /usr/local/lib/cmake/boost_log_setup-1.83.0 /usr/local/lib/cmake/boost_log_setup-1.83.0/boost_log_setup-config-version.cmake /usr/local/lib/cmake/boost_log_setup-1.83.0/libboost_log_setup-variant-shared.cmake /usr/local/lib/cmake/boost_log_setup-1.83.0/libboost_log_setup-variant-static.cmake /usr/local/lib/cmake/boost_log_setup-1.83.0/boost_log_setup-config.cmake /usr/local/lib/cmake/boost_exception-1.83.0 /usr/local/lib/cmake/boost_exception-1.83.0/boost_exception-config.cmake /usr/local/lib/cmake/boost_exception-1.83.0/boost_exception-config-version.cmake /usr/local/lib/cmake/boost_system-1.83.0 /usr/local/lib/cmake/boost_system-1.83.0/libboost_system-variant-shared.cmake /usr/local/lib/cmake/boost_system-1.83.0/boost_system-config-version.cmake /usr/local/lib/cmake/boost_system-1.83.0/libboost_system-variant-static.cmake /usr/local/lib/cmake/boost_system-1.83.0/boost_system-config.cmake /usr/local/lib/cmake/boost_python-1.83.0 /usr/local/lib/cmake/boost_python-1.83.0/boost_python-config.cmake /usr/local/lib/cmake/boost_python-1.83.0/boost_python-config-version.cmake /usr/local/lib/cmake/boost_thread-1.83.0 /usr/local/lib/cmake/boost_thread-1.83.0/libboost_thread-variant-shared.cmake /usr/local/lib/cmake/boost_thread-1.83.0/boost_thread-config.cmake /usr/local/lib/cmake/boost_thread-1.83.0/boost_thread-config-version.cmake /usr/local/lib/cmake/boost_thread-1.83.0/libboost_thread-variant-static.cmake /usr/local/lib/cmake/boost_json-1.83.0 /usr/local/lib/cmake/boost_json-1.83.0/libboost_json-variant-shared.cmake /usr/local/lib/cmake/boost_json-1.83.0/libboost_json-variant-static.cmake /usr/local/lib/cmake/boost_json-1.83.0/boost_json-config.cmake /usr/local/lib/cmake/boost_json-1.83.0/boost_json-config-version.cmake /usr/local/lib/cmake/boost_iostreams-1.83.0 /usr/local/lib/cmake/boost_iostreams-1.83.0/libboost_iostreams-variant-shared.cmake /usr/local/lib/cmake/boost_iostreams-1.83.0/boost_iostreams-config.cmake /usr/local/lib/cmake/boost_iostreams-1.83.0/libboost_iostreams-variant-static.cmake /usr/local/lib/cmake/boost_iostreams-1.83.0/boost_iostreams-config-version.cmake /usr/local/lib/cmake/boost_math_c99-1.83.0 /usr/local/lib/cmake/boost_math_c99-1.83.0/libboost_math_c99-variant-static.cmake /usr/local/lib/cmake/boost_math_c99-1.83.0/boost_math_c99-config-version.cmake /usr/local/lib/cmake/boost_math_c99-1.83.0/libboost_math_c99-variant-shared.cmake /usr/local/lib/cmake/boost_math_c99-1.83.0/boost_math_c99-config.cmake /usr/local/lib/cmake/boost_unit_test_framework-1.83.0 /usr/local/lib/cmake/boost_unit_test_framework-1.83.0/libboost_unit_test_framework-variant-static.cmake /usr/local/lib/cmake/boost_unit_test_framework-1.83.0/boost_unit_test_framework-config-version.cmake /usr/local/lib/cmake/boost_unit_test_framework-1.83.0/libboost_unit_test_framework-variant-shared.cmake /usr/local/lib/cmake/boost_unit_test_framework-1.83.0/boost_unit_test_framework-config.cmake /usr/local/lib/cmake/boost_filesystem-1.83.0 /usr/local/lib/cmake/boost_filesystem-1.83.0/libboost_filesystem-variant-static.cmake /usr/local/lib/cmake/boost_filesystem-1.83.0/boost_filesystem-config.cmake /usr/local/lib/cmake/boost_filesystem-1.83.0/boost_filesystem-config-version.cmake /usr/local/lib/cmake/boost_filesystem-1.83.0/libboost_filesystem-variant-shared.cmake /usr/local/lib/cmake/Boost-1.83.0 /usr/local/lib/cmake/Boost-1.83.0/BoostConfigVersion.cmake /usr/local/lib/cmake/Boost-1.83.0/BoostConfig.cmake /usr/local/lib/cmake/boost_headers-1.83.0 /usr/local/lib/cmake/boost_headers-1.83.0/boost_headers-config.cmake /usr/local/lib/cmake/boost_headers-1.83.0/boost_headers-config-version.cmake /usr/local/lib/cmake/boost_nowide-1.83.0 /usr/local/lib/cmake/boost_nowide-1.83.0/libboost_nowide-variant-static.cmake /usr/local/lib/cmake/boost_nowide-1.83.0/libboost_nowide-variant-shared.cmake /usr/local/lib/cmake/boost_nowide-1.83.0/boost_nowide-config.cmake /usr/local/lib/cmake/boost_nowide-1.83.0/boost_nowide-config-version.cmake /usr/local/lib/cmake/boost_atomic-1.83.0 /usr/local/lib/cmake/boost_atomic-1.83.0/boost_atomic-config-version.cmake /usr/local/lib/cmake/boost_atomic-1.83.0/boost_atomic-config.cmake /usr/local/lib/cmake/boost_atomic-1.83.0/libboost_atomic-variant-shared.cmake /usr/local/lib/cmake/boost_atomic-1.83.0/libboost_atomic-variant-static.cmake /usr/local/lib/cmake/boost_graph-1.83.0 /usr/local/lib/cmake/boost_graph-1.83.0/boost_graph-config-version.cmake /usr/local/lib/cmake/boost_graph-1.83.0/libboost_graph-variant-static.cmake /usr/local/lib/cmake/boost_graph-1.83.0/libboost_graph-variant-shared.cmake /usr/local/lib/cmake/boost_graph-1.83.0/boost_graph-config.cmake /usr/local/lib/cmake/boost_math_tr1-1.83.0 /usr/local/lib/cmake/boost_math_tr1-1.83.0/boost_math_tr1-config-version.cmake /usr/local/lib/cmake/boost_math_tr1-1.83.0/libboost_math_tr1-variant-static.cmake /usr/local/lib/cmake/boost_math_tr1-1.83.0/libboost_math_tr1-variant-shared.cmake /usr/local/lib/cmake/boost_math_tr1-1.83.0/boost_math_tr1-config.cmake /usr/local/lib/cmake/boost_graph_parallel-1.83.0 /usr/local/lib/cmake/boost_graph_parallel-1.83.0/boost_graph_parallel-config-version.cmake /usr/local/lib/cmake/boost_graph_parallel-1.83.0/boost_graph_parallel-config.cmake /usr/local/lib/cmake/boost_stacktrace_noop-1.83.0 /usr/local/lib/cmake/boost_stacktrace_noop-1.83.0/libboost_stacktrace_noop-variant-static.cmake /usr/local/lib/cmake/boost_stacktrace_noop-1.83.0/boost_stacktrace_noop-config-version.cmake /usr/local/lib/cmake/boost_stacktrace_noop-1.83.0/boost_stacktrace_noop-config.cmake /usr/local/lib/cmake/boost_stacktrace_noop-1.83.0/libboost_stacktrace_noop-variant-shared.cmake /usr/local/lib/cmake/boost_mpi-1.83.0 /usr/local/lib/cmake/boost_mpi-1.83.0/boost_mpi-config-version.cmake /usr/local/lib/cmake/boost_mpi-1.83.0/boost_mpi-config.cmake /usr/local/lib/cmake/boost_type_erasure-1.83.0 /usr/local/lib/cmake/boost_type_erasure-1.83.0/libboost_type_erasure-variant-shared.cmake /usr/local/lib/cmake/boost_type_erasure-1.83.0/libboost_type_erasure-variant-static.cmake /usr/local/lib/cmake/boost_type_erasure-1.83.0/boost_type_erasure-config-version.cmake /usr/local/lib/cmake/boost_type_erasure-1.83.0/boost_type_erasure-config.cmake /usr/local/lib/cmake/boost_chrono-1.83.0 /usr/local/lib/cmake/boost_chrono-1.83.0/boost_chrono-config-version.cmake /usr/local/lib/cmake/boost_chrono-1.83.0/libboost_chrono-variant-static.cmake /usr/local/lib/cmake/boost_chrono-1.83.0/libboost_chrono-variant-shared.cmake /usr/local/lib/cmake/boost_chrono-1.83.0/boost_chrono-config.cmake /usr/local/lib/cmake/boost_math_tr1l-1.83.0 /usr/local/lib/cmake/boost_math_tr1l-1.83.0/libboost_math_tr1l-variant-shared.cmake /usr/local/lib/cmake/boost_math_tr1l-1.83.0/boost_math_tr1l-config.cmake /usr/local/lib/cmake/boost_math_tr1l-1.83.0/boost_math_tr1l-config-version.cmake /usr/local/lib/cmake/boost_math_tr1l-1.83.0/libboost_math_tr1l-variant-static.cmake /usr/local/lib/cmake/boost_contract-1.83.0 /usr/local/lib/cmake/boost_contract-1.83.0/libboost_contract-variant-shared.cmake /usr/local/lib/cmake/boost_contract-1.83.0/boost_contract-config-version.cmake /usr/local/lib/cmake/boost_contract-1.83.0/boost_contract-config.cmake /usr/local/lib/cmake/boost_contract-1.83.0/libboost_contract-variant-static.cmake /usr/local/lib/cmake/boost_fiber-1.83.0 /usr/local/lib/cmake/boost_fiber-1.83.0/libboost_fiber-variant-static.cmake /usr/local/lib/cmake/boost_fiber-1.83.0/boost_fiber-config-version.cmake /usr/local/lib/cmake/boost_fiber-1.83.0/libboost_fiber-variant-shared.cmake /usr/local/lib/cmake/boost_fiber-1.83.0/boost_fiber-config.cmake /usr/local/lib/cmake/BoostDetectToolset-1.83.0.cmake /usr/local/lib/cmake/boost_context-1.83.0 /usr/local/lib/cmake/boost_context-1.83.0/boost_context-config.cmake /usr/local/lib/cmake/boost_context-1.83.0/libboost_context-variant-shared.cmake /usr/local/lib/cmake/boost_context-1.83.0/libboost_context-variant-static.cmake /usr/local/lib/cmake/boost_context-1.83.0/boost_context-config-version.cmake /usr/local/lib/cmake/boost_container-1.83.0 /usr/local/lib/cmake/boost_container-1.83.0/libboost_container-variant-shared.cmake /usr/local/lib/cmake/boost_container-1.83.0/boost_container-config-version.cmake /usr/local/lib/cmake/boost_container-1.83.0/boost_container-config.cmake /usr/local/lib/cmake/boost_container-1.83.0/libboost_container-variant-static.cmake /usr/local/lib/cmake/boost_regex-1.83.0 /usr/local/lib/cmake/boost_regex-1.83.0/libboost_regex-variant-shared.cmake /usr/local/lib/cmake/boost_regex-1.83.0/boost_regex-config.cmake /usr/local/lib/cmake/boost_regex-1.83.0/libboost_regex-variant-static.cmake /usr/local/lib/cmake/boost_regex-1.83.0/boost_regex-config-version.cmake /usr/local/lib/cmake/boost_url-1.83.0 /usr/local/lib/cmake/boost_url-1.83.0/boost_url-config-version.cmake /usr/local/lib/cmake/boost_url-1.83.0/libboost_url-variant-static.cmake /usr/local/lib/cmake/boost_url-1.83.0/libboost_url-variant-shared.cmake /usr/local/lib/cmake/boost_url-1.83.0/boost_url-config.cmake /usr/local/lib/cmake/boost_locale-1.83.0 /usr/local/lib/cmake/boost_locale-1.83.0/libboost_locale-variant-static.cmake /usr/local/lib/cmake/boost_locale-1.83.0/boost_locale-config.cmake /usr/local/lib/cmake/boost_locale-1.83.0/libboost_locale-variant-shared.cmake /usr/local/lib/cmake/boost_locale-1.83.0/boost_locale-config-version.cmake /usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0 /usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0/libboost_stacktrace_addr2line-variant-static.cmake /usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0/libboost_stacktrace_addr2line-variant-shared.cmake /usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0/boost_stacktrace_addr2line-config-version.cmake /usr/local/lib/cmake/boost_stacktrace_addr2line-1.83.0/boost_stacktrace_addr2line-config.cmake /usr/local/lib/libboost_fiber.so.1.83.0 /usr/local/lib/libboost_chrono.a /usr/local/lib/libboost_serialization.so /usr/local/lib/libboost_fiber.so /usr/local/lib/libboost_wserialization.a /usr/local/lib/libboost_unit_test_framework.so.1.83.0 /usr/local/lib/libboost_stacktrace_noop.a /usr/local/lib/libboost_iostreams.a /usr/local/lib/libboost_math_c99f.a /usr/local/lib/libboost_exception.a /usr/local/lib/libboost_type_erasure.a /usr/local/lib/libboost_thread.a /usr/local/lib/libboost_chrono.so.1.83.0 /usr/local/lib/libboost_math_c99l.so /usr/local/lib/libboost_log_setup.so.1.83.0 /usr/local/lib/libboost_stacktrace_addr2line.a /usr/local/lib/libboost_container.so.1.83.0 /usr/local/lib/libboost_context.so /usr/local/lib/libboost_program_options.so /usr/local/lib/libboost_stacktrace_basic.so /usr/local/lib/libboost_graph.a /usr/local/lib/libboost_prg_exec_monitor.so.1.83.0 /usr/local/lib/libboost_atomic.a /usr/local/lib/libboost_math_tr1l.a /usr/local/lib/libboost_log.so.1.83.0 /usr/local/lib/libboost_random.a /usr/local/lib/libboost_math_tr1f.so /usr/local/lib/libboost_contract.a /usr/local/lib/libboost_atomic.so /usr/local/lib/libboost_serialization.so.1.83.0 /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_wave.so /usr/local/lib/libboost_program_options.a /usr/local/lib/libboost_context.so.1.83.0 /usr/local/lib/libboost_coroutine.a /usr/local/lib/libboost_json.a /usr/local/lib/libboost_stacktrace_noop.so.1.83.0 /usr/local/lib/libboost_wserialization.so.1.83.0 /usr/local/lib/libboost_math_tr1.a /usr/local/lib/libboost_chrono.so /usr/local/lib/libboost_stacktrace_backtrace.a /usr/local/lib/libboost_stacktrace_addr2line.so.1.83.0 /usr/local/lib/libboost_stacktrace_basic.so.1.83.0 /usr/local/lib/libboost_math_c99l.a /usr/local/lib/libboost_log.so /usr/local/lib/libboost_log.a /usr/local/lib/libboost_math_tr1l.so /usr/local/lib/libboost_container.so /usr/local/lib/libboost_coroutine.so /usr/local/lib/libboost_prg_exec_monitor.so /usr/local/lib/libboost_nowide.a /usr/local/lib/libboost_url.so /usr/local/lib/libboost_url.a /usr/local/lib/libboost_math_c99f.so /usr/local/lib/libboost_stacktrace_backtrace.so.1.83.0 /usr/local/lib/libboost_test_exec_monitor.a /usr/local/lib/libboost_json.so /usr/local/lib/libboost_wserialization.so /usr/local/lib/libboost_system.a /usr/local/lib/libboost_regex.so.1.83.0 /usr/local/lib/libboost_contract.so /usr/local/lib/libboost_stacktrace_noop.so /usr/local/lib/libboost_locale.so /usr/local/lib/libboost_math_tr1.so /usr/local/lib/libboost_timer.so.1.83.0 /usr/local/lib/libboost_random.so /usr/local/lib/libboost_math_tr1l.so.1.83.0 /usr/local/lib/libboost_timer.so </span></pre> </div> <!-- endregion --> <!-- #region clean up --> <h3 id="clean">Clean Up</h3> <p> Many Ubuntu/Debian packages are left behind by the Boost installation program. You can view them as follows: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id254923fbbb64'><button class='copyBtn' data-clipboard-target='#id254923fbbb64' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>apt --dry-run autoremove | grep -Po '^Remv \K[^ ]+' <span class='unselectable'>libboost-mpi-python-dev libboost-mpi-python1.74-dev libboost-mpi-python1.74.0 mpi-default-bin libboost-mpi-dev libboost-mpi1.74-dev mpi-default-dev libopenmpi-dev openmpi-bin libcoarrays-openmpi-dev libopenmpi3 libucx0 libfabric1 ibverbs-providers libboost-atomic-dev libboost-coroutine-dev libboost-coroutine1.74-dev libboost-fiber-dev libboost-fiber1.74-dev libboost-context1.74-dev libboost-type-erasure-dev libboost-type-erasure1.74-dev libboost-thread1.74-dev libboost-log-dev libboost-log1.74-dev libboost-atomic1.74-dev libboost-atomic1.74.0 libboost-chrono-dev libboost-timer-dev libboost-timer1.74-dev libboost-chrono1.74-dev libboost-timer1.74.0 libboost-chrono1.74.0 libboost-container-dev libboost-container1.74-dev libboost-container1.74.0 libboost-context-dev libboost-fiber1.74.0 libboost-coroutine1.74.0 libboost-context1.74.0 libboost-date-time-dev libboost-date-time1.74-dev libboost-date-time1.74.0 libboost-dev libboost-exception-dev libboost-exception1.74-dev libboost-filesystem-dev libboost-wave-dev libboost-wave1.74-dev libboost-filesystem1.74-dev libboost-graph-dev libboost-graph-parallel-dev libboost-graph-parallel1.74-dev libboost-graph-parallel1.74.0 libboost-graph1.74-dev libboost-graph1.74.0 libboost-iostreams-dev libboost-iostreams1.74-dev libboost-locale-dev libboost-locale1.74-dev libboost-log1.74.0 libboost-math-dev libboost-math1.74-dev libboost-math1.74.0 libboost-mpi1.74.0 libboost-nowide-dev libboost-nowide1.74-dev libboost-nowide1.74.0 libboost-numpy-dev libboost-numpy1.74-dev libboost-numpy1.74.0 libboost-program-options-dev libboost-program-options1.74-dev libboost-program-options1.74.0 libboost-python-dev libboost-python1.74-dev libboost-python1.74.0 libboost-random-dev libboost-random1.74-dev libboost-random1.74.0 libboost-regex-dev libboost-regex1.74-dev libboost-serialization-dev libboost-serialization1.74-dev libboost-serialization1.74.0 libboost-stacktrace-dev libboost-stacktrace1.74-dev libboost-stacktrace1.74.0 libboost-system-dev libboost-system1.74-dev libboost-system1.74.0 libboost-test-dev libboost-test1.74-dev libboost-test1.74.0 libboost-thread-dev libboost-tools-dev libboost-type-erasure1.74.0 libboost-wave1.74.0 libboost1.74-dev libboost1.74-tools-dev libcaf-openmpi-3 libpmix-dev libevent-dev libevent-extra-2.1-7 libevent-openssl-2.1-7 libpmix2 libevent-pthreads-2.1-7 libhwloc-dev libhwloc-plugins libhwloc15 libibverbs-dev librdmacm1 libibverbs1 libjs-jquery-ui libmunge2 libnl-route-3-dev libnl-3-dev libnuma-dev libpsm-infinipath1 libpsm2-2 openmpi-common </span></pre> </div> <!-- endregion --> <p> Remove the packages as follows: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id594d3551f3f5'><button class='copyBtn' data-clipboard-target='#id594d3551f3f5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt autoremove <span class='unselectable'>Reading package lists... Done Building dependency tree... Done Reading state information... Done The following packages will be REMOVED: ibverbs-providers libboost-atomic-dev libboost-atomic1.74-dev libboost-atomic1.74.0 libboost-chrono-dev libboost-chrono1.74-dev libboost-chrono1.74.0 libboost-container-dev libboost-container1.74-dev libboost-container1.74.0 libboost-context-dev libboost-context1.74-dev libboost-context1.74.0 libboost-coroutine-dev libboost-coroutine1.74-dev libboost-coroutine1.74.0 libboost-date-time-dev libboost-date-time1.74-dev libboost-date-time1.74.0 libboost-dev libboost-exception-dev libboost-exception1.74-dev libboost-fiber-dev libboost-fiber1.74-dev libboost-fiber1.74.0 libboost-filesystem-dev libboost-filesystem1.74-dev libboost-graph-dev libboost-graph-parallel-dev libboost-graph-parallel1.74-dev libboost-graph-parallel1.74.0 libboost-graph1.74-dev libboost-graph1.74.0 libboost-iostreams-dev libboost-iostreams1.74-dev libboost-locale-dev libboost-locale1.74-dev libboost-log-dev libboost-log1.74-dev libboost-log1.74.0 libboost-math-dev libboost-math1.74-dev libboost-math1.74.0 libboost-mpi-dev libboost-mpi-python-dev libboost-mpi-python1.74-dev libboost-mpi-python1.74.0 libboost-mpi1.74-dev libboost-mpi1.74.0 libboost-nowide-dev libboost-nowide1.74-dev libboost-nowide1.74.0 libboost-numpy-dev libboost-numpy1.74-dev libboost-numpy1.74.0 libboost-program-options-dev libboost-program-options1.74-dev libboost-program-options1.74.0 libboost-python-dev libboost-python1.74-dev libboost-python1.74.0 libboost-random-dev libboost-random1.74-dev libboost-random1.74.0 libboost-regex-dev libboost-regex1.74-dev libboost-serialization-dev libboost-serialization1.74-dev libboost-serialization1.74.0 libboost-stacktrace-dev libboost-stacktrace1.74-dev libboost-stacktrace1.74.0 libboost-system-dev libboost-system1.74-dev libboost-system1.74.0 libboost-test-dev libboost-test1.74-dev libboost-test1.74.0 libboost-thread-dev libboost-thread1.74-dev libboost-timer-dev libboost-timer1.74-dev libboost-timer1.74.0 libboost-tools-dev libboost-type-erasure-dev libboost-type-erasure1.74-dev libboost-type-erasure1.74.0 libboost-wave-dev libboost-wave1.74-dev libboost-wave1.74.0 libboost1.74-dev libboost1.74-tools-dev libcaf-openmpi-3 libcoarrays-openmpi-dev libevent-dev libevent-extra-2.1-7 libevent-openssl-2.1-7 libevent-pthreads-2.1-7 libfabric1 libhwloc-dev libhwloc-plugins libhwloc15 libibverbs-dev libibverbs1 libjs-jquery-ui libmunge2 libnl-3-dev libnl-route-3-dev libnuma-dev libopenmpi-dev libopenmpi3 libpmix-dev libpmix2 libpsm-infinipath1 libpsm2-2 librdmacm1 libucx0 mpi-default-bin mpi-default-dev openmpi-bin openmpi-common 0 upgraded, 0 newly installed, 121 to remove and 4 not upgraded. After this operation, 413 MB disk space will be freed. (Reading database ... 399799 files and directories currently installed.) Removing libboost-mpi-python-dev (1.74.0.3ubuntu7) ... Removing libboost-mpi-python1.74-dev (1.74.0-18.1ubuntu3) ... Removing libboost-mpi-python1.74.0 (1.74.0-18.1ubuntu3) ... Removing mpi-default-bin (1.14) ... Removing libboost-mpi-dev (1.74.0.3ubuntu7) ... Removing libboost-mpi1.74-dev (1.74.0-18.1ubuntu3) ... Removing mpi-default-dev (1.14) ... Removing libopenmpi-dev:amd64 (4.1.4-3ubuntu2) ... Removing libcoarrays-openmpi-dev:amd64 (2.10.1-1) ... update-alternatives: warning: alternative /usr/bin/caf.openmpi (part of link group caf) doesn't exist; removing from list of alternatives update-alternatives: warning: /etc/alternatives/caf is dangling; it will be updated with best choice Removing libboost-atomic-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-coroutine-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-coroutine1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-fiber-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-fiber1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-type-erasure-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-type-erasure1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-log-dev (1.74.0.3ubuntu7) ... Removing libboost-log1.74-dev (1.74.0-18.1ubuntu3) ... Removing libboost-chrono-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-timer-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-timer1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-timer1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-container-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-container1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-container1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-context-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-fiber1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-coroutine1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-date-time-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-exception-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-exception1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-filesystem-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-wave-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-wave1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-filesystem1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-graph-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-graph-parallel-dev (1.74.0.3ubuntu7) ... Removing libboost-graph-parallel1.74-dev (1.74.0-18.1ubuntu3) ... Removing libboost-graph-parallel1.74.0 (1.74.0-18.1ubuntu3) ... Removing libboost-graph1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-graph1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-iostreams-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-iostreams1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-locale-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-locale1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-log1.74.0 (1.74.0-18.1ubuntu3) ... Removing libboost-math-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-math1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-math1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-mpi1.74.0 (1.74.0-18.1ubuntu3) ... Removing libboost-nowide-dev (1.74.0.3ubuntu7) ... Removing libboost-nowide1.74-dev (1.74.0-18.1ubuntu3) ... Removing libboost-nowide1.74.0 (1.74.0-18.1ubuntu3) ... Removing libboost-numpy-dev (1.74.0.3ubuntu7) ... Removing libboost-numpy1.74-dev (1.74.0-18.1ubuntu3) ... Removing libboost-numpy1.74.0 (1.74.0-18.1ubuntu3) ... Removing libboost-program-options-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-program-options1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-program-options1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-python-dev (1.74.0.3ubuntu7) ... Removing libboost-python1.74-dev (1.74.0-18.1ubuntu3) ... Removing libboost-python1.74.0 (1.74.0-18.1ubuntu3) ... Removing libboost-random-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-random1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-random1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-regex-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-regex1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-serialization-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-stacktrace-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-stacktrace1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-stacktrace1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-system-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-test-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-test1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-test1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-thread-dev:amd64 (1.74.0.3ubuntu7) ... Removing libboost-tools-dev (1.74.0.3ubuntu7) ... Removing libboost-type-erasure1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-wave1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost1.74-tools-dev (1.74.0-18.1ubuntu3) ... Removing libcaf-openmpi-3:amd64 (2.10.1-1) ... Removing libpmix-dev:amd64 (4.2.2-1) ... Removing libevent-dev (2.1.12-stable-8ubuntu3) ... Removing libevent-extra-2.1-7:amd64 (2.1.12-stable-8ubuntu3) ... Removing libevent-openssl-2.1-7:amd64 (2.1.12-stable-8ubuntu3) ... Removing libhwloc-dev:amd64 (2.9.0-1) ... Removing libibverbs-dev:amd64 (44.0-2) ... Removing libjs-jquery-ui (1.13.2+dfsg-1) ... Removing libnl-route-3-dev:amd64 (3.7.0-0.2) ... Removing libnl-3-dev:amd64 (3.7.0-0.2) ... Removing libnuma-dev:amd64 (2.0.16-1) ... Removing openmpi-bin (4.1.4-3ubuntu2) ... Removing libopenmpi3:amd64 (4.1.4-3ubuntu2) ... Removing libucx0:amd64 (1.13.1-1) ... Removing libfabric1:amd64 (1.17.0-3) ... Removing ibverbs-providers:amd64 (44.0-2) ... Removing libboost-context1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-thread1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-atomic1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-atomic1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-chrono1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-chrono1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-context1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-date-time1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-date-time1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-serialization1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-serialization1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-system1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost-system1.74.0:amd64 (1.74.0-18.1ubuntu3) ... Removing libboost1.74-dev:amd64 (1.74.0-18.1ubuntu3) ... Removing libpmix2:amd64 (4.2.2-1) ... Removing libevent-pthreads-2.1-7:amd64 (2.1.12-stable-8ubuntu3) ... Removing libhwloc-plugins:amd64 (2.9.0-1) ... Removing libhwloc15:amd64 (2.9.0-1) ... Removing librdmacm1:amd64 (44.0-2) ... Removing libibverbs1:amd64 (44.0-2) ... Removing libmunge2 (0.5.15-2) ... Removing libpsm-infinipath1 (3.3+20.604758e7-6.2) ... update-alternatives: warning: alternative /usr/lib/libpsm1/libpsm_infinipath.so.1.16 (part of link group libpsm_infinipath.so.1) doesn't exist; removing from list of alternatives update-alternatives: warning: /etc/alternatives/libpsm_infinipath.so.1 is dangling; it will be updated with best choice Removing libpsm2-2 (11.2.185-2) ... Removing openmpi-common (4.1.4-3ubuntu2) ... Processing triggers for man-db (2.11.2-1) ... Processing triggers for libc-bin (2.37-0ubuntu2) ... /sbin/ldconfig.real: /usr/lib/wsl/lib/libcuda.so.1 is not a symbolic link </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region conclusion --> <h2 id="go">Go Forth and Get Boosted</h2> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <p> Boost usage as a C++ general-pupose library is widespread throughout a diverse range of industries and requirements. </p> <!-- endregion --> PDF Manipulation 2023-09-08T00:00:00-04:00 https://mslinn.github.io/blog/2023/09/08/pdftk <!-- #region intro --> <h2 id="install">Installation</h2> <p> The <a href='https://www.pdflabs.com/tools/pdftk-the-pdf-toolkit/' target='_blank' rel="nofollow"><code>pdftk</code> website</a> discusses <code>pdktk</code> (a F/OSS command-line program), a Windows GUI, and mentions the book <a href='https://github.com/InspectorDidi/Hacking-Books/blob/master/PDF%20Hacks.pdf' target='_blank' rel="nofollow"><b>PDF Hacks</b></a>. </p> <p> An alternative free GUI is <a href='https://portableapps.com/apps/office/pdftk_builder_portable' target='_blank' rel="nofollow">PDFTK Builder</a>. </p> <p> Other command-line utilties for manipulating PDF documents are: </p> <ul> <li><a href='https://en.wikipedia.org/wiki/Poppler_(software)' target='_blank' rel="nofollow">Poppler</a> (GPL)</li> <li><a href='https://qpdf.readthedocs.io/en/stable/index.html' target='_blank' rel="nofollow">QPDF</a></li> </ul> <p> Install the command-line <code>pdftk</code> program on Ubuntu: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddedd4abbafcf'><button class='copyBtn' data-clipboard-target='#iddedd4abbafcf' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install pdftk</pre> </div> <!-- endregion --> <p> This is the <code>pdftk</code> help text: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9634c22f77ab'><button class='copyBtn' data-clipboard-target='#id9634c22f77ab' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>man pdftk <span class='unselectable'>pdftk.pdftk-java(1) General Commands Manual pdftk.pdftk-java(1)<br/> NAME pdftk - A handy tool for manipulating PDF<br/> SYNOPSIS pdftk &lt;input PDF files | - | PROMPT&gt; [ input_pw &lt;input PDF owner passwords | PROMPT&gt; ] [ &lt;operation&gt; &lt;operation arguments&gt; ] [ output &lt;output filename | - | PROMPT&gt; ] [ encrypt_40bit | encrypt_128bit | encrypt_aes128 ] [ allow &lt;permissions&gt; ] [ owner_pw &lt;owner password | PROMPT&gt; ] [ user_pw &lt;user password | PROMPT&gt; ] [ flatten ] [ need_appearances ] [ compress | uncompress ] [ keep_first_id | keep_final_id ] [ drop_xfa ] [ drop_xmp ] [ replacement_font &lt;font name&gt; ] [ verbose ] [ dont_ask | do_ask ] Where: &lt;operation&gt; may be empty, or: [ cat | shuffle | burst | rotate | generate_fdf | fill_form | background | multibackground | stamp | multistamp | dump_data | dump_data_utf8 | dump_data_fields | dump_data_fields_utf8 | dump_data_annots | update_info | update_info_utf8 | attach_files | unpack_files ]<br/> For Complete Help: pdftk --help<br/> DESCRIPTION If PDF is electronic paper, then pdftk is an electronic staple- remover, hole-punch, binder, secret-decoder-ring, and X-Ray- glasses. Pdftk is a simple tool for doing everyday things with PDF documents. Use it to:<br/> * Merge PDF Documents or Collate PDF Page Scans * Split PDF Pages into a New Document * Rotate PDF Documents or Pages * Decrypt Input as Necessary (Password Required) * Encrypt Output as Desired * Fill PDF Forms with X/FDF Data and/or Flatten Forms * Generate FDF Data Stencils from PDF Forms * Apply a Background Watermark or a Foreground Stamp * Report PDF Metrics, Bookmarks and Metadata * Add/Update PDF Metrics, Bookmarks or Metadata * Attach Files to PDF Pages or the PDF Document * Unpack PDF Attachments * Burst a PDF Document into Single Pages * Uncompress and Re-Compress Page Streams * Repair Corrupted PDF (Where Possible)<br/> OPTIONS A summary of options is included below.<br/> --help, -h Show this summary of options.<br/> &lt;input PDF files | - | PROMPT&gt; A list of the input PDF files. If you plan to combine these PDFs (without using handles) then list files in the order you want them combined. Use - to pass a sin&#8208; gle PDF into pdftk via stdin. Input files can be asso&#8208; ciated with handles, where a handle is one or more up&#8208; per-case letters:<br/> &lt;input PDF handle&gt;=&lt;input PDF filename&gt;<br/> Handles are often omitted. They are useful when speci&#8208; fying PDF passwords or page ranges, later.<br/> For example: A=input1.pdf QT=input2.pdf M=input3.pdf<br/> [input_pw &lt;input PDF owner passwords | PROMPT&gt;] Input PDF owner passwords, if necessary, are associated with files by using their handles:<br/> &lt;input PDF handle&gt;=&lt;input PDF file owner password&gt;<br/> If handles are not given, then passwords are associated with input files by order.<br/> Most pdftk features require that encrypted input PDF are accompanied by the ~owner~ password. If the input PDF has no owner password, then the user password must be given, instead. If the input PDF has no passwords, then no password should be given.<br/> When running in do_ask mode, pdftk will prompt you for a password if the supplied password is incorrect or none was given.<br/> [&lt;operation&gt; &lt;operation arguments&gt;] Available operations are: cat, shuffle, burst, rotate, generate_fdf, fill_form, background, multibackground, stamp, multistamp, dump_data, dump_data_utf8, dump_data_fields, dump_data_fields_utf8, dump_data_an&#8208; nots, update_info, update_info_utf8, attach_files, un&#8208; pack_files. Some operations takes additional arguments, described below.<br/> If this optional argument is omitted, then pdftk runs in &#39;filter&#39; mode. Filter mode takes only one PDF input and creates a new PDF after applying all of the output op&#8208; tions, like encryption and compression.<br/> cat [&lt;page ranges&gt;] Assembles (catenates) pages from input PDFs to create a new PDF. Use cat to merge PDF pages or to split PDF pages from documents. You can also use it to rotate PDF pages. Page order in the new PDF is specified by the order of the given page ranges. Page ranges are described like this:<br/> &lt;input PDF handle&gt;[&lt;begin page number&gt;[-&lt;end page number&gt;[&lt;qualifier&gt;]]][&lt;page rotation&gt;]<br/> Where the handle identifies one of the input PDF files, and the beginning and ending page numbers are one-based references to pages in the PDF file. The qualifier can be even, odd, or ~, and the page rota&#8208; tion can be north, south, east, west, left, right, or down.<br/> If a PDF handle is given but no pages are specified, then the entire PDF is used. If no pages are speci&#8208; fied for any of the input PDFs, then the input PDFs&#39; bookmarks are also merged and included in the output.<br/> If the handle is omitted from the page range, then the pages are taken from the first input PDF.<br/> The even qualifier causes pdftk to use only the even- numbered PDF pages, so 1-6even yields pages 2, 4 and 6 in that order. 6-1even yields pages 6, 4 and 2 in that order.<br/> The odd qualifier works similarly to the even.<br/> Pages can be subtracted from a page range using the ~ qualifier followed by a page range. For instance, 1-20~5-6 and 1-20~5~6 are equivalent to 1-4 7-20, and ~5 yields all pages except page 5. Depending on your shell, you may need to quote this argument because of the ~ at the beginning.<br/> The page rotation setting can cause pdftk to rotate pages and documents. Each option sets the page rota&#8208; tion as follows (in degrees): north: 0, east: 90, south: 180, west: 270, left: -90, right: +90, down: +180. left, right, and down make relative adjustments to a page&#39;s rotation.<br/> If no arguments are passed to cat, then pdftk com&#8208; bines all input PDFs in the order they were given to create the output.<br/> NOTES: * &lt;end page number&gt; may be less than &lt;begin page num&#8208; ber&gt;. * The keyword end may be used to reference the final page of a document instead of a page number. * Reference a single page by omitting the ending page number. * The handle may be used alone to represent the en&#8208; tire PDF document, e.g., B1-end is the same as B. * You can reference page numbers in reverse order by prefixing them with the letter r. For example, page r1 is the last page of the document, r2 is the next- to-last page of the document, and rend is the first page of the document. You can use this prefix in ranges, too, for example r3-r1 is the last three pages of a PDF.<br/> Page Range Examples without Handles: 1\-endeast &ndash; rotate entire document 90 degrees 5 11 20 &ndash; take single pages from input PDF 5-25oddwest &ndash; take odd pages in range, rotate 90 de&#8208; grees 6-1 &ndash; reverse pages in range from input PDF<br/> Page Range Examples Using Handles: Say A=in1.pdf B=in2.pdf, then: A1-21 &ndash; take range from in1.pdf Bend-1odd &ndash; take all odd pages from in2.pdf in re&#8208; verse order A72 &ndash; take a single page from in1.pdf A1-21 Beven A72 &ndash; assemble pages from both in1.pdf and in2.pdf Awest &ndash; rotate entire in1.pdf document 90 degrees B &ndash; use all of in2.pdf A2-30evenleft &ndash; take the even pages from the range, remove 90 degrees from each page&#39;s rotation A A &ndash; catenate in1.pdf with in1.pdf Aevenwest Aoddeast &ndash; apply rotations to even pages, odd pages from in1.pdf Awest Bwest Bdown &ndash; catenate rotated documents<br/> shuffle [&lt;page ranges&gt;] Collates pages from input PDFs to create a new PDF. Works like the cat operation except that it takes one page at a time from each page range to assemble the output PDF. If one range runs out of pages, it con&#8208; tinues with the remaining ranges. Ranges can use all of the features described above for cat, like reverse page ranges, multiple ranges from a single PDF, and page rotation. This feature was designed to help collate PDF pages after scanning paper documents.<br/> burst Splits a single input PDF document into individual pages. Also creates a report named doc_data.txt which is the same as the output from dump_data. The output section can contain a printf-styled format string to name these pages. For example, if you want pages named page_01.pdf, page_02.pdf, etc., pass output page_%02d.pdf to pdftk. If the pattern is omitted, then a default pattern g_%04d.pdf is appended and produces pages named pg_0001.pdf, pg_0002.pdf, etc. Encryption can be applied to the output by appending output options such as owner_pw, e.g.:<br/> pdftk in.pdf burst owner_pw foopass<br/> rotate [&lt;page ranges&gt;] Takes a single input PDF and rotates just the speci&#8208; fied pages. All other pages remain unchanged. The page order remains unchanged. Specify the pages to rotate using the same notation as you would with cat, except you omit the pages that you aren&#39;t rotating:<br/> [&lt;begin page number&gt;[-&lt;end page number&gt;[&lt;quali&#8208; fier&gt;]]][&lt;page rotation&gt;]<br/> The qualifier can be even or odd, and the page rota&#8208; tion can be north, south, east, west, left, right, or down.<br/> Each option sets the page rotation as follows (in de&#8208; grees): north: 0, east: 90, south: 180, west: 270, left: -90, right: +90, down: +180. left, right, and down make relative adjustments to a page&#39;s rotation.<br/> The given order of the pages doesn&#39;t change the page order in the output.<br/> generate_fdf Reads a single input PDF file and generates an FDF file suitable for fill_form out of it to the given output filename or (if no output is given) to stdout. Does not create a new PDF.<br/> fill_form &lt;FDF data filename | XFDF data filename | - | PROMPT&gt; Fills the single input PDF&#39;s form fields with the data from an FDF file, XFDF file or stdin. Enter the data filename after fill_form, or use - to pass the data via stdin, like so:<br/> pdftk form.pdf fill_form data.fdf output form.filled.pdf<br/> If the input FDF file includes Rich Text formatted data in addition to plain text, then the Rich Text data is packed into the form fields as well as the plain text. Pdftk also sets a flag that cues Reader/Acrobat to generate new field appearances based on the Rich Text data. So when the user opens the PDF, the viewer will create the Rich Text appear&#8208; ance on the spot. If the user&#39;s PDF viewer does not support Rich Text, then the user will see the plain text data instead. If you flatten this form before Acrobat has a chance to create (and save) new field appearances, then the plain text field data is what you&#39;ll see.<br/> Also see the flatten, need_appearances, and replace&#8208; ment_font options.<br/> background &lt;background PDF filename | - | PROMPT&gt; Applies a PDF watermark to the background of a single input PDF. Pass the background PDF&#39;s filename after background like so:<br/> pdftk in.pdf background back.pdf output out.pdf<br/> Pdftk uses only the first page from the background PDF and applies it to every page of the input PDF. This page is scaled and rotated as needed to fit the input page. You can use - to pass a background PDF into pdftk via stdin.<br/> If the input PDF does not have a transparent back&#8208; ground (such as a PDF created from page scans) then the resulting background won&#39;t be visible &ndash; use the stamp operation instead.<br/> multibackground &lt;background PDF filename | - | PROMPT&gt; Same as the background operation, but applies each page of the background PDF to the corresponding page of the input PDF. If the input PDF has more pages than the stamp PDF, then the final stamp page is re&#8208; peated across these remaining pages in the input PDF.<br/> stamp &lt;stamp PDF filename | - | PROMPT&gt; This behaves just like the background operation ex&#8208; cept it overlays the stamp PDF page on top of the in&#8208; put PDF document&#39;s pages. This works best if the stamp PDF page has a transparent background.<br/> multistamp &lt;stamp PDF filename | - | PROMPT&gt; Same as the stamp operation, but applies each page of the background PDF to the corresponding page of the input PDF. If the input PDF has more pages than the stamp PDF, then the final stamp page is repeated across these remaining pages in the input PDF.<br/> dump_data Reads a single input PDF file and reports its meta&#8208; data, bookmarks (a/k/a outlines), page metrics (me&#8208; dia, rotation and labels), data embedded by STAMPtk (see STAMPtk&#39;s embed option) and other data to the given output filename or (if no output is given) to stdout. Non-ASCII characters are encoded as XML nu&#8208; merical entities. Does not create a new PDF.<br/> dump_data_utf8 Same as dump_data except that the output is encoded as UTF-8.<br/> dump_data_fields Reads a single input PDF file and reports form field statistics to the given output filename or (if no output is given) to stdout. Non-ASCII characters are encoded as XML numerical entities. Does not create a new PDF.<br/> dump_data_fields_utf8 Same as dump_data_fields except that the output is encoded as UTF-8.<br/> dump_data_annots This operation currently reports only link annota&#8208; tions. Reads a single input PDF file and reports an&#8208; notation information to the given output filename or (if no output is given) to stdout. Non-ASCII charac&#8208; ters are encoded as XML numerical entities. Does not create a new PDF.<br/> update_info &lt;info data filename | - | PROMPT&gt; Changes the bookmarks, page labels, page sizes, page rotations, and metadata in a single PDF&#39;s Info dic&#8208; tionary to match the input data file. The input data file uses the same syntax as the output from dump_data. Non-ASCII characters should be encoded as XML numerical entities.<br/> This operation does not change the metadata stored in the PDF&#39;s XMP stream, if it has one. (For this reason you should include a ModDate entry in your updated info with a current date/timestamp, format: D:YYYYM&#8208; MDDHHmmSS, e.g. D:201307241346 &ndash; omitted data after YYYY revert to default values.)<br/> For example:<br/> pdftk in.pdf update_info in.info output out.pdf<br/> update_info_utf8 &lt;info data filename | - | PROMPT&gt; Same as update_info except that the input is encoded as UTF-8.<br/> attach_files &lt;attachment filenames | PROMPT&gt; [to_page &lt;page number | PROMPT&gt; | relation &lt;relationship&gt;] Packs arbitrary files into a PDF using PDF&#39;s file at&#8208; tachment features. More than one attachment may be listed after attach_files. Attachments are added at the document level unless the optional to_page option is given, in which case the files are attached to the given page number (the first page is 1, the final page is end). Attachments at the document level may be tagged with a relationship among Source, Data, Al&#8208; ternative, Supplement, and Unspecified (default).<br/> For example:<br/> pdftk in.pdf attach_files table1.html table2.html to_page 6 output out.pdf<br/> pdftk in.pdf attach_files in.tex relation Source out&#8208; put out.pdf<br/> unpack_files Copies all of the attachments from the input PDF into the current folder or to an output directory given after output. For example:<br/> pdftk report.pdf unpack_files output ~/atts/<br/> or, interactively:<br/> pdftk report.pdf unpack_files output PROMPT<br/> [output &lt;output filename | - | PROMPT&gt;] The output PDF filename may not be set to the name of an input filename. Use - to output to stdout. When using the dump_data operation, use output to set the name of the output data file. When using the unpack_files opera&#8208; tion, use output to set the name of an output directory. When using the burst operation, you can use output to control the resulting PDF page filenames (described above).<br/> [encrypt_40bit | encrypt_128bit | encrypt_aes128] If an output PDF user or owner password is given, the output PDF encryption algorithm defaults to AES-128. The weaker RC4 40-bit and RC4 128-bit algorithms can be cho&#8208; sen by specifying encrypt_40bit or encrypt_128bit (dis&#8208; couraged).<br/> [allow &lt;permissions&gt;] Permissions are applied to the output PDF only if an en&#8208; cryption strength is specified or an owner or user pass&#8208; word is given. If permissions are not specified, they default to &#39;none,&#39; which means all of the following fea&#8208; tures are disabled.<br/> The permissions section may include one or more of the following features:<br/> Printing Top Quality Printing<br/> DegradedPrinting Lower Quality Printing<br/> ModifyContents Also allows Assembly<br/> Assembly<br/> CopyContents Also allows ScreenReaders<br/> ScreenReaders<br/> ModifyAnnotations Also allows FillIn<br/> FillIn<br/> AllFeatures Allows the user to perform all of the above, and top quality printing.<br/> [owner_pw &lt;owner password | PROMPT&gt;]<br/> [user_pw &lt;user password | PROMPT&gt;] If an encryption strength is given but no passwords are supplied, then the owner and user passwords remain empty, which means that the resulting PDF may be opened and its security parameters altered by anybody.<br/> [compress | uncompress] These are only useful when you want to edit PDF code in a text editor like vim or emacs. Remove PDF page stream compression by applying the uncompress filter. Use the compress filter to restore compression.<br/> [flatten] Use this option to merge an input PDF&#39;s interactive form fields (and their data) with the PDF&#39;s pages. Only one input PDF may be given. Sometimes used with the fill_form operation.<br/> [need_appearances] Sets a flag that cues Reader/Acrobat to generate new field appearances based on the form field values. Use this when filling a form with non-ASCII text to ensure the best presentation in Adobe Reader or Acrobat. It won&#39;t work when combined with the flatten option.<br/> [replacement_font &lt;font name&gt;] Use the specified font to display text in form fields. This option is useful when filling a form with non-ASCII text that is not supported by the fonts included in the input PDF. font name may be either the file name or the family name of a font, but using a file name is more re&#8208; liable. Currently only TrueType fonts with Unicode text are supported.<br/> [keep_first_id | keep_final_id] When combining pages from multiple PDFs, use one of these options to copy the document ID from either the first or final input document into the new output PDF. Otherwise pdftk creates a new document ID for the output PDF. When no operation is given, pdftk always uses the ID from the (single) input PDF.<br/> [drop_xfa] If your input PDF is a form created using Acrobat 7 or Adobe Designer, then it probably has XFA data. Filling such a form using pdftk yields a PDF with data that fails to display in Acrobat 7 (and 6?). The workaround solution is to remove the form&#39;s XFA data, either before you fill the form using pdftk or at the time you fill the form. Using this option causes pdftk to omit the XFA data from the output PDF form.<br/> This option is only useful when running pdftk on a sin&#8208; gle input PDF. When assembling a PDF from multiple in&#8208; puts using pdftk, any XFA data in the input is automati&#8208; cally omitted.<br/> [drop_xmp] Many PDFs store document metadata using both an Info dictionary (old school) and an XMP stream (new school). Pdftk&#39;s update_info operation can update the Info dic&#8208; tionary, but not the XMP stream. The proper remedy for this is to include a ModDate entry in your updated info with a current date/timestamp. The date/timestamp format is: D:YYYYMMDDHHmmSS, e.g. D:201307241346 &ndash; omitted data after YYYY revert to default values. This newer ModDate should cue PDF viewers that the Info metadata is more current than the XMP data.<br/> Alternatively, you might prefer to remove the XMP stream from the PDF altogether &ndash; that&#39;s what this option does. Note that objects inside the PDF might have their own, separate XMP metadata streams, and that drop_xmp does not remove those. It only removes the PDF&#39;s document- level XMP stream.<br/> [verbose] By default, pdftk runs quietly. Append verbose to the end and it will speak up.<br/> [dont_ask | do_ask] Depending on the compile-time settings (see ASK_ABOUT_WARNINGS), pdftk might prompt you for further input when it encounters a problem, such as a bad pass&#8208; word. Override this default behavior by adding dont_ask (so pdftk won&#39;t ask you what to do) or do_ask (so pdftk will ask you what to do).<br/> When running in dont_ask mode, pdftk will over-write files with its output without notice.<br/> EXAMPLES Collate scanned pages pdftk A=even.pdf B=odd.pdf shuffle A B output collated.pdf or if odd.pdf is in reverse order: pdftk A=even.pdf B=odd.pdf shuffle A Bend-1 output col&#8208; lated.pdf<br/> The following examples use actual passwords as command line pa&#8208; rameters, which is discouraged (see the SECURITY CONSIDERATIONS section).<br/> Decrypt a PDF pdftk secured.pdf input_pw foopass output unsecured.pdf<br/> Encrypt a PDF using AES-128 (the default), withhold all permis&#8208; sions (the default) pdftk 1.pdf output 1.128.pdf owner_pw foopass<br/> Same as above, except password &#39;baz&#39; must also be used to open output PDF pdftk 1.pdf output 1.128.pdf owner_pw foo user_pw baz<br/> Same as above, except printing is allowed (once the PDF is open) pdftk 1.pdf output 1.128.pdf owner_pw foo user_pw baz allow printing<br/> Apply RCA 40-bit encryption to output, revoking all permissions (the default). Set the owner PW to &#39;foopass&#39;. pdftk 1.pdf 2.pdf cat output 3.pdf encrypt_40bit owner_pw foopass<br/> Join two files, one of which requires the password &#39;foopass&#39;. The output is not encrypted. pdftk A=secured.pdf 2.pdf input_pw A=foopass cat output 3.pdf<br/> Join in1.pdf and in2.pdf into a new PDF, out1.pdf pdftk in1.pdf in2.pdf cat output out1.pdf or (using handles): pdftk A=in1.pdf B=in2.pdf cat A B output out1.pdf or (using wildcards): pdftk *.pdf cat output combined.pdf<br/> Remove page 13 from in1.pdf to create out1.pdf pdftk in.pdf cat 1-12 14-end output out1.pdf or: pdftk A=in1.pdf cat A1-12 A14-end output out1.pdf<br/> Uncompress PDF page streams for editing the PDF in a text edi&#8208; tor (e.g., vim, emacs) pdftk doc.pdf output doc.unc.pdf uncompress<br/> Repair a PDF&#39;s corrupted XREF table and stream lengths, if pos&#8208; sible pdftk broken.pdf output fixed.pdf<br/> Burst a single PDF document into pages and dump its data to doc_data.txt pdftk in.pdf burst<br/> Burst a single PDF document into encrypted pages. Allow low- quality printing pdftk in.pdf burst owner_pw foopass allow DegradedPrinting<br/> Write a report on PDF document metadata and bookmarks to re&#8208; port.txt pdftk in.pdf dump_data output report.txt<br/> Rotate the first PDF page to 90 degrees clockwise pdftk in.pdf cat 1east 2-end output out.pdf<br/> Rotate an entire PDF document to 180 degrees pdftk in.pdf cat 1-endsouth output out.pdf<br/> NOTES This is a port of pdftk to java. See https://gitlab.com/pdftk-java/pdftk The original program can be found at www.pdftk.com<br/> AUTHOR Original author of pdftk is Sid Steward (sid.steward at pdflabs dot com).<br/> SECURITY CONSIDERATIONS Passing a password as a command line parameter is insecure be&#8208; cause it can get saved into the shell&#39;s history and be accessi&#8208; ble by other users via /proc. Use the keyword PROMPT and input any passwords via standard input instead.<br/> December 7, 2020 pdftk.pdftk-java(1) </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region OCR --> <h2 id="ocr">OCR</h2> <p> <a href='https://https://tools.pdf24.org/en/ocr-pdf' target='_blank' rel="nofollow"><code>https://tools.pdf24.org/en/ocr-pdf</code></a> provides fast, free OCR, without ads. </p> <p> This website works well, and offers a total of 30 PDF operations. The other operations of <a href='https://https://tools.pdf24.org/' target='_blank' rel="nofollow"><code>https://tools.pdf24.org/</code></a> include merging, splitting, compressing, editing, signing, redacting, watermarking, locking, unlocking, etc. </p> <!-- endregion --> <!-- #region ops --> <h2 id="ops">PdfTk Operations</h2> <!-- #region rotate one page --> <h3 id="2south">Rotate page 2 only</h3> <p> Given <code>my_file.pdf</code>, a 2-page document: </p> <ol> <li>Do not transform page 1 (keep it as-is)</li> <li>Rotate page 2 only</li> <li>Save as <code>my_file_fixed.pdf</code></li> </ol> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id56f21000a941'><button class='copyBtn' data-clipboard-target='#id56f21000a941' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk my_file.pdf cat 1 2south output my_file_fixed.pdf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region merge --> <h3 id="merge">Merge PDFs</h3> <p> Concatenate <code>file_a.pdf</code>, <code>file_b.pdf</code> and <code>file_c.pdf</code>, then save as <code>big_file.pdf</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd421b3cdf6de'><button class='copyBtn' data-clipboard-target='#idd421b3cdf6de' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk file_a.pdf file_b.pdf file_c.pdf cat output big_file.pdf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region assemble --> <h3 id="assemble">Assemble Book Excerpt</h3> <p> If you need to create multiple excerpts of a book, one way to proceed is to: </p> <ol> <li> Create a PDF containing 3 pages: the front cover, the page containing the <a href='https://en.wikipedia.org/wiki/Edition_notice' target='_blank' rel="nofollow">edition notice</a>, and the back cover. Let's call this PDF file <code>book_wrapper.pdf</code>. </li> <li> Scan each of the excerpts into separate PDFs. Let's call these PDF files <code>book_excerpt_raw_1.pdf</code>, <code>book_excerpt_raw_2.pdf</code>, etc. </li> <li> Wrap the each excerpt within the front cover and edition notice, and the back cover. To do that, concatenate the first 2 pages of <code>book_wrapper.pdf</code>, <code>book_excerpt_raw_N.pdf</code> and the last page of <code>book_wrapper.pdf</code>, then save as <code>book_excerpt_N.pdf</code>. Following is an example of how to do that for 3 excerpts: </li> </ol> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3c56a11278f9'><button class='copyBtn' data-clipboard-target='#id3c56a11278f9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk book_wrapper.pdf 1-2 \ book_excerpt_raw_1.pdf \ book_wrapper.pdf 3 \ cat output book_excerpt_1.pdf <span class='unselectable'>$ </span>pdftk book_wrapper.pdf 1-2 \ book_excerpt_raw_2.pdf \ book_wrapper.pdf 3 \ cat output book_excerpt_2.pdf <span class='unselectable'>$ </span>pdftk book_wrapper.pdf 1-2 \ book_excerpt_raw_3.pdf \ book_wrapper.pdf 3 \ cat output book_excerpt_3.pdf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region cut --> <h3 id="cut">Cut a large PDF into 2 Parts</h3> <p> Cut <code>bigfile.pdf</code> into two parts: </p> <ol> <li><code>bigfile_part1.pdf</code> (containing pages 1-150)</li> <li><code>bigfile_part2.pdf</code> (containing pages 151-350)</li> </ol> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbf7c0057f466'><button class='copyBtn' data-clipboard-target='#idbf7c0057f466' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk A=bigfile.pdf cat 1-150 output bigfile_part1.pdf <span class='unselectable'>$ </span>pdftk A=bigfile.pdf cat 151-end output bigfile_part2.pdf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Move Pages --> <h2 id="double">Interleaving Double-Sided Originals</h2> <p> If you scan a double-sided document, you might end up with the odd-numbered pages in one PDF document, and the even-numbered pages in another. </p> <p> A problem that you will encouter is that when you turn over the document to scan the even-numbered pages, they will be in reverse order because the pages will be scanned from the back to the front of the document. The following examples show two ways of combining the odd- and even-numbered pages in the proper order. </p> <h3 id="manual_reverse">Manually Reversing Even-Numbered Pages</h3> <p> This example shows how to interleave the pages in both documents to create <code>complete.pdf</code>. </p> <ol> <li>Scan the paper document from front to back, so only the odd pages are saved as <code>odd.pdf</code>.</li> <li>Manually reverse the order of the pages in the paper document.</li> <li>Turn the paper document over.</li> <li>Scan the even pages into <code>even.pdf</code>.</li> <li>Run the following incantation:</li> </ol> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5ef2879b6988'><button class='copyBtn' data-clipboard-target='#id5ef2879b6988' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk A=odd.pdf B=even.pdf shuffle A B output complete.pdf<br> <span class='unselectable'>$ </span>rm even.pdf odd.pdf</pre> </div> <!-- endregion --> <h3 id="auto_reverse">Automatically Reversing Even-Numbered Pages</h3> <p> This example does not require you to manually re-order the pages, because <code>pdftk</code> can perform that task. </p> <ol> <li>Scan the document from front to back, so only the odd pages are saved as <code>odd.pdf</code>.</li> <li>Turn over the paper document.</li> <li>Scan the even pages from back to front into <code>even_reversed.pdf</code>.</li> <li>Run the following incantation:</li> </ol> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9b5e49d6182c'><button class='copyBtn' data-clipboard-target='#id9b5e49d6182c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk even_reversed.pdf cat end-1 output even.pdf<br> <span class='unselectable'>$ </span>pdftk A=odd.pdf B=even.pdf shuffle A B output complete.pdf<br> <span class='unselectable'>$ </span>rm even.pdf even_reversed.pdf odd.pdf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region delete 1 page --> <h3 id="cut">Delete a Page</h3> <p> Delete page 69 from <code>bigfile.pdf</code> and save as <code>smaller.pdf</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id620cb54dbeae'><button class='copyBtn' data-clipboard-target='#id620cb54dbeae' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk A=bigfile.pdf cat ~69 output smaller.pdf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region delete several pages --> <h3 id="cut">Delete Several Pages</h3> <p> Delete pages 69-96 from <code>bigfile.pdf</code> and save as <code>smaller.pdf</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8e85b692b35e'><button class='copyBtn' data-clipboard-target='#id8e85b692b35e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk A=bigfile.pdf cat ~69-96 output smaller.pdf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region remove password --> <h2 id="pwd">Remove Password</h2> <p> Assuming <code>protected.pdf</code> is encrypted with password <code>myPwd</code>, make an unencrypted copy and save as <code>output.pdf</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id544a2f568118'><button class='copyBtn' data-clipboard-target='#id544a2f568118' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pdftk protected.pdf input_pw myPwd output output.pdf</pre> </div> <!-- endregion --> <p> You might need to enclose the password within single quotes. </p> <!-- endregion --> <!-- endregion --> <!-- #region Edit Contents --> <h2 id="edit">Edit Contents</h2> <p> Use LibreOffice Writer to edit the PDF, then export as PDF. </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0c1a2d18ec44'><button class='copyBtn' data-clipboard-target='#id0c1a2d18ec44' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>libreoffice --writer my_file.pdf</pre> </div> <!-- endregion --> <!-- endregion --> Pytest and Visual Studio Code 2023-08-20T00:00:00-04:00 https://mslinn.github.io/blog/2023/08/20/pytest <!-- #region intro --> <p> Python has several well-known testing frameworks. Popular choices for functional and unit testing include <a href='https://docs.pytest.org' target='_blank' rel="nofollow"><code>pytest</code></a>, the <a href='https://robotframework.org/' target='_blank' rel="nofollow">Robot framework</a>, and <a href='https://docs.python.org/3/library/unittest.html' target='_blank' rel="nofollow"><code>unittest</code></a>. <a href='https://github.com/gabrielfalcao/lettuce' target='_blank' rel="nofollow">Lettuce</a> and <a href='https://behave.readthedocs.io/en/latest/' target='_blank' rel="nofollow">Behave</a> are popular for behavior-driven testing. </p> <p> I decided to use <code>pytest</code> for a recent Python project. </p> <p> I found that <code>pytest</code>&rsquo;s test discovery mechanism was finicky, and demanded that Python packages and modules be set up before tests could be run. This is in sharp contrast to writing unit tests for Ruby, for example, using <a href='https://rspec.info/' target='_blank' rel="nofollow"><code>rspec</code></a>, because Ruby modules are so much more flexible, forgiving and malleable. </p> <p> However, getting <code>pytest</code> to work in Visual Studio Code was nearly impossible. Microsoft's documentation leaves a lot unsaid. I share what I learned, and how I made things work in this article. </p> <!-- endregion --> <!-- #region pytest --> <h2 id="pytest">Pytest</h2> <p> You can run <code>pytest</code> from the command line 3 ways. </p> <!-- #region pytest command --> <h3 id="cmd"><span class="code">Pytest</span> Command</h3> <p> The most common way to run <code>pytest</code> is to type the <code>pytest</code> command, like the following, which runs all tests: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8d45fa086ca7'><button class='copyBtn' data-clipboard-target='#id8d45fa086ca7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pytest</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region module --> <h3 id="module"><span class="code">Pytest</span> Module</h3> <p> The second way to run <code>pytest</code> is to invoke the <code>pytest</code> Python module, as follows: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id347b690f89dd'><button class='copyBtn' data-clipboard-target='#id347b690f89dd' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>python -m pytest</pre> </div> <!-- endregion --> <p> Running <code>pytest</code> as a Python module is nearly the same as running the <code>pytest</code> command, except that running it as a module adds the current directory to <code>sys.path</code>, which is standard python behavior, and can be helpful. </p> <p> I had best results running <code>pytest</code> tests from Visual Studio Code when <code>pytest</code> was invoked as a module. </p> <!-- endregion --> <!-- #region calling from python --> <h3 id="pgm">Calling <span class="code">Pytest</span> from Python</h3> <p> Your code can <a href='https://docs.pytest.org/en/7.1.x/how-to/usage.html#calling-pytest-from-python-code' target='_blank' rel="nofollow">call <code>pytest</code></a>. Lots of interesting and advanced development setups could be crafted using this approach. Baby steps. </p> <!-- endregion --> <!-- #region help --> <h3 id="help">Help Message</h3> <p> The help message for <code>pytest</code> is: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcd723d738be6'><button class='copyBtn' data-clipboard-target='#idcd723d738be6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pytest -h <span class='unselectable'>usage: pytest [options] [file_or_dir] [file_or_dir] [...]<br/> positional arguments: file_or_dir<br/> general: -k EXPRESSION Only run tests which match the given substring expression. An expression is a Python evaluatable expression where all names are substring-matched against test names and their parent classes. Example: -k &#39;test_method or test_other&#39; matches all test functions and classes whose name contains &#39;test_method&#39; or &#39;test_other&#39;, while -k &#39;not test_method&#39; matches those that don&#39;t contain &#39;test_method&#39; in their names. -k &#39;not test_method and not test_other&#39; will eliminate the matches. Additionally keywords are matched to classes and functions containing extra names in their &#39;extra_keyword_matches&#39; set, as well as functions which have names assigned directly to them. The matching is case-insensitive. -m MARKEXPR Only run tests matching given mark expression. For example: -m &#39;mark1 and not mark2&#39;. --markers show markers (builtin, plugin and per-project ones). -x, --exitfirst Exit instantly on first error or failed test --fixtures, --funcargs Show available fixtures, sorted by plugin appearance (fixtures with leading &#39;_&#39; are only shown with &#39;-v&#39;) --fixtures-per-test Show fixtures per test --pdb Start the interactive Python debugger on errors or KeyboardInterrupt --pdbcls=modulename:classname Specify a custom interactive Python debugger for use with --pdb.For example: --pdbcls=IPython.terminal.debugger:TerminalPdb --trace Immediately break when running each test --capture=method Per-test capturing method: one of fd|sys|no|tee-sys -s Shortcut for --capture=no --runxfail Report the results of xfail tests as if they were not marked --lf, --last-failed Rerun only the tests that failed at the last run (or all if none failed) --ff, --failed-first Run all tests, but run the last failures first. This may re-order tests and thus lead to repeated fixture setup/teardown. --nf, --new-first Run tests from new files first, then the rest of the tests sorted by file mtime --cache-show=[CACHESHOW] Show cache contents, don&#39;t perform collection or tests. Optional argument: glob (default: &#39;*&#39;). --cache-clear Remove all cache contents at start of test run --lfnf={all,none}, --last-failed-no-failures={all,none} Which tests to run with no previously (known) failures --sw, --stepwise Exit on test failure and continue from last failing test next time --sw-skip, --stepwise-skip Ignore the first failing test but stop on the next failing test. Implicitly enables --stepwise.<br/> Reporting: --durations=N Show N slowest setup/test durations (N=0 for all) --durations-min=N Minimal duration in seconds for inclusion in slowest list. Default: 0.005. -v, --verbose Increase verbosity --no-header Disable header --no-summary Disable summary -q, --quiet Decrease verbosity --verbosity=VERBOSE Set verbosity. Default: 0. -r chars Show extra test summary info as specified by chars: (f)ailed, (E)rror, (s)kipped, (x)failed, (X)passed, (p)assed, (P)assed with output, (a)ll except passed (p/P), or (A)ll. (w)arnings are enabled by default (see --disable-warnings), &#39;N&#39; can be used to reset the list. (default: &#39;fE&#39;). --disable-warnings, --disable-pytest-warnings Disable warnings summary -l, --showlocals Show locals in tracebacks (disabled by default) --no-showlocals Hide locals in tracebacks (negate --showlocals passed through addopts) --tb=style Traceback print mode (auto/long/short/line/native/no) --show-capture={no,stdout,stderr,log,all} Controls how captured stdout/stderr/log is shown on failed tests. Default: all. --full-trace Don&#39;t cut any tracebacks (default is to cut) --color=color Color terminal output (yes/no/auto) --code-highlight={yes,no} Whether code should be highlighted (only if --color is also enabled). Default: yes. --pastebin=mode Send failed|all info to bpaste.net pastebin service --junit-xml=path Create junit-xml style report file at given path --junit-prefix=str Prepend prefix to classnames in junit-xml output<br/> pytest-warnings: -W PYTHONWARNINGS, --pythonwarnings=PYTHONWARNINGS Set which warnings to report, see -W option of Python itself --maxfail=num Exit after first num failures or errors --strict-config Any warnings encountered while parsing the `pytest` section of the configuration file raise errors --strict-markers Markers not registered in the `markers` section of the configuration file raise errors --strict (Deprecated) alias to --strict-markers -c FILE, --config-file=FILE Load configuration from `FILE` instead of trying to locate one of the implicit configuration files. --continue-on-collection-errors Force test execution even if collection errors occur --rootdir=ROOTDIR Define root directory for tests. Can be relative path: &#39;root_dir&#39;, &#39;./root_dir&#39;, &#39;root_dir/another_dir/&#39;; absolute path: &#39;/home/user/root_dir&#39;; path with variables: &#39;$HOME/root_dir&#39;.<br/> collection: --collect-only, --co Only collect tests, don&#39;t execute them --pyargs Try to interpret all arguments as Python packages --ignore=path Ignore path during collection (multi-allowed) --ignore-glob=path Ignore path pattern during collection (multi-allowed) --deselect=nodeid_prefix Deselect item (via node id prefix) during collection (multi-allowed) --confcutdir=dir Only load conftest.py&#39;s relative to specified dir --noconftest Don&#39;t load any conftest.py files --keep-duplicates Keep duplicate tests --collect-in-virtualenv Don&#39;t ignore tests in a local virtualenv directory --import-mode={prepend,append,importlib} Prepend/append to sys.path when importing test modules and conftest files. Default: prepend. --doctest-modules Run doctests in all .py modules --doctest-report={none,cdiff,ndiff,udiff,only_first_failure} Choose another output format for diffs on doctest failure --doctest-glob=pat Doctests file matching pattern, default: test*.txt --doctest-ignore-import-errors Ignore doctest ImportErrors --doctest-continue-on-failure For a given doctest, continue to run after the first failure<br/> test session debugging and configuration: --basetemp=dir Base temporary directory for this test run. (Warning: this directory is removed if it exists.) -V, --version Display pytest version and information about plugins. When given twice, also display information about plugins. -h, --help Show help message and configuration info -p name Early-load given plugin module name or entry point (multi-allowed). To avoid loading of plugins, use the `no:` prefix, e.g. `no:doctest`. --trace-config Trace considerations of conftest.py files --debug=[DEBUG_FILE_NAME] Store internal tracing debug information in this log file. This file is opened with &#39;w&#39; and truncated as a result, care advised. Default: pytestdebug.log. -o OVERRIDE_INI, --override-ini=OVERRIDE_INI Override ini option with &quot;option=value&quot; style, e.g. `-o xfail_strict=True -o cache_dir=cache`. --assert=MODE Control assertion debugging tools. &#39;plain&#39; performs no assertion debugging. &#39;rewrite&#39; (the default) rewrites assert statements in test modules on import to provide assert expression information. --setup-only Only setup fixtures, do not execute tests --setup-show Show setup of fixtures while executing tests --setup-plan Show what fixtures and tests would be executed but don&#39;t execute anything<br/> logging: --log-level=LEVEL Level of messages to catch/display. Not set by default, so it depends on the root/parent log handler&#39;s effective level, where it is &quot;WARNING&quot; by default. --log-format=LOG_FORMAT Log format used by the logging module --log-date-format=LOG_DATE_FORMAT Log date format used by the logging module --log-cli-level=LOG_CLI_LEVEL CLI logging level --log-cli-format=LOG_CLI_FORMAT Log format used by the logging module --log-cli-date-format=LOG_CLI_DATE_FORMAT Log date format used by the logging module --log-file=LOG_FILE Path to a file when logging will be written to --log-file-level=LOG_FILE_LEVEL Log file logging level --log-file-format=LOG_FILE_FORMAT Log format used by the logging module --log-file-date-format=LOG_FILE_DATE_FORMAT Log date format used by the logging module --log-auto-indent=LOG_AUTO_INDENT Auto-indent multiline messages passed to the logging module. Accepts true|on, false|off or an integer. --log-disable=LOGGER_DISABLE Disable a logger by name. Can be passed multiple times.<br/> [pytest] ini-options in the first pytest.ini|tox.ini|setup.cfg|pyproject.toml file found:<br/> markers (linelist): Markers for test functions empty_parameter_set_mark (string): Default marker for empty parametersets norecursedirs (args): Directory patterns to avoid for recursion testpaths (args): Directories to search for tests when no files or directories are given on the command line filterwarnings (linelist): Each line specifies a pattern for warnings.filterwarnings. Processed after -W/--pythonwarnings. usefixtures (args): List of default fixtures to be used with this project python_files (args): Glob-style file patterns for Python test module discovery python_classes (args): Prefixes or glob names for Python test class discovery python_functions (args): Prefixes or glob names for Python test function and method discovery disable_test_id_escaping_and_forfeit_all_rights_to_community_support (bool): Disable string escape non-ASCII characters, might cause unwanted side effects(use at your own risk) console_output_style (string): Console output: &quot;classic&quot;, or with additional progress information (&quot;progress&quot; (percentage) | &quot;count&quot; | &quot;progress-even-when-capture-no&quot; (forces progress even when capture=no) xfail_strict (bool): Default for the strict parameter of xfail markers when not given explicitly (default: False) tmp_path_retention_count (string): How many sessions should we keep the `tmp_path` directories, according to `tmp_path_retention_policy`. tmp_path_retention_policy (string): Controls which directories created by the `tmp_path` fixture are kept around, based on test outcome. (all/failed/none) enable_assertion_pass_hook (bool): Enables the pytest_assertion_pass hook. Make sure to delete any previously generated pyc cache files. junit_suite_name (string): Test suite name for JUnit report junit_logging (string): Write captured log messages to JUnit report: one of no|log|system-out|system-err|out-err|all junit_log_passing_tests (bool): Capture log information for passing tests to JUnit report: junit_duration_report (string): Duration time to report: one of total|call junit_family (string): Emit XML for schema: one of legacy|xunit1|xunit2 doctest_optionflags (args): Option flags for doctests doctest_encoding (string): Encoding used for doctest files cache_dir (string): Cache directory path log_level (string): Default value for --log-level log_format (string): Default value for --log-format log_date_format (string): Default value for --log-date-format log_cli (bool): Enable log display during test run (also known as &quot;live logging&quot;) log_cli_level (string): Default value for --log-cli-level log_cli_format (string): Default value for --log-cli-format log_cli_date_format (string): Default value for --log-cli-date-format log_file (string): Default value for --log-file log_file_level (string): Default value for --log-file-level log_file_format (string): Default value for --log-file-format log_file_date_format (string): Default value for --log-file-date-format log_auto_indent (string): Default value for --log-auto-indent pythonpath (paths): Add paths to sys.path faulthandler_timeout (string): Dump the traceback of all threads if a test takes more than TIMEOUT seconds to finish addopts (args): Extra command line options minversion (string): Minimally required pytest version required_plugins (args): Plugins that must be present for pytest to run<br/> Environment variables: PYTEST_ADDOPTS Extra command line options PYTEST_PLUGINS Comma-separated plugins to load during startup PYTEST_DISABLE_PLUGIN_AUTOLOAD Set to disable plugin auto-loading PYTEST_DEBUG Set to enable debug tracing of pytest&#39;s internals<br/><br/> to see available markers type: pytest --markers to see available fixtures type: pytest --fixtures (shown according to specified file_or_dir or current dir if not specified; fixtures with leading &#39;_&#39; are only shown with the &#39;-v&#39; option </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region helpful options --> <h3 id="options">Helpful Options</h3> <p> It is handy to run new and failing tests, but to skip previously passing tests. The <code>&#8209;&#8209;nf</code> and <code>&#8209;&#8209;lf</code> options, respectively, invoke that mode of behavior: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb79544a40dd3'><button class='copyBtn' data-clipboard-target='#idb79544a40dd3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pytest --nf --lf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- endregion --> <!-- #region setup --> <h2 id="setup">Setup</h2> <!-- #region config --> <h3 id="config">Configuration</h3> <p> <code>Pytest</code> needed to know that the source code for my Python project is stored in the <code>src</code> directory. I created <code>pyproject.toml</code> as follows, which configured <code>pytest</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>pyproject.toml</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id25bf4876b24e'><button class='copyBtn' data-clipboard-target='#id25bf4876b24e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[tool.pytest.ini_options] addopts = [ "--import-mode=importlib", ] pythonpath = "src"</pre> </div> <!-- endregion --> <p> The way in which <code>pytest</code> discovers tests is <a href='https://docs.pytest.org/en/7.1.x/explanation/pythonpath.htm' target='_blank' rel="nofollow">complex</a>. <a href='https://www.merriam-webster.com/dictionary/TL%3BDR' target='_blank' rel="nofollow">TL;DR</a>: specify the newest and best mechanism for test discovery by using the <code>&#8209;&#8209;import-mode=importlib</code> option as shown above. </p> <p> To always run <code>pytest</code> with the <code>&#8209;&#8209;nf</code> and <code>&#8209;&#8209;lf</code> options, add those options to <code>pyproject.toml</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>pyproject.toml</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id44d109306c08'><button class='copyBtn' data-clipboard-target='#id44d109306c08' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[tool.pytest.ini_options] addopts = [ "--import-mode=importlib", <span class='bg_yellow'>"--lf",</span> <span class='bg_yellow'>"--nf",</span> ] pythonpath = "src"</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Test Directories --> <h3 id="dirs">Test Directories</h3> <p> Flexibility generally adds complexity. <code>Pytest</code> allows for a lot of flexibility regarding the location of where tests can reside. Unfortunately, Python&rsquo;s package and module mechanism in combination with the <code>pytest</code> discovery mechanism for tests can lead to a frustrating experience when setting up a new project. </p> <p> You can read about <a href='https://pytest.org/en/7.4.x/explanation/goodpractices.html' target='_blank' rel="nofollow">good practices for <code>pytest</code></a>. Those recommendations are intimidating for first-time <code>pytest</code> programmers. I did not follow all of them. </p> <p> I found the simplest approach was to create a subdirectory for the bulk of the Python project code. If a file called <code>__init__.py</code> is created in that directory (use <a href='https://man7.org/linux/man-pages/man1/touch.1.html' target='_blank' rel="nofollow"><code>touch</code></a> for this) then the subdirectory becomes a submodule of your Python program. For this type of <code>pytest</code> setup, each submodule needs a <code>tests</code> directory. </p> <p> Here is how you could use <code>touch</code> to define an imaginary subdirectory/submodule called <code>my_submodule</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4d4a4ced9958'><button class='copyBtn' data-clipboard-target='#id4d4a4ced9958' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mkdir -p src/my_submodule/<br> <span class='unselectable'>$ </span>touch src/my_submodule/__init__.py</pre> </div> <!-- endregion --> <p> My project structure was as follows. <a href='https://linux.die.net/man/1/tree' target='_blank' rel="nofollow"><code>Tree</code> docs are here.</a> </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>pytest project structure</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer tree' id='id6371a8682850'><button class='copyBtn' data-clipboard-target='#id6371a8682850' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>tree -I .git -I __pycache__ <span class='unselectable'>. ├── README.md ├── dl ├── dl.config ├── pyproject.toml ├── requirements.txt └── src ├── __main__.py └── dl ├── __init__.py ├── argument_parse.py ├── dl_config.py ├── media_file.py ├── remote.py ├── tests │   ├── __init__.py │   ├── test_dl_config.py │   └── test_uti.py └── util.py 3 directories, 15 files </span></pre> </div> <!-- endregion --> <p> <code>src/dl/tests/__init__.py</code> was an empty marker file, but the <code>___init__.py</code> file in the <a href='https://docs.python.org/3/tutorial/modules.html' target='_blank' rel="nofollow">parent module</a> defined <code>import</code>s for the submodule: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>src/dl/__init__.py</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2032c248b28e'><button class='copyBtn' data-clipboard-target='#id2032c248b28e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>from dl.argument_parse import ArgParse from dl.dl_config import DLConfig from dl.media_file import MediaFile from dl.remote import Remote from dl.util import *</pre> </div> <!-- endregion --> <!-- endregion --> <!-- endregion --> <!-- #region sample test --> <h2 id="test">Sample Test</h2> <p> Here is a sample <code>pytest</code> file, containing 2 unit tests. It is simple and clear. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>src/dl/tests/test_util.py</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4c68c5bb11c6'><button class='copyBtn' data-clipboard-target='#id4c68c5bb11c6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>import pytest from pathlib import Path from dl import samba_mount, samba_parse class TestUtil: def test_samba_parse(self): remote_drive, local_path = samba_parse('e:/media/renders') assert remote_drive=='e' assert local_path=='/media/renders' def test_samba_mount(self): samba_root = samba_mount('bear', 'e', False) assert samba_root==Path('/mnt/bear/e') assert samba_root.is_mount()</pre> </div> <!-- endregion --> <h2 id="rap">Secret Sauce</h2> <p> If you have been paying attention, you should realize that one option for running new and failing tests is to type the following from the command line. We will use this method of invocation in the next section to integrate with Visual Studio Code. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id007c457d1976'><button class='copyBtn' data-clipboard-target='#id007c457d1976' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>python -m pytest --nf --lf</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region vscode --> <h2 id="vscode">Visual Studio Code Integration</h2> <p> Visual Studio Code has a Testing explorer and a Run and Debug explorer. I find the existance of two very similar explorers annoying, because you often want to debug a test. Yes, you can do that by pushing the right button in the Testing Explorer ... if it works. This did not work for me. </p> <p> <a href='https://marketplace.visualstudio.com/items?itemName=Shopify.ruby-extensions-pack' target='_blank' rel="nofollow">Shopify Ruby</a>, the latest and greatest Ruby extension for Visual Studio Code, uses the Run and Debug explorer to run and debug tests as well as entire programs. This makes sense to me. </p> <p> <a href='https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance' target='_blank' rel="nofollow">Microsoft Pylance</a> (for Python) follows a different convention, and uses the Run and Debug explorer to run and debug Python programs. You must use the Testing explorer for running or debugging Python unit tests. Annoying. </p> <p> In the Testing explorer, I was able to see my <code>dl</code> module, but no tests were discovered. This was really annoying, and this is where more documentation from Microsoft would have been helpful. After wasting lots of time trying to solve this problem, I discovered a solution. </p> <h3 id="">Invoke <span class="code">Pytest</span> As a Module from VSCode</h3> <p> Run and Debug configurations that invoke the <code>pytest</code> module work perfectly. Here is an example <code>.vscode/launch.json</code> that demonstrates this: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>.vscode/launch.json</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8564f2a96ca1'><button class='copyBtn' data-clipboard-target='#id8564f2a96ca1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{ "version": "0.2.0", "configurations": [ { "name": "PyTest All", "type": "python", "request": "launch", "module": "pytest", "justMyCode": true }, { "args": ["--nf", "--lf"], "name": "PyTest New and Failing", "type": "python", "request": "launch", "module": "pytest", "justMyCode": true }, ] }</pre> </div> <!-- endregion --> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <p> You can set breakpoints in your code and debug your tests this way. </p> <!-- endregion --> C and C++ Online and On Ubuntu 2023-05-12T00:00:00-04:00 https://mslinn.github.io/blog/2023/05/12/c <!-- #region intro --> <div class="one_column right quartersize"> <div class='imgWrapper imgFlex inline' style=' '> <figure> <a href='/blog/c/Lattice_C_8086_1982.pdf' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/c/lattice_c.svg" type="image/svg"> <!---<source srcset="/blog/c/lattice_c.avif" type="image/avif">--> <source srcset="/blog/c/lattice_c.webp" type="image/webp"> <source srcset="/blog/c/lattice_c.apng" type="image/apng"> <source srcset="/blog/c/lattice_c.png" type="image/png"> <source srcset="/blog/c/lattice_c.jpg" type="image/jpeg"> <source srcset="/blog/c/lattice_c.jpeg" type="image/jpeg"> <source srcset="/blog/c/lattice_c.jfif" type="image/jpeg"> <source srcset="/blog/c/lattice_c.pjpeg" type="image/jpeg"> <source srcset="/blog/c/lattice_c.pjp" type="image/jpeg"> <source srcset="/blog/c/lattice_c.gif" type="image/gif"> <source srcset="/blog/c/lattice_c.tif" type="image/tiff"> <source srcset="/blog/c/lattice_c.tiff" type="image/tiff"> <source srcset="/blog/c/lattice_c.bmp" type="image/bmp"> <source srcset="/blog/c/lattice_c.ico" type="image/x-icon"> <source srcset="/blog/c/lattice_c.cur" type="image/x-icon"> <img alt='Lattice C' class="imgImg rounded shadow" src="/blog/c/lattice_c.png" style='width: 100%; ' title='Lattice C' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="/blog/c/Lattice_C_8086_1982.pdf" target='_blank' > Lattice C </a> </figcaption> </figure> </div> <div class='imgWrapper imgFlex inline' style=' '> <figure> <picture class='imgPicture'> <source srcset="/blog/c/k&r.svg" type="image/svg"> <!---<source srcset="/blog/c/k&r.avif" type="image/avif">--> <source srcset="/blog/c/k&r.webp" type="image/webp"> <source srcset="/blog/c/k&r.apng" type="image/apng"> <source srcset="/blog/c/k&r.png" type="image/png"> <source srcset="/blog/c/k&r.jpg" type="image/jpeg"> <source srcset="/blog/c/k&r.jpeg" type="image/jpeg"> <source srcset="/blog/c/k&r.jfif" type="image/jpeg"> <source srcset="/blog/c/k&r.pjpeg" type="image/jpeg"> <source srcset="/blog/c/k&r.pjp" type="image/jpeg"> <source srcset="/blog/c/k&r.gif" type="image/gif"> <source srcset="/blog/c/k&r.tif" type="image/tiff"> <source srcset="/blog/c/k&r.tiff" type="image/tiff"> <source srcset="/blog/c/k&r.bmp" type="image/bmp"> <source srcset="/blog/c/k&r.ico" type="image/x-icon"> <source srcset="/blog/c/k&r.cur" type="image/x-icon"> <img alt='K&R Book' class="imgImg rounded shadow" src="/blog/c/k&r.png" style='width: 100%; ' title='K&R Book' /> </picture> <figcaption class='imgFigCaption '> K&R Book </figcaption> </figure> </div> <div class='imgWrapper imgFlex inline' style=' '> <figure> <a href='https://developerinsider.co/download-turbo-c-for-windows-7-8-8-1-and-windows-10-32-64-bit-full-screen/' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/c/turbo_c_2.svg" type="image/svg"> <!---<source srcset="/blog/c/turbo_c_2.avif" type="image/avif">--> <source srcset="/blog/c/turbo_c_2.webp" type="image/webp"> <source srcset="/blog/c/turbo_c_2.apng" type="image/apng"> <source srcset="/blog/c/turbo_c_2.png" type="image/png"> <source srcset="/blog/c/turbo_c_2.jpg" type="image/jpeg"> <source srcset="/blog/c/turbo_c_2.jpeg" type="image/jpeg"> <source srcset="/blog/c/turbo_c_2.jfif" type="image/jpeg"> <source srcset="/blog/c/turbo_c_2.pjpeg" type="image/jpeg"> <source srcset="/blog/c/turbo_c_2.pjp" type="image/jpeg"> <source srcset="/blog/c/turbo_c_2.gif" type="image/gif"> <source srcset="/blog/c/turbo_c_2.tif" type="image/tiff"> <source srcset="/blog/c/turbo_c_2.tiff" type="image/tiff"> <source srcset="/blog/c/turbo_c_2.bmp" type="image/bmp"> <source srcset="/blog/c/turbo_c_2.ico" type="image/x-icon"> <source srcset="/blog/c/turbo_c_2.cur" type="image/x-icon"> <img alt='Borland Turbo C' class="imgImg rounded shadow" src="/blog/c/turbo_c_2.png" style='width: 100%; ' title='Borland Turbo C' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://developerinsider.co/download-turbo-c-for-windows-7-8-8-1-and-windows-10-32-64-bit-full-screen/" target='_blank' > Borland Turbo C </a> </figcaption> </figure> </div> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/zamples.svg" type="image/svg"> <!---<source srcset="/blog/images/zamples.avif" type="image/avif">--> <source srcset="/blog/images/zamples.webp" type="image/webp"> <source srcset="/blog/images/zamples.apng" type="image/apng"> <source srcset="/blog/images/zamples.png" type="image/png"> <source srcset="/blog/images/zamples.jpg" type="image/jpeg"> <source srcset="/blog/images/zamples.jpeg" type="image/jpeg"> <source srcset="/blog/images/zamples.jfif" type="image/jpeg"> <source srcset="/blog/images/zamples.pjpeg" type="image/jpeg"> <source srcset="/blog/images/zamples.pjp" type="image/jpeg"> <source srcset="/blog/images/zamples.gif" type="image/gif"> <source srcset="/blog/images/zamples.tif" type="image/tiff"> <source srcset="/blog/images/zamples.tiff" type="image/tiff"> <source srcset="/blog/images/zamples.bmp" type="image/bmp"> <source srcset="/blog/images/zamples.ico" type="image/x-icon"> <source srcset="/blog/images/zamples.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/zamples.png" style='width: 100%; ' /> </picture> </div> </div> <p> Recently I decided to brush up on my C programming. </p> <p> I first discovered C during a visit in 1978 to the <a href='https://cs.berkeley.edu/' target='_blank' rel="nofollow">Computer Science department of University of California, Berkeley</a>, where I also encounted UNIX for the first time. </p> <p> From about 1983 to 1985 I was the unofficial Canadian distributor of the <a href='/blog/c/Lattice_C_8086_1982.pdf'>Lattice C compiler</a>. I used that compiler for most of the programming work I was doing on PCs at the time. FedEx Express Canada did not come into being until 1987, UPS was even worse with the Canadian border than it is now, and Canada Post took forever to bring packages across the border. I bought in bulk at a discount, and sold at list price. Distribution is all about time and place. </p> <p> I also sold the first edition of the K&R book (&ldquo;The C Programming Language&rdquo;) via mail order, and I provided technical support, which brought in interesting consulting work. </p> <p> The Borland Turbo C compiler, released in 1990, was the original IDE. It was eventually replaced by Borland C++ compiler, which also compiled C. In 2005 I became the product marketing manager for most of Borland&rsquo;s compilers. One of my responsibilities was to give free copies to book authors and members of the trade press. Borland C++ shipped in a massive box that was responsible for a lot of dead trees. </p> <p> The C language has evolved since K&R days. The current and previous iterations are well summarized on <a href='https://en.cppreference.com/w/c/language' target='_blank' rel="nofollow"><code>cppreference.com</code></a>, which also provides an online C/C++ IDE for sample code. </p> <!-- endregion --> <!-- #region Revisions --> <h2 id="revisions">Revision History</h2> <p> 1978: <b>K&R book</b> first published. </p> <p> 1989: <b>C89</b>, also known as ANSI X3. 159-1989, or ANSI C. </p> <p>1995: <b>NA1</b> (aka C94 and C95) added support for wide-characters, wide-strings, and international keyboard layouts. </p> <p>1999: <b>C99</b> added restricted pointers, variable-length arrays, flexible array members, complex numbers support, type-generic math, <code>long long int</code>, extended identifiers, hexadecimal floating-point constants, compound literals, designated initializers, single line comments, extended integer type, the ability to mix declarations and code, variadic macros, new math functions, inline functions, boolean types, <code>_Pragma</code> and standard pragmas, and <code>VA_COPY</code>. C99 removed implicit function declaration. </p> <p>2011: <b>C11</b> added concurrency support, type-generic expressions, alignment-related facilities, static assertion, Unicode support, floating-point characteristic macros, no-return functions, anonymous structures and unions, bound-checking, and reentrancy functions. </p> <p>2018: <b>C18</b> addressed defects in C11 without introducing new language features. </p> <p>2023: The most recent publicly available draft of the next release, <a href='https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3096.pdf' target='_blank' rel="nofollow"><b>C23</b></a>, was released on April 1, 2023. Unfortunately, that document was written for insiders, and is very difficult to read. Happily, JeanHeyd Meneide, the Project Editor for ISO/IEC JTC1 SC22 WG14 - Programming Languages, C, wrote an <a href='https://thephd.dev/c23-is-coming-here-is-what-is-on-the-menu' target='_blank' rel="nofollow">easy-to-read <wbr>summary of C23</a>. </p> <h2 id="linux">Linux Is Written in C</h2> <p> It has been <a href='https://www.zdnet.com/article/linus-torvalds-prepares-to-move-the-linux-kernel-to-modern-c/' target='_blank' rel="nofollow">reported</a> that Linux was originally written in C89, but Linux moved to C11 in 2022. I am curious as to why C18 is not used, since C18 just contains bug fixes for C11. Most of the Linux kernel code is written using the <a href='https://gcc.gnu.org/onlinedocs/gcc-10.2.0/gcc/C-Extensions.html#C-Extensions' target='_blank' rel="nofollow">GNU extensions of GCC</a>. </p> <!-- endregion --> <!-- #region online --> <h2 id="online">Online IDEs</h2> <p> I pioneered online programming with SMX Explorer in 1994, <a href='/resume/history/jspexplorer/developerWorks/'>JSP Explorer</a> in 2000, and supported most available languages in 2002 with <a href='/blog/2004/12/09/live-code-examples-for-j2se-6-mustang.html'>Zamples.com</a>. </p> <p> Now it seems that every day another online programming option becomes available. Many are free: </p> <ul> <li> <a href='https://coliru.stacked-crooked.com/' target='_blank' rel="nofollow">Coliru</a> is used by <code>cppreference.com</code>; <a href='https://github.com/StackedCrooked/coliru' target='_blank' rel="nofollow">GitHub project</a> </li> <li> <a href='https://github.com/features/codespaces' target='_blank' rel="nofollow">GitHub Codespaces</a> is tightly integrated with Microsoft Azure. </li> <li><a href='https://www.jetbrains.com/space/' target='_blank' rel="nofollow">JetBrains Space</a></li> <li> <a href='https://www.onlinegdb.com/online_c_compiler' target='_blank' rel="nofollow">OnlineGDB</a> claims to be to first Online IDE, but the <a href='https://web.archive.org/web/20160815000000*/OnlineGDB.com' target='_blank' rel="nofollow">WayBack Machine</a> shows that this site was created in 2016. </li> <li><a href='https://www.programiz.com/c-programming/online-compiler/' target='_blank' rel="nofollow">Programiz</a></li> <li><a href='https://www.tutorialspoint.com/compile_c_online.php' target='_blank' rel="nofollow">TutorialsPoint CodingGround</a></li> <li><a href='https://replit.com/languages/c' target='_blank' rel="nofollow">Replit</a></li> <li><a href='https://onecompiler.com/c' target='_blank' rel="nofollow">OneCompiler</a></li> <li>... and <a href='https://www.google.com/search?q=online+c+programming' target='_blank' rel="nofollow">many more</a> ... <a href='https://arne-mertz.de/2017/05/online-compilers/' target='_blank' rel="nofollow">even more!</a></li> </ul> <!-- endregion --> <!-- #region install --> <h2 id="install">Installing C/C++ on WSL/Ubuntu</h2> <p> Online programming is fine for learning, but I prefer to be independent and self-reliant whenever practical for doing actual work. </p> <h3 id="cli">Command-Line Toolchain</h3> <p> To program in C or C++ on Ubuntu using the traditional Linux command-line tool&shy;chain, install the <a href='https://packages.debian.org/sid/build-essential' target='_blank' rel="nofollow"><code>build-essential</code></a> Debian meta-package on Ubuntu. It includes the <a href='https://gcc.gnu.org/' target='_blank' rel="nofollow">GNU Compiler Collection (GCC)</a>, which is a collection of compilers and libraries for C, C++, Objective-C, Fortran, Ada, Go, and D programming languages. It also includes <a href='https://linux.die.net/man/1/make' target='_blank' rel="nofollow"><code>make</code></a> and <a href='https://packages.debian.org/sid/libc6.1-dev' target='_blank' rel="nofollow"><code>libc6.1-dev</code></a>, the GNU C development libraries and header files. </p> <p> You probably should also install <a href='https://www.sourceware.org/gdb/' target='_blank' rel="nofollow"><code>gdb</code></a>, the GNU Project Debugger, and the <a href='https://packages.debian.org/unstable/manpages-dev' target='_blank' rel="nofollow">manual pages</a> about using GNU/Linux for development. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd83142e0ea46'><button class='copyBtn' data-clipboard-target='#idd83142e0ea46' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install build-essential gdb manpages-dev</pre> </div> <!-- endregion --> <p> Now you are able to compile and run C programs at the command line: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide70443f46faf'><button class='copyBtn' data-clipboard-target='#ide70443f46faf' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mkdir test && cd test <span class='unselectable'>$ </span>cat &gt;test.c &lt;&lt;EOF #include &lt;stdio.h&gt; int main(void) { printf("Hello, World!\n"); } EOF <span class='unselectable'>$ </span>gcc test.c -o test && ./test <span class='unselectable'>Hello, World! </span></pre> </div> <!-- endregion --> <h3 id="vscode">Visual Studio Code</h3> <p> Visual Studio Code is a high-quality (and free) IDE. To work with C and C++ in Visual Studio Code, install the <a href='https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools' target='_blank' rel="nofollow">Microsoft C/C++</a> extension. </p> <!-- endregion --> Don't Poke the Bear 2023-03-03T00:00:00-05:00 https://mslinn.github.io/blog/2023/03/03/lego-tm <!-- #region --> <h2 style="margin-bottom: 2em; margin-top: 2em;"> To the developers of the excellent &lsquo;<a href='https://github.com/go-acme/lego/' class='code>lego</span>' target='_blank' rel="nofollow"><span</a>&rsquo; GitHub project </h2> <div class='imgWrapper imgFlex inline' style=' '> <a href='https://github.com/go-acme/lego' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/ddns/lego-logo.svg" type="image/svg"> <!---<source srcset="/blog/images/ddns/lego-logo.avif" type="image/avif">--> <source srcset="/blog/images/ddns/lego-logo.webp" type="image/webp"> <source srcset="/blog/images/ddns/lego-logo.apng" type="image/apng"> <source srcset="/blog/images/ddns/lego-logo.png" type="image/png"> <source srcset="/blog/images/ddns/lego-logo.jpg" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.jpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.jfif" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.pjpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.pjp" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.gif" type="image/gif"> <source srcset="/blog/images/ddns/lego-logo.tif" type="image/tiff"> <source srcset="/blog/images/ddns/lego-logo.tiff" type="image/tiff"> <source srcset="/blog/images/ddns/lego-logo.bmp" type="image/bmp"> <source srcset="/blog/images/ddns/lego-logo.ico" type="image/x-icon"> <source srcset="/blog/images/ddns/lego-logo.cur" type="image/x-icon"> <img class="imgImg " src="/blog/images/ddns/lego-logo.png" style='width: 100%; ' /> </picture> </a> </div> <p> The name of this project is problematic, and you <a href='https://github.com/go-acme/lego/issues/470' target='_blank' rel="nofollow">publicly dismissed</a> the issue when it was brought to your attention. While ignorance of the law is no defense; demonstrably flaunting another party&rsquo;s rights never has positive consequences. </p> <p> <a href='https://www.lego.com/en-sg/service/help/fun-for-fans/behind-the-scenes/our-company/commercial-use-of-lego-products-bltb4ad5090ec5e4b08' target='_blank' rel="nofollow">The LEGO Group</a> has not taken legal action yet, but the laws of most countries (including the US) are that <a href='https://www.forbes.com/sites/oliverherzfeld/2013/02/28/failure-to-enforce-trademarks-if-you-snooze-do-you-lose/?sh=1d67ec326c22' target='_blank' rel="nofollow">if a trademark is not defended, then the trademark is at risk of being lost</a>. </p> <p> You should expect to receive a legal notice at some point. Not because the trademark owners are evil, but because they have a duty to their company, and the well-being of their employees, and a duty to their shareholders. </p> <p> Your disregard for the LEGO Group's rights constituted a real and present danger to them, their employees and shareholders. They are compelled to address threats against their well-being. </p> <div class='imgWrapper imgBlock right fullsize' style=' '> <figure> <a href='https://www.flickr.com/photos/angietigger/6894972004/' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/bearpokelego.svg" type="image/svg"> <!---<source srcset="/blog/images/bearpokelego.avif" type="image/avif">--> <source srcset="/blog/images/bearpokelego.webp" type="image/webp"> <source srcset="/blog/images/bearpokelego.apng" type="image/apng"> <source srcset="/blog/images/bearpokelego.png" type="image/png"> <source srcset="/blog/images/bearpokelego.jpg" type="image/jpeg"> <source srcset="/blog/images/bearpokelego.jpeg" type="image/jpeg"> <source srcset="/blog/images/bearpokelego.jfif" type="image/jpeg"> <source srcset="/blog/images/bearpokelego.pjpeg" type="image/jpeg"> <source srcset="/blog/images/bearpokelego.pjp" type="image/jpeg"> <source srcset="/blog/images/bearpokelego.gif" type="image/gif"> <source srcset="/blog/images/bearpokelego.tif" type="image/tiff"> <source srcset="/blog/images/bearpokelego.tiff" type="image/tiff"> <source srcset="/blog/images/bearpokelego.bmp" type="image/bmp"> <source srcset="/blog/images/bearpokelego.ico" type="image/x-icon"> <source srcset="/blog/images/bearpokelego.cur" type="image/x-icon"> <img alt='&lsquo;lego bear&rsquo; by angie <br> CC BY-NC-SA 2.0' class="imgImg rounded shadow" src="/blog/images/bearpokelego.png" style='width: 100%; ' title='&lsquo;lego bear&rsquo; by angie <br> CC BY-NC-SA 2.0' /> </picture> </a> <figcaption class='imgFigCaption fullsize'> <a href="https://www.flickr.com/photos/angietigger/6894972004/" target='_blank' > &lsquo;lego bear&rsquo; by angie <br> CC BY-NC-SA 2.0 </a> </figcaption> </figure> </div> <h2 id="think">Think</h2> <p> Not only must a trademark holder defend their trademark, they must be seen to defend it. That means taking action that is publicly visible. For example, they might include GitHub and maybe even Microsoft in the legal action against you. </p> <div class="pullQuote"> No-one would have reason to defend you </div> <p> The LEGO Group would likely include GitHub/Microsoft in the case, even though they know that <a href='https://www.eff.org/issues/cda230' target='_blank' rel="nofollow">Section 230</a> would probably be found to apply (in due course), and GitHub/Microsoft would not be found to be liable &ndash; after they spent several million dollars in a successful legal defense. </p> <p> It is easy to understand the motivation for the LEGO Group deliberately &lsquo;losing&rsquo; at the attempt to ensnare GitHub/Microsoft in this court case. You would have caused GitHub/Microsoft an unnecessary and pointless waste of time and resources, and it would be shown that you knew better. Remember, you have already been publicly been warned that there would likely be consequences &ndash; yet you chose to ignore the warning. </p> <p> This would likely motivate GitHub/Microsoft to protect themselves against similar misguided behavior of other GitHub users. Restrictive policies and procedures would likely get implemented, and the entire world would know that the resulting degraded GitHub experience was your fault. You would be vilified by every other GitHub user. Not a great thing for your resume. </p> <p> ... And GitHub/Microsoft might ask you to pay their legal expenses. </p> <h2 id="family">Do You Have Families?</h2> <p> Please understand that because you are acting as an informal collection of individuals, everyone who ever contributed to this project is exposed to signficant financial risk. The LEGO Group would probably not attempt to narrow down the guilty individuals &ndash; identifying the guilty and assigning punishment would be the outcomes of the court case. Instead, they would likely include every contributor in the litigation; that means every committer. </p> <p> You are risking the financial well-being of your families ... just because you like a cute play on words? </p> <p class="pullQuoteFull"> Rational people would not expose themselves to significant risk without any prospect of signficant benefit, and when not faced with dire necessity. </p> <p> If you presently balk at paying a lawyer to advise you on this matter, that cost will be as nothing compared to the world of hurt you are likely to find yourselves in if you do not change the name of your project. </p> <p> From a technical aspect, you have done excellent work on your project, and you are freely providing the world a valuable benefit. Unfortunately, you trampled the rights of another in doing so... and for no reason. Please act on this before the inevitable negative consequences arise. Do not poke the bear because you are curious to see what it will do. </p> <div class='imgWrapper imgFlex right' style='width: 45%; '> <picture class='imgPicture'> <source srcset="/blog/images/bearpoke.svg" type="image/svg"> <!---<source srcset="/blog/images/bearpoke.avif" type="image/avif">--> <source srcset="/blog/images/bearpoke.webp" type="image/webp"> <source srcset="/blog/images/bearpoke.apng" type="image/apng"> <source srcset="/blog/images/bearpoke.png" type="image/png"> <source srcset="/blog/images/bearpoke.jpg" type="image/jpeg"> <source srcset="/blog/images/bearpoke.jpeg" type="image/jpeg"> <source srcset="/blog/images/bearpoke.jfif" type="image/jpeg"> <source srcset="/blog/images/bearpoke.pjpeg" type="image/jpeg"> <source srcset="/blog/images/bearpoke.pjp" type="image/jpeg"> <source srcset="/blog/images/bearpoke.gif" type="image/gif"> <source srcset="/blog/images/bearpoke.tif" type="image/tiff"> <source srcset="/blog/images/bearpoke.tiff" type="image/tiff"> <source srcset="/blog/images/bearpoke.bmp" type="image/bmp"> <source srcset="/blog/images/bearpoke.ico" type="image/x-icon"> <source srcset="/blog/images/bearpoke.cur" type="image/x-icon"> <img class="imgImg " src="/blog/images/bearpoke.png" style='width: 100%; ' /> </picture> </div> <p class="left alert rounded shadow" style="width: 50%;"> The wheels of corporate law grind slowly, and huge amounts of money are spent. According to the path you are now on, the committers of this project may be subjected to immense financial and mental burdens. Broken families and suicides may result. <br><br> Please heed this warning. If nothing bad has ever happened to you yet in life, rest assured that <a href='https://brewminate.com/clotho-lachesis-and-atropos-the-three-sisters-of-fate-in-ancient-greek-mythology/' target='_blank' rel="nofollow">Lachesis, Atropos and Clotho</a> will inevitably make themselves known. Do not let arrogance and pride ruin you. </p> <div class="clear pullQuoteFull" style="margin-top: 2.5em;"> I am trying to help you people.<br> Because you have done good work,<br> and it would be a shame to see you suffer. </div> <h2 id="disclaim" class="clear">Disclaimer</h2> <p> I have no relationship with the LEGO Group, or any individuals associated with them. Please consider this as a notice freely offered for public service. </p> <p> Although <a href='/softwareexpert/'>I have experience working as an expert witness in the US Federal legal system as a software expert</a>, I am not a lawyer and I have no legal training. However, I have seen corporate legal machinery at work many times, up close. I have spent thousands of hours working with corporate attorneys in the US and Europe. My clients have included Adobe, Amazon/AWS, Apple, and SAP. </p> <!-- endregion --> Letsencrypt/ACME Wildcard SSL Certificates by Lego 2023-03-02T00:00:00-05:00 https://mslinn.github.io/blog/2023/03/02/lego <style> .redButton { color:white; background-color:#e04433; padding: 2px; font-family: Helvetica, Arial, Sans-Serif; font-weight: bold; } </style> <!-- #region intro --> <div class='imgWrapper imgFlex right' style='width: 40%; '> <a href='https://letsencrypt.org/' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/ddns/letsencrypt-logo.svg" type="image/svg"> <!---<source srcset="/blog/images/ddns/letsencrypt-logo.avif" type="image/avif">--> <source srcset="/blog/images/ddns/letsencrypt-logo.webp" type="image/webp"> <source srcset="/blog/images/ddns/letsencrypt-logo.apng" type="image/apng"> <source srcset="/blog/images/ddns/letsencrypt-logo.png" type="image/png"> <source srcset="/blog/images/ddns/letsencrypt-logo.jpg" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.jpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.jfif" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.pjpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.pjp" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.gif" type="image/gif"> <source srcset="/blog/images/ddns/letsencrypt-logo.tif" type="image/tiff"> <source srcset="/blog/images/ddns/letsencrypt-logo.tiff" type="image/tiff"> <source srcset="/blog/images/ddns/letsencrypt-logo.bmp" type="image/bmp"> <source srcset="/blog/images/ddns/letsencrypt-logo.ico" type="image/x-icon"> <source srcset="/blog/images/ddns/letsencrypt-logo.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/ddns/letsencrypt-logo.png" style='width: 100%; ' /> </picture> </a> </div> <p> Letsencrypt certificates are only valid for <a href='https://letsencrypt.org/docs/faq/#what-is-the-lifetime-for-let-s-encrypt-certificates-for-how-long-are-they-valid' target='_blank' rel="nofollow">90 days</a>. It is common practice to renew them every 60 days, a task that one should automate when the website is first published. </p> <p> I wrote about <a href='/blog/2022/06/15/certbot.html'>setting up wildcard SSL certificates with Nginx</a> 9 months ago. That article included a demonstration of how to create an SSL certificate manually, but did not discuss how to maintain it. </p> <p> This article discusses how to automatically generate and maintain the SSL certificate using <a href='https://github.com/go-acme/lego' target='_blank' rel="nofollow"><code>lego</code></a>, a Letsencrypt/ACME client and library written in Go that has become popular since I last wrote about this topic. </p> <p> <code>Lego</code> handles many moving parts transparently. It is <i>much</i> simpler than using DNS delegates with <a href='https://github.com/joohoi/acme-dns' target='_blank' rel="nofollow"><code>acme-dns</code></a>. </p> <p> We&rsquo;ll start by briefly discussing some background information. </p> <!-- endregion --> <!-- #region Letsencrypt certbot and wildcard ssl certs --> <h2 id="certbot" class="clear">Letsencrypt&rsquo;s <span class="code">Certbot</span> and Wildcard SSL Certificates</h2> <p> You must prove to Letsencrypt that you control the DNS for a domain before it issues a wildcard SSL certificate for that domain. <a href='https://certbot.eff.org/' target='_blank' rel="nofollow">Letsencrypt&rsquo;s <code>certbot</code></a> currently uses the <a href='https://cert-manager.io/docs/configuration/acme/dns01/' target='_blank' rel="nofollow"><code>DNS-01</code> challenge</a> for this purpose. </p> <h3 id="DNS-01">DNS-01 Challenge</h3> <p> The DNS-01 challenge requires that DNS <code>TXT</code> records for the domain be created with specific values as part of the authentication mechanism. Several of these <code>TXT</code> records can co-exist among the DNS records for a domain. The name of the <code>TXT</code> records is always <code>_acme-challenge</code>. </p> <p> When <code>certbot</code> requests that a new wildcard SSL certificate be created by Letsencrypt, it sends DNS-01 <code>TXT</code> queries to the primary DNS. The Letsencrypt service verifies that the required <code>_acme-challenge</code> <code>TXT</code> records are available with the correct values. </p> <p> <a href='https://www.eff.org/deeplinks/2018/02/technical-deep-dive-securing-automation-acme-dns-challenge-validation' target='_blank' rel="nofollow">A Technical Deep Dive: Securing the Automation of ACME DNS Challenge Validation</a>, published by the Electronic Frontier Foundation, explains how this works in detail. </p> <p> <code>Certbot</code> plugins have been created for various DNS providers to make this process easier. <code>Lego</code> is a higher-level program that makes the entire process even easier. </p> <!-- endregion --> <!-- #region agenda --> <h2 id="agenda">Agenda</h2> <p> The high-level outline of the remainder of this blog is: </p> <ol> <li><a href='#go'>Install <code>go</code> language support.</a></li> <li><a href='#install'>Install <code>lego</code>.</a></li> <li><a href='#gen'>Generate the wildcard SSL certificate.</a></li> <li><a href='#provide'>Provide the certificate</a> to your production web server.</li> <li><a href='#restart'>Restart the webserver</a> and verify the new certificate is served.</li> <li><a href='#crontab'>Add a new entry to <code>crontab</code></a> to automate the SSL certificate generation.</li> </ol> <!-- endregion --> <!-- #region Install Go Language Support --> <h2 id="go">Install Go Language Support</h2> <p> <code>Lego</code> is written in Go. Ensure that <a href='https://go.dev/doc/install' target='_blank' rel="nofollow">Go language support</a> is installed. For Ubuntu Linux (and the default WSL distro), type: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddb89a2d6e5de'><button class='copyBtn' data-clipboard-target='#iddb89a2d6e5de' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install golang-go <span class='unselectable'>The following NEW packages will be installed: golang-1.18-go golang-1.18-src golang-go golang-src 0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded. Need to get 82.2 MB of archives. After this operation, 436 MB of additional disk space will be used. Do you want to continue? [Y/n] Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 golang-1.18-src all 1.18.1-1ubuntu1 [16.2 MB] Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 golang-1.18-go amd64 1.18.1-1ubuntu1 [66.0 MB] Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 golang-src all 2:1.18~0ubuntu2 [4438 B] Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 golang-go amd64 2:1.18~0ubuntu2 [41.8 kB] Fetched 82.2 MB in 2s (33.6 MB/s) Selecting previously unselected package golang-1.18-src. (Reading database ... 173763 files and directories currently installed.) Preparing to unpack .../golang-1.18-src_1.18.1-1ubuntu1_all.deb ... Unpacking golang-1.18-src (1.18.1-1ubuntu1) ... Selecting previously unselected package golang-1.18-go. Preparing to unpack .../golang-1.18-go_1.18.1-1ubuntu1_amd64.deb ... Unpacking golang-1.18-go (1.18.1-1ubuntu1) ... Selecting previously unselected package golang-src. Preparing to unpack .../golang-src_2%3a1.18~0ubuntu2_all.deb ... Unpacking golang-src (2:1.18~0ubuntu2) ... Selecting previously unselected package golang-go:amd64. Preparing to unpack .../golang-go_2%3a1.18~0ubuntu2_amd64.deb ... Unpacking golang-go:amd64 (2:1.18~0ubuntu2) ... Setting up golang-1.18-src (1.18.1-1ubuntu1) ... Setting up golang-src (2:1.18~0ubuntu2) ... Setting up golang-1.18-go (1.18.1-1ubuntu1) ... Setting up golang-go:amd64 (2:1.18~0ubuntu2) ... Processing triggers for man-db (2.10.2-1) ... Scanning processes... Scanning processor microcode... Scanning linux images... Failed to retrieve available kernel versions. Failed to check for processor microcode upgrades. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. </span></pre> </div> <p> View the version of <code>go</code> that was installed: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idba9934dd636e'><button class='copyBtn' data-clipboard-target='#idba9934dd636e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>go version <span class='unselectable'>go version go1.19.2 linux/amd64 </span></pre> </div> <!-- endregion --> <!-- #region Installing Lego --> <h2 id="install">Installing Lego</h2> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/ddns/lego-logo.svg" type="image/svg"> <!---<source srcset="/blog/images/ddns/lego-logo.avif" type="image/avif">--> <source srcset="/blog/images/ddns/lego-logo.webp" type="image/webp"> <source srcset="/blog/images/ddns/lego-logo.apng" type="image/apng"> <source srcset="/blog/images/ddns/lego-logo.png" type="image/png"> <source srcset="/blog/images/ddns/lego-logo.jpg" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.jpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.jfif" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.pjpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.pjp" type="image/jpeg"> <source srcset="/blog/images/ddns/lego-logo.gif" type="image/gif"> <source srcset="/blog/images/ddns/lego-logo.tif" type="image/tiff"> <source srcset="/blog/images/ddns/lego-logo.tiff" type="image/tiff"> <source srcset="/blog/images/ddns/lego-logo.bmp" type="image/bmp"> <source srcset="/blog/images/ddns/lego-logo.ico" type="image/x-icon"> <source srcset="/blog/images/ddns/lego-logo.cur" type="image/x-icon"> <img class="imgImg " src="/blog/images/ddns/lego-logo.png" style='width: 100%; ' /> </picture> </div> <p> The official <code>lego</code> installation instructions are <a href='https://go-acme.github.io/lego/installation/' target='_blank' rel="nofollow">here</a>. They do not mention Ubuntu instructions, but the <a href='https://ubuntu.pkgs.org/22.04/ubuntu-universe-amd64/lego_4.1.3-3ubuntu1_amd64.deb.html' target='_blank' rel="nofollow"><code>lego</code> package for Ubuntu</a> can be installed in the usual way: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6a258c0738b6'><button class='copyBtn' data-clipboard-target='#id6a258c0738b6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install lego <span class='unselectable'>Reading package lists... Done Building dependency tree... Done Reading state information... Done The following NEW packages will be installed: lego 0 upgraded, 1 newly installed, 0 to remove and 7 not upgraded. Need to get 5322 kB of archives. After this operation, 19.7 MB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 lego amd64 4.1.3-3ubuntu1.22.04.1 [5322 kB] Fetched 5322 kB in 1s (10.2 MB/s) Selecting previously unselected package lego. (Reading database ... 86044 files and directories currently installed.) Preparing to unpack .../lego_4.1.3-3ubuntu1.22.04.1_amd64.deb ... Unpacking lego (4.1.3-3ubuntu1.22.04.1) ... Setting up lego (4.1.3-3ubuntu1.22.04.1) ... </span></pre> </div> <p> Here is the <code>lego</code> help message: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4a6361dabaae'><button class='copyBtn' data-clipboard-target='#id4a6361dabaae' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>lego -h <span class='unselectable'>NAME: lego - Let&#39;s Encrypt client written in Go<br/> USAGE: lego [global options] command [command options] [arguments...]<br/> VERSION: dev<br/> COMMANDS: run Register an account, then create and install a certificate revoke Revoke a certificate renew Renew a certificate dnshelp Shows additional help for the &#39;--dns&#39; global option list Display certificates and accounts information. help, h Shows a list of commands or help for one command<br/> GLOBAL OPTIONS: --domains value, -d value Add a domain to the process. Can be specified multiple times. --server value, -s value CA hostname (and optionally :port). The server certificate must be trusted in order to avoid further modifications to the client. (default: &quot;https://acme-v02.api.letsencrypt.org/directory&quot;) --accept-tos, -a By setting this flag to true you indicate that you accept the current Let&#39;s Encrypt terms of service. --email value, -m value Email used for registration and recovery contact. --csr value, -c value Certificate signing request filename, if an external CSR is to be used. --eab Use External Account Binding for account registration. Requires --kid and --hmac. --kid value Key identifier from External CA. Used for External Account Binding. --hmac value MAC key from External CA. Should be in Base64 URL Encoding without padding format. Used for External Account Binding. --key-type value, -k value Key type to use for private keys. Supported: rsa2048, rsa4096, rsa8192, ec256, ec384. (default: &quot;ec256&quot;) --filename value (deprecated) Filename of the generated certificate. --path value Directory to use for storing the data. (default: &quot;/mnt/_/work/lego/.lego&quot;) [$LEGO_PATH] --http Use the HTTP challenge to solve challenges. Can be mixed with other types of challenges. --http.port value Set the port and interface to use for HTTP based challenges to listen on.Supported: interface:port or :port. (default: &quot;:80&quot;) --http.proxy-header value Validate against this HTTP header when solving HTTP based challenges behind a reverse proxy. (default: &quot;Host&quot;) --http.webroot value Set the webroot folder to use for HTTP based challenges to write directly in a file in .well-known/acme-challenge. This disables the built-in server and expects the given directory to be publicly served with access to .well-known/acme-challenge --http.memcached-host value Set the memcached host(s) to use for HTTP based challenges. Challenges will be written to all specified hosts. --tls Use the TLS challenge to solve challenges. Can be mixed with other types of challenges. --tls.port value Set the port and interface to use for TLS based challenges to listen on. Supported: interface:port or :port. (default: &quot;:443&quot;) --dns value Solve a DNS challenge using the specified provider. Can be mixed with other types of challenges. Run &#39;lego dnshelp&#39; for help on usage. --dns.disable-cp By setting this flag to true, disables the need to wait the propagation of the TXT record to all authoritative name servers. --dns.resolvers value Set the resolvers to use for performing recursive DNS queries. Supported: host:port. The default is to use the system resolvers, or Google&#39;s DNS resolvers if the system&#39;s cannot be determined. --http-timeout value Set the HTTP timeout value to a specific value in seconds. (default: 0) --dns-timeout value Set the DNS timeout value to a specific value in seconds. Used only when performing authoritative name servers queries. (default: 10) --pem Generate a .pem file by concatenating the .key and .crt files together. --cert.timeout value Set the certificate timeout value to a specific value in seconds. Only used when obtaining certificates. (default: 30) --help, -h show help --version, -v print the version </span></pre> </div> <!-- endregion --> <!-- #region Generating the Wildcard SSL Certificate --> <h2 id="gen">Generating the Wildcard SSL Certificate</h2> <!-- #region /etc/environment --> <h3 id="env"><span class="code">/etc/environment</span></h3> <p> My projects are defined in a directory tree, which is pointed to by an environment variable called <code>work</code>. This is helpful when working across many machines, each of which has a different directory layout. </p> <p> I define the <code>work</code> environment variable in the <a href='https://askubuntu.com/a/866240/58760' target='_blank' rel="nofollow">system-wide <code>/etc/environment</code></a> file, where it affects all users, all scripts, and crontab entries: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>/etc/environment</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id163fd0f90f18'><button class='copyBtn' data-clipboard-target='#id163fd0f90f18' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin" work=/mnt/_/work sites=/mnt/_/www</pre> </div> <p> <code>$work</code> is used in the script shown <a href='#certGenScalaCourses'>below</a>, and that script will be launched via <code>crontab</code>, which is why <code>/etc/environment</code> is used instead of <code>$HOME/.bashrc</code>. </p> <p class="alert rounded shadow"> WSL does not run <code>systemd</code> services, and WSL2 only runs them if they are enabled. You can easily convert WSL instances into WSL2 instances. <br><br> Unless <code>systemd</code> starts automatically at boot time, <code>/etc/environment</code> will not be processed. <br><br> Read Microsoft&rsquo;s announcement to learn more: <a href='https://devblogs.microsoft.com/commandline/systemd-support-is-now-available-in-wsl/' target='_blank' rel="nofollow"><code>Systemd</code> support is now available in WSL!</a> </p> <!-- endregion --> <!-- #region generation --> <h3 id="dir">Generation</h3> <p> When working with the <code>lego</code> CLI with DNS authentication, a directory is needed to hold all the files that are required. I created a directory for this purpose at <code>$work/lego</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id37f6b574f8b0'><button class='copyBtn' data-clipboard-target='#id37f6b574f8b0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mkdir $work/lego<br> <span class='unselectable'>$ </span>cd $work/lego</pre> </div> <p> The <code>lego --dns namecheap</code> option connects to Namecheap, which is my DNS provider, and uses the Namecheap DNS for the <code>DNS-01</code> challenge. Lego supports <a href='https://go-acme.github.io/lego/dns/#dns-providers' target='_blank' rel="nofollow">many other DNS providers</a>. </p> <!-- #region Namecheap Users --> <div class="alert rounded shadow"> <h2>Namecheap Users</h2> <p> The Namecheap API documentation is <a href='https://www.namecheap.com/support/api/intro/' target='_blank' rel="nofollow">here</a>. For some reason, that page reloads to a different page, which is annoying. Press the <kbd>Esc</kbd> key right after the page opens, so you can read it. </p> <p> Please also read the <a href='https://www.namecheap.com/support/knowledgebase/article.aspx/9739/63/api-faq/#t' target='_blank' rel="nofollow">FAQ</a>.<br> Enable the Namecheap API <a href='https://ap.www.namecheap.com/settings/tools/apiaccess/' target='_blank' rel="nofollow">here</a>.<br> You could enable the Namecheap sandbox API <a href='https://ap.www.sandbox.namecheap.com/settings/tools/apiaccess/' target='_blank' rel="nofollow">here</a>. </p> <p> Here is a short script that displays your current public IP address: </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,whatismyip' download='whatismyip' title='Click on the file name to download the file'>whatismyip</a> </div> <pre data-lt-active="false" class="pre_tag maxOneScreenHigh copyContainer" id="id915bed728195"><button class='copyBtn' data-clipboard-target='#id915bed728195'title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash dig +short myip.opendns.com @resolver1.opendns.com </pre> <p style="margin-top: 1em;"> If your modem goes offline, Namecheap&rsquo;s DNS will assign a new IP address the next time the modem reconnects. That will cause the scripts shown below to stop working until you manually provide the new address to the Namecheap API. Annoying! Be sure to plug your modem into a UPS to maintain the IP address even during short outages, power brownouts and voltage dropouts. A sine-wave UPS is strongly preferred, since modems are sensitive to low-quality power. </p> <p> The Namecheap API needs 2 environment variables for authentication (<code>NAMECHEAP_API_USER</code> and <code>NAMECHEAP_API_KEY</code>). Here is how I defined those environment variables so they were available for the <code>lego</code> process launched by <code>sudo</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id07629d2b4546'><button class='copyBtn' data-clipboard-target='#id07629d2b4546' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>export NAMECHEAP_API_USER=asdf<br> <span class='unselectable'>$ </span>export NAMECHEAP_API_KEY=asdfasdf</pre> </div> </div> <!-- endregion --> <p> To generate a wildcard certificate, you must specify the <code>--domains</code> option twice: once with the domain name, and once for all subdomains, like this: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide1169bc5b38d'><button class='copyBtn' data-clipboard-target='#ide1169bc5b38d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>lego \ --accept-tos \ --dns namecheap \ --domains="scalacourses.com" \ --domains="*.scalacourses.com" \ --email="mslinn@scalacourses.com" \ <span class="bg_yellow">run</span> <span class='unselectable'>2023/03/01 13:40:03 [INFO] [*.scalacourses.com] acme: Obtaining bundled SAN certificate 2023/03/01 13:40:03 [INFO] [*.scalacourses.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/207409481036 2023/03/01 13:40:03 [INFO] [*.scalacourses.com] acme: use dns-01 solver 2023/03/01 13:40:03 [INFO] [*.scalacourses.com] acme: Preparing to solve DNS-01 2023/03/01 13:40:04 [INFO] [*.scalacourses.com] acme: Trying to solve DNS-01 2023/03/01 13:40:04 [INFO] [*.scalacourses.com] acme: Checking DNS record propagation using [172.19.176.1:53] 2023/03/01 13:40:19 [INFO] Wait for propagation [timeout: 1h0m0s, interval: 15s] 2023/03/01 13:40:20 [INFO] [*.scalacourses.com] acme: Waiting for DNS record propagation. 2023/03/01 13:40:35 [INFO] [*.scalacourses.com] acme: Waiting for DNS record propagation. 2023/03/01 13:40:56 [INFO] [*.scalacourses.com] The server validated our request 2023/03/01 13:40:56 [INFO] [*.scalacourses.com] acme: Cleaning DNS-01 challenge 2023/03/01 13:40:56 [INFO] [*.scalacourses.com] acme: Validations succeeded; requesting certificates 2023/03/01 13:40:57 [INFO] [*.scalacourses.com] Server responded with a certificate. </span></pre> </div> <!-- endregion --> <p> The wildcard SSL certificate and associated files are generated into the <code>.lego/certificates</code> directory. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id200bb2c83d30'><button class='copyBtn' data-clipboard-target='#id200bb2c83d30' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls -alF .lego/certificates <span class='unselectable'>total 12 drwx------ 1 mslinn mslinn 4096 Mar 1 13:40 ./ drwx------ 1 mslinn mslinn 4096 Mar 1 13:15 ../ -rw------- 1 mslinn mslinn 5325 Mar 1 13:40 scalacourses.com.crt -rw------- 1 mslinn mslinn 3751 Mar 1 13:40 scalacourses.com.issuer.crt -rw------- 1 mslinn mslinn 239 Mar 1 13:40 scalacourses.com.json -rw------- 1 mslinn mslinn 227 Mar 1 13:40 scalacourses.com.key </span></pre> </div> <!-- endregion --> <ul> <li> <code>scalacourses.com.crt</code> is the server certificate that nginx needs, already combined with the CA certificate. </li> <li><code>scalacourses.com.key</code> is the private key for the server certificate.</li> <li><code>scalacourses.com.issuer.crt</code> is the issuing Certificate Authority&apos;s certificate.</li> <li> <code>scalacourses.com.json</code> contains JSON encoded meta information. It looked like this: <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>&dollar;work/lego/.lego/certificates/scalacourses.com.json</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9bcc5d9a0f93'><button class='copyBtn' data-clipboard-target='#id9bcc5d9a0f93' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>{ "domain": "*.scalacourses.com", "certUrl": "https://acme-v02.api.letsencrypt.org/acme/cert/3939493849340275024024", "certStableUrl": "https://acme-v02.api.letsencrypt.org/acme/cert/738378373923492384932874932" }</pre> </div> </li> </ul> <p> Using the value for <code>certStableUrl</code> in the above JSON file, we can view the generated (and combined) certificate that was saved on <code>letsencrypt.org</code>. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide281035b9712'><button class='copyBtn' data-clipboard-target='#ide281035b9712' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>wget -O aw.crt \ https://acme-v02.api.letsencrypt.org/acme/cert/03b099f52b4841db17e035c5c2b390f0219b</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region certGenScalaCourses script --> <h2 id="certGenScalaCourses"><span class="code">certGenScalaCourses</span> Script</h2> <p> 60 days from now, when the certificate should be renewed, the last parameter passed to <code>lego</code> in the above command line should be changed from <code class="bg_yellow">run</code> to <code>renew</code>. Here is a bash script that you can edit for this purpose; at the end of this article this script is incorporated into a new <code>crontab</code> entry: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>&dollar;work/lego/certGenScalaCourses</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide7be83bb0763'><button class='copyBtn' data-clipboard-target='#ide7be83bb0763' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash export NAMECHEAP_API_USER=asdf export NAMECHEAP_API_KEY=asdfasdf ACTION=run if [ -f .lego/certificates/scalacourses.com.crt ]; then ACTION=renew fi cd $work/lego lego \ --accept-tos \ --dns namecheap \ --domains="scalacourses.com" \ --domains="*.scalacourses.com" \ --email="mslinn@scalacourses.com" \ "$ACTION" sudo systemctl reload nginx</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Viewing a Certificate --> <h2 id="view">Viewing a Certificate</h2> <p> For Ubuntu 22.10, the filetype associations are defined in a hierarchy of files called <code>mimeapps.list</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida126176b1709'><button class='copyBtn' data-clipboard-target='#ida126176b1709' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>locate mimeapps.list <span class='unselectable'>~/.config/mimeapps.list ~/.local/share/applications/mimeapps.list /snap/core/14447/usr/share/applications/mimeapps.list /snap/core/14784/usr/share/applications/mimeapps.list /snap/core18/2679/usr/share/applications/mimeapps.list /snap/core18/2697/usr/share/applications/mimeapps.list /snap/core20/1778/usr/share/applications/mimeapps.list /snap/core20/1822/usr/share/applications/mimeapps.list /snap/core22/509/usr/share/applications/mimeapps.list /snap/core22/522/usr/share/applications/mimeapps.list /usr/share/gdm/greeter/applications/mimeapps.list </span></pre> </div> <!-- endregion --> <p> <code>/snap/<wbr>core22/<wbr>509/<wbr>usr/<wbr>share/<wbr>applications/<wbr>mimeapps.list</code> contains the default association for <code>*.crt</code> files: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>/snap/core22/509/usr/share/applications/mimeapps.list</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id36c23605d2f3'><button class='copyBtn' data-clipboard-target='#id36c23605d2f3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[Default Applications] x-scheme-handler/http=xdg-open.desktop x-scheme-handler/https=xdg-open.desktop x-scheme-handler/mailto=xdg-open.desktop x-scheme-handler/help=xdg-open.desktop</pre> </div> <!-- endregion --> <p> The above associates a program called <a href='https://linux.die.net/man/1/xdg-open' target='_blank' rel="nofollow"><code>xdg-open</code></a> with CRT files. </p> <p> Double-clicking on a <code>crt</code> file displayed by an Ubuntu file manager such as Nautilus causes <code>xdg-open</code> to open the file. You can also use the command line to open the file in <code>xdg-open</code>. For example, typing the following causes <code>aw.crt</code> to open: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0d5193640958'><button class='copyBtn' data-clipboard-target='#id0d5193640958' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>xdg-open aw.crt</pre> </div> <p> This is what <code>xdg-open</code> displays: </p> <div class='imgWrapper imgFlex center' style='width: 478px; '> <picture class='imgPicture'> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.svg" type="image/svg"> <!---<source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.avif" type="image/avif">--> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.webp" type="image/webp"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.apng" type="image/apng"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.png" type="image/png"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.jpg" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.jpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.jfif" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.pjpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.pjp" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.gif" type="image/gif"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.tif" type="image/tiff"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.tiff" type="image/tiff"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.bmp" type="image/bmp"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.ico" type="image/x-icon"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_2.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/ddns/ubuntuViewFileOfSslCert_2.png" style='width: 100%; ' /> </picture> </div> <p> As you can see, this is a combined certificate, containing not just one certificate, but an entire certificate chain. Clicking on any of the red <kbd class="redButton">&gt; Details</kbd> buttons causes more information to be displayed about the corresponding certificate in the chain. </p> <!-- endregion --> <!-- #region warning --> <div class="alert rounded shadow"> <h2>Warning</h2> <p> If you forget to include the domain, and just specify the wildcard for subdomains, the certificate will not be valid for the domain. For example, if above I had only specified one <code>--domains</code> option, like this: <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6f92e9cc6354'><button class='copyBtn' data-clipboard-target='#id6f92e9cc6354' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>--domains="scalacourses.com"</pre> </div> ... instead of <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id631e12725d3d'><button class='copyBtn' data-clipboard-target='#id631e12725d3d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>--domains="scalacourses.com" \<br>--domains="*.scalacourses.com"</pre> </div> ... then that would be an error. You can notice your error three ways: </p> <ol> <li> The name of the generated certificate will start with an underscore (<code>_</code>), for example <code>_.scalacourses.<wbr>com.<wbr>crt</code>, instead of being named <code>scalacourses.<wbr>com.<wbr>crt</code>. </li> <li> When you examine the certificate, the domain listed will show the subdomain wildcard, as shown below, with a leading underscore asterisk (<code>*</code>).<br> <div class='imgWrapper imgFlex center' style='width: 478px; '> <picture class='imgPicture'> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.svg" type="image/svg"> <!---<source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.avif" type="image/avif">--> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.webp" type="image/webp"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.apng" type="image/apng"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.png" type="image/png"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.jpg" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.jpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.jfif" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.pjpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.pjp" type="image/jpeg"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.gif" type="image/gif"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.tif" type="image/tiff"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.tiff" type="image/tiff"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.bmp" type="image/bmp"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.ico" type="image/x-icon"> <source srcset="/blog/images/ddns/ubuntuViewFileOfSslCert_1.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/ddns/ubuntuViewFileOfSslCert_1.png" style='width: 100%; margin-top: 2em;' /> </picture> </div> </li> <li style="margin-bottom: 0; padding-bottom: 0;"> When you open the certificate, the section entitled <b>Subject Alternative Names</b> will just contain the subdomain wildcard, like this:<br><br> <b>Subject Alternative Names</b><br> <code>DNS: *.scalacourses.com</code> <br><br>Or just the domain, without a wildcard:<br><br> <b>Subject Alternative Names</b><br> <code>DNS: scalacourses.com</code><br><br> Instead of both being present, like this:<br><br> <b>Subject Alternative Names</b><br> <code>DNS: *.scalacourses.com<br> DNS: scalacourses.com</code> </li> </ol> </div> <!-- endregion --> <!-- #region Provide the certificate to Nginx --> <h2 id="provide">Provide the certificate to Nginx</h2> <p> <a href='/blog/2022/07/08/reverse-proxy.html#vs'>Previously</a>, before working with <code>lego</code>, I used raw <code>certbot</code> to generate the SSL wildcard certificates. That process generated files in <code>~/.certbot/<wbr>scalacourses.com/<wbr>config/<wbr>live/<wbr>scalacourses.com</code>: </p> <ul> <li> <code>fullchain.pem</code> (the <a href='http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate' target='_blank' rel="nofollow"><code>ssl_certificate</code></a>) </li> <li> <code>privkey.pem</code> (the <a href='https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate_key' target='_blank' rel="nofollow"><code>ssl_certificate_key</code></a>) </li> </ul> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>From /etc/nginx/sites-enabled/scalacourses.com</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0059a2492725'><button class='copyBtn' data-clipboard-target='#id0059a2492725' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>ssl_certificate /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/fullchain.pem; ssl_certificate_key /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/privkey.pem;</pre> </div> <p> Now I need to modify <code>/etc/<wbr>nginx/<wbr>sites-enabled/<wbr>scalacourses.com</code> to reference the files generated by <code>lego</code>: </p> <ul> <li> <code>scalacourses.com.crt</code> (the <a href='http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate' target='_blank' rel="nofollow"><code>ssl_certificate</code></a>) </li> <li> <code>scalacourses.com.key</code> (the <a href='https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate_key' target='_blank' rel="nofollow"><code>ssl_certificate_key</code></a>) </li> </ul> <p> If the webserver had access to the directory containing the new wildcard certificate, it would be simplest to provide the full path to the certificate and its key, like this: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Modified portion of /etc/nginx/sites-enabled/scalacourses.com</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id995bbf96608b'><button class='copyBtn' data-clipboard-target='#id995bbf96608b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>ssl_certificate /mnt/_/work/lego/.lego/certificates/scalacourses.com.crt; ssl_certificate_key /mnt/_/work/lego/.lego/certificates/scalacourses.com.key;</pre> </div> <!-- endregion --> <!-- #region Restart Nginx --> <h2 id="restart">Restart Nginx</h2> <p> I tested the <code>nginx</code> configuration: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb3dbebf4653c'><button class='copyBtn' data-clipboard-target='#idb3dbebf4653c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo nginx -t <span class='unselectable'>nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful </span></pre> </div> <p> Then I reloaded nginx: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id733ddd4b69d6'><button class='copyBtn' data-clipboard-target='#id733ddd4b69d6' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl reload nginx</pre> </div> <p> The new SSL certificate needs to be checked to make sure it was installed properly. I just visually examine the start and expire dates: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id79e8238915fb'><button class='copyBtn' data-clipboard-target='#id79e8238915fb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -Lvs https://www.scalacourses.com \ 2>&1 1>/dev/null | \ grep '\(start\|expire\) date:' <span class='unselectable'>* <span style="color:red">start date</span>: Mar 1 17:40:56 2023 GMT * <span style="color:red">expire date</span>: May 30 17:40:55 2023 GMT </span></pre> </div> <span style='font-size: 3em;;'>&#x1F601;</span> <!-- endregion --> <!-- #region Add a New Entry to crontab --> <h2 id="crontab" class="clear">Add a New Entry to <span class="code">Crontab</span></h2> <p> Automating the regeneration of the SSL wildcard certificate is easy. Edit your <code>crontab</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf00a82f862e1'><button class='copyBtn' data-clipboard-target='#idf00a82f862e1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>crontab -e</pre> </div> <p> Add the following reference to the <a href='#certGenScalaCourses'><code>certGenScalaCourses</code> script</a> that I gave you earlier to <code>crontab</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>crontab Entry</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6293f37d90ca'><button class='copyBtn' data-clipboard-target='#id6293f37d90ca' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># Runs every 2 months / 60 days (more or less) 30 0 1 */2 * $work/lego/certGenScalaCourses</pre> </div> <p> The above only works because <a href='#env'>we defined the <code>work</code> environment variable</a> in <code>/etc/<wbr>environment</code>. </p> <!-- endregion --> <!-- #region checking with With A Web Browser --> <h2 id="chrome">Checking the Certificate With A Web Browser</h2> <p> Microsoft Windows caches SSL certificates, which can prevent your web browser from detecting newly updated SSL certificates. To clear the Windows SSL cache: </p> <ol> <li>Press the <kbd>Windows</kbd> key, type <code>Internet Options</code>, and press <kbd>Enter</kbd>.</li> <li>Select the <b>Content</b> tab.</li> <li>Click the <kbd>Clear SSL state</kbd> button.</li> <li>Click the <kbd>OK</kbd> button.</li> </ol> <!-- endregion --> HTML Hyphens 2023-01-23T00:00:00-05:00 https://mslinn.github.io/blog/2023/01/23/html-hyphen <!-- #region intro --> <p> Web browsers now have built-in automatic hyphenation support, but it is turned off by default for compatibity with older browsers. You can enable hyphenation with two words of CSS, like this: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>your_stylesheet.css</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4118b46b8930'><button class='copyBtn' data-clipboard-target='#id4118b46b8930' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>body { <span class='bg_yellow'>hyphens: auto;</span> }</pre> </div> <!-- endregion --> <!-- #region W3 CSS3 Hyphenation Standard --> <h2 id="w3c">W3 CSS3 Hyphenation Standard</h2> <div class='quote'> <div class='quoteText clearfix'> Hyphenation occurs when the line breaks at a valid hyphenation opportunity, which is a type of soft wrap opportunity that exists within a word where hyphenation is allowed. <br><br> In CSS, hyphenation opportunities are controlled with the <code>hyphens</code> property. CSS Text Level 3 does not define the exact rules for hyphenation; however, UAs are strongly encouraged to optimize their choice of break points and to chose language-appropriate hyphenation points. <div class="jekyll_pre" > <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id32d73f4142ee'><button class='copyBtn' data-clipboard-target='#id32d73f4142ee' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>Possible hyphens values: none | manual | auto Initial value: manual</pre> </div> </div><div class='quoteAttribution'> &nbsp;&ndash; <a href='https://www.w3.org/TR/css-text-3/#hyphenation' rel='nofollow' target='_blank'>W3 CSS3 Standard</a></div> </div> <p> However, the W3 CSS3 hyphenation standard also says: </p> <div class='quote'> <div class='quoteText clearfix'> Correct automatic hyphenation requires a hyphenation resource appropriate to the language of the text being broken. The UA must therefore only automatically hyphenate text for which the content language is known and for which it has an appropriate hyphenation resource. <br><br> Authors should correctly tag their content’s language (e.g. using the HTML <code>lang</code> attribute or XML <code>xml:lang</code> attribute) in order to obtain correct automatic hyphenation. </div><div class='quoteAttribution'> &nbsp;&ndash; <a href='https://www.w3.org/TR/css-text-3/#hyphenation' rel='nofollow' target='_blank'>W3 CSS3 Standard</a></div> </div> <p> Let&rsquo;s put this into practice now. </p> <!-- endregion --> <!-- #region Your Website Stylesheet --> <h2 id="css">Your Website Stylesheet</h2> <p> You may not want certain passages to be hyphenated. Defining a CSS style that disables automatic hyphenation for portions of a document is easy; you can still use <code>&ampshy;</code> in those portions of the document to manually define optional hyphenation points. </p> <p> Here is a suggestion for your stylesheet, so it has the hyphenation support you need. Note that hypenation for <kbd>kbd</kbd> tags is completely disabled. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>your_stylesheet.css</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idac233b730607'><button class='copyBtn' data-clipboard-target='#idac233b730607' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>body, .hyphen { hyphens: auto; } .nohyphen { hyphens: manual; } kbd { hyphens: none; }</pre> </div> <!-- endregion --> <!-- #region Complete Example --> <h2 id="example">Complete Example</h2> <p> The following example enables automatic hyphenation of the entire HTML <code>body</code>, according to US English rules. The HTML document contains nested tags, some of which disable or enable automatic hyphenation. One line demonstrates manual hyphenation, through the use of <a href='https://developer.mozilla.org/en-US/docs/Web/CSS/hyphens' target='_blank' rel="nofollow"><code>&amp;shy;</code> entities</a>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>index.html</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' style='margin-bottom: 1em;' id='idb6ffeec59097'><button class='copyBtn' data-clipboard-target='#idb6ffeec59097' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>&lt;html <span class="bg_yellow">lang="en-US"</span>> &lt;head> &lt;style> body, .hyphen { hyphens: auto; } .nohyphen { hyphens: manual; } &lt;/style> &lt;/head> &lt;body> &lt;p>This element is automatically hyphenated.&lt;/p> &lt;p <span class="bg_yellow">class="nohyphen"</span>>This element is not hyphenated.&lt;/p> &lt;ol <span class="bg_yellow">class="nohyphen"</span>> &lt;li>This elem<span class="bg_yellow">&amp;shy;</span>ent is manual<span class="bg_yellow">&amp;shy;</span>ly hyphen<span class="bg_yellow">&amp;shy;</span>ated.&lt;/li> &lt;li <span class="bg_yellow">class="hyphen"</span>>This element is automatically hyphenated.&lt;/li> &lt;/ol> &lt;/body> &lt;/html></pre> </div> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <p>Easy!</p> <!-- endregion --> <!-- #region Alternative Manual Hyphenation --> <h2 id="alt" class="clear">Alternative Manual Hyphenation</h2> <!-- #region implicit --> <p> Invisible hypenation is sometimes desirable. For example, if I have a long directory path, like <code>$HOME/.rbenv/versions/$RUBY_VERSION/lib/ruby/gems/$RUBY_VERSION/$GEM_NAME-$GEM_VERSION</code> I might want to tell the web browser where the string could be broken. </p> <p> You have two options &mdash; they are actually different syntaxes for the same <a href='https://en.wikipedia.org/wiki/Zero-width_space' target='_blank' rel="nofollow"><i>zero-width space</i></a>. </p> <!-- endregion --> <!-- #region Zero-Width Space HTML Entity --> <h3 id="alt">Zero-Width Space HTML Entity</h3> <p> <code>&amp;#8203;</code> is the HTML entity for a unicode character called the zero-width space (<a href='https://unicode-table.com/en/200B/' target='_blank' rel="nofollow"><code>ZWSP</code></a>). </p> <p> For example, <code>$HOME/.rbenv/<span class="bg_yellow">&amp;#8203;</span>versions/<span class="bg_yellow">&amp;#8203;</span>$RUBY_VERSION/<span class="bg_yellow">&amp;#8203;</span>lib/<span class="bg_yellow">&amp;#8203;</span>ruby/<span class="bg_yellow">&amp;#8203;</span>gems/<span class="bg_yellow">&amp;#8203;</span>$RUBY_VERSION/<span class="bg_yellow">&amp;#8203;</span>$GEM_NAME-<span class="bg_yellow">&amp;#8203;</span>$GEM_VERSION</code> <br><br>renders as:<br><br> <code>$HOME/.rbenv/versions/$RUBY_VERSION/&#8203;lib/&#8203;ruby/&#8203;gems/&#8203;$RUBY_VERSION/&#8203;$GEM_NAME-&#8203;$GEM_VERSION</code> </p> <!-- endregion --> <!-- #region Line Break Opportunity Tag --> <h3 id="alt">Line Break Opportunity Tag</h3> <p> You could also use the <a href='https://developer.mozilla.org/en-US/docs/Web/HTML/Element/wbr' target='_blank' rel="nofollow">line break opportunity tag</a>, <code>&lt;wbr&gt;</code> . </p> <p> For example, <code>$HOME/.rbenv/<span class="bg_yellow">&lt;wbr&gt;</span>versions/<span class="bg_yellow">&lt;wbr&gt;</span>$RUBY_VERSION/<span class="bg_yellow">&lt;wbr&gt;</span>lib/<span class="bg_yellow">&lt;wbr&gt;</span>ruby/<span class="bg_yellow">&lt;wbr&gt;</span>gems/<span class="bg_yellow">&lt;wbr&gt;</span>$RUBY_VERSION/<span class="bg_yellow">&lt;wbr&gt;</span>$GEM_NAME-<span class="bg_yellow">&lt;wbr&gt;</span>$GEM_VERSION</code> <br><br>renders as:<br><br> <code>$HOME/.rbenv/versions/$RUBY_VERSION/<wbr>lib/<wbr>ruby/<wbr>gems/<wbr>$RUBY_VERSION/<wbr>$GEM_NAME-<wbr>$GEM_VERSION</code> </p> <!-- endregion --> <!-- #region Both Options Yield the Same Rendered Text --> <h3 id="same">Both Options Yield the Same Rendered Text</h3> <p> I prefer to use &lt;wbr&gt;. <a href='https://dictionary.cambridge.org/dictionary/english/ymmv' target='_blank' rel="nofollow">YMMV</a>. </p> <!-- endregion --> <!-- endregion --> <!-- #region Non-Breaking Hyphen --> <h2 id="nbh">Non-Breaking Hyphen</h2> <p> The non-breaking hyphen HTML entity (<code>&amp;#8209;</code>) displays a hyphen character that does not break. </p> <p> For CSS, use <code>content: "\2011"</code> to produce a non-breaking hyphen. </p> <!-- endregion --> A Curmudgeon’s Social Networking 2023-01-07T00:00:00-05:00 https://mslinn.github.io/blog/2023/01/07/curmudgeon <p> I am an unapologetic curmudgeon. I am not here to please anyone else, I am only here to earnestly be myself to the best of my ability. </p> <p> If we do not continuously demonstrate that we endeavor to treat each other right, then we should be dismissed and forgotten. </p> <p> Bullshit walks; money drives away, unsatisfied. </p> <p> Fairness and equity are essential in all things without reservation. Otherwise, go away, life is short and you are just noise. </p> <h2 id="social">Facebook and LinkedIn</h2> <p> Twitter is now a circus. I don't need or want another circus in my communication. Do not look for me on Twitter any longer. </p> <p> If someone connects with me on Facebook or LinkedIn, I expect them to interact with me in person, on the phone, and to want to engage with me. Failing that, I disconnect from them without hesitation. </p> <p> Yes, please connect with me. Tell me your truth. Listen to me. I will listen to you. </p> <h2 id="action">Action Over Empty Words</h2> <p> Even better, we might endeavor to do things together. Imagining, conceptualizing, and building something real with others gives me joy. I want to share that joy. </p> <p> People who talk about what &lsquo;others&rsquo; should do, but are unwilling to act themselves, are a waste of time. They irritate me. Only approach me if you are an action-oriented individual. </p> <div class='quote'> The most difficult thing is the decision to act, the rest is merely tenacity. <span class='quoteAttribution'> &nbsp;&ndash; Amelia Earhart </span> </div> <p> Peace and love. <span style="font-size: 36pt;"><a href='https://en.wiktionary.org/wiki/Sigma' target='_blank' rel="nofollow">&#963;</a></span> </p> Working With Volumes and Directories Under Ubuntu 2022-12-01T00:00:00-05:00 https://mslinn.github.io/blog/2022/12/01/ubuntu-files <p> I've been reorganizing directories across several volumes on an Ubuntu server. This article documents how I used the commands I found to be most useful: <a href='https://linux.die.net/man/1/rsync' target='_blank' rel="nofollow"><code>rsync</code></a>, <a href='https://linux.die.net/man/1/ncdu' target='_blank' rel="nofollow"><code>ncdu</code></a>, and <a href='https://linux.die.net/man/1/fdupes' target='_blank' rel="nofollow"><code>fdupes</code></a>. The latter two programs are not installed by default. You can install them this way: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id74767bb48d64'><button class='copyBtn' data-clipboard-target='#id74767bb48d64' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install fdupes ncdu</pre> </div> <h2 id="ncdu">Display Directory Sizes</h2> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id599592e33c8e'><button class='copyBtn' data-clipboard-target='#id599592e33c8e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ncdu /directory</pre> </div> <h2 id="fdupes">Find Duplicate Files</h2> <p> This displays duplicate files and interactively asks which duplicate files to delete. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7ab898b0bbaf'><button class='copyBtn' data-clipboard-target='#id7ab898b0bbaf' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>fdupes -r1Sd /directory</pre> </div> <h2 id="comp">Compare Two Folders For Missing Files</h2> <p> This tip was inspired by an answer on <a href='https://unix.stackexchange.com/questions/524074/compare-two-folders-for-missing-files' target='_blank' rel="nofollow"><code>unix.stackexchange.com</code></a>. </p> <h3 id="dry">Dry Run</h3> <p> The <code>-n</code> option displays the names of missing files in the destination directory, but makes no changes to the destination file system. If that option is not specified, then the missing files are copied. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id66d14db1d948'><button class='copyBtn' data-clipboard-target='#id66d14db1d948' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sync -ri --ignore-existing -n /srcDir<span class="bg_yellow">/</span> /destDir</pre> </div> <p> There are a few things to note: </p> <ul> <li>The first directory is the source, and it must end with a slash (<kbd class="bg_yellow">/</kbd>).</li> <li>The second directory is the target, and it must NOT end with a slash.</li> </ul> <h3 id="doit">Copy Missing Files</h3> <p> Simply do not provide the <code>-n</code> (dry run) option: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0c0bf909d7df'><button class='copyBtn' data-clipboard-target='#id0c0bf909d7df' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>rsync -ri --ignore-existing /srcDir<span class="bg_yellow">/</span> /destDir</pre> </div> JiraCLI, a Feature-rich Interactive Jira Command Line 2022-08-12T00:00:00-04:00 https://mslinn.github.io/blog/2022/08/12/jiracli <!-- #region intro --> <p> When first released in 2002, <a href='https://www.atlassian.com/software/jira' target='_blank' rel="nofollow">Jira</a> was merely a bug tracker. Since then, it has gained features and is now used as a project management tool. Jira only provides a web user interface. </p> <p> <a href='https://github.com/ankitpokhrel/jira-cli' target='_blank' rel="nofollow">JiraCLI</a>, an independent F/OSS project, provides command-line explorers for Jira issues, epics, and sprints. </p> <!-- endregion --> <!-- #region Installation --> <h2 id="install">Installation</h2> <p> JiraCLI is a Go program, so it is quick and easy to install on Ubuntu once Go is installed. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idadc1a0a88bb7'><button class='copyBtn' data-clipboard-target='#idadc1a0a88bb7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install golang-go<br> <span class='unselectable'>$ </span>go install github.com/ankitpokhrel/jira-cli/cmd/jira@latest <span class='unselectable'>go: downloading github.com/ankitpokhrel/jira-cli v1.0.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/spf13/cobra v1.5.0 go: downloading github.com/spf13/viper v1.12.0 go: downloading github.com/zalando/go-keyring v0.2.1 go: downloading github.com/briandowns/spinner v1.18.1 go: downloading github.com/fatih/color v1.13.0 go: downloading github.com/mitchellh/go-homedir v1.1.0 go: downloading github.com/AlecAivazis/survey/v2 v2.3.5 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 go: downloading github.com/cpuguy83/go-md2man/v2 v2.0.2 go: downloading github.com/spf13/pflag v1.0.5 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/atotto/clipboard v0.1.4 go: downloading github.com/charmbracelet/glamour v0.5.0 go: downloading github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d go: downloading github.com/cli/safeexec v1.0.0 go: downloading github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 go: downloading github.com/kentaro-m/blackfriday-confluence v0.0.0-20220126124413-8e85477b49b3 go: downloading github.com/russross/blackfriday/v2 v2.1.0 go: downloading github.com/mattn/go-colorable v0.1.12 go: downloading github.com/mattn/go-isatty v0.0.14 go: downloading github.com/gdamore/tcell/v2 v2.5.1 go: downloading github.com/rivo/tview v0.0.0-20220610163003-691f46d6f500 go: downloading github.com/fsnotify/fsnotify v1.5.4 go: downloading github.com/mitchellh/mapstructure v1.5.0 go: downloading github.com/spf13/afero v1.8.2 go: downloading github.com/spf13/cast v1.5.0 go: downloading github.com/spf13/jwalterweatherman v1.1.0 go: downloading github.com/godbus/dbus/v5 v5.1.0 go: downloading golang.org/x/text v0.3.7 go: downloading golang.org/x/term v0.0.0-20220526004731-065cf7ba2467 go: downloading golang.org/x/sys v0.0.0-20220615213510-4f61da869c0c go: downloading github.com/subosito/gotenv v1.4.0 go: downloading github.com/hashicorp/hcl v1.0.0 go: downloading gopkg.in/ini.v1 v1.66.6 go: downloading github.com/magiconair/properties v1.8.6 go: downloading github.com/pelletier/go-toml/v2 v2.0.2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/gdamore/encoding v1.0.0 go: downloading github.com/pelletier/go-toml v1.9.5 go: downloading github.com/lucasb-eyer/go-colorful v1.2.0 go: downloading github.com/mattn/go-runewidth v0.0.13 go: downloading github.com/muesli/termenv v0.12.0 go: downloading github.com/yuin/goldmark v1.4.12 go: downloading github.com/yuin/goldmark-emoji v1.0.1 go: downloading github.com/rivo/uniseg v0.2.0 go: downloading github.com/alecthomas/chroma v0.10.0 go: downloading github.com/microcosm-cc/bluemonday v1.0.18 go: downloading github.com/muesli/reflow v0.3.0 go: downloading github.com/olekukonko/tablewriter v0.0.5 go: downloading github.com/aymerick/douceur v0.2.0 go: downloading golang.org/x/net v0.0.0-20220621193019-9d032be2e588 go: downloading github.com/dlclark/regexp2 v1.4.0 go: downloading github.com/gorilla/css v1.0.0 </span></pre> </div> <!-- endregion --> <p> Add <code>$HOME/go/bin</code> to the <code>PATH</code> so <code>jira</code> can be found: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd628dc395d04'><button class='copyBtn' data-clipboard-target='#idd628dc395d04' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>echo 'PATH=$HOME/go/bin:$PATH' >> ~/.bashrc<br> <span class='unselectable'>$ </span>source ~/.bashrc<br> <span class='unselectable'>$ </span>which jira <span class='unselectable'>/home/mslinn/go/bin/jira </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Usage --> <h2 id="usage">Usage</h2> <p>This is the <code>jira </code> help message:</p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id77ddcbd746d5'><button class='copyBtn' data-clipboard-target='#id77ddcbd746d5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira <span class='unselectable'>Interactive Jira CLI.<br/> USAGE jira [flags]<br/> MAIN COMMANDS board Board manages Jira boards in a project epic Epic manage epics in a project issue Issue manage issues in a project open Open issue in a browser project Project manages Jira projects sprint Sprint manage sprints in a project board<br/> OTHER COMMANDS completion Output shell completion code for the specified shell (bash or zsh) help Help about any command init Init initializes jira config man Help generate man(7) pages for Jira CLI. me Displays configured jira user version Print the app version information<br/> FLAGS -c, --config string Config file (default is /home/mslinn/.config/.jira/.config.yml) --debug Turn on debug output -h, --help help for jira -p, --project string Jira project to look into (defaults to /home/mslinn/.config/.jira/.config.yml)<br/> LEARN MORE Use &#39;jira &lt;command&gt; &lt;subcommand&gt; --help&#39; for more information about a command. </span></pre> </div> <!-- endregion --> <p> You need an Jira API token before you can use JiraCLI. Get it from <a href='https://id.atlassian.com/manage-profile/security/api-tokens' target='_blank' rel="nofollow">here</a>. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/atlassian_api_token.svg" type="image/svg"> <!---<source srcset="/blog/images/atlassian_api_token.avif" type="image/avif">--> <source srcset="/blog/images/atlassian_api_token.webp" type="image/webp"> <source srcset="/blog/images/atlassian_api_token.apng" type="image/apng"> <source srcset="/blog/images/atlassian_api_token.png" type="image/png"> <source srcset="/blog/images/atlassian_api_token.jpg" type="image/jpeg"> <source srcset="/blog/images/atlassian_api_token.jpeg" type="image/jpeg"> <source srcset="/blog/images/atlassian_api_token.jfif" type="image/jpeg"> <source srcset="/blog/images/atlassian_api_token.pjpeg" type="image/jpeg"> <source srcset="/blog/images/atlassian_api_token.pjp" type="image/jpeg"> <source srcset="/blog/images/atlassian_api_token.gif" type="image/gif"> <source srcset="/blog/images/atlassian_api_token.tif" type="image/tiff"> <source srcset="/blog/images/atlassian_api_token.tiff" type="image/tiff"> <source srcset="/blog/images/atlassian_api_token.bmp" type="image/bmp"> <source srcset="/blog/images/atlassian_api_token.ico" type="image/x-icon"> <source srcset="/blog/images/atlassian_api_token.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/atlassian_api_token.png" style='width: 100%; ' /> </picture> </div> <p> You need to configure the program before you use it. Set the <code>JIRA_API_TOKEN</code> environment variable before running <code>jira init</code>. If you do not, then you will get an error like <code>Received unexpected response '401 Unauthorized' from jira.</code> in the next step. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0f19340fb3ff'><button class='copyBtn' data-clipboard-target='#id0f19340fb3ff' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>export JIRA_API_TOKEN=asdfasdfasdf <span class='unselectable'>$ </span>jira init <span class='unselectable'>? Installation type: Cloud ? Link to Jira server: https://xxx.atlassian.net/ ? Login email: mslinn@mslinn.com ⠼ Verifying login details... ? Default project: My ? Default board: [Use arrows to move, type to filter, ? for more help] > [Search...] ---------- My Scrum Board None ✓ Configuration generated: /home/mslinn/.config/.jira/.config.yml </span></pre> </div> <!-- endregion --> <p> This is the configuration file that was generated: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>/home/mslinn/.config/.jira/.config.yml/</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddcb154997f0d'><button class='copyBtn' data-clipboard-target='#iddcb154997f0d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>board: id: 350 name: My Scrum Board type: scrum epic: name: customfield_10009 link: customfield_10008 installation: Cloud issue: fields: custom: - name: Epic Link key: customfield_10008 schema: datatype: any - name: Epic Name key: customfield_10009 schema: datatype: string types: - id: "3" name: Task handle: Task subtask: false - id: "5" name: Sub-task handle: Sub-task subtask: true - id: "7" name: Story handle: Story subtask: false - id: "1" name: Bug handle: Bug subtask: false - id: "6" name: Epic handle: Epic subtask: false login: mslinn@mslinn.com project: key: My type: classic server: https://xxx.atlassian.net</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Example Commands --> <h2 id="ex">Example Commands</h2> <p> Help for the <code>jira issue</code> subcommand: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3d1b01abbb2c'><button class='copyBtn' data-clipboard-target='#id3d1b01abbb2c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue -h <span class='unselectable'>Issue manage issues in a given project. See available commands below.<br/> USAGE jira issue [flags]<br/> MAIN COMMANDS assign Assign issue to a user clone Clone duplicates an issue comment Manage issue comments create Create an issue in a project delete Delete an issue edit Edit an issue in a project link Link connects two issues list List lists issues in a project move Transition an issue to a given state unlink Unlink disconnects two issues from each other view View displays contents of an issue worklog Manage issue worklog<br/> FLAGS -h, --help help for issue<br/> INHERITED FLAGS -c, --config string Config file (default is /home/mslinn/.config/.jira/.config.yml) --debug Turn on debug output -p, --project string Jira project to look into (defaults to /home/mslinn/.config/.jira/.config.yml)<br/> ALIASES issues<br/> LEARN MORE Use &#39;jira &lt;command&gt; &lt;subcommand&gt; --help&#39; for more information about a command. </span></pre> </div> <!-- endregion --> <p> Help for the <code>jira issue list</code> sub-subcommand: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0d8728ae7d3c'><button class='copyBtn' data-clipboard-target='#id0d8728ae7d3c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue list -h <span class='unselectable'>List lists issues in a given project.<br/> You can combine different flags to create a unique query. For instance,<br/> # Issues that are of high priority, is in progress, was created this month, and has given labels jira issue list -yHigh -s&quot;In Progress&quot; --created month -lbackend -l&quot;high prio&quot;<br/> Issues are displayed in an interactive list view by default. You can use a --plain flag to display output in a plain text mode. A --no-headers flag will hide the table headers in plain view. A --no-truncate flag will display all available fields in plain mode.<br/> USAGE jira issue list [flags]<br/> FLAGS -t, --type string Filter issues by type -R, --resolution string Filter issues by resolution type -s, --status string Filter issues by status -y, --priority string Filter issues by priority -r, --reporter string Filter issues by reporter (email or display name) -a, --assignee string Filter issues by assignee (email or display name) -C, --component string Filter issues by component -l, --label stringArray Filter issues by label -P, --parent string Filter issues by parent --history Issues you accessed recently -w, --watching Issues you are watching --created string Filter issues by created date Accepts: today, week, month, year, or a date in yyyy-mm-dd and yyyy/mm/dd format, or a period format using w = weeks, d = days, h = hours, m = minutes. eg: -10d Created filter will have precedence over created-after and created-before filter --updated string Filter issues by updated date Accepts: today, week, month, year, or a date in yyyy-mm-dd and yyyy/mm/dd format, or a period format using w = weeks, d = days, h = hours, m = minutes. eg: -10d Updated filter will have precedence over updated-after and updated-before filter --created-after string Filter by issues created after certain date --updated-after string Filter by issues updated after certain date --created-before string Filter by issues created before certain date --updated-before string Filter by issues updated before certain date -q, --jql string Run a raw JQL query in a given project context --order-by string Field to order the list with (default &quot;created&quot;) --reverse Reverse the display order (default &quot;DESC&quot;) --paginate string Paginate the result. Max 100 at a time, format: &lt;from&gt;:&lt;limit&gt; where &lt;from&gt; is optional (default &quot;0:100&quot;) --plain Display output in plain mode --no-headers Don&#39;t display table headers in plain mode. Works only with --plain --no-truncate Show all available columns in plain mode. Works only with --plain --columns string Comma separated list of columns to display in the plain mode. Accepts: TYPE, KEY, SUMMARY, STATUS, ASSIGNEE, REPORTER, PRIORITY, RESOLUTION, CREATED, UPDATED -h, --help help for list<br/> INHERITED FLAGS -c, --config string Config file (default is /home/mslinn/.config/.jira/.config.yml) --debug Turn on debug output -p, --project string Jira project to look into (defaults to /home/mslinn/.config/.jira/.config.yml)<br/> EXAMPLES $ jira issue list<br/> # Limit list to 20 items $ jira issue list --paginate 20<br/> # Get 50 items starting from 10 $ jira issue list --paginate 10:50<br/> # List issues in a plain table view without headers $ jira issue list --plain --no-headers<br/> # List some columns of the issue in a plain table view $ jira issue list --plain --columns key,assignee,status<br/> # List issues in a plain table view and show all fields $ jira issue list --plain --no-truncate<br/> # List issues of type &quot;Epic&quot; in status &quot;Done&quot; $ jira issue list -tEpic -sDone<br/> # List issues in status other than &quot;Open&quot; and is assigned to no one $ jira issue list -s~Open -ax<br/> # List issues from all projects $ jira issue list -q&quot;project IS NOT EMPTY&quot;<br/> ALIASES lists ls<br/> LEARN MORE Use &#39;jira &lt;command&gt; &lt;subcommand&gt; --help&#39; for more information about a command. </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Issues Updated on a Certain Date --> <h3 id="updated">Issues Updated on a Certain Date</h3> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1b41c664d5a7'><button class='copyBtn' data-clipboard-target='#id1b41c664d5a7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue list --updated 2021-11-17</pre> </div> <!-- endregion --> <!-- #region Issues Selected by Complex Criteria --> <h3 id="updated">Issues Selected by Complex Criteria</h3> <p> The following command will yield a list of high-priority issues, created this month, with status <code>To Do</code>, that are assigned to you, and have the label <code>backend</code>. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id596ad36ce4d3'><button class='copyBtn' data-clipboard-target='#id596ad36ce4d3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue list -yHigh -s"To Do" --created month -lbackend -a$(jira me)</pre> </div> <p> Output might look something like the following, which was redacted: </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/jira_issues.svg" type="image/svg"> <!---<source srcset="/blog/images/jira_issues.avif" type="image/avif">--> <source srcset="/blog/images/jira_issues.webp" type="image/webp"> <source srcset="/blog/images/jira_issues.apng" type="image/apng"> <source srcset="/blog/images/jira_issues.png" type="image/png"> <source srcset="/blog/images/jira_issues.jpg" type="image/jpeg"> <source srcset="/blog/images/jira_issues.jpeg" type="image/jpeg"> <source srcset="/blog/images/jira_issues.jfif" type="image/jpeg"> <source srcset="/blog/images/jira_issues.pjpeg" type="image/jpeg"> <source srcset="/blog/images/jira_issues.pjp" type="image/jpeg"> <source srcset="/blog/images/jira_issues.gif" type="image/gif"> <source srcset="/blog/images/jira_issues.tif" type="image/tiff"> <source srcset="/blog/images/jira_issues.tiff" type="image/tiff"> <source srcset="/blog/images/jira_issues.bmp" type="image/bmp"> <source srcset="/blog/images/jira_issues.ico" type="image/x-icon"> <source srcset="/blog/images/jira_issues.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/jira_issues.png" style='width: 100%; ' /> </picture> </div> <!-- endregion --> <!-- #region Issue Ids Only --> <h3 id="ids"><!-- #region Issue Ids Only --></h3> <p> The <code>--plain</code> and <code>--no-headers</code> options are useful for driving scripts. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id35d065ff1cfe'><button class='copyBtn' data-clipboard-target='#id35d065ff1cfe' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue list --updated 2021-11-17 --plain --no-headers --columns KEY</pre> </div> <!-- endregion --> <!-- #region View Issue --> <h3 id="detail">View Issue</h3> <p> The following returns the available information about an issue, in a plain-text format. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idfe5be108f6d5'><button class='copyBtn' data-clipboard-target='#idfe5be108f6d5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>jira issue view ISSUE-1 --comments 9999 --plain</pre> </div> <!-- endregion --> <!-- #region Verdict --> <h2 id="verdict">Verdict</h2> <p> JiraCLI is useful project! <span style='font-size: 3em;;'>&#x1F601;</span> </p> <!-- endregion --> ImageMagick Slicing on Ubuntu/WSL 2022-07-28T00:00:00-04:00 https://mslinn.github.io/blog/2022/07/28/imagemagick-slicing <!-- #region intro --> <p> Lawyers like the Microsoft Office software suite; so when I am working on a court case as an expert, I endeavor to provide my clients with Word documents that contain necessary information. I like working in WSL/WSL2 because I can use Windows programs and Ubuntu programs together effectively. </p> <!-- endregion --> <!-- #region Grab Image, Then Slice --> <h2 id="steps">Grab Image, Then Slice</h2> <p> Recently, I used <a href='https://www.techsmith.com/screen-capture.html' target='_blank' rel="nofollow">SnagIt</a>, a Windows program, to capture large web pages as single images. Some of these images were quite tall. </p> <p> The CSS for the web pages made some content invisible on the printed page. Yes, I could have injected CSS using a Chrome plugin like <a href='https://chrome.google.com/webstore/detail/my-style/ljdhjpmbnkbengahefamnhmegbdifhlb' target='_blank' rel="nofollow">My Style</a> to ensure that all content will be printed, like this: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Injected Style</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id34f809e06a94'><button class='copyBtn' data-clipboard-target='#id34f809e06a94' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>@media print { * { display: initial; visibility: visible; } }</pre> </div> <!-- endregion --> <p> I decided to use screen grabs, which would guarantee that the contents of my report would exactly match what had been displayed on the screen, without injecting anything into the web pages. </p> <p> <a href='https://imagemagick.org/index.php' target='_blank' rel="nofollow">ImageMagick</a> is preinstalled on Ubuntu Desktop. I used ImageMagick to slice the image captures into smaller page-sized images, so they could be inserted into a Word document. </p> <!-- endregion --> <!-- #region The Computer Worked Hard --> <h2 id="grunt">The Computer Worked Hard</h2> <p> Grabbing such large web pages was a lot of work for my desktop computer. The only programs active during the screen grab process were the Google Chrome browser and SnagIt. I found that 10GB RAM and 30% of the GPU capability (an NVidia GTX 1660 Super) was used. </p> <p> The screen grab failed if I did not start scrolling from the top of the web page; while it is possible to scrub up and down smaller web pages in order to grab portions of interest, this fails for large pages. </p> <p> I also found that scrolling too fast caused the screen grabbing process to fail. Clicking and holding the bottom scroll arrowhead at the bottom right of the screen seemed to result in a smooth and optimal scrolling speed. This meant that grabbing large web pages took a few minutes as the page slowly scrolled downward. </p> <!-- endregion --> <!-- #region Setting Up the Conversion --> <h2 id="setup">Setting Up the Conversion</h2> <p> The Word documents I usually work with are formatted for North American standards. This means one-inch margins on letter-sized paper (8.5" x 11"), which gives a working area of 6.5" x 9", yielding an aspect ratio of 0.72. </p> <p> The tall captured images needed to be sliced into rectangles that fit efficiently into Word documents. The computations are as follows. </p> <ol> <li> Determine the width of a screen grab and save it into <code>W</code>. The ImageMagick <code>identify</code> command does not provide a newline after its output, however I have inserted one for readability: <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd78a1d243c06'><button class='copyBtn' data-clipboard-target='#idd78a1d243c06' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>identify -ping -format '%w' ../IMG2005.png <span class='unselectable'>1536 </span> <span class='unselectable'>$ </span>export W="$( identify -ping -format '%w' ../IMG2005.png )"</pre> </div> </li> <li> Determine the height of a screen grab and save it into <code>H</code>: <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida8c9c41af03d'><button class='copyBtn' data-clipboard-target='#ida8c9c41af03d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>identify -ping -format '%h' ../IMG2005.png <span class='unselectable'>$ </span>export H="$( identify -ping -format '%h' ../IMG2005.png )"</pre> </div> </li> <li> The width can be divided by the aspect ratio to obtain the desired height of each slice so they can be inserted optimally into the Word documents. I used the <a href='https://linux.die.net/man/1/bc' target='_blank' rel="nofollow"><code>bc</code> calculator</a> provided with <code>Bash</code> to divide <code>W / ASPECT_RATIO</code>. The <code>H2</code> integer variable contains the computed height for the images. <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6d654d3d9447'><button class='copyBtn' data-clipboard-target='#id6d654d3d9447' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>export ASPECT_RATIO=0.72 <span class='unselectable'>$ </span>export H2="$( echo "scale=0 ; $W / $ASPECT_RATIO" | bc )"</pre> </div> </li> <li> Now the image called <code>IMG2005.png</code> can be sliced using ImageMagick’s convert command. The slices are stored into a subdirectory called <code>slices</code>, with file names like <code>IMG2005-1.jpg</code>, <code>IMG2005-2.jpg</code>, etc. <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2b577165d1e8'><button class='copyBtn' data-clipboard-target='#id2b577165d1e8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>convert IMG2005.png -crop ${W}x${H2} \ -quality 100% -scene 0 slices/IMG2005-%d.jpg</pre> </div> </li> </ol> <!-- endregion --> <!-- #region Automating the Conversion --> <h2 id="script">Automating the Conversion</h2> <p> I wrote the following bash script, which incorporates the above computations. It slices all the images in a directory and saves the results to a second directory. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>sliceImages</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd65870be46a3'><button class='copyBtn' data-clipboard-target='#idd65870be46a3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash<br> function help { if [ "$1" ]; then echo "Error: $1"; fi echo " $(basename $0): slice all images in the given directory and place them into a specified directory, which will be created if required. " exit 1 }<br> function setup { export ASPECT_RATIO=0.72 export W="$( identify -ping -format '%w' "$1" )" export H="$( identify -ping -format '%h' "$1" )" export H2="$( echo "scale=0 ; $W / $ASPECT_RATIO" | bc )" }<br> function convert1 { FULLNAME=$(basename -- "$1") FILENAME="${FULLNAME%.*}" FILETYPE="${FULLNAME##*.}"<br> convert "$1" \ -crop "${W}x${H2}" \ -quality 100% \ -scene 0 \ "$DIR_OUTPUT/$FILENAME-%d.png" }<br><br> if [ -z "$1" ]; then help "No directory path for images to be converted was provided."; fi export DIR_INPUT="$( realpath $1 )"<br> if [ -z "$2" ]; then help "No directory path for the image slices to be saved into was provided."; fi export DIR_OUTPUT="$( realpath $2 )"<br> mkdir -p "$DIR_OUTPUT"<br> find $DIR_INPUT -type f -exec file --mime-type {} \+ | awk -F: '{if ($2 ~/image\//) print $1}' | while read FILE; do setup "$FILE" echo "Slicing $FILE into ${W}x${H2} pixels" convert1 "$FILE" done</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Overcoming ImageMagick Processing Limits --> <h2 id="limits">Overcoming ImageMagick Processing Limits</h2> <p> Some of the web pages that I needed to grab were quite long, which resulted in those images requiring more computational resources than the default Imagemagick configuration allows. This caused errors such as the following to appear: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id1e1e498217eb'><button class='copyBtn' data-clipboard-target='#id1e1e498217eb' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>convert-im6.q16: no images defined `/mnt/c/images/slices/IMG1466-%d.png' @ error/convert.c/ConvertImageCommand/3229. convert-im6.q16: cache resources exhausted `/mnt/c/images/IMG1091.png' @ error/cache.c/OpenPixelCache/4095.</pre> </div> <p> Imagemagick defines computational resources limits in <code>/etc/ImageMagick-6/policy.xml</code>. The default maximum memory is 256 KB, the default maximum allowable height is 16,000 pixels (16KP), and the default maximum area is 128M pixels. These values are defined by the following entries: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>/etc/ImageMagick-6/policy.xml</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idcb2c06c8aaf2'><button class='copyBtn' data-clipboard-target='#idcb2c06c8aaf2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>&lt;policy domain="resource" name="memory" value="256MiB"/> &lt;policy domain="resource" name="height" value="16KP"/> &lt;policy domain="resource" name="area" value="128MP"/></pre> </div> <p> I changed the maximum memory limit to 2 GB RAM, the maximum height limit to 10,000,000 pixels (10MP), and the maximum area limit to 2G pixels with these entries: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>/etc/ImageMagick-6/policy.xml</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0eda698095f4'><button class='copyBtn' data-clipboard-target='#id0eda698095f4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>&lt;policy domain="resource" name="memory" value="2GiB"/> &lt;policy domain="resource" name="height" value="10MP"/> &lt;policy domain="resource" name="area" value="2GP"/></pre> </div> <p> Alternatively, I could have simply commented out the limits, as shown in highlighted text below. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>/etc/ImageMagick-6/policy.xml</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf17f2436cbd1'><button class='copyBtn' data-clipboard-target='#idf17f2436cbd1' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class="bg_yellow">&lt;!--</span> &lt;policy domain="resource" name="memory" value="256MiB"/> &lt;policy domain="resource" name="height" value="16KP"/> &lt;policy domain="resource" name="area" value="128MP"/> <span class="bg_yellow">--></span></pre> </div> <p> The largest web page to be sliced was converted to a very tall image, which was 83,703 pixels high. It was sliced into 40 images. </p> <!-- endregion --> <!-- #region Word Macro --> <h2 id="macro">Word Macro</h2> <p> A Word macro is also needed to insert the images into the currently open Word document in alphabetical order. I modified <a href='https://software-solutions-online.com/word-vba-insert-images/' target='_blank' rel="nofollow">this one</a>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Microsoft Word Macro</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idde6f0c90fa63'><button class='copyBtn' data-clipboard-target='#idde6f0c90fa63' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>Sub insertImages() Dim intResult As Integer Dim strPath As String Dim strFolderPath As String Dim objFSO As Object Dim objFolder As Object Dim objFile As Object Dim i As Integer<br> intResult = Application.FileDialog(msoFileDialogFolderPicker).Show 'Check if user canceled the dialog If intResult <> 0 Then 'dispaly message box strFolderPath = Application.FileDialog(msoFileDialogFolderPicker).SelectedItems(1) 'Create an instance of the FileSystemObject Set objFSO = CreateObject("Scripting.FileSystemObject") 'Get the folder object Set objFolder = objFSO.GetFolder(strFolderPath) i = 1 'loops through each file in the directory and prints their names and path For Each objFile In objFolder.Files 'get file path strPath = objFile.Path 'insert the image Selection.InlineShapes.AddPicture FileName:= _ strPath, LinkToFile:=False, _ SaveWithDocument:=True Next objFile End If End Sub</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Done! --> <h2 id="done">Done!</h2> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <p> Thanks to the above automation, I was able to deliver the Word documents containing the sliced web pages to my client soon after they were requested. </p> <!-- endregion --> Optimizing Plex Media Server on Ubuntu 2022-07-23T00:00:00-04:00 https://mslinn.github.io/blog/2022/07/23/optimizing-plex <p> <a href='https://www.plex.tv/media-server-downloads/#plex-media-server' target='_blank' rel="nofollow">Plex Media Server</a> is a free, full-featured media manager that is available for Linux, Windows, Mac, and is built into some NAS devices. I run it along with other processes on an Ubuntu server in my apartment. </p> <div class='imgWrapper imgFlex inline' style=' '> <figure> <a href='https://support.plex.tv/articles/200288286-what-is-plex/' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/plex/plex-works-matrix-1.svg" type="image/svg"> <!---<source srcset="/blog/images/plex/plex-works-matrix-1.avif" type="image/avif">--> <source srcset="/blog/images/plex/plex-works-matrix-1.webp" type="image/webp"> <source srcset="/blog/images/plex/plex-works-matrix-1.apng" type="image/apng"> <source srcset="/blog/images/plex/plex-works-matrix-1.png" type="image/png"> <source srcset="/blog/images/plex/plex-works-matrix-1.jpg" type="image/jpeg"> <source srcset="/blog/images/plex/plex-works-matrix-1.jpeg" type="image/jpeg"> <source srcset="/blog/images/plex/plex-works-matrix-1.jfif" type="image/jpeg"> <source srcset="/blog/images/plex/plex-works-matrix-1.pjpeg" type="image/jpeg"> <source srcset="/blog/images/plex/plex-works-matrix-1.pjp" type="image/jpeg"> <source srcset="/blog/images/plex/plex-works-matrix-1.gif" type="image/gif"> <source srcset="/blog/images/plex/plex-works-matrix-1.tif" type="image/tiff"> <source srcset="/blog/images/plex/plex-works-matrix-1.tiff" type="image/tiff"> <source srcset="/blog/images/plex/plex-works-matrix-1.bmp" type="image/bmp"> <source srcset="/blog/images/plex/plex-works-matrix-1.ico" type="image/x-icon"> <source srcset="/blog/images/plex/plex-works-matrix-1.cur" type="image/x-icon"> <img alt='From Plex.tv documentation' class="imgImg rounded shadow" src="/blog/images/plex/plex-works-matrix-1.png" style='width: 100%; ' title='From Plex.tv documentation' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://support.plex.tv/articles/200288286-what-is-plex/" target='_blank' > From Plex.tv documentation </a> </figcaption> </figure> </div> <p> We can learn a lot about the Plex Media Server processes using the information provided by <code>systemctl status</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id940ba8a52979'><button class='copyBtn' data-clipboard-target='#id940ba8a52979' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl status plexmediaserver <span class='unselectable'>● plexmediaserver.service - Plex Media Server Loaded: loaded (/lib/systemd/system/plexmediaserver.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2022-07-22 12:13:32 EDT; 23h ago Process: 2307 ExecStartPre=/bin/sh -c /usr/bin/test -d "${PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR}" || /bin/mkdir -p "${PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR}" (code=exited, status=0/SUCCESS) Main PID: 2329 (Plex Media Serv) Tasks: 45 (limit: 38369) Memory: 25.2G CPU: 42min 54.673s CGroup: /system.slice/plexmediaserver.service ├─2329 "/usr/lib/plexmediaserver/Plex Media Server" ├─3404 "Plex Plug-in [com.plexapp.system]" /usr/lib/plexmediaserver/Resources/Plug-ins-6b0e31a64/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.27.1.5916-6b0e31a64 /usr/lib/plexmediaserver/Resources/Plug-ins-6b0e31a64/System.bundle └─3521 "/usr/lib/plexmediaserver/Plex Tuner Service" /usr/lib/plexmediaserver/Resources/Tuner/Private /usr/lib/plexmediaserver/Resources/Tuner/Shared 1.27.1.5916-6b0e31a64 32600 </span></pre> </div> <p> The above tells us that Plex Media Server runs under a <a href='https://man7.org/linux/man-pages/man7/cgroups.7.html' target='_blank' rel="nofollow"><code>cgroup</code></a>, which is &ldquo;a Linux kernel feature which allow processes to be organized into hierarchical groups whose usage of various types of resources can then be limited and monitored&rdquo;. The <code>cgroup</code> is defined in <code>/lib/systemd/system/plexmediaserver.service</code>. Here is mine: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>/lib/systemd/system/plexmediaserver.service</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id504ca5d74a41'><button class='copyBtn' data-clipboard-target='#id504ca5d74a41' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># DO NOT EDIT THIS FILE DIRECTLY! # # Plex Media Server's variables can be customized by creating an 'overide.conf' # file using 'systemctl edit plexmediaserver' which will create the following; # /etc/systemd/system/plexmediaserver.service.d/override.conf # # An example of the override.conf would be as follows if you wished to edit # your user, group, temp directory, or app support directory (without the leading #) # # [Service] # Environment="TMPDIR=/path/to/new/tmp" # Environment="PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR=/home/myusername/Library/Application Support" # User=myusername # Group=mygroup # [Unit] Description=Plex Media Server After=network.target network-online.target [Service] Environment="PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR=/var/lib/plexmediaserver/Library/Application Support" Environment=PLEX_MEDIA_SERVER_HOME=/usr/lib/plexmediaserver Environment=PLEX_MEDIA_SERVER_MAX_PLUGIN_PROCS=6 ExecStartPre=/bin/sh -c '/usr/bin/test -d "${PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR}" || /bin/mkdir -p "${PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR}"' ExecStart=/bin/sh -c '\ export PLEX_MEDIA_SERVER_INFO_VENDOR="$(grep ^NAME= /etc/os-release | awk -F= "{print \\$2}" | tr -d \\" )"; \ export PLEX_MEDIA_SERVER_INFO_DEVICE="PC"; \ export PLEX_MEDIA_SERVER_INFO_MODEL="$(uname -m)"; \ export PLEX_MEDIA_SERVER_INFO_PLATFORM_VERSION="$(grep ^VERSION= /etc/os-release | awk -F= "{print \\$2}" | tr -d \\" )"; \ exec "/usr/lib/plexmediaserver/Plex Media Server"' Type=simple User=plex Group=plex Restart=on-failure RestartSec=5 StartLimitInterval=60s StartLimitBurst=3 SyslogIdentifier=Plex Media Server StandardOutput=journal StandardError=journal [Install] WantedBy=multi-user.target</pre> </div> <h2 id="dirs">Directories and Files</h2> <p> On Linux, Plex Media Server uses the following directories and files by default. These are defined by the <code>cgroup</code> information shown above. </p> <table> <tr> <td> Programs </td> <td> <code>/usr/lib/plexmediaserver/</code> </td> </tr> <tr> <td> Service Configuration </td> <td> <code>/usr/lib/plexmediaserver/lib/plexmediaserver.service</code> </td> </tr> <tr> <td> Content library metadata </td> <td> <code>/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/</code> </td> </tr> </table> <h2 id="priority">Adjusting CPU Priority</h2> <p> By default, the Plex processes run at nice factor 35 (priority 20), which is standard. When transcoding this means that most other processes, such as web servers, must wait. Transcoding can take a while, which means that the websites served from the same server time out. </p> <p> There are several ways to control how much CPU resource is allocated to a Linux process. I ran my Plex processes under <a href='https://linux.die.net/man/1/nice' target='_blank' rel="nofollow">nice</a> so other processes can continue unimpeded. <code>Nice</code> only throttles the process it manages when other processes request a time slice. If the system load is low then <code>nice</code> does not throttle the process that is supervises. </p> <p> All the Plex processes run under Linux as user <code>plex</code>, group <code>plex</code>. It is easy to change the priority of these processes using <code>nice</code>. The following changes all currently running Plex processes to priority 22:</p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id33ea7443f73d'><button class='copyBtn' data-clipboard-target='#id33ea7443f73d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo renice 22 -u plex</pre> </div> <p> I noticed that the video stream would stall at this priority, so I raised it by 10% (10% "less nice" equates to renicing by -2). <a href='https://www.tecmint.com/set-linux-process-priority-using-nice-and-renice-commands/' target='_blank' rel="nofollow">Aaron Kili</a> wrote a nice explanation of <code>renice</code>. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7d5d6ed5fed8'><button class='copyBtn' data-clipboard-target='#id7d5d6ed5fed8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo renice -2 -u plex</pre> </div> <p> One way to ensure that Plex Media Server always runs at normal priority is to add this to the root <code>crontab</code>: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>sudo crontab -e</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2ede87fcf230'><button class='copyBtn' data-clipboard-target='#id2ede87fcf230' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>@reboot sleep 60 && renice 18 -u plex</pre> </div> <h2 id="rygel">Rygel</h2> <p> While playing with Plex on Ubuntu I noticed that <a href='https://wiki.gnome.org/Projects/Rygel' target='_blank' rel="nofollow"><code>rygel</code></a> became active when Plex started transcoding. Coincidence? I did not know about <code>rygel</code> before. The <a href='https://github.com/GNOME/rygel' target='_blank' rel="nofollow">GitHub repo</a>. </p> Uncomplicated Firewall on Ubuntu 2022-07-16T00:00:00-04:00 https://mslinn.github.io/blog/2022/07/16/ufw <p> All websites, especially ecommerce sites, need to be secure. A properly set up firewall is an essential component for a secure server. </p> <p> Ubuntu 22.04 uses the <a href='https://wiki.ubuntu.com/UncomplicatedFirewall' target='_blank' rel="nofollow">Uncomplicated Firewall <code>ufw</code> firewall</a> frontend by default. <code>Ufw</code> has been provided for Ubuntu since v8.04 (Hardy Heron). </p> <h2 id="tldr">Quick Setup</h2> <p> Enable <code>ufw</code> as follows: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id7d501577c564'><button class='copyBtn' data-clipboard-target='#id7d501577c564' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo ufw enable <span class='unselectable'>Firewall is active and enabled on system startup </span></pre> </div> <p> Enable the <a href='https://github.com/ageis/ufw-application-profiles' target='_blank' rel="nofollow"><code>ufw</code> application profiles</a> for <code>ssh</code> and nginx (HTTP/HTTPS) like this: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc65f3e2d342c'><button class='copyBtn' data-clipboard-target='#idc65f3e2d342c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo ufw allow OpenSSH <span class='unselectable'>Output Rule added Rule added (v6) </span> <span class='unselectable'>$ </span>sudo ufw allow 'Nginx Full' <span class='unselectable'>Output Rule added Rule added (v6) </span></pre> </div> <h3 id="ufwDetails">Diving Deeper</h3> <p> The following is mostly true: </p> <div class='quote'> <div class='quoteText clearfix'> The default firewall on Ubuntu 22.04 Jammy Jellyfish is <code>ufw</code>, which is short for “uncomplicated firewall.” <code>Ufw</code> is a frontend for the typical Linux <code>iptables</code> commands, but it is developed in such a way that basic firewall tasks can be performed without the knowledge of <code>iptables</code>. <br><br> Additionally, <code>ufw</code> can be managed from a graphical interface. In this tutorial, you will learn how to enable and disable the <code>ufw</code> firewall on Ubuntu 22.04 Jammy Jellyfish from both command line and GUI. </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://linuxconfig.org/how-to-enable-disable-firewall-on-ubuntu-22-04-lts-jammy-jellyfish-linux' rel='nofollow' target='_blank'>How to enable/disable firewall on Ubuntu 22.04 LTS Jammy Jellyfish Linux</a></div> </div> <p> The above makes no mention of how Ubuntu 22.04 replaced <code>iptables</code> with <code>nftables</code>, as described below. </p> <div class='quote'> <div class='quoteText clearfix'> <h2><span class="code">nftables</span> as the default firewall backend</h2> Firewalling on Linux consists of two components &ndash; the firewall mechanism within the Linux kernel, and the tools used to configure this from userspace. The Linux kernel has traditionally supported two different subsystems for firewall policies – <code>iptables</code> / <code>xtables</code> and the newer <code>nftables</code>. <br><br> <code>Nftables</code> brings significant benefits both in terms of performance and flexibility when creating and deploying firewall rules, particularly for dual stack IPv4/IPv6 systems. <br><br> The traditional <code>iptables</code> userspace management tool now configures the <code>nftables</code> kernel backend, whilst the new <code>nft</code> userspace tool is also present to allow the creation of more flexible rules not supported by the traditional iptables paradigm. </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://ubuntu.com/blog/whats-new-in-security-for-ubuntu-22-04-lts/docs-internal-guid-117e0493-7fff-ea13-3537-bde70965f89d' rel='nofollow' target='_blank'>What’s new in Security for Ubuntu 22.04 LTS?</a></div> </div> <p> Digital Ocean has a <a href='https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-18-04' target='_blank' rel="nofollow">good <code>ufw</code> tutorial</a>. </p> Using Nginx As a Reverse Proxy With SSL 2022-07-08T00:00:00-04:00 https://mslinn.github.io/blog/2022/07/08/reverse-proxy <div class="right"> <div class='imgWrapper imgFlex inline' style=' '> <figure> <a href='https://www.ScalaCourses.com' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/svg"> <!---<source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/avif">--> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/webp"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/apng"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/png"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/jpeg"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/jpeg"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/jpeg"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/jpeg"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/jpeg"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/gif"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/tiff"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/tiff"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/bmp"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/x-icon"> <source srcset="/assets/images/ScalaCoursesLogo207x207.png" type="image/x-icon"> <img alt='ScalaCourses.com' class="imgImg rounded shadow" src="/assets/images/ScalaCoursesLogo207x207.png" style='width: 100%; ' title='ScalaCourses.com' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://www.ScalaCourses.com" target='_blank' > ScalaCourses.com </a> </figcaption> </figure> </div> </div> <p> Recently I <a href='https://mslinn.com/blog/2022/05/26/aws-hijacking.html' target='_blank' rel="nofollow">migrated</a> ScalaCourses.com from AWS EC2/S3/CloudFront to a server in my apartment, which has fiber optic internet service. The server&rsquo;s motherboard is an ASUS Sabertooth x79 with an Intel i7 4820, 32 GB DDR3 RAM, and a 4TB SATA SSD. This motherboard is unable to boot from NVMe drives, so SATA is necessary. At the time of this writing, the server runs Ubuntu 22.04. </p> <p> The backup server is currently being set up. I am repurposing an old Hackintosh as a fallback to the Ubuntu server. That motherboard is another ASUS Sabertooth x79, with 32 GB DDR3 RAM and two 2 TB SATA drives. </p> <h2 id="why">Why Am I Doing This?</h2> <p> Once again, I control my hardware, my software, my network, and all ancillary services. After several years of using PaaS vendor servers, I am now reverting to running ScalaCourses.com on my own hardware and software, using my own network. </p> <div class="pullQuote"> No longer will I subject myself to the unlimited financial liability that current PaaS vendors expose their customers to. </div> <h2 id="defs">Definitions</h2> <p> A <i>proxy</i> is a person or process serving as an authorized agent or substitute for another. In computer science, a more specific term is <i>forward proxy</i>. <br><br> A <i>proxy server</i> is a server process that acts as an intermediary between a client requesting a resource, and the process that provides the resource. <br><br> A <i>reverse proxy</i> is a process that sits in front of other processes, and forwards client requests to them. The term <i>forwarding process</i> is similar to <i>reverse proxy</i>. <br><br> The definitions for <i>proxy</i>, <i>forward proxy</i> and <i>reverse proxy</i> all sound identical. The key difference between a reverse proxy and a forward proxy is that a <b>forward proxy enables computers isolated on a private network</b> to connect to the public internet, while a <b>reverse proxy enables computers on the internet</b> to access a private subnet. </p> <h2 id="details">Dealing With Details</h2> <p> The old <a href='https://en.wikipedia.org/wiki/Pound_(networking)' target='_blank' rel="nofollow">Pound</a> v2.8-2 <a href='https://www.cloudflare.com/en-ca/learning/cdn/glossary/reverse-proxy/' target='_blank' rel="nofollow">reverse proxy</a> that was the front end for the old Play Framework app that runs <a href='https://scalacourses.com' target='_blank' rel="nofollow">ScalaCourses.com</a> is no longer viable, and the new version 3 of Pound is incomplete. Depending on configuration, reverse proxies can provide extra security from external attacks on a website, decrypt https requests to http, and act as stream editors for the content. </p> <p> This site still uses AWS CloudFront, for the moment. I am researching alternatives, with the goal of closing my AWS account... unless they introduce a way to cap financial liability before I fully migrate off AWS. </p> <h2 id="vs">Apache httpd vs. Nginx</h2> <p> Two popular reverse proxies are <a href='https://httpd.apache.org/docs/trunk/mod/mod_proxy_http2.html' target='_blank' rel="nofollow">Apache <code>mod_proxy_http2</code></a> and <a href='https://www.nginx.com/' target='_blank' rel="nofollow">nginx</a>. Anything Pound can do, both nginx and Apache httpd/2 can do. While they both work well, Apache httpd has an older code base, in fact, many of my websites ran on Apache httpd at the turn of the century. </p> <p> Nginx is relatively newer than Apache httpd. It is performant, well supported, well documented, and widely available. I decided to use nginx as the reverse proxy because I had found that nginx worked well on previous projects. </p> <p> Installing nginx and the Letsencrypt software is easy on Debian distros such as Ubuntu: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id91b834e505f7'><button class='copyBtn' data-clipboard-target='#id91b834e505f7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install nginx certbot python3-certbot-nginx</pre> </div> <h2 id="vs">Verifying the Nginx Build</h2> <p> When acting as a proxy server, nginx requires the <code>http_sub_module</code> to translate HTTP content. This allows the website content to be stream edited, so various links and paths that are not translated properly can be fixed up. </p> <p> The nginx package provided by Ubuntu includes the <code>&#8209;&#8209;with&#8209;http_sub_module</code> option. You can verify that your instance of nginx was built with the <code>&#8209;&#8209;with&#8209;http_sub_module</code> option as follows: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id77ca9c957214'><button class='copyBtn' data-clipboard-target='#id77ca9c957214' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>nginx -V <span class='unselectable'>nginx version: nginx/1.18.0 (Ubuntu) built with OpenSSL 3.0.2 15 Mar 2022 TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -ffile-prefix-map=/build/nginx-9P0wNJ/nginx-1.18.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --add-dynamic-module=/build/nginx-9P0wNJ/nginx-1.18.0/debian/modules/http-geoip2 --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module <span class="bg_yellow">--with-http_sub_module</span> </span></pre> </div> <h2 id="vs">Nginx SSL Configuration</h2> <p> Two files are needed for Letsencrypt to be able to create SSL certificates for nginx: </p> <ol> <li> The contents of <a href='https://github.com/certbot/certbot/blob/master/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf' target='_blank' rel="nofollow"><code>/etc/letsencrypt/options-ssl-nginx.conf</code></a> need to be stored into <code>/etc/letsencrypt/options-ssl-nginx.conf</code>. I later discovered this file was also available locally at <code>/usr/lib/python3/dist-packages/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf</code>. </li> <li> The contents of <a href='https://github.com/certbot/certbot/blob/master/certbot/certbot/ssl-dhparams.pem' target='_blank' rel="nofollow"><code>/etc/letsencrypt/ssl-dhparams.pem</code></a> need to be stored into <code>/etc/letsencrypt/ssl-dhparams.pem</code>. I later discovered that this file was also available locally at <code>/usr/lib/python3/dist-packages/certbot/ssl-dhparams.pem</code>. </li> </ol> <h2 id="ssl">Making a Wildcard SSL Certificate</h2> <p> I knew an SSL wildcard certificate would be needed, so I made one using Letsencrypt. Please read <a href='/blog/2022/06/15/certbot.html'>Creating and Renewing Letsencrypt Wildcard SSL Certificates</a> for details. </p> <p class="alert rounded shadow"> Update &ndash; 8 months after writing this article, I wrote about a better way to generate wildcard SSL certificates: <a href='/blog/2023/03/02/lego.html'>Wildcard SSL Certificates for Let&apos;s Encrypt with Lego</a>. </p> <h2 id="vs">Defining the Nginx Website Reverse Proxy</h2> <p> Please see my article entitled <a href='/blog/2021/03/20/cors.html'>Cross-Origin Resource Sharing (CORS)</a> for a discussion of how to configure servers whose content needs to be proxied. </p> <p> I saved the following configuration in a new file called <code>/etc/nginx/sites-available/scalacourses.com</code>. Note that a single <code>server</code> block answers on ports 80 and 443, using IPv4 and IPv6, for SSL and non-SSL requests, for the apex domain (<code>scalacourses.com</code>) and the <code>www.scalacourses.com</code> subdomain. No redirects are used. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id40415029056f'><button class='copyBtn' data-clipboard-target='#id40415029056f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>server { listen 80; listen [::]:80; listen 443 ssl; listen [::]:443 ssl; server_name scalacourses.com www.scalacourses.com; ssl_certificate /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/fullchain.pem; ssl_certificate_key /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/privkey.pem; ssl_trusted_certificate /home/mslinn/.certbot/scalacourses.com/config/live/scalacourses.com/chain.pem; include /etc/letsencrypt/options-ssl-nginx.conf; ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; root /var/www/html; index index.html; # This gets served if the proxied website is down location / { proxy_pass http://localhost:9000/; proxy_set_header Accept-Encoding ""; # sub_filter requires this sub_filter 'https://localhost:9000/authenticate/' 'https://www.scalacourses.com/authenticate/'; sub_filter_once off; } }</pre> </div> <p> I disabled the <code>default</code> site: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id15c66c2011ac'><button class='copyBtn' data-clipboard-target='#id15c66c2011ac' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo rm /etc/nginx/sites-enabled/default</pre> </div> <p> I enabled the new <code>scalacourses.com</code> site: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id40792e888a68'><button class='copyBtn' data-clipboard-target='#id40792e888a68' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo ln /etc/nginx/sites-{available,enabled}/scalacourses.com</pre> </div> <p> The <code>nginx</code> configuration was tested for syntax: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2c85012eec40'><button class='copyBtn' data-clipboard-target='#id2c85012eec40' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo nginx -t <span class='unselectable'>nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful </span></pre> </div> <p> The nginx configuration was reloaded: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide32c55161428'><button class='copyBtn' data-clipboard-target='#ide32c55161428' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl reload nginx</pre> </div> <h2 id="flush">Flush DNS Cache</h2> <p> Ensure that DNS requests receive up-to-date values by flushing the DNS cache. The following works on Ubuntu, but not when running on WSL/WSL2. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idee21e5568b94'><button class='copyBtn' data-clipboard-target='#idee21e5568b94' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo resolvectl flush-caches</pre> </div> <p> The following works on Windows 10: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Command and PowerShell consoles</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida4350769d60b'><button class='copyBtn' data-clipboard-target='#ida4350769d60b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>C:\Users\Mike Slinn> </span>ipconfig /flushdns</pre> </div> <h2 id="verify">Verifying nginx Works</h2> <p> I verified that <code>nginx</code> was listening on ports 80 and 443. I highlighted the <code>nginx</code> process number &ndash; you might need to scroll the output below to the right to see it. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf3b6b832166f'><button class='copyBtn' data-clipboard-target='#idf3b6b832166f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo netstat -tulpn | grep ':\(443\|80\)' <span class='unselectable'>tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN - tcp6 0 0 :::80 :::* LISTEN - tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN <span class="bg_yellow">87487</span>/nginx: master </span></pre> </div> <span> <p> The executable for process 87487 can be found by: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id8a73377cc0df'><button class='copyBtn' data-clipboard-target='#id8a73377cc0df' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ls -l /proc/87487/exe <span class='unselectable'>lrwxrwxrwx 1 mslinn mslinn 0 Jul 7 09:13 /proc/5166/exe -> <span class="bg_yellow">/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java*</span> </span></pre> </div> <p> If more detail is desired, get it from the process list. Enclosing a character of the process id within square brackets is an old trick for only showing the desired process. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3bea6b2fd467'><button class='copyBtn' data-clipboard-target='#id3bea6b2fd467' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ps aux | grep [8]7487 <span class='unselectable'>mslinn 87487 2.4 1.3 6596476 439456 ? Sl 09:13 0:20 java -Xms1024m -Xmx1024m -Dhttp.port=9000 /path/to/jar </span></pre> </div> <p> If there is any problem getting things to work, it is often helpful to monitor the logs as you click on the web pages: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id05fa4de0a3c7'><button class='copyBtn' data-clipboard-target='#id05fa4de0a3c7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo tail -f /var/log/nginx/*.log</pre> </div> <h2 id="persist">Finishing Up</h2> <p> Make nginx start each time the system starts as follows. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide961e165f849'><button class='copyBtn' data-clipboard-target='#ide961e165f849' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl enable nginx <span class='unselectable'>Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable nginx </span> <span class='unselectable'>$ </span>sudo update-rc.d nginx defaults</pre> </div> <h3 id="performance">Performance Test</h3> <p> <a href='https://tools.keycdn.com/performance?url=https://www.scalacourses.com' target='_blank' rel="nofollow">KeyCDN</a> offfers free website performance statistics. The result column labeled <b>TTFB</b> means &ldquo;time to first byte&rdquo;, which is the length of time required for the website content to begin being received by a user's web browser. </p> <p> I am in Montreal, Canada. Response time for TTFB reported by KeyCDN varies, from a minimum of 68ms in New York, to a maximum of 815ms in Bangalore. This range of response times is typical for websites that have a centralized process. The speed of light is finite, after all. </p> <h3 id="monitor">Free Availability Monitoring</h3> <p> <a href='https://hetrixtools.com/' target='_blank' rel="nofollow">HetrixTools</a> offers free availability monitoring for up to 10 websites. It was quick and easy to set up. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/hetrix.svg" type="image/svg"> <!---<source srcset="/blog/images/hetrix.avif" type="image/avif">--> <source srcset="/blog/images/hetrix.webp" type="image/webp"> <source srcset="/blog/images/hetrix.apng" type="image/apng"> <source srcset="/blog/images/hetrix.png" type="image/png"> <source srcset="/blog/images/hetrix.jpg" type="image/jpeg"> <source srcset="/blog/images/hetrix.jpeg" type="image/jpeg"> <source srcset="/blog/images/hetrix.jfif" type="image/jpeg"> <source srcset="/blog/images/hetrix.pjpeg" type="image/jpeg"> <source srcset="/blog/images/hetrix.pjp" type="image/jpeg"> <source srcset="/blog/images/hetrix.gif" type="image/gif"> <source srcset="/blog/images/hetrix.tif" type="image/tiff"> <source srcset="/blog/images/hetrix.tiff" type="image/tiff"> <source srcset="/blog/images/hetrix.bmp" type="image/bmp"> <source srcset="/blog/images/hetrix.ico" type="image/x-icon"> <source srcset="/blog/images/hetrix.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/hetrix.png" style='width: 100%; ' /> </picture> </div> <h3 id="live">We Are Live!</h3> <p> <a href='https://www.scalacourses.com' target='_blank' rel="nofollow"><code>ScalaCourses.com</code></a> now serves Scala students from its newly refurbished server! <span style='font-size: 3em;;'>&#x1F601;</span> </p> Trialing mslinn.com on Linode Storage 2022-07-01T00:00:00-04:00 https://mslinn.github.io/blog/2022/07/01/trialing-linode-storage <!-- #region intro --> <p> This is another article in my ongoing saga of moving off AWS, which <a href='/blog/2022/05/26/aws-hijacking.html'>does not integrate security with real-time billing</a>. I recently learned this the hard way: when my AWS account was hijacked, in less than 15 minutes, a huge bill was incurred. </p> <div class="pullQuote"> &ldquo;Pay-as-you-go&rdquo; is shorthand for &ldquo;there is nothing you can do to limit your financial liability&rdquo; </div> <p> The world of pain that I experienced after the breach was inflicted by broken and wasteful AWS remedial processes, and an ineffective AWS management structure. This type of issue only is enabled because of deficiencies in the AWS architecture. Those AWS architectural deficiencies feel like the result of an exploitive mindset: </p> <div class='quote'> <div class='quoteText clearfix'> Unlimited financial liability is our customer&rsquo;s problem, not ours &ndash; as a result, exploits are quite profitable for us. </div><div class='quoteAttribution'> &nbsp;&ndash; From a mythical retrospective discussion at an AWS offsite. </div> </div> <div class='quote'> <div class='quoteText clearfix'> At this moment that feature of setting limits does not exist, Azure is not able to safeguard customers from unlimited financial liability. </div><div class='quoteAttribution'> &nbsp;&ndash; From an email sent to me from Microsoft Azure support staff on 2022-06-22. </div> </div> <!-- endregion --> <!-- #region Demand Limits to Financial Liability From PaaS Vendors --> <div class="alert rounded shadow"> <h2>Demand Limits to Financial Liability From PaaS Vendors</h2> <p> PaaS vendors currently provide accounts with all services ready to go, without limit. That is good for the vendor's bottom line, but highly dangerous for their customers. </p> <p> All PaaS customers should demand the ability for themselves to be able to set firm limits on budgeted expenses, along with the ability to deny all services not explicitly authorized. </p> <p> You can buy insurance against losses resulting from various calamities, but you cannot limit your financial liability with PaaS vendors. </p> <p> <span style="font-size:larger; font-weight: bold;">Yet.</span> </p> </div> <!-- endregion --> <!-- #region Website Hosting Market --> <h2 id="market">Website Hosting Market</h2> <p> Recently, I have spent a lot of time looking at options for hosting websites. I found the following types of products: </p> <style> .row th { text-align: right; } </style> <table class="table"> <tr> <td></td> <th>WordPress</th> <th>General Web Server</th> <th>VM</th> <th>S3 Compatible</th> </tr> <tr class="row"> <th>CDN</th> <td>No</td> <td>No</td> <td>No</td> <td>Yes</td> </tr> <tr class="row"> <th>Speed</th> <td>Slow to Medium</td> <td>Slow to Medium</td> <td>Slow to Fast</td> <td>Fast</td> </tr> <tr class="row"> <th>Reliability</th> <td>Fair</td> <td>Fair</td> <td>Depends on you</td> <td>Good</td> </tr> <tr class="row"> <th>Financial Liability</th> <td>Fixed monthly cost</td> <td>Fixed monthly cost</td> <td class="bg_yellow">Depends</td> <td class="bg_yellow">Unlimited</td> </tr> <tr class="row"> <th>Storage</th> <td>Small</td> <td>Small</td> <td>Depends</td> <td>Unlimited</td> </tr> <tr class="row"> <th>CLI & API</th> <td>No</td> <td>No</td> <td>Depends</td> <td>Yes</td> </tr> </table> <p> This website requires about 220 GB of storage. It has numerous images, in 2 versions: <a href='/blog/2020/08/15/converting-all-images-to-webp-format.html'><code>webp</code></a> and <code>png</code>. So for this website, &lsquo;small&rsquo; means less than 250 GB storage. </p> <p> I decided to give the S3-compatible product <a href='https://www.linode.com/docs/products/storage/object-storage/' target='_blank' rel="nofollow">Linode Storage</a> a try. </p> <!-- endregion --> <!-- #region Why Linode --> <h2 id="why">Why Linode?</h2> <p> First, the positive: <a href='https://www.linode.com/' target='_blank' rel="nofollow">Linode</a> is a pioneer in virtual computing. It has been on my short list of places for hosting my pet projects for many years. Prices have always been quite competitive, products solid, with good support ... and smart people have answered the phone when I have called. Being able to easily speak with a capable human provides huge value. </p> <p> On the other hand, as I shall demonstrate in this article, Linode&rsquo;s documentation presents as a significant barrier to customers considering adopting their services. I had to work a lot harder than I should have to get my evaluation website up and running. Hopefully, this document is complete enough, so others can follow along and host their static websites on Linode Storage. </p> <!-- endregion --> <!-- #region Acquired by Akamai --> <h2 id="akamai">Acquired by Akamai</h2> <p> <a href='https://www.akamai.com/newsroom/press-release/akamai-completes-acquisition-of-linode' target='_blank' rel="nofollow">Linode was acquired by Akamai</a> 3 months ago. Akamai is the original CDN, and their network is gigantic. </p> <p> Linode does not yet offer a CDN product, but the person I spoke to at Linode when I started writing this article suggested that a CDN product from Linode based on Akamai&rsquo;s network might be available soon. He also said that they were planning to working on a mechanism for limiting financial liability to their customers in 2022. </p> <p> This was music to my financially risk-averse ears! </p> <div class="notepaper shadow" style="width: 80%;"> <p> The remainder of this article will take you through all the steps necessary to: </p> <ol> <li>Install and configure software tools on a WSL / WSL2 / Ubuntu computer</li> <li>Make an S3-compatible bucket and set it up to hold a website</li> <li>Upload the website, and easily handle mimetype issues</li> <li>Generate and install a free 4096-bit SSL wildcard certificate</li> <li>Get a security rating from Qualys / SSL Labs</li> </ol> </div> <!-- endregion --> <!-- #region Trialing Linode Storage --> <h2 id="setup">Trialing Linode Storage</h2> <!-- #region Installing s3cmd --> <h3 id="s3cmd">Installing <span class="code">s3cmd</span></h3> <p> I installed the recommended S3-compatible command-line program <code>s3cmd</code> on WSL2 / Ubuntu like this: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ideb9145bec584'><button class='copyBtn' data-clipboard-target='#ideb9145bec584' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install s3cmd <span class='unselectable'>Reading package lists... Done Building dependency tree... Done Reading state information... Done The following additional packages will be installed: python3-magic The following NEW packages will be installed: python3-magic s3cmd 0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 133 kB of archives. After this operation, 584 kB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-magic all 2:0.4.24-2 [12.6 kB] Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 s3cmd all 2.2.0-1 [120 kB] Fetched 133 kB in 0s (278 kB/s) Selecting previously unselected package python3-magic. (Reading database ... 169821 files and directories currently installed.) Preparing to unpack .../python3-magic_2%3a0.4.24-2_all.deb ... Unpacking python3-magic (2:0.4.24-2) ... Selecting previously unselected package s3cmd. Preparing to unpack .../archives/s3cmd_2.2.0-1_all.deb ... Unpacking s3cmd (2.2.0-1) ... Setting up python3-magic (2:0.4.24-2) ... Setting up s3cmd (2.2.0-1) ... Processing triggers for man-db (2.10.2-1) ... Scanning processes... Scanning processor microcode... Scanning linux images... Failed to retrieve available kernel versions. Failed to check for processor microcode upgrades. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. </span></pre> </div> <!-- endregion --> <p> Here is the <code>s3cmd</code> help message: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide078f034bd31'><button class='copyBtn' data-clipboard-target='#ide078f034bd31' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd <span class='unselectable'>Usage: s3cmd [options] COMMAND [parameters]<br/> S3cmd is a tool for managing objects in Amazon S3 storage. It allows for making and removing &quot;buckets&quot; and uploading, downloading and removing &quot;objects&quot; from these buckets.<br/> Options: -h, --help show this help message and exit --configure Invoke interactive (re)configuration tool. Optionally use as &#39;--configure s3://some-bucket&#39; to test access to a specific bucket instead of attempting to list them all. -c FILE, --config=FILE Config file name. Defaults to $HOME/.s3cfg --dump-config Dump current configuration after parsing config files and command line options and exit. --access_key=ACCESS_KEY AWS Access Key --secret_key=SECRET_KEY AWS Secret Key --access_token=ACCESS_TOKEN AWS Access Token -n, --dry-run Only show what should be uploaded or downloaded but don&#39;t actually do it. May still perform S3 requests to get bucket listings and other information though (only for file transfer commands) -s, --ssl Use HTTPS connection when communicating with S3. (default) --no-ssl Don&#39;t use HTTPS. -e, --encrypt Encrypt files before uploading to S3. --no-encrypt Don&#39;t encrypt files. -f, --force Force overwrite and other dangerous operations. --continue Continue getting a partially downloaded file (only for [get] command). --continue-put Continue uploading partially uploaded files or multipart upload parts. Restarts parts/files that don&#39;t have matching size and md5. Skips files/parts that do. Note: md5sum checks are not always sufficient to check (part) file equality. Enable this at your own risk. --upload-id=UPLOAD_ID UploadId for Multipart Upload, in case you want continue an existing upload (equivalent to --continue- put) and there are multiple partial uploads. Use s3cmd multipart [URI] to see what UploadIds are associated with the given URI. --skip-existing Skip over files that exist at the destination (only for [get] and [sync] commands). -r, --recursive Recursive upload, download or removal. --check-md5 Check MD5 sums when comparing files for [sync]. (default) --no-check-md5 Do not check MD5 sums when comparing files for [sync]. Only size will be compared. May significantly speed up transfer but may also miss some changed files. -P, --acl-public Store objects with ACL allowing read for anyone. --acl-private Store objects with default ACL allowing access for you only. --acl-grant=PERMISSION:EMAIL or USER_CANONICAL_ID Grant stated permission to a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all --acl-revoke=PERMISSION:USER_CANONICAL_ID Revoke stated permission for a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all -D NUM, --restore-days=NUM Number of days to keep restored file available (only for &#39;restore&#39; command). Default is 1 day. --restore-priority=RESTORE_PRIORITY Priority for restoring files from S3 Glacier (only for &#39;restore&#39; command). Choices available: bulk, standard, expedited --delete-removed Delete destination objects with no corresponding source file [sync] --no-delete-removed Don&#39;t delete destination objects [sync] --delete-after Perform deletes AFTER new uploads when delete-removed is enabled [sync] --delay-updates *OBSOLETE* Put all updated files into place at end [sync] --max-delete=NUM Do not delete more than NUM files. [del] and [sync] --limit=NUM Limit number of objects returned in the response body (only for [ls] and [la] commands) --add-destination=ADDITIONAL_DESTINATIONS Additional destination for parallel uploads, in addition to last arg. May be repeated. --delete-after-fetch Delete remote objects after fetching to local file (only for [get] and [sync] commands). -p, --preserve Preserve filesystem attributes (mode, ownership, timestamps). Default for [sync] command. --no-preserve Don&#39;t store FS attributes --exclude=GLOB Filenames and paths matching GLOB will be excluded from sync --exclude-from=FILE Read --exclude GLOBs from FILE --rexclude=REGEXP Filenames and paths matching REGEXP (regular expression) will be excluded from sync --rexclude-from=FILE Read --rexclude REGEXPs from FILE --include=GLOB Filenames and paths matching GLOB will be included even if previously excluded by one of --(r)exclude(-from) patterns --include-from=FILE Read --include GLOBs from FILE --rinclude=REGEXP Same as --include but uses REGEXP (regular expression) instead of GLOB --rinclude-from=FILE Read --rinclude REGEXPs from FILE --files-from=FILE Read list of source-file names from FILE. Use - to read from stdin. --region=REGION, --bucket-location=REGION Region to create bucket in. As of now the regions are: us-east-1, us-west-1, us-west-2, eu-west-1, eu- central-1, ap-northeast-1, ap-southeast-1, ap- southeast-2, sa-east-1 --host=HOSTNAME HOSTNAME:PORT for S3 endpoint (default: s3.amazonaws.com, alternatives such as s3-eu- west-1.amazonaws.com). You should also set --host- bucket. --host-bucket=HOST_BUCKET DNS-style bucket+hostname:port template for accessing a bucket (default: %(bucket)s.s3.amazonaws.com) --reduced-redundancy, --rr Store object with &#39;Reduced redundancy&#39;. Lower per-GB price. [put, cp, mv] --no-reduced-redundancy, --no-rr Store object without &#39;Reduced redundancy&#39;. Higher per- GB price. [put, cp, mv] --storage-class=CLASS Store object with specified CLASS (STANDARD, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER or DEEP_ARCHIVE). [put, cp, mv] --access-logging-target-prefix=LOG_TARGET_PREFIX Target prefix for access logs (S3 URI) (for [cfmodify] and [accesslog] commands) --no-access-logging Disable access logging (for [cfmodify] and [accesslog] commands) --default-mime-type=DEFAULT_MIME_TYPE Default MIME-type for stored objects. Application default is binary/octet-stream. -M, --guess-mime-type Guess MIME-type of files by their extension or mime magic. Fall back to default MIME-Type as specified by --default-mime-type option --no-guess-mime-type Don&#39;t guess MIME-type and use the default type instead. --no-mime-magic Don&#39;t use mime magic when guessing MIME-type. -m MIME/TYPE, --mime-type=MIME/TYPE Force MIME-type. Override both --default-mime-type and --guess-mime-type. --add-header=NAME:VALUE Add a given HTTP header to the upload request. Can be used multiple times. For instance set &#39;Expires&#39; or &#39;Cache-Control&#39; headers (or both) using this option. --remove-header=NAME Remove a given HTTP header. Can be used multiple times. For instance, remove &#39;Expires&#39; or &#39;Cache- Control&#39; headers (or both) using this option. [modify] --server-side-encryption Specifies that server-side encryption will be used when putting objects. [put, sync, cp, modify] --server-side-encryption-kms-id=KMS_KEY Specifies the key id used for server-side encryption with AWS KMS-Managed Keys (SSE-KMS) when putting objects. [put, sync, cp, modify] --encoding=ENCODING Override autodetected terminal and filesystem encoding (character set). Autodetected: UTF-8 --add-encoding-exts=EXTENSIONs Add encoding to these comma delimited extensions i.e. (css,js,html) when uploading to S3 ) --verbatim Use the S3 name as given on the command line. No pre- processing, encoding, etc. Use with caution! --disable-multipart Disable multipart upload on files bigger than --multipart-chunk-size-mb --multipart-chunk-size-mb=SIZE Size of each chunk of a multipart upload. Files bigger than SIZE are automatically uploaded as multithreaded- multipart, smaller files are uploaded using the traditional method. SIZE is in Mega-Bytes, default chunk size is 15MB, minimum allowed chunk size is 5MB, maximum is 5GB. --list-md5 Include MD5 sums in bucket listings (only for &#39;ls&#39; command). -H, --human-readable-sizes Print sizes in human readable form (eg 1kB instead of 1234). --ws-index=WEBSITE_INDEX Name of index-document (only for [ws-create] command) --ws-error=WEBSITE_ERROR Name of error-document (only for [ws-create] command) --expiry-date=EXPIRY_DATE Indicates when the expiration rule takes effect. (only for [expire] command) --expiry-days=EXPIRY_DAYS Indicates the number of days after object creation the expiration rule takes effect. (only for [expire] command) --expiry-prefix=EXPIRY_PREFIX Identifying one or more objects with the prefix to which the expiration rule applies. (only for [expire] command) --progress Display progress meter (default on TTY). --no-progress Don&#39;t display progress meter (default on non-TTY). --stats Give some file-transfer stats. --enable Enable given CloudFront distribution (only for [cfmodify] command) --disable Disable given CloudFront distribution (only for [cfmodify] command) --cf-invalidate Invalidate the uploaded filed in CloudFront. Also see [cfinval] command. --cf-invalidate-default-index When using Custom Origin and S3 static website, invalidate the default index file. --cf-no-invalidate-default-index-root When using Custom Origin and S3 static website, don&#39;t invalidate the path to the default index file. --cf-add-cname=CNAME Add given CNAME to a CloudFront distribution (only for [cfcreate] and [cfmodify] commands) --cf-remove-cname=CNAME Remove given CNAME from a CloudFront distribution (only for [cfmodify] command) --cf-comment=COMMENT Set COMMENT for a given CloudFront distribution (only for [cfcreate] and [cfmodify] commands) --cf-default-root-object=DEFAULT_ROOT_OBJECT Set the default root object to return when no object is specified in the URL. Use a relative path, i.e. default/index.html instead of /default/index.html or s3://bucket/default/index.html (only for [cfcreate] and [cfmodify] commands) -v, --verbose Enable verbose output. -d, --debug Enable debug output. --version Show s3cmd version (2.2.0) and exit. -F, --follow-symlinks Follow symbolic links as if they are regular files --cache-file=FILE Cache FILE containing local source MD5 values -q, --quiet Silence output on stdout --ca-certs=CA_CERTS_FILE Path to SSL CA certificate FILE (instead of system default) --ssl-cert=SSL_CLIENT_CERT_FILE Path to client own SSL certificate CRT_FILE --ssl-key=SSL_CLIENT_KEY_FILE Path to client own SSL certificate private key KEY_FILE --check-certificate Check SSL certificate validity --no-check-certificate Do not check SSL certificate validity --check-hostname Check SSL certificate hostname validity --no-check-hostname Do not check SSL certificate hostname validity --signature-v2 Use AWS Signature version 2 instead of newer signature methods. Helpful for S3-like systems that don&#39;t have AWS Signature v4 yet. --limit-rate=LIMITRATE Limit the upload or download speed to amount bytes per second. Amount may be expressed in bytes, kilobytes with the k suffix, or megabytes with the m suffix --no-connection-pooling Disable connection re-use --requester-pays Set the REQUESTER PAYS flag for operations -l, --long-listing Produce long listing [ls] --stop-on-error stop if error in transfer --content-disposition=CONTENT_DISPOSITION Provide a Content-Disposition for signed URLs, e.g., &quot;inline; filename=myvideo.mp4&quot; --content-type=CONTENT_TYPE Provide a Content-Type for signed URLs, e.g., &quot;video/mp4&quot;<br/> Commands: Make bucket s3cmd mb s3://BUCKET Remove bucket s3cmd rb s3://BUCKET List objects or buckets s3cmd ls [s3://BUCKET[/PREFIX]] List all object in all buckets s3cmd la Put file into bucket s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX] Get file from bucket s3cmd get s3://BUCKET/OBJECT LOCAL_FILE Delete file from bucket s3cmd del s3://BUCKET/OBJECT Delete file from bucket (alias for del) s3cmd rm s3://BUCKET/OBJECT Restore file from Glacier storage s3cmd restore s3://BUCKET/OBJECT Synchronize a directory tree to S3 (checks files freshness using size and md5 checksum, unless overridden by options, see below) s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR or s3://BUCKET[/PREFIX] s3://BUCKET[/PREFIX] Disk usage by buckets s3cmd du [s3://BUCKET[/PREFIX]] Get various information about Buckets or Files s3cmd info s3://BUCKET[/OBJECT] Copy object s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2] Modify object metadata s3cmd modify s3://BUCKET1/OBJECT Move object s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2] Modify Access control list for Bucket or Files s3cmd setacl s3://BUCKET[/OBJECT] Modify Bucket Policy s3cmd setpolicy FILE s3://BUCKET Delete Bucket Policy s3cmd delpolicy s3://BUCKET Modify Bucket CORS s3cmd setcors FILE s3://BUCKET Delete Bucket CORS s3cmd delcors s3://BUCKET Modify Bucket Requester Pays policy s3cmd payer s3://BUCKET Show multipart uploads s3cmd multipart s3://BUCKET [Id] Abort a multipart upload s3cmd abortmp s3://BUCKET/OBJECT Id List parts of a multipart upload s3cmd listmp s3://BUCKET/OBJECT Id Enable/disable bucket access logging s3cmd accesslog s3://BUCKET Sign arbitrary string using the secret key s3cmd sign STRING-TO-SIGN Sign an S3 URL to provide limited public access with expiry s3cmd signurl s3://BUCKET/OBJECT &lt;expiry_epoch|+expiry_offset&gt; Fix invalid file names in a bucket s3cmd fixbucket s3://BUCKET[/PREFIX] Create Website from bucket s3cmd ws-create s3://BUCKET Delete Website s3cmd ws-delete s3://BUCKET Info about Website s3cmd ws-info s3://BUCKET Set or delete expiration rule for the bucket s3cmd expire s3://BUCKET Upload a lifecycle policy for the bucket s3cmd setlifecycle FILE s3://BUCKET Get a lifecycle policy for the bucket s3cmd getlifecycle s3://BUCKET Remove a lifecycle policy for the bucket s3cmd dellifecycle s3://BUCKET List CloudFront distribution points s3cmd cflist Display CloudFront distribution point parameters s3cmd cfinfo [cf://DIST_ID] Create CloudFront distribution point s3cmd cfcreate s3://BUCKET Delete CloudFront distribution point s3cmd cfdelete cf://DIST_ID Change CloudFront distribution point parameters s3cmd cfmodify cf://DIST_ID Display CloudFront invalidation request(s) status s3cmd cfinvalinfo cf://DIST_ID[/INVAL_ID]<br/> For more information, updates and news, visit the s3cmd website: http://s3tools.org </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Signup --> <h3 id="signup">Signup</h3> <p> I went to the <a href='https://login.linode.com/signup' target='_blank' rel="nofollow">Linode signup page</a> and signed up with my email. Using third parties for authentication means they track you more easily, and introduces an unnecessary dependency. </p> <p> The email signup procedure requires a mobile phone for SMS-based MFA. I would rather use a <a href='https://datatracker.ietf.org/doc/html/rfc6238' target='_blank' rel="nofollow">TOTP authenticator app</a> instead of SMS for MFA. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/linode/signup.svg" type="image/svg"> <!---<source srcset="/blog/images/linode/signup.avif" type="image/avif">--> <source srcset="/blog/images/linode/signup.webp" type="image/webp"> <source srcset="/blog/images/linode/signup.apng" type="image/apng"> <source srcset="/blog/images/linode/signup.png" type="image/png"> <source srcset="/blog/images/linode/signup.jpg" type="image/jpeg"> <source srcset="/blog/images/linode/signup.jpeg" type="image/jpeg"> <source srcset="/blog/images/linode/signup.jfif" type="image/jpeg"> <source srcset="/blog/images/linode/signup.pjpeg" type="image/jpeg"> <source srcset="/blog/images/linode/signup.pjp" type="image/jpeg"> <source srcset="/blog/images/linode/signup.gif" type="image/gif"> <source srcset="/blog/images/linode/signup.tif" type="image/tiff"> <source srcset="/blog/images/linode/signup.tiff" type="image/tiff"> <source srcset="/blog/images/linode/signup.bmp" type="image/bmp"> <source srcset="/blog/images/linode/signup.ico" type="image/x-icon"> <source srcset="/blog/images/linode/signup.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/linode/signup.png" style='width: 100%; ' /> </picture> </div> <p> I want to ensure that I limit my financial liability. One way of protecting myself is to limit the maximum amount that can be charged to the payment mechanism. Because PayPal has no maximum transaction limit, it is not a suitable choice for a service with unlimited financial liability. However, credit cards can have transaction limits set, and Google Pay has the following limits: </p> <div class='quote'> If you <a href='https://support.google.com/googlepay/answer/10623602' target='_blank' rel="nofollow">set up your Google Pay balance to make contactless payments</a>, there are some transaction limits: <br><br> Maximum single transaction amount: $2,000 USD.<br> Daily maximum total transaction amount: $2,500 USD.<br> Up to 15 transactions per day.<br> Additional limits on the dollar amount or frequency of transactions may be imposed in accordance with the Google Pay Terms of Service. <span class='quoteAttribution'> &nbsp;&ndash; From <a href='https://support.google.com/googlepay/answer/10187490#:~:text=If%20you%20set%20up%20your,to%2015%20transactions%20per%20day.' rel='nofollow' target='_blank'>Google Pay Help</a></span> </div> <p> <a href='https://support.google.com/googlepay/answer/10187490' target='_blank' rel="nofollow">Here</a> are additional limits for Google Pay. Because of prior good experiences with how credit card processors handled fraud, I decided to use a credit card with a low limit instead of Google Pay. </p> <!-- endregion --> <!-- #region Generating Linode Access Keys --> <h3 id="keys">Generating Linode Access Keys</h3> <p> I followed the <a href='https://www.linode.com/docs/products/storage/object-storage/guides/access-keys/' target='_blank' rel="nofollow">directions</a>: </p> <ol> <li>Logged into the <a href='https://cloud.linode.com/' target='_blank' rel="nofollow">Cloud Manager</a>.</li> <li>Selected the <b>Object Storage</b> menu item in the sidebar and clicked on the <b>Access Keys</b> tab.</li> <li>Clicked on the <b>Create Access</b> Key button, which displays the <b>Create Access Key</b> panel.</li> <li>Typed in the name <code>mslinn</code> as the label for the new access key.</li> <li>Clicked the <kbd>Submit</kbd> button to create the access key.</li> <li>The new access key and its secret key are displayed. This is the only time that the secret key is visible. I stored the access key and the secret key in my password manager, <a href='https://lastpass.com'><code>lastpass.com</code></a>.</li> </ol> <!-- endregion --> <!-- #region Configuring s3cmd --> <h3 id="s3cmd_cfg">Configuring <span class="code">s3cmd</span></h3> <p> <code>s3cmd</code> is a Python program that works with all S3-compatible APIs, such as those provided by AWS, Linode Storage, Google Cloud Storage and DreamHost DreamObjects. I intend to use its <a href='https://s3tools.org/s3cmd-sync' target='_blank' rel="nofollow"><code>sync</code></a> subcommand to synchronize the <code>www.mslinn.com</code> bucket contents in Linode Storage with the most recently generated version of this website by Jekyll. </p> <p> <code>s3cmd</code> needs to be <a href='https://s3tools.org/kb/item14.htm' target='_blank' rel="nofollow">configured</a> for <a href='https://www.linode.com/docs/products/storage/object-storage/guides/s3cmd' target='_blank' rel="nofollow">Linode</a> before it can be used. Some configuration settings for Linode are non-obvious. The following should work for most users, with the caution that the access key and secret key are of course unique for every user. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idb3223be406ba'><button class='copyBtn' data-clipboard-target='#idb3223be406ba' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd --configure <span class='unselectable'><br>Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options.<br/> Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. Access Key [asdfasdf]: </span>asdfasdfasdf <span class='unselectable'>Secret Key [asdfasdfasdf]: </span>asdfasdfasdf <span class='unselectable'>Default Region [US]:<br/> Use &quot;s3.amazonaws.com&quot; for S3 Endpoint and not modify it to the target Amazon S3. S3 Endpoint [s3.amazonaws.com]: </span>us-east-1.linodeobjects.com<br/> <span class='unselectable'>Use &quot;%(bucket)s.s3.amazonaws.com&quot; to the target Amazon S3. &quot;%(bucket)s&quot; and &quot;%(location)s&quot; vars can be used if the target S3 system supports dns based buckets. DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: </span>%(bucket)s.us-east-1.linodeobjects.com<br/> <span class='unselectable'>Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: Path to GPG program [/usr/bin/gpg]:<br/> When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP, and can only be proxied with Python 2.7 or newer Use HTTPS protocol [Yes]:<br/> On some networks all internet access must go through a HTTP proxy. Try setting it here if you can&#39;t connect to S3 directly HTTP Proxy server name:<br/> New settings: Access Key: </span>asdfasdfasdf <span class='unselectable'>Secret Key: </span>asdfasdfasdf <span class='unselectable'>Default Region: US S3 Endpoint: </span>https://us-east-1.linodeobjects.com <span class='unselectable'>DNS-style bucket+hostname:port template for accessing a bucket: </span>%(bucket)s.us-east-1.linodeobjects.com <span class='unselectable'>Encryption password: Path to GPG program: /usr/bin/gpg Use HTTPS protocol: True HTTP Proxy server name: HTTP Proxy server port: 0<br/> Test access with supplied credentials? [Y/n] n Save settings? [y/N] y Configuration saved to '/home/mslinn/.s3cfg' </span></pre> </div> <!-- endregion --> <p> I was unable to successfully test access. Two different errors appeared at different times: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id777e1895c6b4'><button class='copyBtn' data-clipboard-target='#id777e1895c6b4' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>Please wait, attempting to list all buckets... ERROR: Test failed: 403 (SignatureDoesNotMatch) ERROR: Test failed: [Errno -2] Name or service not known<br/></pre> </div> <!-- endregion --> <p> Eventually I saved the configuration without testing as shown above. This created a file called <code>$HOME/.s3cfg</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>~/.s3cfg</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd65f59aad3e8'><button class='copyBtn' data-clipboard-target='#idd65f59aad3e8' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[default] access_key = asdfasdfasdf access_token = add_encoding_exts = add_headers = bucket_location = US ca_certs_file = cache_file = check_ssl_certificate = True check_ssl_hostname = True cloudfront_host = cloudfront.amazonaws.com connection_max_age = 5 connection_pooling = True content_disposition = content_type = default_mime_type = binary/octet-stream delay_updates = False delete_after = False delete_after_fetch = False delete_removed = False dry_run = False enable_multipart = True encoding = UTF-8 encrypt = False expiry_date = expiry_days = expiry_prefix = follow_symlinks = False force = False get_continue = False gpg_command = /usr/bin/gpg gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_passphrase = guess_mime_type = True host_base = us-east-1.linodeobjects.com host_bucket = %(bucket)s.us-east-1.linodeobjects.com human_readablesizes = False invalidate_default_index_on_cf = False invalidate_default_index_root_on_cf = True invalidate_on_cf = False kms_key = limit = -1 limitrate = 0 list_md5 = False log_target_prefix = long_listing = False max_delete = -1 mime_type = multipart_chunksize_mb = 15 multipart_copy_chunksize_mb = 1024 multipart_max_chunks = 10000 preserve_attrs = True progress_meter = True proxy_host = proxy_port = 0 public_url_use_https = False put_continue = False recursive = False recv_chunk = 65536 reduced_redundancy = False requester_pays = False restore_days = 1 restore_priority = Standard secret_key = asdfasdfasdf send_chunk = 65536 server_side_encryption = False signature_v2 = False signurl_use_https = False simpledb_host = sdb.amazonaws.com skip_existing = False socket_timeout = 300 ssl_client_cert_file = ssl_client_key_file = stats = False stop_on_error = False storage_class = throttle_max = 100 upload_id = urlencoding_mode = normal use_http_expect = False use_https = True use_mime_magic = True verbosity = INFO website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/ website_error = website_index = index.html</pre> </div> <!-- endregion --> <p> According to the docs (<a href='https://www.linode.com/docs/products/storage/object-storage/guides/urls/' target='_blank' rel="nofollow">Access Buckets and Files through URLs</a>), the configuration file needs <code>website_endpoint: http://%(bucket)s.website-[cluster-url]/</code>. However, this is an error; instead of <code>cluster-url</code>, which might be <code>https://us-east-1.linodeobjects.com</code>, <code>cluster-id</code> should be used, which for me was simply <code>us-east-1</code>. <span class="bg_yellow">Thus the endpoint for me, and everyone with a bucket at <code>us-east-1</code> in Newark, New Jersey should be:<br> <code>http://%(bucket)s.<span class="bg_yellow">website-</span>us-east-1.linodeobjects.com/</code></span> </p> <p> After editing the file to manually change the value of <code>website_endpoint</code>, I tried to list the buckets, and that worked: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbffba5a196e3'><button class='copyBtn' data-clipboard-target='#idbffba5a196e3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd ls <span class='unselectable'>2022-07-01 17:32 s3://mslinn </span></pre> </div> <!-- endregion --> <!-- #region linode-cli --> <h3 id="linode-cli"><span class="code">linode-cli</span></h3> <p> Next I tried <a href='https://github.com/linode/linode-cli' target='_blank' rel="nofollow"><code>linode-cli</code></a>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id51b3a188cd7b'><button class='copyBtn' data-clipboard-target='#id51b3a188cd7b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pip install linode-cli <span class='unselectable'>Collecting linode-cli Downloading linode_cli-5.21.0-py2.py3-none-any.whl (204 kB) &#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473; 204.2/204.2 kB 4.8 MB/s eta 0:00:00 Collecting terminaltables Downloading terminaltables-3.1.10-py2.py3-none-any.whl (15 kB) Collecting PyYAML Downloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB) &#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473;&#9473; 682.2/682.2 kB 16.8 MB/s eta 0:00:00 Requirement already satisfied: requests in /home/mslinn/venv/default/lib/python3.10/site-packages (from linode-cli) (2.28.0) Requirement already satisfied: charset-normalizer~=2.0.0 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests-&gt;linode-cli) (2.0.12) Requirement already satisfied: idna&lt;4,&gt;=2.5 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests-&gt;linode-cli) (3.3) Requirement already satisfied: urllib3&lt;1.27,&gt;=1.21.1 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests-&gt;linode-cli) (1.26.9) Requirement already satisfied: certifi&gt;=2017.4.17 in /home/mslinn/venv/default/lib/python3.10/site-packages (from requests-&gt;linode-cli) (2022.6.15) Installing collected packages: terminaltables, PyYAML, linode-cli Successfully installed PyYAML-6.0 linode-cli-5.21.0 terminaltables-3.1.10 </span></pre> </div> <!-- endregion --> <p> The documentation fails to mention that the first time <code>linode-cli</code> is executed, and every time it is invoked with the <code>configure</code> subcommand, it attempts to open a web browser. WSL and WSL2 will not respond appropriately unless an X client is already running, for example <a href='https://apps.microsoft.com/store/detail/gwsl/9NL6KD1H33V3' target='_blank' rel="nofollow">GWSL</a>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id010ac8a0c852'><button class='copyBtn' data-clipboard-target='#id010ac8a0c852' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>linode-cli configure <span class='unselectable'><br>Welcome to the Linode CLI. This will walk you through some initial setup.<br/> The CLI will use its web-based authentication to log you in. If you prefer to supply a Personal Access Token, use <code>linode-cli configure --token</code><br/> Press enter to continue. This will open a browser and proceed with authentication. A browser should open directing you to this URL to authenticate:<br/> https://login.linode.com/oauth/authorize?client_id=asdfasdfasdf&amp;response_type=token&amp;scopes=*&amp;redirect_uri=http://localhost:45789<br/> If you are not automatically directed there, please copy/paste the link into your browser to continue..<br/><br/> Configuring mslinn<br/><br/> Default Region for operations. Choices are: 1 - ap-west 2 - ca-central 3 - ap-southeast 4 - us-central 5 - us-west 6 - us-southeast 7 - us-east 8 - eu-west 9 - ap-south 10 - eu-central 11 - ap-northeast<br/> Default Region (Optional): 7<br/> Default Type of Linode to deploy. Choices are: 1 - g6-nanode-1 2 - g6-standard-1 3 - g6-standard-2 4 - g6-standard-4 5 - g6-standard-6 6 - g6-standard-8 7 - g6-standard-16 8 - g6-standard-20 9 - g6-standard-24 10 - g6-standard-32 11 - g7-highmem-1 12 - g7-highmem-2 13 - g7-highmem-4 14 - g7-highmem-8 15 - g7-highmem-16 16 - g6-dedicated-2 17 - g6-dedicated-4 18 - g6-dedicated-8 19 - g6-dedicated-16 20 - g6-dedicated-32 21 - g6-dedicated-48 22 - g6-dedicated-50 23 - g6-dedicated-56 24 - g6-dedicated-64 25 - g1-gpu-rtx6000-1 26 - g1-gpu-rtx6000-2 27 - g1-gpu-rtx6000-3 28 - g1-gpu-rtx6000-4<br/> Default Type of Linode (Optional):<br/> Default Image to deploy to new Linodes. Choices are: 1 - linode/almalinux8 2 - linode/almalinux9 3 - linode/alpine3.12 4 - linode/alpine3.13 5 - linode/alpine3.14 6 - linode/alpine3.15 7 - linode/alpine3.16 8 - linode/arch 9 - linode/centos7 10 - linode/centos-stream8 11 - linode/centos-stream9 12 - linode/debian10 13 - linode/debian11 14 - linode/debian9 15 - linode/fedora34 16 - linode/fedora35 17 - linode/fedora36 18 - linode/gentoo 19 - linode/kali 20 - linode/debian11-kube-v1.20.15 21 - linode/debian9-kube-v1.20.7 22 - linode/debian9-kube-v1.21.1 23 - linode/debian11-kube-v1.21.12 24 - linode/debian9-kube-v1.22.2 25 - linode/debian11-kube-v1.22.9 26 - linode/debian11-kube-v1.23.6 27 - linode/opensuse15.3 28 - linode/opensuse15.4 29 - linode/rocky8 30 - linode/slackware14.2 31 - linode/slackware15.0 32 - linode/ubuntu16.04lts 33 - linode/ubuntu18.04 34 - linode/ubuntu20.04 35 - linode/ubuntu21.10 36 - linode/ubuntu22.04 37 - linode/centos8 38 - linode/slackware14.1 39 - linode/ubuntu21.04<br/> Default Image (Optional): Active user is now mslinn<br/> Config written to /home/mslinn/.config/linode-cli </span></pre> </div> <!-- endregion --> <p> <code>linode-cli</code> configuration created this file: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>~/.config/linode-cli</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide9af10e0692f'><button class='copyBtn' data-clipboard-target='#ide9af10e0692f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>[DEFAULT] default-user = mslinn [mslinn] token = asdfasdfasdfasdfasdfasdf region = us-east</pre> </div> <!-- endregion --> <p> Right after I did the above, I was surprised to get the following email from Linode: </p> <div class="quote"> Hello,<br><br> This is a notification to inform you that a device has been trusted to skip authentication on your Linode Account (mslinn). The request came from the following IP address: 70.53.179.203. <br><br> The device will not be prompted for a username or password for 30 days. <br><br> If this action did not originate from you, we recommend logging in and changing your password immediately. Also, as an extra layer of security, we highly recommend enabling two-factor authentication on your account. Please see the link below for more information on how to enable two-factor authentication: <br><br> <a href='https://www.linode.com/docs/security/linode-manager-security-controls/#enable-two-factor-authentication' target='_blank' rel="nofollow">Two-Factor Authentication</a> <br><br> If you have any questions or concerns, please do not hesitate to contact us 24/7 by opening a support ticket from within the Linode Manager, giving us a call, or emailing <a href='mailto:support@linode.com'>support@linode.com</a>. <br><br> <a href='https://www.linode.com/contact' target='_blank' rel="nofollow">Contact</a> <br><br> Thank you,<br> The Linode.com Team<br> <br><br> <a href='mailto:support@linode.com'>Contact Us</a> </div> <p class="alert rounded shadow"> I began to suspect that Linode&rsquo;s documentation had major deficiencies at this point. Instead of merely configuring a command-line client, my IP was whitelisted unexpectedly. I have nothing polite to say about this. </p> <p> After poking around a bit, I discovered my <a href='https://cloud.linode.com/profile/tokens' target='_blank' rel="nofollow">API Tokens page</a> on Linode. It seems that the <code>linode-cli</code> created a personal access token with a label containing the name of the computer used: <code>Linode CLI @ camille</code>. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/linode/apiTokens.svg" type="image/svg"> <!---<source srcset="/blog/images/linode/apiTokens.avif" type="image/avif">--> <source srcset="/blog/images/linode/apiTokens.webp" type="image/webp"> <source srcset="/blog/images/linode/apiTokens.apng" type="image/apng"> <source srcset="/blog/images/linode/apiTokens.png" type="image/png"> <source srcset="/blog/images/linode/apiTokens.jpg" type="image/jpeg"> <source srcset="/blog/images/linode/apiTokens.jpeg" type="image/jpeg"> <source srcset="/blog/images/linode/apiTokens.jfif" type="image/jpeg"> <source srcset="/blog/images/linode/apiTokens.pjpeg" type="image/jpeg"> <source srcset="/blog/images/linode/apiTokens.pjp" type="image/jpeg"> <source srcset="/blog/images/linode/apiTokens.gif" type="image/gif"> <source srcset="/blog/images/linode/apiTokens.tif" type="image/tiff"> <source srcset="/blog/images/linode/apiTokens.tiff" type="image/tiff"> <source srcset="/blog/images/linode/apiTokens.bmp" type="image/bmp"> <source srcset="/blog/images/linode/apiTokens.ico" type="image/x-icon"> <source srcset="/blog/images/linode/apiTokens.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/linode/apiTokens.png" style='width: 100%; ' /> </picture> </div> <p> Clicking on the <kbd>View Scopes</kbd> button above displays the permissions granted to the token: </p> <div class='imgWrapper imgFlex center halfsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/linode/cliPermissions.svg" type="image/svg"> <!---<source srcset="/blog/images/linode/cliPermissions.avif" type="image/avif">--> <source srcset="/blog/images/linode/cliPermissions.webp" type="image/webp"> <source srcset="/blog/images/linode/cliPermissions.apng" type="image/apng"> <source srcset="/blog/images/linode/cliPermissions.png" type="image/png"> <source srcset="/blog/images/linode/cliPermissions.jpg" type="image/jpeg"> <source srcset="/blog/images/linode/cliPermissions.jpeg" type="image/jpeg"> <source srcset="/blog/images/linode/cliPermissions.jfif" type="image/jpeg"> <source srcset="/blog/images/linode/cliPermissions.pjpeg" type="image/jpeg"> <source srcset="/blog/images/linode/cliPermissions.pjp" type="image/jpeg"> <source srcset="/blog/images/linode/cliPermissions.gif" type="image/gif"> <source srcset="/blog/images/linode/cliPermissions.tif" type="image/tiff"> <source srcset="/blog/images/linode/cliPermissions.tiff" type="image/tiff"> <source srcset="/blog/images/linode/cliPermissions.bmp" type="image/bmp"> <source srcset="/blog/images/linode/cliPermissions.ico" type="image/x-icon"> <source srcset="/blog/images/linode/cliPermissions.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/linode/cliPermissions.png" style='width: 100%; ' /> </picture> </div> <p class="alert rounded shadow"> This personal access token is all-powerful. The <code>linode-cli</code> quietly created a personal access token with all permissions enabled, and did not warn the user, and did not say how to reduce or limit the permissions. <br><br> Security is enhanced when the minimum permission necessary is provided to accomplish necessary tasks. Instead, this token silently maximizes the financial risk to the person or entity who pays for the account. </p> <!-- endregion --> <!-- #region Creating the Website Bucket --> <h3 id="bucket">Creating the Website Bucket</h3> <p class="alert rounded shadow"> The name of the bucket must exactly match the name of the subdomain that the website is served from. If you try to serve content from a misnamed bucket, you will not be able to apply an SSL certificate; instead, the certificate will be issued by <code>linodeobjects.com</code>. </p> <p> I wanted to test using the subdomain <code>linode.mslinn.com</code>, and then if all was well, switch <code>www.mslinn.com</code> over to Linode Storage. Because buckets cannot be renamed, I had to make 2 buckets with identical contents, one called <code>linode.mslinn.com</code> and the other called <code>www.mslinn.com</code>. If I decide to stay with Linode, I will delete the bucket I am using for testing, called <code>linode.mslinn.com</code>, and run this website from the <code>www.mslinn.com</code> bucket. </p> <p> I created a website-enabled bucket called <code>www.mslinn.com</code> as follows: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf405efc35b35'><button class='copyBtn' data-clipboard-target='#idf405efc35b35' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd mb --acl-public s3://www.mslinn.com <span class='unselectable'>Bucket 's3://www.mslinn.com/' created </span> <span class='unselectable'>$ </span>s3cmd ls <span class='unselectable'>2022-07-01 17:32 s3://www.mslinn.com </span> <span class='unselectable'>$ </span>s3cmd ws-create \ --ws-index=index.html \ --ws-error=404.html \ s3://www.mslinn.com <span class='unselectable'>Bucket 's3://www.mslinn.com/': website configuration created. </span></pre> </div> <!-- endregion --> <p> The <code>s3cmd ws-info</code> subcommand displays the Linode Object Storage bucket URL as the <b>Website endpoint</b>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id59a52a4f3aa0'><button class='copyBtn' data-clipboard-target='#id59a52a4f3aa0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd ws-info s3://www.mslinn.com <span class='unselectable'>Bucket s3://www.mslinn.com/: Website configuration Website endpoint: http://www.mslinn.com.website-us-east-1.linodeobjects.com/ Index document: index.html Error document: 404.html </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Defining a CNAME for the Bucket --> <h3 id="subdomain">Defining a CNAME for the Bucket</h3> <p> Although Linode provides a <a href='https://www.linode.com/docs/guides/dns-manager/' target='_blank' rel="nofollow">DNS Manager</a>, while I am testing out Linode I wanted to continue using Namecheap for DNS, so I navigated to <a href='https://ap.www.namecheap.com/Domains/DomainControlPanel/mslinn.com/advancedns'><code>ap.www.namecheap.com/Domains/DomainControlPanel/mslinn.com/advancedns</code></a> and created CNAMEs like this: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id118b3f6ead93'><button class='copyBtn' data-clipboard-target='#id118b3f6ead93' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>linode.mslinn.com CNAME linode.mslinn.com.<span class="bg_yellow">website-</span>us-east-1.linodeobjects.com www.mslinn.com CNAME www.mslinn.com.<span class="bg_yellow">website-</span>us-east-1.linodeobjects.com</pre> </div> <p class="alert rounded shadow"> Warning: if the <code>CNAME</code> does not point to a URL with the word <code>website</code> in it, for example <code>www.mslinn.com.us-east-1.linodeobjects.com</code>, the default index file and the error file will not automatically be served when expected. </p> <!-- endregion --> <!-- #region Making a Free SSL Certificate --> <h3 id="cert">Making a Free SSL Certificate</h3> <p> <a href='https://www.ssllabs.com/ssltest/' target='_blank' rel="nofollow">Qualys / SSL Labs</a> rates the SSL security provided by AWS CloudFront using the AWS-generated SSL certificates with a &ldquo;B&rdquo; grade. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/sslReport.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/sslReport.avif" type="image/avif">--> <source srcset="/blog/images/aws/sslReport.webp" type="image/webp"> <source srcset="/blog/images/aws/sslReport.apng" type="image/apng"> <source srcset="/blog/images/aws/sslReport.png" type="image/png"> <source srcset="/blog/images/aws/sslReport.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/sslReport.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/sslReport.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/sslReport.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/sslReport.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/sslReport.gif" type="image/gif"> <source srcset="/blog/images/aws/sslReport.tif" type="image/tiff"> <source srcset="/blog/images/aws/sslReport.tiff" type="image/tiff"> <source srcset="/blog/images/aws/sslReport.bmp" type="image/bmp"> <source srcset="/blog/images/aws/sslReport.ico" type="image/x-icon"> <source srcset="/blog/images/aws/sslReport.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/sslReport.png" style='width: 100%; ' /> </picture> </div> <p> Lets see if Linode Object Storage can host a more secure website when provided with a 4096-bit certificate generated by <code>certbot</code>/<code>letsencrypt</code>. The conclusion of this article will reveal the answer. </p> <p> Please read <a href='/blog/2022/06/15/certbot.html'>Creating and Renewing Letsencrypt Wildcard SSL Certificates</a> for details. </p> <!-- endregion --> <!-- #region Linode DNS Authentication --> <h4 id="linodeDns">Linode DNS Authentication</h4> <p> Eventually, if I stick with Linode, I could use the <a href='https://certbot-dns-linode.readthedocs.io/en/stable/' target='_blank' rel="nofollow"><code>certbot</code> DNS plugin for Linode</a>, but I will not use it right now because I am trialing Linode. Use of this plugin requires a configuration file containing Linode API credentials, obtained from your Linode account&rsquo;s <a href='https://cloud.linode.com/profile/tokens' target='_blank' rel="nofollow">Applications &amp; API Tokens</a> page. I stored the credentials in <code>~/.certbot/linode.ini</code>: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>~/.certbot/linode.ini</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id2b446b2b657d'><button class='copyBtn' data-clipboard-target='#id2b446b2b657d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># Linode API credentials used by Certbot dns_linode_key = 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ64 dns_linode_version = [<blank>|3|4]</pre> </div> <!-- endregion --> <p> Here is how to create an SSL certificate <code>certbot</code> using Linode DNS authentication: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='iddd0b6cc8e869'><button class='copyBtn' data-clipboard-target='#iddd0b6cc8e869' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><!-- #region --> <span class='unselectable'>$ </span>certbot certonly \ --dns-linode \ --dns-linode-credentials ~/.certbot/linode.ini \ --rsa-key-size 4096 \ -d mslinn.com \ -d *.mslinn.com \ --config-dir ~/.certbot/mslinn.com/config \ --logs-dir ~/.certbot/mslinn.com/logs \ --work-dir ~/.certbot/mslinn.com/work</pre> </div> <!-- endregion --> <!-- endregion --> <!-- endregion --> <!-- #region Installing the Certificate --> <h2 id="certUp">Installing the Certificate</h2> <!-- #region implicit --> <p> Now I created 2 certificates: one with the full chain of responsibility (<code>fullchain.pem</code>) and one containing my private key (<code>privkey.pem</code>). I used bash environment variables called <code>CERT_DIR</code>, <code>CERT</code> and <code>KEY</code> to make the incantation convenient to type in. <code>CERT</code> and <code>KEY</code> contain the contents of the files <code>fullchain.pem</code> and <code>privkey.pem</code>, respectively. As you can see, the result was a valid (wildcard) SSL certificate. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf7a69ee24ed7'><button class='copyBtn' data-clipboard-target='#idf7a69ee24ed7' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>CERT_DIR=/home/mslinn/.certbot/mslinn.com/config/live/mslinn.com <span class='unselectable'>$ </span>CERT="$( cat $CERT_DIR/fullchain.pem )" <span class='unselectable'>$ </span>KEY="$( cat $CERT_DIR/privkey.pem )" <span class='unselectable'>$ </span>linode-cli object-storage ssl-upload \ us-east-1 www.mslinn.com \ --certificate "$CERT" \ --private_key "$KEY" <div class="tree"><span class='unselectable'>┌──────┐ │ ssl │ ├──────┤ │ True │ └──────┘ </span></div></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region Syncing the Website Bucket --> <h3 id="sync">Syncing the Website Bucket</h3> <p> I wrote a bash script to build this Jekyll-generated website. Jekyll places the generated website in a directory called <code>_site</code>. I <a href='https://s3tools.org/s3cmd-sync' target='_blank' rel="nofollow">synchronized</a> the contents of the freshly generated <code>mslinn.com</code> website, stored in <code>_site/</code>, with the bucket as follows. Note the trailing slash on <code>_site<span class="bg_yellow">/</span></code>, <a href='https://s3tools.org/s3cmd-sync' target='_blank' rel="nofollow">it is significant</a>. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id4b5aefc3e5de'><button class='copyBtn' data-clipboard-target='#id4b5aefc3e5de' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd sync \ --acl-public --delete-removed --guess-mime-type --quiet \ _site<span class="bg_yellow">/</span> \ s3://www.mslinn.com <span class='unselectable'>INFO: No cache file found, creating it. INFO: Compiling list of local files... INFO: Running stat() and reading/calculating MD5 values on 2157 files, this may take some time... INFO: [1000/2157] INFO: [2000/2157] INFO: Retrieving list of remote files for s3://www.mslinn.com/_site ... INFO: Found 2157 local files, 0 remote files INFO: Verifying attributes... INFO: Summary: 2137 local files to upload, 20 files to remote copy, 0 remote files to delete ... lots more output ... Done. Uploaded 536630474 bytes in 153.6 seconds, 3.33 MB/s. </span></pre> </div> <!-- endregion --> <p> Originally, I did not provide the <code>--acl-public</code> option to <code>s3cmd sync</code>. That meant the files in the bucket were all private &ndash; the opposite of what is required for a website. Here is the incantation for making all the files in the bucket publicly readable: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id3b065ef17e73'><button class='copyBtn' data-clipboard-target='#id3b065ef17e73' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>s3cmd setacl s3://www.mslinn.com/ \ --acl-public --recursive --quiet</pre> </div> <p> Linode&rsquo;s S3 implementation faithfully mirrors a problem that AWS S3 has. CSS files are automatically assigned the MIME type <code>text/plain</code>, instead of <code>text/css</code>. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ide5749612875a'><button class='copyBtn' data-clipboard-target='#ide5749612875a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl -sI http://linode.mslinn.com/assets/css/style.css | \ grep Content-Type <span class='unselectable'>Content-Type: text/plain </span></pre> </div> <p> I re-uploaded the CSS files with the proper mime type using this incantation: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id0ad51a5577e0'><button class='copyBtn' data-clipboard-target='#id0ad51a5577e0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>cd _site <span class='unselectable'>$ </span>find . -name *.css -exec \ s3cmd put -P --mime-type=text/css {} s3://www.mslinn.com/{} \;</pre> </div> <p> It should be possible to provide a MIME-type mapping somehow! Linode, if you are reading this, please provide an extension to the S3 functionality, so customers could go beyond AWS's bugs. </p> <!-- endregion --> <!-- endregion --> <!-- #region The Result --> <h2 id="result">The Result</h2> <p> Success! The same content can be fetched with 3 different URLs: </p> <ul> <li> <a href='https://https://linode.mslinn.com' target='_blank' rel="nofollow"><code>https://linode.mslinn.com</code></a> uses the free wildcard certificate that was created above. </li> <li> <a href='http://linode.mslinn.com.website-us-east-1.linodeobjects.com' target='_blank' rel="nofollow"><code>http://linode.mslinn.com.website-us-east-1.linodeobjects.com</code></a> works without an SSL certificate. </li> <li> <a href='https://linode.mslinn.com.website-us-east-1.linodeobjects.com' target='_blank' rel="nofollow"><code>https://linode.mslinn.com.website-us-east-1.linodeobjects.com</code></a> uses a certificate from <code>us-east-1.linodeobjects.com</code>, which shows as an error, as it should. The error would be prevented if the certificate was a wildcard certificate issued by <code>linodeobjects.com</code> instead. </li> </ul> <p> Qualys / SSL Labs rated the site on Linode Storage with the Let&rsquo;s Encrypt SSL certificate: and gave it the same rating as when the site was hosted on AWS S3 / CloudFront. </p> <div class='imgWrapper imgFlex inline' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/linode/sslReportLinode.svg" type="image/svg"> <!---<source srcset="/blog/images/linode/sslReportLinode.avif" type="image/avif">--> <source srcset="/blog/images/linode/sslReportLinode.webp" type="image/webp"> <source srcset="/blog/images/linode/sslReportLinode.apng" type="image/apng"> <source srcset="/blog/images/linode/sslReportLinode.png" type="image/png"> <source srcset="/blog/images/linode/sslReportLinode.jpg" type="image/jpeg"> <source srcset="/blog/images/linode/sslReportLinode.jpeg" type="image/jpeg"> <source srcset="/blog/images/linode/sslReportLinode.jfif" type="image/jpeg"> <source srcset="/blog/images/linode/sslReportLinode.pjpeg" type="image/jpeg"> <source srcset="/blog/images/linode/sslReportLinode.pjp" type="image/jpeg"> <source srcset="/blog/images/linode/sslReportLinode.gif" type="image/gif"> <source srcset="/blog/images/linode/sslReportLinode.tif" type="image/tiff"> <source srcset="/blog/images/linode/sslReportLinode.tiff" type="image/tiff"> <source srcset="/blog/images/linode/sslReportLinode.bmp" type="image/bmp"> <source srcset="/blog/images/linode/sslReportLinode.ico" type="image/x-icon"> <source srcset="/blog/images/linode/sslReportLinode.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/linode/sslReportLinode.png" style='width: 100%; ' /> </picture> </div> <p> BTW, you can test the date range of a certificate with this incantation: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id182cd0f50e59'><button class='copyBtn' data-clipboard-target='#id182cd0f50e59' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>curl https://linode.mslinn.com -vI --stderr - | grep 'date:' <span class='unselectable'>* start date: Nov 25 17:46:07 2022 GMT * expire date: Feb 23 17:46:06 2023 GMT </span></pre> </div> <p> The site feels quite responsive. <span style='font-size: 3em;;'>&#x1F601;</span> </p> <!-- endregion --> Considering Microsoft Azure for Static Websites 2022-06-20T00:00:00-04:00 https://mslinn.github.io/blog/2022/06/20/azure <p> After realizing that <a href='/blog/2022/06/02/azure-security.html'>AWS does not integrate security with real-time billing</a>, and being presented with a huge bill after my account was hijacked, I decided to see if Microsoft Azure offered me better financial security. </p> <p> In particular, I am looking for two things: </p> <ol> <li> <b>A more secure authentication mechanism</b> &ndash; If AWS credentials are compromised, they can be used from anywhere in the world. </li> <li> <b>Restricting scope and scale of authorized services</b> &ndash; With AWS, if an IAM user has a role that allows EC2 instances to be launched, any number of any size EC2 instances can be launched. There is no way to cap the number or type of EC2 instances, and AWS real-time billing is not integrated with the authentication mechanism. Furthermore, AWS has a peculiar set of busy work tasks for victims of account hijacking that appears to be targeted more towards increasing support revenue than address root issues. This means that the bad guys have no limit to the financial damage they can incur on their victims. </li> </ol> <p class="alert rounded shadow"> For PaaS vendors such as AWS, Azure, Digital Ocean, Cloudflare, ScaleWay, etc: &ldquo;pay-as-you-go&rdquo; is shorthand for &ldquo;there is nothing you can do to limit your financial liability&rdquo;. </p> <p> I set out to discover if Azure also suffers from these same problems. </p> <h2 id="policies_roles">Azure Roles and Policies</h2> <div class='quote'> <h2>Policies</h2> <p> <a href='https://docs.microsoft.com/en-us/azure/governance/policy/overview' target='_blank' rel="nofollow">Azure policies</a> can be used to define the desired behavior for your organization's <a href='https://docs.microsoft.com/en-us/azure/virtual-machines/windows/policy' target='_blank' rel="nofollow">Windows VMs</a> and <a href='https://docs.microsoft.com/en-us/azure/virtual-machines/linux/policy' target='_blank' rel="nofollow">Linux VMs</a>. By using policies, an organization can enforce various conventions and rules throughout the enterprise. Enforcement of the desired behavior can help mitigate risk while contributing to the success of the organization. </p> <h3>Azure role-based access control</h3> <p> Using <a href='https://docs.microsoft.com/en-us/azure/role-based-access-control/overview' target='_blank' rel="nofollow">Azure role-based access control (Azure RBAC)</a>, you can segregate duties within your team and grant only the amount of access to users on your VM that they need to perform their jobs. Instead of giving everybody unrestricted permissions on the VM, you can allow only certain actions. You can configure access control for the VM in the <a href='https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal' target='_blank' rel="nofollow">Azure portal</a>, using the <a href='https://docs.microsoft.com/en-us/cli/azure/role' target='_blank' rel="nofollow">Azure CLI</a>, or <a href='https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-powershell' target='_blank' rel="nofollow">Azure PowerShell</a>. </p> <span class='quoteAttribution'> &nbsp;&ndash; From <a href='https://docs.microsoft.com/en-us/azure/virtual-machines/security-policy' rel='nofollow' target='_blank'>Azure Virtual Machines Documentation</a></span> </div> <p> Ekran Systems publishes a <a href='https://www.ekransystem.com/en/blog/rbac-vs-abac' target='_blank' rel="nofollow">good description</a> of various flavors of RBAC, and the more complex ABAC. Okta, Inc. also publishes a <a href='https://www.okta.com/identity-101/role-based-access-control-vs-attribute-based-access-control/' target='_blank' rel="nofollow">good overview of RBAC and ABAC</a>. I have no connection with either company. </p> <p> I soon discovered the following passage in the Azure documentation, which appeared to exactly match the second item in the wish list at the top of this article: </p> <div class='quote'> Azure RBAC focuses on managing user <a href='https://docs.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations' target='_blank' rel="nofollow">actions</a> at different scopes. If control of an action is required, then Azure RBAC is the correct tool to use. Even if an individual has access to perform an action, if the result is a non-compliant resource, Azure Policy still blocks the create or update. <br><br> The combination of Azure RBAC and Azure Policy provides full scope control in Azure. <span class='quoteAttribution'> &nbsp;&ndash; From <a href='https://docs.microsoft.com/en-us/azure/governance/policy/overview#azure-policy-and-azure-rbac' rel='nofollow' target='_blank'>Azure documentation</a></span> </div> <p class="alert rounded shadow"> <a href='https://docs.aws.amazon.com/prescriptive-guidance/latest/saas-multitenant-api-access-authorization/access-control-types.html' target='_blank' rel="nofollow">AWS RBAC</a> does not provide functionality equivalent to that provided by Microsoft Azure RBAC plus Azure Policy, even when combined with other AWS functionality. </p> <h2 id="rbacAgain">No Limits On Other Azure Services</h2> <p> The financial limitations that Azure allows you to impose are only available for virtual machines. Users are subject to unlimited financial liability from other Azure services. It seems I have <strike>two</strike>three choices: </p> <ol> <li>Figure out how to set up RBAC and accept that other services might run up a huge bill.</li> <li>Look for a fixed-price hosting service</li> <li>Give <a href="#spendLimit">Azure Spending Limit</a> a try</li> </ol> <p> Before I sign off today, I encountered a blocking issue with Azure that I'd like to mention. </p> <h2 id="ad">Active Directory?</h2> <p> I want to update my website by running a command-line command that runs something analogous to <a href='https://linux.die.net/man/1/rsync' target='_blank' rel="nofollow"><code>rsync</code></a>. I spent quite some time looking for how this might be done with Azure. </p> <p> Before long, it seemed that Active Directory was the best way forward, however setting it up is daunting. Apparently a <a href='https://docs.microsoft.com/en-us/azure/active-directory/develop/app-objects-and-service-principals' target='_blank' rel="nofollow">service principal</a> is better suited for scripts that run on-premises, like my home office. I got the impression that I need to register an app; unsure what app might be required for hosting a web site on Azure Blob Storage with Azure CDN. </p> <p> These Active Directory docs look like a lot of abstract, generalized information that is way more complex than I need. I found a <a href='https://docs.microsoft.com/en-us/azure/storage/blobs/authorize-access-azure-active-directory' target='_blank' rel="nofollow">streamlined article</a> for my use case. Not sure what this means: &ldquo;Only storage accounts created with the Azure Resource Manager deployment model support Azure AD authorization.&rdquo; This streamlined document is not very streamlined. For me, this issue is a significant barrier to adoption. </p> <h2 id="spendLimit">Update: Azure Spending Limit</h2> <p> I just bumped into an article entitled <a href='https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/spending-limit' target='_blank' rel="nofollow">Azure spending limit</a>. Perhaps this might be useful. I tried to find out more from Microsoft, but at first they said I would have to subscribe to an expensive support plan before anyone would speak to me. That is not how I wish to proceed. Later, I opened a ticket inquiring about Azure customers facing unlimited financial liability. </p> <h2>Update 2022-06-29</h2> <p> The ticket was escalated twice, then Microsoft sent me the following official response (emphasis is mine): </p> <div class='quote'> Correct, at this moment that feature of setting limits does not exist, <b class="bg_yellow">Azure is not able to safeguard customers from unlimited financial liability</b>. <br><br> We are a support team that we provide help with current account or billing issues, in addition to that, I would like to mention that Azure products are constantly being updated, also Azure is constantly improving to offer better services to our customers. For this reason, I would like to invite you to leave your feedback in the Azure feedback forum (<a href='https://feedback.azure.com/d365community/forum/79b1327d-d925-ec11-b6e6-000d3a4f06a4' target='_blank' rel="nofollow">General Feedback · Community (azure.com)</a>), so your feedback is forwarded through the right channel and to the Microsoft internal resources that work on it. </div> <p> I responded by indicating that I would close my Azure account. </p> <p class="alert rounded shadow"> While the practice of imposing unlimited financial liability on customers is common today for PaaS vendors, it is unnecessary and predatory. If in the future I need to use a product or service that incurs unlimited financial liability for a client, I will require the client to accept the liability. </p> Creating and Renewing Letsencrypt Wildcard SSL Certificates 2022-06-15T00:00:00-04:00 https://mslinn.github.io/blog/2022/06/15/certbot <!-- #region intro --> <p> The process of making wildcard SSL certificates can be hard to get right at first. However, I use wildcard SSL certificates for most of my internet domains, so it is important to me. This article distills the essence of what I have found to be important when creating these certificates. </p> <p class="alert rounded shadow"> Update &ndash; 9 months after writing this article, I wrote about a better way to generate wildcard SSL certificates: <a href='/blog/2023/03/02/lego.html'>Wildcard SSL Certificates for Letsencrypt with Lego</a>. </p> <!-- endregion --> <!-- #region cert --> <div class='imgWrapper imgFlex right quartersize' style=' '> <a href='https://letsencrypt.org/' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/ddns/letsencrypt-logo.svg" type="image/svg"> <!---<source srcset="/blog/images/ddns/letsencrypt-logo.avif" type="image/avif">--> <source srcset="/blog/images/ddns/letsencrypt-logo.webp" type="image/webp"> <source srcset="/blog/images/ddns/letsencrypt-logo.apng" type="image/apng"> <source srcset="/blog/images/ddns/letsencrypt-logo.png" type="image/png"> <source srcset="/blog/images/ddns/letsencrypt-logo.jpg" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.jpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.jfif" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.pjpeg" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.pjp" type="image/jpeg"> <source srcset="/blog/images/ddns/letsencrypt-logo.gif" type="image/gif"> <source srcset="/blog/images/ddns/letsencrypt-logo.tif" type="image/tiff"> <source srcset="/blog/images/ddns/letsencrypt-logo.tiff" type="image/tiff"> <source srcset="/blog/images/ddns/letsencrypt-logo.bmp" type="image/bmp"> <source srcset="/blog/images/ddns/letsencrypt-logo.ico" type="image/x-icon"> <source srcset="/blog/images/ddns/letsencrypt-logo.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/ddns/letsencrypt-logo.png" style='width: 100%; ' /> </picture> </a> </div> <h2 id="cert">Making a Free Wildcard SSL Certificate</h2> <p> <a href='https://certbot.eff.org/' target='_blank' rel="nofollow"><code>Certbot</code></a> powers <a href='https://www.eff.org/' target='_blank' rel="nofollow">EFF</a>&rsquo;s <a href='https://letsencrypt.org/' target='_blank' rel="nofollow">Letsencrypt</a> capability. Source code is on <a href='https://github.com/certbot/certbot' target='_blank' rel="nofollow">GitHub</a>. </p> <p> Install <code>certbot</code> on Debian distros such as Ubuntu as follows: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable clear' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id9998090e83cc'><button class='copyBtn' data-clipboard-target='#id9998090e83cc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>yes | sudo apt install certbot <span class='unselectable'>Preparing to unpack .../06-python3-zope.event_4.4-3_all.deb ... Unpacking python3-zope.event (4.4-3) ... Selecting previously unselected package python3-zope.component. Preparing to unpack .../07-python3-zope.component_4.3.0-3_all.deb ... Unpacking python3-zope.component (4.3.0-3) ... Selecting previously unselected package python3-certbot. Preparing to unpack .../08-python3-certbot_1.21.0-1build1_all.deb ... Unpacking python3-certbot (1.21.0-1build1) ... Selecting previously unselected package python3-icu. Preparing to unpack .../09-python3-icu_2.8.1-0ubuntu2_amd64.deb ... Unpacking python3-icu (2.8.1-0ubuntu2) ... Selecting previously unselected package certbot. Preparing to unpack .../10-certbot_1.21.0-1build1_all.deb ... Unpacking certbot (1.21.0-1build1) ... Setting up python3-configargparse (1.5.3-1) ... Setting up python3-requests-toolbelt (0.9.1-1) ... Setting up python3-parsedatetime (2.6-2) ... Setting up python3-icu (2.8.1-0ubuntu2) ... Setting up python3-zope.event (4.4-3) ... Setting up python3-zope.hookable (5.1.0-1build1) ... Setting up python3-josepy (1.10.0-1) ... Setting up python3-zope.component (4.3.0-3) ... Setting up python3-acme (1.21.0-1) ... Setting up python3-certbot (1.21.0-1build1) ... Setting up certbot (1.21.0-1build1) ... Created symlink /etc/systemd/system/timers.target.wants/certbot.timer → /lib/systemd/system/certbot.timer. Processing triggers for man-db (2.10.2-1) ... Scanning processes... Scanning processor microcode... Scanning linux images... Failed to retrieve available kernel versions. Failed to check for processor microcode upgrades. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. </span></pre> </div> <!-- endregion --> <p> <code>certbot</code> has 2 subcommands of interest: <code>certonly</code> (used when creating a certificate for the first time), and <code>renew</code> (used when updating a pre-existing certificate). The creation process requires the user to do things before it can complete, so it can only be run interactively. </p> <p class="alert shadow rounded"> The files and directories creating by the process of creating a new SSL certificate should not be deleted. Contrary to what the Letsencrypt <code>certbot</code> documentation says, I have found that these files and directories allow the renewal process to proceed without requiring user interaction, so it can be scripted. </p> <!-- endregion --> <!-- #region help --> <h2 id="help"><span class='code'>Certbot</span> Help Messages</h2> <p> Here is the help message for the <code>certbot certonly</code> subcommand: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id69dbe1f10202'><button class='copyBtn' data-clipboard-target='#id69dbe1f10202' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>certbot certonly --help <span class='unselectable'>- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...<br/> Certbot can obtain and install HTTPS/TLS/SSL certificates. By default, it will attempt to use a webserver both for obtaining and installing the certificate. The most common SUBCOMMANDS and flags are:<br/> obtain, install, and renew certificates: (default) run Obtain &amp; install a certificate in your current webserver certonly Obtain or renew a certificate, but do not install it renew Renew all previously obtained certificates that are near expiry enhance Add security enhancements to your existing configuration -d DOMAINS Comma-separated list of domains to obtain a certificate for<br/> (the certbot apache plugin is not installed) --standalone Run a standalone webserver for authentication (the certbot nginx plugin is not installed) --webroot Place files in a server&#39;s webroot folder for authentication --manual Obtain certificates interactively, or using shell script hooks<br/> -n Run non-interactively --test-cert Obtain a test certificate from a staging server --dry-run Test &quot;renew&quot; or &quot;certonly&quot; without saving any certificates to disk<br/> manage certificates: certificates Display information about certificates you have from Certbot revoke Revoke a certificate (supply --cert-name or --cert-path) delete Delete a certificate (supply --cert-name)<br/> manage your account: register Create an ACME account unregister Deactivate an ACME account update_account Update an ACME account --agree-tos Agree to the ACME server&#39;s Subscriber Agreement -m EMAIL Email address for important account notifications<br/> More detailed help:<br/> -h, --help [TOPIC] print this message, or detailed help on a topic; the available TOPICS are:<br/> all, automation, commands, paths, security, testing, or any of the subcommands or plugins (certonly, renew, install, register, nginx, apache, standalone, webroot, etc.) -h all print a detailed help page including all topics --version print the version number - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </span></pre> </div> <!-- endregion --> <p> Here is the help message for the <code>certbot certonly</code> subcommand: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idd1e5b2edc29f'><button class='copyBtn' data-clipboard-target='#idd1e5b2edc29f' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>certbot renew --help <span class='unselectable'>- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -<br/> certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...<br/> Certbot can obtain and install HTTPS/TLS/SSL certificates. By default, it will attempt to use a webserver both for obtaining and installing the certificate. The most common SUBCOMMANDS and flags are:<br/> obtain, install, and renew certificates: (default) run Obtain &amp; install a certificate in your current webserver certonly Obtain or renew a certificate, but do not install it renew Renew all previously obtained certificates that are near expiry enhance Add security enhancements to your existing configuration -d DOMAINS Comma-separated list of domains to obtain a certificate for<br/> (the certbot apache plugin is not installed) --standalone Run a standalone webserver for authentication (the certbot nginx plugin is not installed) --webroot Place files in a server&#39;s webroot folder for authentication --manual Obtain certificates interactively, or using shell script hooks<br/> -n Run non-interactively --test-cert Obtain a test certificate from a staging server --dry-run Test &quot;renew&quot; or &quot;certonly&quot; without saving any certificates to disk<br/> manage certificates: certificates Display information about certificates you have from Certbot revoke Revoke a certificate (supply --cert-name or --cert-path) delete Delete a certificate (supply --cert-name)<br/> manage your account: register Create an ACME account unregister Deactivate an ACME account update_account Update an ACME account show_account Display account details --agree-tos Agree to the ACME server&#39;s Subscriber Agreement -m EMAIL Email address for important account notifications<br/> More detailed help:<br/> -h, --help [TOPIC] print this message, or detailed help on a topic; the available TOPICS are:<br/> all, automation, commands, paths, security, testing, or any of the subcommands or plugins (certonly, renew, install, register, nginx, apache, standalone, webroot, etc.) -h all print a detailed help page including all topics --version print the version number - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </span></pre> </div> <!-- endregion --> <p> Complete documentation for <code>certbot</code> is <a href='https://eff-certbot.readthedocs.io/en/stable/intro.html' target='_blank' rel="nofollow">here</a>. </p> <!-- endregion --> <!-- #region dns --> <h2 id="genericDns">Generic DNS Authentication</h2> <p> I used <code>certbot</code> to create a special file within the website as follows, without using <code>sudo</code>. I want a <a href='https://en.wikipedia.org/wiki/Wildcard_certificate' target='_blank' rel="nofollow">wildcard SSL certificate</a>, which <a href='https://certbot.eff.org/faq#does-let-s-encrypt-issue-wildcard-certificates' target='_blank' rel="nofollow">requires certbot to use DNS authentication</a>. </p> <p> BTW, the following options include two <code>-d</code> options, one for the domain apex (<code>mslinn.com</code>) and one for the subdomains (<code>*.mslinn.com</code>) </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id535e65243410'><button class='copyBtn' data-clipboard-target='#id535e65243410' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>certbot certonly \ --manual \ --agree-tos \ --preferred-challenges dns-01 \ --rsa-key-size 4096 \ -d mslinn.com -d *.mslinn.com \ --config-dir ~/.certbot/mslinn.com/config \ --logs-dir ~/.certbot/mslinn.com/logs \ --work-dir ~/.certbot/mslinn.com/work <span class='unselectable'>Saving debug log to /home/mslinn/.certbot/mslinn.com/logs/letsencrypt.log Enter email address (used for urgent renewal and security notices) (Enter &#39;c&#39; to cancel): </span> mslinn@mslinn.com <br/><span class='unselectable'>- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Would you be willing, once your first certificate is successfully issued, to share your email address with the Electronic Frontier Foundation, a founding partner of the Let&#39;s Encrypt project and the non-profit organization that develops Certbot? We&#39;d like to send you email about our work encrypting the web, EFF news, campaigns, and ways to support digital freedom. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (Y)es/(N)o: </span>n <span class='unselectable'>Account registered. Requesting a certificate for mslinn.com and *.mslinn.com<br/> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Please deploy a DNS TXT record under the name:<br/> _acme-challenge.mslinn.com.<br/> with the following value:<br/> n6o3qMw5N7qxAzV4uNQePKxJjaw0f-Bo32CybWr-lhE<br/> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Press Enter to Continue (This must be set up in addition to the previous challenges; do not remove, replace, or undo the previous challenge tasks yet. Note that you might be asked to create multiple distinct TXT records with the same name. This is permitted by DNS standards.) Before continuing, verify the TXT record has been deployed. Depending on the DNS provider, this may take some time, from a few seconds to multiple minutes. You can check if it has finished deploying with aid of online tools, such as the Google Admin Toolbox: https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.mslinn.com. Look for one or more bolded line(s) below the line ';ANSWER'. It should show the value(s) you&rsquo;ve just added. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Press Enter to Continue Successfully received certificate. Certificate is saved at: /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/fullchain.pem Key is saved at: /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/privkey.pem This certificate expires on 2022-09-29. These files will be updated when the certificate renews. NEXT STEPS: - This certificate will not be renewed automatically. Autorenewal of --manual certificates requires the use of an authentication hook script (--manual-auth-hook) but one was not provided. To renew this certificate, repeat this same certbot command before the certificate&rsquo;s expiry date. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - If you like Certbot, please consider supporting our work by: * Donating to ISRG / Let&rsquo;s Encrypt: https://letsencrypt.org/donate * Donating to EFF: https://eff.org/donate-le - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </span></pre> </div> <!-- endregion --> <p> I made a bash script (<code>~/.local/bin/sslCert</code>) to create SSL certificates, which performs the same steps as the above, but it works for any site, and can renew certificates as well as create them: </p> <div class="codeLabel"><a href='data:text/plain;charset=UTF-8,sslCert' download='sslCert' title='Click on the file name to download the file'>~/.local/bin/sslCert</a> </div> <pre data-lt-active="false" class="pre_tag maxOneScreenHigh copyContainer" id="id505ac6c3d8eb">#!/bin/bash function help &#123; echo "Creates or updates an wildcard SSL certificate Usage: $(basename $0) DOMAIN Example: $(basename $0) scalacourses.com The relevant files are stored in ~/.certbot/DOMAIN/ Creating a new certificate requires an interactive user session; updating a certificate can be done in a cron job. Currently, the renew verb is capable of either renewing all installed certificates that are due to be renewed or renewing a single certificate specified by its name. If you would like to renew specific certificates by their domains, use the certonly command instead. The renew verb may provide other options for selecting certificates to renew in the future. Ask for help or search for solutions at https://community.letsencrypt.org See the logfile /home/mslinn/.certbot/ancientwarmth.com/logs/letsencrypt.log or re-run Certbot with -v for more details. BTW, you can test the date range of the certificate with this incantation: curl https://domain.com -vI --stderr - | grep 'date:' " exit 1 &#125; if [ -z "$1" ]; then help; fi DOMAIN="$1" if [ -d "$HOME/.certbot/$DOMAIN" ]; then # Renew existing certificate CMD=certonly # renew OPTIONS=--force-renew else # Make new certificate (interactively) CMD=certonly unset OPTIONS fi certbot "$CMD" $OPTIONS \ --agree-tos \ --config-dir "$HOME/.certbot/$DOMAIN/config" \ -d "$DOMAIN" -d "*.$DOMAIN" \ --email mslinn@mslinn.com \ --logs-dir "$HOME/.certbot/$DOMAIN/logs" \ --manual \ --preferred-challenges dns-01 \ --rsa-key-size 4096 \ --work-dir "$HOME/.certbot/$DOMAIN/work" </pre> <p> Here is the help message for the script: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id5134b8610249'><button class='copyBtn' data-clipboard-target='#id5134b8610249' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sslCert <span class='unselectable'>Creates or updates an wildcard SSL certificate Usage: sslCert DOMAIN Example: sslCert scalacourses.com The relevant files are stored in ~/.certbot/DOMAIN/ Creating a new certificate requires an interactive user session; updating a certificate can be done in a cron job. </span></pre> </div> <!-- endregion --> <p> The following <code>crontab</code> entry causes the script to run every 60 days. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>crontab</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idc5e051fb3e64'><button class='copyBtn' data-clipboard-target='#idc5e051fb3e64' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>0 0 1 */2 * .local/bin/sslCert mslinn.com</pre> </div> <p> The next time the script was run, output looked like: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id84d8f1886630'><button class='copyBtn' data-clipboard-target='#id84d8f1886630' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sslCert mslinn.com <span class='unselectable'>Saving debug log to /home/mslinn/.certbot/mslinn.com/logs/letsencrypt.log Renewing an existing certificate for mslinn.com and *.mslinn.com<br/> Successfully received certificate. Certificate is saved at: /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/fullchain.pem Key is saved at: /home/mslinn/.certbot/mslinn.com/config/live/mslinn.com/privkey.pem This certificate expires on 2022-12-13. These files will be updated when the certificate renews.<br/> NEXT STEPS: - This certificate will not be renewed automatically. Autorenewal of --manual certificates requires the use of an authentication hook script (--manual-auth-hook) but one was not provided. To renew this certificate, repeat this same certbot command before the certificate&#39;s expiry date.<br/> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - If you like Certbot, please consider supporting our work by: * Donating to ISRG / Let&#39;s Encrypt: https://letsencrypt.org/donate * Donating to EFF: https://eff.org/donate-le - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </span></pre> </div> <!-- endregion --> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <!-- endregion --> <!-- #region checking with With A Web Browser --> <h2 id="chrome">Checking the Certificate With A Web Browser</h2> <p> Microsoft Windows caches SSL certificates, which can prevent your web browser from detecting newly updated SSL certificates. To clear the Windows SSL cache: </p> <ol> <li>Press the <kbd>Windows</kbd> key, type <code>Internet Options</code>, and press <kbd>Enter</kbd>.</li> <li>Select the <b>Content</b> tab.</li> <li>Click the <kbd>Clear SSL state</kbd> button.</li> <li>Click the <kbd>OK</kbd> button.</li> </ol> <!-- endregion --> Considering Cloudflare R2 for Static Websites 2022-06-09T00:00:00-04:00 https://mslinn.github.io/blog/2022/06/09/cloudflare-r2 <p> After deciding to <a href='/blog/2022/05/26/aws-hijacking.html'>close my AWS account</a>, I started looking for alternatives to host my static websites. Without getting into my selection criteria, my short list of possible hosting companies was <a href='https://azure.microsoft.com/' target='_blank' rel="nofollow">Microsoft Azure</a>, <a href='https://www.cloudflare.com' target='_blank' rel="nofollow">Cloudflare</a>, <a href='https://www.digitalocean.com/' target='_blank' rel="nofollow">Digital Ocean</a>, <a href='https://www.linode.com' target='_blank' rel="nofollow">Linode</a>, and <a href='https://www.netlify.com/' target='_blank' rel="nofollow">Netlify</a>. Many other options exist. </p> <p> Cloudflare has a free tier that has no time limit. It includes an S3 work-alike called R2, SSL and a built-in world-wide CDN that works automatically. 250 GB of storage and 1 TB/month of transfer are provided at no charge, forever. There are also no ingress or egress charges. <a href='https://www.fool.com/investing/2022/05/02/this-tumbling-cloud-stock-is-still-way-too-expensi/' target='_blank' rel="nofollow">Cloudflare&rsquo;s edge network</a> now spans 275 cities around the world, with nearly all Internet users within 50 milliseconds of a Cloudflare server. </p> <p> This article documents my experience with Cloudflare. If you don't care about details, and want to know my verdict on Cloudflare R2, <a href='#verdict'>skip to the end</a>. </p> <p class="alert rounded shadow"> For PaaS vendors such as AWS, Azure, Digital Ocean, Cloudflare, ScaleWay, etc.: &ldquo;pay-as-you-go&rdquo; is shorthand for &ldquo;there is nothing you can do to limit your financial liability&rdquo;. </p> <h2 id="history">This is what I did</h2> <p> After creating an account, I followed the directions at <a href='https://developers.cloudflare.com/r2/get-started/' target='_blank' rel="nofollow">R2 get started guide</a>. </p> <h2 id="wrangler">Wrangler</h2> <p> The directions told me to set up <a href='https://github.com/cloudflare/wrangler2' target='_blank' rel="nofollow">Wrangler v2</a>, a command-line interface for transferring files to R2. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id39b5431f71af'><button class='copyBtn' data-clipboard-target='#id39b5431f71af' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>npm install -g wrangler <span class='unselectable'>npm WARN config global `--global`, `--local` are deprecated. Use `--location=global` instead. npm WARN deprecated rollup-plugin-inject@3.0.2: This package has been deprecated and is no longer maintained. Please use @rollup/plugin-inject.<br/> added 56 packages, and audited 57 packages in 8s<br/> 10 high severity vulnerabilities<br/> To address issues that do not require attention, run: npm audit fix<br/> To address all issues, run: npm audit fix --force<br/> Run `npm audit` for details. </span></pre> </div> <p> The above messages do not inspire confidence. Wrangler is written using Node. I believe that Node has more security issues than any other computer language, particularly in <a href='https://turingpoint.de/en/blog/node-package-manager-security-everything-about-npm-package-security/' target='_blank' rel="nofollow">package management</a>. <a href='https://cheatsheetseries.owasp.org/cheatsheets/NPM_Security_Cheat_Sheet.html' target='_blank' rel="nofollow">The OWasp recommendations</a> do not address the <a href='https://news.ycombinator.com/item?id=29245080' target='_blank' rel="nofollow">fundamental security vulnerabilities</a> in the Node package management infrastructure. </p> <h2 id="rclone">RClone</h2> <p> I looked for alternatives to Wrangler and found <a href='https://rclone.org/docs/' target='_blank' rel="nofollow">RClone</a>. RClone is a command-line program to manage files on cloud storage. It has <a href='https://rclone.org/docs/#subcommands' target='_blank' rel="nofollow">many subcommands</a>, including two types of sync. Although the RClone documentation does not mention Cloudflare, the <a href='https://developers.cloudflare.com/r2/examples/rclone/' target='_blank' rel="nofollow">Cloudflare docs described how to set up RClone</a>. </p> <h2 id="limits">Limits</h2> <p> <a href='https://developers.cloudflare.com/workers/platform/limits/' target='_blank' rel="nofollow">Platform limits</a> are important. Not only are the technical limits important for defining inputs and outputs, users should be particularly interested in spend limits, so they are not subject to unlimited financial liability. </p> <div class='quote'> <div class='quoteText clearfix'> Workers Paid plan is separate from any other Cloudflare plan (Free, Professional, Business) you may have. If you are an Enterprise customer, reach out to your account team to confirm pricing details. <br><br> Only requests that hit a Worker will count against your limits and your bill. </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://developers.cloudflare.com/workers/platform/pricing/#fine-print' rel='nofollow' target='_blank'>Cloudflare Workers Pricing Fine Print</a></div> </div> <h2 id="setup">Setup</h2> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/cloudflare/bucket0.svg" type="image/svg"> <!---<source srcset="/blog/images/cloudflare/bucket0.avif" type="image/avif">--> <source srcset="/blog/images/cloudflare/bucket0.webp" type="image/webp"> <source srcset="/blog/images/cloudflare/bucket0.apng" type="image/apng"> <source srcset="/blog/images/cloudflare/bucket0.png" type="image/png"> <source srcset="/blog/images/cloudflare/bucket0.jpg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/bucket0.jpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/bucket0.jfif" type="image/jpeg"> <source srcset="/blog/images/cloudflare/bucket0.pjpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/bucket0.pjp" type="image/jpeg"> <source srcset="/blog/images/cloudflare/bucket0.gif" type="image/gif"> <source srcset="/blog/images/cloudflare/bucket0.tif" type="image/tiff"> <source srcset="/blog/images/cloudflare/bucket0.tiff" type="image/tiff"> <source srcset="/blog/images/cloudflare/bucket0.bmp" type="image/bmp"> <source srcset="/blog/images/cloudflare/bucket0.ico" type="image/x-icon"> <source srcset="/blog/images/cloudflare/bucket0.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/cloudflare/bucket0.png" style='width: 100%; ' /> </picture> </div> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/cloudflare/analytics0.svg" type="image/svg"> <!---<source srcset="/blog/images/cloudflare/analytics0.avif" type="image/avif">--> <source srcset="/blog/images/cloudflare/analytics0.webp" type="image/webp"> <source srcset="/blog/images/cloudflare/analytics0.apng" type="image/apng"> <source srcset="/blog/images/cloudflare/analytics0.png" type="image/png"> <source srcset="/blog/images/cloudflare/analytics0.jpg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/analytics0.jpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/analytics0.jfif" type="image/jpeg"> <source srcset="/blog/images/cloudflare/analytics0.pjpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/analytics0.pjp" type="image/jpeg"> <source srcset="/blog/images/cloudflare/analytics0.gif" type="image/gif"> <source srcset="/blog/images/cloudflare/analytics0.tif" type="image/tiff"> <source srcset="/blog/images/cloudflare/analytics0.tiff" type="image/tiff"> <source srcset="/blog/images/cloudflare/analytics0.bmp" type="image/bmp"> <source srcset="/blog/images/cloudflare/analytics0.ico" type="image/x-icon"> <source srcset="/blog/images/cloudflare/analytics0.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/cloudflare/analytics0.png" style='width: 100%; ' /> </picture> </div> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/cloudflare/billing0.svg" type="image/svg"> <!---<source srcset="/blog/images/cloudflare/billing0.avif" type="image/avif">--> <source srcset="/blog/images/cloudflare/billing0.webp" type="image/webp"> <source srcset="/blog/images/cloudflare/billing0.apng" type="image/apng"> <source srcset="/blog/images/cloudflare/billing0.png" type="image/png"> <source srcset="/blog/images/cloudflare/billing0.jpg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/billing0.jpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/billing0.jfif" type="image/jpeg"> <source srcset="/blog/images/cloudflare/billing0.pjpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/billing0.pjp" type="image/jpeg"> <source srcset="/blog/images/cloudflare/billing0.gif" type="image/gif"> <source srcset="/blog/images/cloudflare/billing0.tif" type="image/tiff"> <source srcset="/blog/images/cloudflare/billing0.tiff" type="image/tiff"> <source srcset="/blog/images/cloudflare/billing0.bmp" type="image/bmp"> <source srcset="/blog/images/cloudflare/billing0.ico" type="image/x-icon"> <source srcset="/blog/images/cloudflare/billing0.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/cloudflare/billing0.png" style='width: 100%; ' /> </picture> </div> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/cloudflare/workers0.svg" type="image/svg"> <!---<source srcset="/blog/images/cloudflare/workers0.avif" type="image/avif">--> <source srcset="/blog/images/cloudflare/workers0.webp" type="image/webp"> <source srcset="/blog/images/cloudflare/workers0.apng" type="image/apng"> <source srcset="/blog/images/cloudflare/workers0.png" type="image/png"> <source srcset="/blog/images/cloudflare/workers0.jpg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers0.jpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers0.jfif" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers0.pjpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers0.pjp" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers0.gif" type="image/gif"> <source srcset="/blog/images/cloudflare/workers0.tif" type="image/tiff"> <source srcset="/blog/images/cloudflare/workers0.tiff" type="image/tiff"> <source srcset="/blog/images/cloudflare/workers0.bmp" type="image/bmp"> <source srcset="/blog/images/cloudflare/workers0.ico" type="image/x-icon"> <source srcset="/blog/images/cloudflare/workers0.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/cloudflare/workers0.png" style='width: 100%; ' /> </picture> </div> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/cloudflare/workers1.svg" type="image/svg"> <!---<source srcset="/blog/images/cloudflare/workers1.avif" type="image/avif">--> <source srcset="/blog/images/cloudflare/workers1.webp" type="image/webp"> <source srcset="/blog/images/cloudflare/workers1.apng" type="image/apng"> <source srcset="/blog/images/cloudflare/workers1.png" type="image/png"> <source srcset="/blog/images/cloudflare/workers1.jpg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers1.jpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers1.jfif" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers1.pjpeg" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers1.pjp" type="image/jpeg"> <source srcset="/blog/images/cloudflare/workers1.gif" type="image/gif"> <source srcset="/blog/images/cloudflare/workers1.tif" type="image/tiff"> <source srcset="/blog/images/cloudflare/workers1.tiff" type="image/tiff"> <source srcset="/blog/images/cloudflare/workers1.bmp" type="image/bmp"> <source srcset="/blog/images/cloudflare/workers1.ico" type="image/x-icon"> <source srcset="/blog/images/cloudflare/workers1.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/cloudflare/workers1.png" style='width: 100%; ' /> </picture> </div> <h2 id="cache">Caching</h2> <p> <a href='https://dash.cloudflare.com/de87d0e0cbc74c70e8feb87e9671cdd0/mslinn.com/caching/configuration' target='_blank' rel="nofollow">Cache control</a>. </p> <h2 id="pages">Pages</h2> <div class='quote'> <div class='quoteText clearfix'>> Cloudflare Pages is a JAMstack platform for frontend developers to collaborate and deploy websites. </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://pages.cloudflare.com' rel='nofollow' target='_blank'>pages.cloudflare.com</a></div> </div> <p> Cloudflare Pages supports <a href='https://developers.cloudflare.com/pages/framework-guides/deploy-a-jekyll-site/' target='_blank' rel="nofollow">Jekyll sites</a>. However, it looks like Cloudflare Pages builds the site in the cloud. While this might be a useful mechanism for many, my Jekyll builds need to access my local machine, and use my <a href="/jekyll/3000-jekyll-plugins.html">Jekyll plugins</a>. </p> <p> <a href='https://developers.cloudflare.com/workers/platform/sites/start-from-existing' target='_blank' rel="nofollow">Cloudflare Workers Sites</a> suits my use case. The <a href='https://developers.cloudflare.com/workers/platform/sites/start-from-existing' target='_blank' rel="nofollow">Start From Existing</a> documentation looks appropriate, except it is written for using Wrangler, which I view as a security threat. </p> <h2 id="verdict">The Verdict: No to Cloudflare</h2> <p class="alert rounded shadow"> CloudFlare does not offer a spend limit for accounts on paid plans. This is unacceptable. I tried to remove my credit card, but found I could not. I then deleted my user account, and saw: </p> <p class="warning shadow rounded"> It could take up to 12 months to delete your information completely. </p> <p> There is no way to frame this as an example of how Cloudflare is looking out for the best interests of their customers. </p> Upgrading PostgreSQL Ubuntu 2022-05-28T00:00:00-04:00 https://mslinn.github.io/blog/2022/05/28/psql-upgrade <!-- #region intro --> <p> This article describes my recommended steps to follow when upgrading a Postgres database forced upon you when upgrading Ubuntu, and the new OS includes a new release of Postgres. </p> <p> I use PostgreSQL as the database for <a href='https://ScalaCourses.com'><code>ScalaCourses.com</code></a>. The site is served from an Ubuntu instance, and there is a backup instance. </p> <p> When upgrading to Ubuntu 22.04, PostgreSQL upgraded from v13 to v14; I used <code>pg_upgrade</code>, which required temporarily swapping the ports that Postgres used. </p> <p> However, I used <code>pg_upgradecluster</code> when upgrading to Ubuntu 23.04; and PostgreSQL upgraded from v14 to v15. Unfortunately, the documented procedure to use <code>pg_upgradecluster</code> has no consideration for <code>systemd</code>, and this makes the upgrade slightly more awkward than it might otherwise be. </p> <p> While both approaches have issues, I found that it is easier and faster to upgrade a Postgres installation using <code>pg_upgradecluster</code> than using <code>pg_upgrade</code>. </p> <!-- endregion --> <!-- #region Upgrade Backup System First --> <h2 id="upgradeubuntu">Upgrade Backup System First</h2> <p> One of the benefits of a backup system is that you can work through upgrade problems on the backup, before doing the same with the live instance. </p> <p> <a href='https://gorails.com/guides/upgrading-postgresql-version-on-ubuntu-server' target='_blank' rel="nofollow">This article</a> suggests to upgrade the entire system to Ubuntu 23.04, then upgrade the active Postgres database cluster. The alternative would be to add a new PPA for Postgres, and just update that program before upgrading the entire system; this merely substitutes one set of potential issues for another. The article also demonstrates renaming the extra database cluster that the Postgres upgrade creates, but that just creates cruft since it is not required for anything. </p> <p> I decided to delete the new database cluster instead, since upgrading the old database cluster is all that is required. As you will read in the <a href='#ug'>Upgrade Procedure</a> section, this is also the advice given during the upgrade procedure. </p> <p> I decided to first <a href='https://help.ubuntu.com/community/LunarUpgrades' target='_blank' rel="nofollow">upgrade the Ubuntu OS</a>, then migrate the Postgres database cluster. </p> <!-- endregion --> <!-- #region Upgrade Removed NVidia Support --> <h2 id="nvidia">Upgrade Removed NVidia Support</h2> <p> After running <code>do-release upgrade</code> on the production system, the system worked fine, except the console had no graphics. I fixed it by automatically installing all video drivers: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idbad0a8c3060d'><button class='copyBtn' data-clipboard-target='#idbad0a8c3060d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo ubuntu-drivers autoinstall</pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region pg_lsclusters --> <h2 id="pg_lsclusters">Examining the Postgres Databases</h2> <p> <code>pg_lsclusters</code> displays the available PostgreSQL database clusters. Prior to upgrading to Postgres 15, I had never deleted the old versions. They had quietly accumulated over the 11 years that I had been running ScalaCourses: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id07fa5f1a6f64'><button class='copyBtn' data-clipboard-target='#id07fa5f1a6f64' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pg_lsclusters <span class='unselectable'>Ver Cluster Port Status Owner Data directory Log file 9.5 main 5431 online postgres /var/lib/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log 9.6 main 5433 online postgres /var/lib/postgresql/9.6/main /var/log/postgresql/postgresql-9.6-main.log 10 main 5434 online postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log 12 main 5432 online postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log 13 main 5435 online postgres /var/lib/postgresql/13/main /var/log/postgresql/postgresql-13-main.log 14 main 5436 online postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log </span></pre> </div> <p> Listing the databases in the active cluster showed an unnecessary database: </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida3ac183b7faa'><button class='copyBtn' data-clipboard-target='#ida3ac183b7faa' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo -iu postgres psql -l <span class='unselectable'>List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges --------------+----------+----------+-------------+-------------+----------------------- bix_dev | postgres | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | postgres | postgres | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | scalacourses | postgres | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | template0 | postgres | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres template1 | postgres | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres (5 rows) </span></pre> </div> <p> I do not care about the <code>bix_dev</code> database, so I decided to delete it. The database that I care about is called <code>scalacourses</code>. </p> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idf9a274df29a5'><button class='copyBtn' data-clipboard-target='#idf9a274df29a5' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo -iu postgres psql -c 'drop database bix_dev;' <span class='unselectable'>psql (14.7 (Ubuntu 14.7-0ubuntu0.22.10.1)) Type "help" for help. DROP DATABASE </span></pre> </div> <!-- endregion --> <!-- #region Upgrade Procedure --> <h2 id="ug">Upgrade Procedure</h2> <p> Ubuntu is normally upgraded by running <code>do-release-upgrade</code>. Near the end of that process, the following message appeared. </p> <p> At this point <code>postgresql-15</code> and <code>postgresql-client-15</code> had just been installed. The message did not reflect the true state of the system, which caused me some consternation for a moment. </p> <p class="quote shadow rounded"> The PostgreSQL version 14 is obsolete, but the server or client packages are still installed. Please install the latest packages (<code>postgresql-15</code> and <code>postgresql-client-15</code>) and upgrade the existing clusters with pg_upgradecluster (<a href='https://manpages.ubuntu.com/manpages/trusty/man8/pg_upgradecluster.8.html' target='_blank' rel="nofollow">see manpage</a>). <br><br> Please be aware that the installation of postgresql-15 will automatically create a default cluster 15/main. If you want to upgrade the 14/main cluster, you need to remove the already existing 15 cluster (<code>pg_dropcluster --stop 15 main</code>, see manpage for details). <br><br> The old server and client packages are no longer supported. After the existing clusters are upgraded, the <code>postgresql-14</code> and <code>postgresql-client-14</code> packages should be removed. <br><br> Please see /usr/share/doc/postgresql-common/README.Debian.gz for details. </p> <p> Following is <code>/usr/share/doc/postgresql-common/README.Debian.gz</code>, which was referenced by the above message. Most of that file is uninteresting, however two sections are important, and we will look at them. </p> <!-- #region README.Debian.gz --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>/usr/share/doc/postgresql-common/README.Debian.gz</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id71294e43ee28'><button class='copyBtn' data-clipboard-target='#id71294e43ee28' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>PostgreSQL for Debian =====================<br/> PostgreSQL is a fully featured object-relational database management system. It supports a large part of the SQL standard and is designed to be extensible by users in many aspects. Its features include ACID transactions, foreign keys, views, sequences, subqueries, triggers, outer joins, multiversion concurrency control, and user-defined types and functions.<br/> Since the on-disk data format of all major PostgreSQL versions (like 9.6, 11, etc.) is incompatible to each other, Debian&#39;s PostgreSQL packaging architecture is designed to maintain clusters of different major versions in parallel.<br/> This postgresql-common package provides the common infrastructure and all frontend programs that users and administrators use. The version specific server and client programs are shipped in postgresql-*-&lt;version&gt; packages.<br/> For a detailed description of the architecture, please see<br/> /usr/share/doc/postgresql-common/README.md.gz<br/> First steps for the impatient ----------------------------- Eventually you will not get around reading at least some parts of the manual, but if you want to get straight into playing SQL, here are the steps to create a database user and a database for the Unix user &#39;joe&#39;:<br/> 1. Install a database server with the major version of your choice (&#39;postgresql-XY&#39;, e. g. &#39;postgresql-11&#39;). Preferably the latest version, which you can get by installing the metapackage &#39;postgresql&#39;. This will automatically create a default cluster &#39;main&#39; with the database superuser &#39;postgres&#39;.<br/> 2. Get a shell for the database superuser &#39;postgres&#39;. If your system has an active root user, use su:<br/> # su -s /bin/bash postgres<br/> If your system uses sudo to get administrative rights, use sudo instead:<br/> joe$ sudo -u postgres bash<br/> 3. In this postgres shell, create a database user with the same name as your Unix login:<br/> $ createuser -DRS joe<br/> For details about the options, see createuser(1).<br/> 4. Create a database &quot;joework&quot; which is owned by &quot;joe&quot;:<br/> $ createdb -O joe joework<br/> For details about the options, see createdb(1).<br/> 5. Exit the postgres shell.<br/> 6. As user joe, you should now be able to connect to your database with<br/> $ psql joework<br/> Cluster management ------------------ For managing clusters, the following commands are provided (each with its own manual page):<br/> pg_createcluster - Create a new cluster or integrate an existing one into the postgresql-common architecture. pg_dropcluster - Completely remove a cluster. pg_ctlcluster - Control the server process of a cluster (start, stop, restart). pg_lsclusters - Show a list of all existing clusters and their status. pg_upgradecluster - Migrate a cluster from one major version to another one. pg_renamecluster - Rename a cluster.<br/> Please note that you can of course also use the upstream tools for creating clusters, such as initdb(1). However, please note that in this case you cannot expect *any* of above pg_* tools to work, since they use different configuration settings (SSL, data directories, etc.) and file locations (e. g. /etc/postgresql/11/main/postgresql.conf). If in doubt, then do *not* use initdb, but only pg_createcluster. Since merely installing postgresql-NN will already set up a default cluster which is ready to work, most people do not need to bother about initdb or pg_createcluster at all.<br/> Port assignment --------------- Please note that the pg_* tools automatically manage the server ports unless you specify them manually. The first cluster which is ever created (by any major version) will run on the default port 5432, and each new cluster will use the next higher free one.<br/> E. g. if you first install &quot;postgresql-11&quot; on a clean system, the default 11/main cluster will run on port 5432. If you then create another 11 cluster, or install the &quot;postgresql-12&quot; package, that new one will run on 5433.<br/> Please use &quot;pg_lsclusters&quot; for displaying the cluster &lt;-&gt; port mapping, and please have a look at the pg_createcluster manpage (the --port option) for details.<br/> Default clusters and upgrading ------------------------------ When installing a postgresql-NN package from scratch, a default cluster &#39;main&#39; will automatically be created. This operation is equivalent to doing &#39;pg_createcluster NN main --start&#39;.<br/> Due to this default cluster, an immediate attempt to upgrade an earlier &#39;main&#39; cluster to a new version will fail and you need to remove the newer default cluster first. E. g., if you have postgresql-9.6 installed and want to upgrade to 11, you first install postgresql-11:<br/> apt-get install postgresql-11<br/> Then drop the default 11 cluster that was just created:<br/> pg_dropcluster 11 main --stop<br/> And then upgrade the 9.6 cluster to the latest installed version (e. g. 11):<br/> pg_upgradecluster 9.6 main<br/> SSL --- The PostgreSQL server packages support SSL, which provides encrypted and authenticated network communication. SSL should be used if you have an untrusted network between a database server and a client and these exchange security sensitive data like passwords or confidential database contents.<br/> When a cluster is created with pg_createcluster, SSL support will automatically be enabled. postgresql-common makes use of the &#39;snakeoil&#39; SSL certificate that is generated by the ssl-cert package, so that SSL works out of the box (ssl_cert_file, ssl_key_file). In addition, if /etc/postgresql-common/root.crt exists, it will be used as CA certificate file (ssl_ca_file).<br/> /etc/postgresql-common/root.crt is a dummy file by default, so that client-side authentication is not performed. To enable it, you should add some root certificates to it. A reasonable choice is to just symlink the file to /etc/ssl/certs/ssl-cert-snakeoil.pem; in this case, client certificates need to be signed by the snakeoil certificate, which might be desirable in many cases. See<br/> /usr/share/doc/postgresql-doc-11/html/ssl-tcp.html<br/> for details (in package postgresql-doc).<br/> Further documentation --------------------- All commands shipped by postgresql-common have detailed manpages. See postgresql-common(7) for the documentation of the database client program wrapping, and user_clusters(5) and postgresqlrc(5) for the cluster configuration.<br/> The documentation of the database server and client functions, SQL commands, modules, etc. documented is shipped in the per-version packages postgresql-doc-&lt;version&gt;.</pre> </div> <!-- endregion --> <p> The <b>Cluster management</b> section in the above file provides some background information for our purposes, so I repeat it here: </p> <!-- #region Cluster management --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Cluster management</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id537dfe241784'><button class='copyBtn' data-clipboard-target='#id537dfe241784' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>For managing clusters, the following commands are provided (each with its own manual page): pg_createcluster - Create a new cluster or integrate an existing one into the postgresql-common architecture. pg_dropcluster - Completely remove a cluster. pg_ctlcluster - Control the server process of a cluster (start, stop, restart). pg_lsclusters - Show a list of all existing clusters and their status. pg_upgradecluster - Migrate a cluster from one major version to another one. pg_renamecluster - Rename a cluster. Please note that you can of course also use the upstream tools for creating clusters, such as initdb(1). However, please note that in this case you cannot expect *any* of above pg_* tools to work, since they use different configuration settings (SSL, data directories, etc.) and file locations (e. g. /etc/postgresql/11/main/postgresql.conf). If in doubt, then do *not* use initdb, but only pg_createcluster. Since merely installing postgresql-NN will already set up a default cluster which is ready to work, most people do not need to bother about initdb or pg_createcluster at all.</pre> </div> <!-- endregion --> <p> The <b>Default clusters and upgrading</b> section is more important, however it contained errors. I fixed the errors in the following instructions. </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Default clusters and upgrading</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='idec07fee50e52'><button class='copyBtn' data-clipboard-target='#idec07fee50e52' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>When installing a postgresql-NN package from scratch, a default cluster 'main' will automatically be created. This operation is equivalent to doing 'pg_createcluster NN main --start'. Due to this default cluster, an immediate attempt to upgrade an earlier 'main' cluster to a new version will fail and you need to remove the newer default cluster first. E. g., if you have postgresql-9.6 installed and want to upgrade to 11, you first install postgresql-11: yes | sudo apt install postgresql-11 Then drop the default 11 cluster that was just created: sudo -iu postgres pg_dropcluster 11 main --stop And then upgrade the 9.6 cluster to the latest installed version (e. g. 11): sudo -iu postgres pg_upgradecluster 9.6 main</pre> </div> <!-- endregion --> <p> <a href='https://freedesktop.org/wiki/Software/systemd/' target='_blank' rel="nofollow">Systemd</a> can significantly affect the upgrade process, and unfortunately it is completely ignored by the above instructions. I muddled through, as you will see. </p> <!-- endregion --> <!-- #region transcript --> <h2 id="trans">Transcript</h2> <p> The following is a transcript of what I actually did: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id6921cab192ee'><button class='copyBtn' data-clipboard-target='#id6921cab192ee' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo -iu postgres pg_dropcluster --stop 15 main <span class='unselectable'>Warning: stopping the cluster using pg_ctlcluster will mark the systemd unit as failed. Consider using systemctl: sudo systemctl stop postgresql@15-main Warning: systemd was not informed about the removed cluster yet. Operations like "service postgresql start" might fail. To fix, run: sudo systemctl daemon-reload </span> <span class='unselectable'>$ </span>sudo -iu postgres pg_upgradecluster 14 main <span class='unselectable'>Stopping old cluster... Warning: stopping the cluster using pg_ctlcluster will mark the systemd unit as failed. Consider using systemctl: sudo systemctl stop postgresql@14-main Restarting old cluster with restricted connections... Notice: extra pg_ctl/postgres options given, bypassing systemctl for start operation Creating new PostgreSQL cluster 15/main ... /usr/lib/postgresql/15/bin/initdb -D /var/lib/postgresql/15/main --auth-local peer --auth-host scram-sha-256 --no-instructions --encoding UTF8 --lc-collate en_CA.UTF-8 --lc-ctype en_CA.UTF-8 The files belonging to this database system will be owned by user "postgres". This user must also own the server process. The database cluster will be initialized with locale "en_CA.UTF-8". The default text search configuration will be set to "english". Data page checksums are disabled. fixing permissions on existing directory /var/lib/postgresql/15/main ... ok creating subdirectories ... ok selecting dynamic shared memory implementation ... posix selecting default max_connections ... 100 selecting default shared_buffers ... 128MB selecting default time zone ... America/Toronto creating configuration files ... ok running bootstrap script ... ok performing post-bootstrap initialization ... ok syncing data to disk ... ok Warning: systemd does not know about the new cluster yet. Operations like "service postgresql start" will not handle it. To fix, run: sudo systemctl daemon-reload Copying old configuration files... Copying old start.conf... Copying old pg_ctl.conf... Starting new cluster... Notice: extra pg_ctl/postgres options given, bypassing systemctl for start operation Roles, databases, schemas, ACLs... set_config ------------ (1 row) set_config ------------ (1 row) set_config ------------ (1 row) set_config ------------ (1 row) Fixing hardcoded library paths for stored procedures... Upgrading database template1... Analyzing database template1... Fixing hardcoded library paths for stored procedures... Upgrading database postgres... Analyzing database postgres... Fixing hardcoded library paths for stored procedures... Upgrading database scalacourses... Analyzing database scalacourses... Stopping target cluster... Stopping old cluster... Disabling automatic startup of old cluster... Starting upgraded cluster on port 5432... Warning: the cluster will not be running as a systemd service. Consider using systemctl: sudo systemctl start postgresql@15-main Success. Please check that the upgraded cluster works. If it does, you can remove the old cluster with pg_dropcluster 14 main Ver Cluster Port Status Owner Data directory Log file 14 main 5433 down postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log Ver Cluster Port Status Owner Data directory Log file 15 main 5432 online postgres /var/lib/postgresql/15/main /var/log/postgresql/postgresql-15-main.log </span></pre> </div> <!-- endregion --> <p> When I attempted to following the above instructions for systemd, errors appeared: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id221234ef5428'><button class='copyBtn' data-clipboard-target='#id221234ef5428' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo systemctl daemon-reload <span class='unselectable'>$ </span>sudo systemctl start postgresql@15-main <span class='unselectable'>Job for postgresql@15-main.service failed because the service did not take the steps required by its unit configuration. See "systemctl status postgresql@15-main.service" and "journalctl -xeu postgresql@15-main.service" for details. </span></pre> </div> <!-- endregion --> <p> Great, an obscure failure message. <a href='https://askubuntu.com/questions/1311254/postgres-12-on-ubuntu-20-04-replication-checkpoint-has-wrong-magic-539122744-in' target='_blank' rel="nofollow">Others have fussed with this problem.</a> I expected that rebooting would be much faster than trying to fix it, and that did prove to be the case: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id884d887cf87e'><button class='copyBtn' data-clipboard-target='#id884d887cf87e' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo reboot</pre> </div> <!-- endregion --> <p> After rebooting, the upgraded Postgres cluster worked fine: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id756b286e1430'><button class='copyBtn' data-clipboard-target='#id756b286e1430' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>pg_lsclusters <span class='unselectable'>Ver Cluster Port Status Owner Data directory Log file 14 main 5433 down postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log 15 main 5432 online postgres /var/lib/postgresql/15/main /var/log/postgresql/postgresql-15-main.log </span> <span class='unselectable'>$ </span>sudo -iu postgres psql -l <span class='unselectable'>List of databases Name | Owner | Encoding | Collate | Ctype | ICU Locale | Locale Provider | Access privileges --------------+--------------+----------+-------------+-------------+------------+-----------------+----------------------- postgres | postgres | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | | libc | scalacourses | scalacourses | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | | libc | template0 | postgres | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | | libc | =c/postgres + | | | | | | | postgres=CTc/postgres template1 | postgres | UTF8 | en_CA.UTF-8 | en_CA.UTF-8 | | libc | =c/postgres + | | | | | | | postgres=CTc/postgres (4 rows) </span></pre> </div> <!-- endregion --> <p> Here is how I deleted the older database clusters: </p> <!-- #region --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='ida574c17f1601'><button class='copyBtn' data-clipboard-target='#ida574c17f1601' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>for X in 9.5 9.6 10 12 13; do sudo -iu postgres pg_dropcluster --stop $X main end</pre> </div> <!-- endregion --> <span style='font-size: 3em; float: right; margin-left: 5px;;'>&#x1F601;</span> <p> All done! </p> <!-- endregion --> Montréal International vs. Bill 96 2022-05-27T00:00:00-04:00 https://mslinn.github.io/blog/2022/05/27/mi-bill96 <p> Last night I attended the 25th Annual General Meeting of <a href='https://www.montrealinternational.com/en/' target='_blank' rel="nofollow">Montréal International</a>, an energetic organization dedicated to attracting foreign business to establish a presence in the greater Montréal region. My key takeaways were: </p> <ol> <li> I spoke with about 20 Montréal International employees; all of them were bilingual or trilingual, intelligent, motivated and well-informed. Their talent and commitment is palpable. Clearly, the hiring process is successful. </li> <li> The event was marred by long-winded, self-congratulatory speeches by senior management, and former senior managers who were invited to a panel discussion. This lack of discipline caused the event to significantly run overtime. </li> <li> The proceedings were conducted entirely in French. For an organization that calls itself &ldquo;Montréal International&rdquo;, this is unforgivable. </li> <li> I asked every employee I spoke with if <a href='https://qcgn.ca/bill-96/' target='_blank' rel="nofollow">Quebec&rsquo;s new Bill 96</a> impacted their goals. All of them expressed grave concern, and said that their mandate to attract foreign business had become extremely difficult as a result. However, not one mention was made by anyone on stage about Bill 96. </li> </ol> <p> Montréal International&rsquo;s mandate is severely impacted by Quebec's Bill 96. If any organization has an intrinsic mandate to demonstrate public leadership towards repealing or modifying Bill 96 so it respects basic human rights, it would be Montréal International. Instead, Montréal International&rsquo;s leadership, which consists entirely of old white French-speaking alpha males, is complicit in their silence, and by their actions. </p> Limit Your Financial Vulnerability From AWS Account Hijacking 2022-05-26T00:00:00-04:00 https://mslinn.github.io/blog/2022/05/26/aws-hijacking <p class="alert shadow rounded"> I am a free agent, independent and critical. <a href='/softwareexpert/index.html'>You can buy my time as a software expert</a>, but not my opinion. Just because Amazon / AWS hired me in the past to help them against a patent troll does not mean they own me. Quite the contrary. </p> <p> In the time you take a quick shower, your AWS account can be highjacked and tens of thousands of dollars in fees can be incurred. EC2 is the service that provides the greatest financial risk. You can take steps to limit your liability, and this article shows you how. </p> <h2 id="backlash">Out-of-Control Cloud Costs</h2> <p> Out-of-control costs are causing cloud customers to reduce or eliminate their cloud use. <a href='https://techcrunch.com/2023/03/20/the-cloud-backlash-has-begun-why-big-data-is-pulling-compute-back-on-premises/' target='_blank' rel="nofollow">The cloud backlash has begun: Why big data is pulling compute back on premises</a> &ndash; Thomas Robinson, published by TechCrunch 2023-03-20. </p> <p> <a href='https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47e0' target='_blank' rel="nofollow">Why we&rsquo;re leaving the cloud</a>, by David Heinemeier Hansson, creator of Ruby on Rails and CTO of 37signals. Mr. Hansson also wrote <a href='https://world.hey.com/dhh/we-stand-to-save-7m-over-five-years-from-our-cloud-exit-53996caa' target='_blank' rel="nofollow">We stand to save $7m over five years from our cloud exit</a>. </p> <h2 id="nolimits">AWS IAM Users and Roles Have No Budget Limitations</h2> <div class="pullQuote"> AWS customers are exposed to unlimited financial liability </div> <p> I encountered two main issues when attempting to secure against AWS account hijacking: </p> <ol> <li> <b>AWS does not provide a mechanism to guard against launching expensive services.</b> For example, if an IAM user has a role that allows EC2 instances to be launched, then that user can launch an unlimited number of EC2 instances of any size. <br><br> The most expensive Amazon EC2 is currently the <a href='https://aws.amazon.com/ec2/instance-types/p4/' target='_blank' rel="nofollow"><code>p4de.24xlarge</code></a>, which costs $40.96 USD per hour on-demand in the US East (N. Virginia) region. The instance comes with 96 vCPUs, 1152 GiB of RAM and eight 1TB NVMe SSDs, and has eight NVIDIA A100 Tensor Core GPUs. <br><br> Using stolen credentials that allow EC2 instances to be launched, an attacker using scripts could spin up an armada of <code>p4de.24xlarge</code> instances and incur eye-popping costs in minutes. </li> <li> AWS Budgets provides a convenient way to shut down pre-designated services when a total budgetary amount is exceeded. <b>However, the AWS AMI security model is not integrated into real-time cost monitoring.</b> Thus, there is no way to automatically shut down newly launched service instances that exceed a pre-authorized budgetary amount. </li> </ol> <div class="alert rounded shadow"> <h2 id="vercel"><a href='https://vercel.com/pricing' target='_blank' rel="nofollow">Vercel&rsquo;s Pricing Policy Incorporates Limits</a></h2> <p> <i>We provide customers with tools to observe, control, and alert on their infrastructure spend with Spend Management. You can define a spend amount (e.g. $40) and receive email, web, and SMS notifications as you reach that amount. When reaching 100%, you can optionally automatically pause all projects with a hard limit. Over the past year, we've added features to the platform based on your feedback to help automatically prevent runaway spend, including recursion protection, improved function defaults, hard spend limits, Attack Challenge Mode, and more.</i> </p> <p> AWS, Azure, Google App Engine, IBM Cloud and Linode all need a pricing policy like Vercel&rsquo;s. </p> </div> <h2 id="orElse">Shared Security Responsibility</h2> <p> The <a href='https://aws.amazon.com/agreement/' target='_blank' rel="nofollow">AWS Customer Agreement</a> and <a href='https://aws.amazon.com/compliance/shared-responsibility-model/' target='_blank' rel="nofollow">Shared Responsibility Model</a> describe how AWS shares responsibility for security with users. If you follow the AWS security recommendations, described next, you will not be held accountable for extra charges if your AWS account is hijacked. </p> <p> The AWS Security Blog published a nice overview, entitled <a href='https://aws.amazon.com/blogs/security/getting-started-follow-security-best-practices-as-you-configure-your-aws-resources/' target='_blank' rel="nofollow">Getting Started: Follow Security Best Practices as You Configure Your AWS Resources</a>. Some of the same information is also provided in <a href='https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html' target='_blank' rel="nofollow">Security best practices in IAM</a>. </p> <p class="alert rounded shadow"> Even though the AWS instructions do not prevent bad guys taking over your account, follow them anyway so you won't be held liable for the costs incurred if your AWS account is hijacked. </p> <h2 id="recommendations">AWS Security Recommendations</h2> <p> Security needs to have depth. Any single measure can fail, and given enough scale, all measures will eventually fail. Layer your security measures so that one failure, no matter how grave, will not be fatal. </p> <p> AWS recommends the following. I have highlighted what AWS personnel have described to me as being the quickest and easiest path; this article walks through those steps to the extent that I have been able. </p> <ol> <li> Set up at least two of the following services to monitor cost and usage: <ol> <li> <a href='https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-managing-costs.html' class='bg_yellow>Managing' target='_blank' rel="nofollow"><span your costs with AWS Budgets </span></a> </li> <li> <a href='https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/gs_monitor_estimated_charges_with_cloudwatch.html#gs_creating_billing_alarm' class='bg_yellow>Create' target='_blank' rel="nofollow"><span a billing alarm Using CloudWatch</span></a>. </li> <li> <a href='https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html' target='_blank' rel="nofollow">CloudTrail User Guide</a>. </li> <li> <a href='https://docs.aws.amazon.com/waf/latest/developerguide/what-is-aws-waf.html' target='_blank' rel="nofollow">Web Application Firewall (WAF)</a>. </li> <li> <a href='https://docs.aws.amazon.com/awssupport/latest/user/get-started-with-aws-trusted-advisor.html' target='_blank' rel="nofollow">Trusted Advisor</a>. <span class="bg_yellow">I found that Trusted Advisor was somewhat easier to use than CloudWatch.</span> </li> </ol> For more information about managing your AWS cost and usage, see the <a href='https://docs.aws.amazon.com/cost-management/latest/userguide/what-is-costmanagement.html' target='_blank' rel="nofollow">AWS Cost Management User Guide</a>. </li> <li>Set up at least one of the following security best practices: <ol> <li> <a href='https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa_enable.html' class='bg_yellow>Using' target='_blank' rel="nofollow"><span multi-factor authentication (MFA) in AWS</span></a>. </li> <li> <a href='https://docs.aws.amazon.com/securityhub/index.html' target='_blank' rel="nofollow">AWS Security Hub</a>. </li> <li> <a href='https://docs.aws.amazon.com/guardduty/index.html' target='_blank' rel="nofollow">Amazon GuardDuty</a>. </li> </ol> </ol> <h2 id="easiestFirst">Do The Easiest Things First</h2> <p> I usually prefer to do the easiest things first. <a href='https://www.lifehack.org/articles/productivity/easy-tasks-difficult-tasks-first-which-one-more-productive.html' target='_blank' rel="nofollow">Not everyone agrees.</a> Doing even one highlighted item above provides a significant security improvement, so why wait? </p> <div class='quote'> <div class='quoteText clearfix'> The best is the enemy of the good. </div><div class='quoteAttribution'> &nbsp;&ndash; Voltaire </div> </div> <div class='quote'> <div class='quoteText clearfix'> Better a diamond with a flaw than a pebble without. </div><div class='quoteAttribution'> &nbsp;&ndash; Confucius </div> </div> <div class='quote'> <div class='quoteText clearfix'> Striving to better, oft we mar what&rsquo;s well. </div><div class='quoteAttribution'> &nbsp;&ndash; Shakespeare </div> </div> <h2 id="root">Root Credentials</h2> <p> If a bad guy has your root credentials, they can change budget limits. For greatest security, <a href='https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/security_credentials' target='_blank' rel="nofollow">delete your root credentials</a> by selecting <b>Access Keys (access key ID and secret access key)</b> as shown in the image below. However, if you do so, <a href='https://docs.aws.amazon.com/general/latest/gr/root-vs-iam.html#aws_tasks-that-require-root' target='_blank' rel="nofollow">many things become impossible</a>. </p> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/rootKeys.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/rootKeys.avif" type="image/avif">--> <source srcset="/blog/images/aws/rootKeys.webp" type="image/webp"> <source srcset="/blog/images/aws/rootKeys.apng" type="image/apng"> <source srcset="/blog/images/aws/rootKeys.png" type="image/png"> <source srcset="/blog/images/aws/rootKeys.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/rootKeys.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/rootKeys.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/rootKeys.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/rootKeys.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/rootKeys.gif" type="image/gif"> <source srcset="/blog/images/aws/rootKeys.tif" type="image/tiff"> <source srcset="/blog/images/aws/rootKeys.tiff" type="image/tiff"> <source srcset="/blog/images/aws/rootKeys.bmp" type="image/bmp"> <source srcset="/blog/images/aws/rootKeys.ico" type="image/x-icon"> <source srcset="/blog/images/aws/rootKeys.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/rootKeys.png" style='width: 100%; ' /> </picture> </div> <h3 id="mfa">Enabling MFA</h3> <div class='imgWrapper imgBlock right' style='width: 10em; '> <figure> <a href='https://www.amazon.com/Yubico-YubiKey-NFC-Authentication-USB/dp/B07HBD71HL' target='_blank' class='imgImgUrl'><picture class='imgPicture'> <source srcset="/blog/images/aws/yubikey.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/yubikey.avif" type="image/avif">--> <source srcset="/blog/images/aws/yubikey.webp" type="image/webp"> <source srcset="/blog/images/aws/yubikey.apng" type="image/apng"> <source srcset="/blog/images/aws/yubikey.png" type="image/png"> <source srcset="/blog/images/aws/yubikey.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/yubikey.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/yubikey.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/yubikey.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/yubikey.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/yubikey.gif" type="image/gif"> <source srcset="/blog/images/aws/yubikey.tif" type="image/tiff"> <source srcset="/blog/images/aws/yubikey.tiff" type="image/tiff"> <source srcset="/blog/images/aws/yubikey.bmp" type="image/bmp"> <source srcset="/blog/images/aws/yubikey.ico" type="image/x-icon"> <source srcset="/blog/images/aws/yubikey.cur" type="image/x-icon"> <img alt='Yubikey NFC' class="imgImg rounded shadow" src="/blog/images/aws/yubikey.png" style='width: 100%; ' title='Yubikey NFC' /> </picture> </a> <figcaption class='imgFigCaption '> <a href="https://www.amazon.com/Yubico-YubiKey-NFC-Authentication-USB/dp/B07HBD71HL" target='_blank' > Yubikey NFC </a> </figcaption> </figure> </div> <p> The easiest highlighted item above is to enable multifactor authentication (MFA). </p> <p> I use a virtual MFA device, <a href='https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa.html#enable-virt-mfa-for-root' target='_blank' rel="nofollow">enabled for my root account</a> Google Authenticator for <a href='https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2' target='_blank' rel="nofollow">Android</a> and <a href='https://apps.apple.com/us/app/google-authenticator/id388497605' target='_blank' rel="nofollow">iOS</a>. Google Authenticator features a <a href='https://tools.ietf.org/html/rfc6238' target='_blank' rel="nofollow">RFC 6238</a> standards-based TOTP (time-based one-time password) algorithm. </p> <p> I use a <a href='https://www.amazon.com/Yubico-YubiKey-NFC-Authentication-USB/dp/B07HBD71HL' target='_blank' rel="nofollow">YubiKey</a> to provide MFA for the AMI user ID that I use to log into the AWS console for everyday work. </p> <p class="pullQuote"> Only use root credentials when absolutely necessary. </p> <h2 id="trustedAdvisor">Trusted Advisor</h2> <p> After adding MFA to root credentials, I found that using <a href='https://docs.aws.amazon.com/awssupport/latest/user/get-started-with-aws-trusted-advisor.html' target='_blank' rel="nofollow">AWS Trusted Advisor</a> was the next easiest thing to try to make my AWS account secure. </p> <p> Once you visit the <a href='https://console.aws.amazon.com/trustedadvisor/home' target='_blank' rel="nofollow">Trusted Advisor Console</a>, and agree to enable Trusted Advisor, <a href='https://docs.aws.amazon.com/awssupport/latest/user/security-trusted-advisor.html' target='_blank' rel="nofollow">IAM permissions are added to the AWS IAM user</a> that you are logged in as. </p> <p> Usage seems straightforward; however, I found the information for securing S3 buckets puzzling at first: </p> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/trustedAdvisor1.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/trustedAdvisor1.avif" type="image/avif">--> <source srcset="/blog/images/aws/trustedAdvisor1.webp" type="image/webp"> <source srcset="/blog/images/aws/trustedAdvisor1.apng" type="image/apng"> <source srcset="/blog/images/aws/trustedAdvisor1.png" type="image/png"> <source srcset="/blog/images/aws/trustedAdvisor1.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor1.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor1.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor1.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor1.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor1.gif" type="image/gif"> <source srcset="/blog/images/aws/trustedAdvisor1.tif" type="image/tiff"> <source srcset="/blog/images/aws/trustedAdvisor1.tiff" type="image/tiff"> <source srcset="/blog/images/aws/trustedAdvisor1.bmp" type="image/bmp"> <source srcset="/blog/images/aws/trustedAdvisor1.ico" type="image/x-icon"> <source srcset="/blog/images/aws/trustedAdvisor1.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/trustedAdvisor1.png" style='width: 100%; ' /> </picture> </div> <p> The column labeled <b>ACL Allows List</b> means that buckets flagged with <b>Yes</b> have an ACL that allows objects to be listed. AWS recommends that S3 buckets not allow objects to be listed by the public. To correct this: </p> <ol> <li>Click on the bucket name.</li> <li>Click on the <b>Permissions</b> tab in the page that opens next.</li> <li>Scroll down to <b>Access Control List</b>.</li> <li>Click on the <kbd>Edit</kbd> button.</li> <li>Disable the items marked in red, as shown below.</li> </ol> <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/trustedAdvisor2.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/trustedAdvisor2.avif" type="image/avif">--> <source srcset="/blog/images/aws/trustedAdvisor2.webp" type="image/webp"> <source srcset="/blog/images/aws/trustedAdvisor2.apng" type="image/apng"> <source srcset="/blog/images/aws/trustedAdvisor2.png" type="image/png"> <source srcset="/blog/images/aws/trustedAdvisor2.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor2.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor2.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor2.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor2.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/trustedAdvisor2.gif" type="image/gif"> <source srcset="/blog/images/aws/trustedAdvisor2.tif" type="image/tiff"> <source srcset="/blog/images/aws/trustedAdvisor2.tiff" type="image/tiff"> <source srcset="/blog/images/aws/trustedAdvisor2.bmp" type="image/bmp"> <source srcset="/blog/images/aws/trustedAdvisor2.ico" type="image/x-icon"> <source srcset="/blog/images/aws/trustedAdvisor2.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/trustedAdvisor2.png" style='width: 100%; ' /> </picture> </div> <p> Referring back to the previous screenshot, the <b>Policy Allows Access</b> column flags all S3 buckets used to serve webpages as insecure. To serve HTML pages and assets stored in an S3 bucket, the permissions must be set to allow objects to be read by everyone. Trusted Advisor flags these buckets as insecure, which does not make sense to me. </p> <p> Perhaps there is a better security policy for when webpages are served via CloudFront, but as is often the case with AWS documentation, it is difficult to piece together how to optimally configure S3 permissions with CloudFront options. </p> <h3 id="evolution">Best Practices Evolve With Technology</h3> <p> The AWS documentation is written as if the products described have always existed in their present state. In the course of researching this article, I discovered that AWS provides revision histories for their services. For example, the revision history of S3 is documented in the <a href='https://docs.aws.amazon.com/AmazonS3/latest/userguide/WhatsNew.html' target='_blank' rel="nofollow">AWS S3 User Guide</a>, and the CloudFront revision history is documented in the <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/WhatsNew.html' target='_blank' rel="nofollow">AWS CloudFront Developer Guide</a>. Revision histories are useful for experienced technologists to review periodically, so they can keep up with best practices as they evolve. </p> <p> I started using AWS S3 to serve websites in 2013, and at that time, CloudFront did not exist yet. As the two services matured, best practices, which were at first unknown, evolved and interactions between them became more complex. In particular, CloudFront <code>http</code>/<code>https</code> promotion, how origins are specified, and S3 permissions, all interact in ways that were not well documented for several years. This was a source of security problems and unwanted downtime for me. </p> <div class='imgWrapper imgFlex right' style='width: 12em; margin-right: 1em;'> <picture class='imgPicture'> <source srcset="/blog/images/aws/roadToHell.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/roadToHell.avif" type="image/avif">--> <source srcset="/blog/images/aws/roadToHell.webp" type="image/webp"> <source srcset="/blog/images/aws/roadToHell.apng" type="image/apng"> <source srcset="/blog/images/aws/roadToHell.png" type="image/png"> <source srcset="/blog/images/aws/roadToHell.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/roadToHell.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/roadToHell.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/roadToHell.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/roadToHell.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/roadToHell.gif" type="image/gif"> <source srcset="/blog/images/aws/roadToHell.tif" type="image/tiff"> <source srcset="/blog/images/aws/roadToHell.tiff" type="image/tiff"> <source srcset="/blog/images/aws/roadToHell.bmp" type="image/bmp"> <source srcset="/blog/images/aws/roadToHell.ico" type="image/x-icon"> <source srcset="/blog/images/aws/roadToHell.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/roadToHell.png" style='width: 100%; ' /> </picture> </div> <p class="alert rounded shadow"> I have 15 buckets that are used to serve web pages. On my <a href='https://us-east-1.console.aws.amazon.com/support/home?region=us-east-1&skipRegion=true#/' target='_blank' rel="nofollow">AWS Support Center console</a>, Trusted Advisor shows that &ldquo;You have a yellow check affecting 15 of 24 resources&rdquo;; however, the Trusted Advisor console displays those items as red triangles with exclamation marks. <br><br> This is confusing, and became frustrating when an AWS security support technician suggested I disregard anything that seemed annoying. For an IT/security person, that advice is the quickest way to hell that I know. <br><br><br> </p> <h3 id="rough">Rough Spots That Need Love</h3> <p> While researching this article I found: </p> <div class='quote'> <div class='quoteText clearfix'> Under <b>Origin Settings</b>, for <b>Origin Domain Name</b>, choose the Amazon S3 bucket that you created previously. <span class="bg_yellow">For <b>S3 bucket access</b>, select <b>Yes, use OAI (bucket can restrict access to only CloudFront)</b></span>. For the <b>Origin access identity</b>, you can choose from the list, or choose <b>Create new OAI</b> (both will work). For <b>Bucket policy</b>, select <b>Yes, update the bucket policy</b>. </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/getting-started-cloudfront-overview.html#getting-started-cloudfront-distribution' rel='nofollow' target='_blank'>Use an Amazon CloudFront distribution to serve a static website</a></div> </div> <p> However, I found conflicting information near the top of this article: </p> <div class='quote'> <div class='quoteText clearfix'> <h2>Important</h2> <p> If you use an Amazon S3 bucket configured as a website endpoint, you must set it up with CloudFront as a custom origin. <span class="bg_yellow">You can&rsquo;t use the origin access identity feature described in this topic.</span> However, you can restrict access to content on a custom origin by setting up custom headers and configuring your origin to require them. For more information, see <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-overview.html#forward-custom-headers-restrict-access' target='_blank' rel="nofollow">Restricting access to files on custom origins</a>. </p> </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3.html' rel='nofollow' target='_blank'>Restricting access to Amazon S3 content by using an origin access identity (OAI)</a></div> </div> <p> As if this is not confusing enough, <a href='https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/DownloadDistS3AndCustomOrigins.html#concept_S3Origin_website' target='_blank' rel="nofollow">Using various origins with CloudFront distributions</a> states that in order for Amazon S3 redirects to work, a non-standard format for the origin must be used:<br><br> <code>http://<span style="color: red;">bucket-name</span>.s3-website-<span style="color: red;">region</span>.amazonaws.com</code> <br><br> As I discussed in a <a href='/jekyll/10500-redirects.html'>previous article</a>, AWS S3 buckets support two types of redirects, but there is no indication of which type might or might not work with the non-standard origin format. <br><br> Also, there is no mention of whether <code>https</code> would work or not, or if these disparate instructions even work together. As usual, there is lots of uncertainty about how things actually work on AWS, and for how long before things change. </p> <p> As per standard operating procedure with AWS, one would have to muck about extensively to determine which of the conflicting statements above is generally prevalent, and under what circumstances. I expect that one article is newer than the other, and it likely supercedes the older article somewhat to some degree under as-yet unknown circumstances. If I get lucky, the Trusted Advisor warnings will go away. Trusted Advisor should reference whichever article is currently correct. </p> <h3 id="2022-05-31">Update 2022-05-31</h3> <p> I received the following email from Trusted Advisor. Unfortunately, the contents of the email were different from what I found on the Trusted Advisor console. Perhaps this is because a higher-level, and more expensive, service contract would be required to see those messages. If so, then this is not the way to gain happy customers. The cost of the higher service level would be 4 times more than my total AWS costs at present, clearly not something that I would want to do. </p> <div class="quote"> Dear AWS customer:<br><br> AWS Trusted Advisor currently shows alerts for 3 checks (1 red and 2 yellow) and $0 of potential monthly savings based on your usage.<br><br> Here is a summary of status changes for this week:<br><br> The alert severity of these checks has improved:<br><br> From red to yellow:<br> &nbsp;&nbsp;&nbsp;Amazon S3 Bucket Permissions<br><br> From red to green:<br> &nbsp;&nbsp;&nbsp;VPC<br> &nbsp;&nbsp;&nbsp;Security Groups - Specific Ports Unrestricted<br><br> From yellow to green:<br> &nbsp;&nbsp;&nbsp;VPC Internet Gateways<br><br> The alert status of 5 checks has not changed. For details and recommendations, visit AWS Trusted Advisor.<br><br> Sign in to the Trusted Advisor console to view your check results.<br><br> Best,<br><br> AWS Support<br> Was this report helpful? <a href='https://www.amazon.com/gp/f.html' target='_blank' rel="nofollow">Send us your feedback.</a> </div> <p> To add insult to injury, the final "Send us your feedback" just links to a generic form for AWS sales, where one can <a href='https://writingexplained.org/idiom-dictionary/go-pound-sand' target='_blank' rel="nofollow">pound sand</a>. </p> <p class="alert rounded shadow"> There is abundant evidence to demonstrate that whereas <code>amazon.com</code> is consumer-centric, AWS marches to a very different drummer. Perhaps small businesses are not presently viewed by AWS as a significant growth opportunity. My personal experience is that the current risk/value proposition provided by AWS is no longer attractive for that demographic. </p> <h3 id="2022-05-31">Update 2022-06-03</h3> <p> The credit card I used to pay for my AWS usage had a very high limit. In retrospect that was unwise. </p> <p> AWS took weeks to agree to refund the charge on my credit card, and then they said it would take another week for the reversed charges to appear. This meant that if I only paid for the other charges on that card, the month-to-month credit card interest on the fraudulent AWS charges would have been problematic. </p> <p> Just before the credit card billing period ended, I called Visa and asked what my options were. Apparently I am not the first person who had this problem with AWS. The answer was immediate: report the charges as fraud. OK, that made sense. This meant my card was immediately cancelled, the charges were forgiven by Visa, and a new card with a new number would arrive in about a week. </p> <h2 id="budgets">Managing Costs With AWS Budgets</h2> <p> Short-lived hijacking is the most serious financial threat, and AWS Budgets was seemingly not designed to guard against that. Ticking off this item will enable AWS to absolve you of financial responsibility for being hacked, but you should not expect that it will do much to slow down an attacker. </p> <p> AWS Budgets are easy to set up in a passive manner, but they are clumsy to work with and not effective for preventing new services from being launched that would exceed the budget. The cost limiting functionality is not automatic, it must be set up manually. AWS documentation does not provide detailed how-to instructions for common scenarios. Tens of thousands of words of documentation must be digested before the full capabilities can be harnessed. </p> <p> Another significant issue is that only three types of actions are supported: </p> <ol> <li> <b>IAM Policy</b> &ndash; I am unclear on how to set this up so the bad guys cannot launch new services, and I am not certain this is possible. </li> <li> <a href='https://us-east-1.console.aws.amazon.com/organizations/v2/home/policies/service-control-policy' target='_blank' rel="nofollow"><b>Service Control Policy</b></a> &ndash; Hopefully this could prevent new services from being launched that would exceed the budget; however, my head exploded when I researched this. <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/serviceControlPolicies1.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/serviceControlPolicies1.avif" type="image/avif">--> <source srcset="/blog/images/aws/serviceControlPolicies1.webp" type="image/webp"> <source srcset="/blog/images/aws/serviceControlPolicies1.apng" type="image/apng"> <source srcset="/blog/images/aws/serviceControlPolicies1.png" type="image/png"> <source srcset="/blog/images/aws/serviceControlPolicies1.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies1.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies1.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies1.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies1.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies1.gif" type="image/gif"> <source srcset="/blog/images/aws/serviceControlPolicies1.tif" type="image/tiff"> <source srcset="/blog/images/aws/serviceControlPolicies1.tiff" type="image/tiff"> <source srcset="/blog/images/aws/serviceControlPolicies1.bmp" type="image/bmp"> <source srcset="/blog/images/aws/serviceControlPolicies1.ico" type="image/x-icon"> <source srcset="/blog/images/aws/serviceControlPolicies1.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/serviceControlPolicies1.png" style='width: 100%; ' /> </picture> </div> Attempting to use this option to prevent new, expensive services from being launched would seem to introduce lots of complexity. Perhaps AWS did not consider this use case, or deemed it unimportant. </li> <li> <b>Automate Instances to Stop for EC2 or RDS</b> &ndash; If you run a <a href='https://aws.amazon.com/ec2/instance-types/' target='_blank' rel="nofollow"><code>t3.nano</code></a> instance, for example, bad guys can only incur a certain amount of financial damage. Instead of shutting down the EC2 or RDS instances that I want to run, my main concern is to prevent bad guys from launching expensive new services. This option is therefore unsuitable for my use case. </li> </ol> <p> If AWS Budgets provides an easy way of preventing new services to be launched that would exceed the budget, I could not find it after several hours of reading. </p> <h3 id="passive">Passively Monitoring Budget Spend</h3> <p> Simply follow <a href='https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-create.html' target='_blank' rel="nofollow">the directions</a> and accept the defaults. Daily budgets do not support enabling forecasted alerts, or daily budget planning, so select <b>Monthly Budgets</b>. There is no need to set up SNS alerts or AWS Chatbot alerts, email notification works just as well for most users. </p> <h3 id="passive">Attaching Actions</h3> <p> Following are my notes from my attempt to enable cost limiting. I was unable to attach an action that prevented expensive new services from being started, and it is unclear if resolving the problem will, in fact, provide the desired benefit. Perhaps someone reading these notes will be able to tell me about an approach that might work for a reasonable amount of effort. </p> <p class="alert rounded shadow"> Because AWS will absolve you of financial liability if you create a Budget without any actions, don't stress about the pointless exercise involved in making a Budget that cannot protect you. Just do it and give AWS more time to figure out a better way of preventing account hijacking. <br><br> You can stop reading this article now, unless the details interest you. </p> <p class="alert rounded shadow"> For PaaS vendors such as AWS, Azure, Digital Ocean, Cloudflare, ScaleWay, etc: &ldquo;pay-as-you-go&rdquo; is shorthand for &ldquo;there is nothing you can do to limit your financial liability&rdquo;. </p> <ol> <li>When you get to the step labeled <b>Attach actions - Optional</b>, choose <b>Add Action</b>.</li> <li> You must choose between using an existing IAM role for AWS Budgets to use, or you can create a new IAM role expressly for this purpose. IAM roles are free. I like the idea of a dedicated IAM role, so I clicked on <a href='https://us-east-1.console.aws.amazon.com/iamv2/home?region=us-east-1#/roles' target='_blank' rel="nofollow">manually create an IAM role</a>.<br> <div class='imgWrapper imgFlex inline' style='width: 75%; '> <picture class='imgPicture'> <source srcset="/blog/images/aws/budgetIamRole1.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/budgetIamRole1.avif" type="image/avif">--> <source srcset="/blog/images/aws/budgetIamRole1.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole1.apng" type="image/apng"> <source srcset="/blog/images/aws/budgetIamRole1.png" type="image/png"> <source srcset="/blog/images/aws/budgetIamRole1.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole1.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole1.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole1.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole1.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole1.gif" type="image/gif"> <source srcset="/blog/images/aws/budgetIamRole1.tif" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole1.tiff" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole1.bmp" type="image/bmp"> <source srcset="/blog/images/aws/budgetIamRole1.ico" type="image/x-icon"> <source srcset="/blog/images/aws/budgetIamRole1.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/budgetIamRole1.png" style='width: 100%; margin-left: 1.2em;' /> </picture> </div> </li> <li> A new browser tab opened, and I clicked on the blue <kbd>Create role</kbd> button. </li> <li> Now a confusing page appeared: <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/budgetIamRole2.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/budgetIamRole2.avif" type="image/avif">--> <source srcset="/blog/images/aws/budgetIamRole2.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole2.apng" type="image/apng"> <source srcset="/blog/images/aws/budgetIamRole2.png" type="image/png"> <source srcset="/blog/images/aws/budgetIamRole2.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole2.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole2.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole2.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole2.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole2.gif" type="image/gif"> <source srcset="/blog/images/aws/budgetIamRole2.tif" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole2.tiff" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole2.bmp" type="image/bmp"> <source srcset="/blog/images/aws/budgetIamRole2.ico" type="image/x-icon"> <source srcset="/blog/images/aws/budgetIamRole2.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/budgetIamRole2.png" style='width: 100%; ' /> </picture> </div> </li> <li> I selected the default <b>Trusted entity type</b>, AWS Service, then from the pulldown menu labeled <b>Use cases for other AWS services:</b> I selected <b>Budgets</b>. </li> <li> A new radio button appeared underneath the pull-down, also labeled <b>Budgets</b>, and I clicked on that. This does not win any usability awards. </li> <li> I clicked on the button labeled <kbd>Next</kbd>. </li> <li> <div class='quote'> <div class='quoteText clearfix'> Managed policy name: <code><a href='https://docs.aws.amazon.com/cost-management/latest/userguide/billing-permissions-ref.html#budget-managedIAM-SSM' target='_blank' rel="nofollow"><code>AWSBudgetsActionsRolePolicyForResourceAdministrationWithSSM</code></a></code><br><br> This managed policy is focused on specific actions that AWS Budgets takes on your behalf when completing a specific action. This policy gives AWS Budgets broad permission to control AWS resources. For example, starts and stops Amazon EC2 or Amazon RDS instances by running AWS Systems Manager (SSM) scripts. </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://docs.aws.amazon.com/cost-management/latest/userguide/billing-permissions-ref.html#budget-managedIAM-SSM' rel='nofollow' target='_blank'>Using identity-based policies (IAM policies) for AWS Cost Management</a></div> </div> On the page, I applied <code>AWSBudgetsActionsRolePolicyForResourceAdministrationWithSSM</code>, then I clicked on <kbd>Next</kbd>. <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/budgetIamRole3.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/budgetIamRole3.avif" type="image/avif">--> <source srcset="/blog/images/aws/budgetIamRole3.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole3.apng" type="image/apng"> <source srcset="/blog/images/aws/budgetIamRole3.png" type="image/png"> <source srcset="/blog/images/aws/budgetIamRole3.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole3.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole3.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole3.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole3.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole3.gif" type="image/gif"> <source srcset="/blog/images/aws/budgetIamRole3.tif" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole3.tiff" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole3.bmp" type="image/bmp"> <source srcset="/blog/images/aws/budgetIamRole3.ico" type="image/x-icon"> <source srcset="/blog/images/aws/budgetIamRole3.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/budgetIamRole3.png" style='width: 100%; ' /> </picture> </div> </li> <li> On the final page, I named the IAM role <b>Budget</b> and clicked on <kbd>Create role</kbd>. <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/budgetIamRole4.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/budgetIamRole4.avif" type="image/avif">--> <source srcset="/blog/images/aws/budgetIamRole4.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole4.apng" type="image/apng"> <source srcset="/blog/images/aws/budgetIamRole4.png" type="image/png"> <source srcset="/blog/images/aws/budgetIamRole4.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole4.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole4.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole4.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole4.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole4.gif" type="image/gif"> <source srcset="/blog/images/aws/budgetIamRole4.tif" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole4.tiff" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole4.bmp" type="image/bmp"> <source srcset="/blog/images/aws/budgetIamRole4.ico" type="image/x-icon"> <source srcset="/blog/images/aws/budgetIamRole4.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/budgetIamRole4.png" style='width: 100%; ' /> </picture> </div> </li> <li> Back in the <b>Billing Management Console</b>, I refreshed the list of available IAM roles, then selected the new role called <b>Budget</b>. <div class='imgWrapper imgFlex inline' style='width: 75%; '> <picture class='imgPicture'> <source srcset="/blog/images/aws/budgetIamRole5.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/budgetIamRole5.avif" type="image/avif">--> <source srcset="/blog/images/aws/budgetIamRole5.webp" type="image/webp"> <source srcset="/blog/images/aws/budgetIamRole5.apng" type="image/apng"> <source srcset="/blog/images/aws/budgetIamRole5.png" type="image/png"> <source srcset="/blog/images/aws/budgetIamRole5.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole5.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole5.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole5.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole5.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/budgetIamRole5.gif" type="image/gif"> <source srcset="/blog/images/aws/budgetIamRole5.tif" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole5.tiff" type="image/tiff"> <source srcset="/blog/images/aws/budgetIamRole5.bmp" type="image/bmp"> <source srcset="/blog/images/aws/budgetIamRole5.ico" type="image/x-icon"> <source srcset="/blog/images/aws/budgetIamRole5.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/budgetIamRole5.png" style='width: 100%; ' /> </picture> </div> </li> <li> I selected <b>Service Control Policy</b> from the pull-down that appeared next, labeled <b>Which action type should be applied when the budget threshold has been exceeded?</b>. I have no idea how to proceed, or even if going in this direction might provide the desired results. <div class='imgWrapper imgFlex inline fullsize' style=' '> <picture class='imgPicture'> <source srcset="/blog/images/aws/serviceControlPolicies2.svg" type="image/svg"> <!---<source srcset="/blog/images/aws/serviceControlPolicies2.avif" type="image/avif">--> <source srcset="/blog/images/aws/serviceControlPolicies2.webp" type="image/webp"> <source srcset="/blog/images/aws/serviceControlPolicies2.apng" type="image/apng"> <source srcset="/blog/images/aws/serviceControlPolicies2.png" type="image/png"> <source srcset="/blog/images/aws/serviceControlPolicies2.jpg" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies2.jpeg" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies2.jfif" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies2.pjpeg" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies2.pjp" type="image/jpeg"> <source srcset="/blog/images/aws/serviceControlPolicies2.gif" type="image/gif"> <source srcset="/blog/images/aws/serviceControlPolicies2.tif" type="image/tiff"> <source srcset="/blog/images/aws/serviceControlPolicies2.tiff" type="image/tiff"> <source srcset="/blog/images/aws/serviceControlPolicies2.bmp" type="image/bmp"> <source srcset="/blog/images/aws/serviceControlPolicies2.ico" type="image/x-icon"> <source srcset="/blog/images/aws/serviceControlPolicies2.cur" type="image/x-icon"> <img class="imgImg rounded shadow" src="/blog/images/aws/serviceControlPolicies2.png" style='width: 100%; ' /> </picture> </div> </li> </ol> Microsoft Clarity Lets Me Watch You Click and Scroll 2022-03-31T00:00:00-04:00 https://mslinn.github.io/blog/2022/03/31/clarity <!-- #region intro --> <p> I spent a fair bit of time designing the plugins for this Jekyll-powered website for <a href='https://moz.com/beginners-guide-to-seo' target='_blank' rel="nofollow">SEO</a>. These plugins are published as open source; click on the word Jekyll in the sentence above this paragraph to learn more. You are welcome! </p> <p> I do not use Google Analytics because it slows down page load time dramatically. I do use <a href='https://search.google.com' target='_blank' rel="nofollow">Google Search Console</a>, however, and recently started trying out <a href='https://www.bing.com/webmasters/' target='_blank' rel="nofollow">Microsoft Bing Webmaster Tools</a>. </p> <!-- endregion --> <!-- #region clarity --> <h2 id="clarity">Microsoft Clarity and Hotjar</h2> <p> The day before April Fool's Day in 2022, I stumbled across <a href='https://clarity.microsoft.com/' target='_blank' rel="nofollow">Microsoft Clarity</a>, a free product that I knew nothing about. I was curious to know what benefit it might provide. </p> <p> I learned that one of the things it can do is provide videos of actual user sessions as they interact with the website. Those videos are fantastic. </p> <p> As a writer of a lot of online content, I have spent hours watching you, dear readers. Individually yet anonymously. Watching people read teaches you a lot about what is important to them. As a writer, knowing your audience allows you to be more relevant. BTW, I do not know who or where my readers are, just the country they are in. </p> <!-- endregion --> <!-- #region update 1 --> <h2 id="upd">Update 2023-07-10</h2> <p class="pullQuoteFull"> I no longer need to watch the videos.<br> Microsoft Clarity's AI generates summaries.<br> Often it yields incredibly actionable information, but the <b>Key Takeaways</b> section is generally way off the mark. YMMV. </p> <p> Today I noticed the <b>insights</b> tab, visible when viewing an entry for a &lsquo;hit&rsquo;. Seems like Microsoft has employed AI to generate the following impressive summary of the user's behavior. I added punctuation, a light edit and a whimsical pad of notepaper. </p> <div class="notepaper"> <p style="text-align: left; "> Visited URL matches regex: <code>^https://www\.mslinn\.com/softwareexpert/index\.html(\?.*)?$</code><br> <b>Last 7 days<br><br> Session insights</b><br> Some key takeaways from this session are:<br><br> The user was interested in the software expert witness and computer expert services offered by Mike Slinn, as they visited the homepage from Google and spent about four minutes there. <br><br> The user was also curious about the technology expert article series, especially the ones related to Git and version control systems. They visited the article index page multiple times and clicked on several articles, spending about 15 minutes in total on this topic. <br><br> The user was most engaged with the article on libgit2, a library that provides low-level access to Git operations. They spent about one and a half minutes on this article and clicked on a link to git-fame, a tool that shows the contribution statistics of a Git repository. <br><br> The user frequently resized their browser window, which may indicate that they were using a mobile device or adjusting their screen for better readability. The user also switched between tabs or applications often, as indicated by the page hidden and page visible events. This may suggest that they were multitasking or comparing information from different sources. <br><br> 01:47 / 47:31 </p> </div> <!-- endregion --> <!-- #region update 2 --> <h2 id="update2">Update 2023-07-18</h2> <p> Today I stumbled across the <a href='https://chrome.google.com/webstore/detail/microsoft-clarity-live/cjfdbemmaeeohgibnhdhlakiahifjjcf/related' target='_blank' rel="nofollow">Clarity Live</a> plugin for the Google Chrome web browser. There is no such plugin for Firefox, sadly. </p> <div class='quote'> <div class='quoteText clearfix'> View instant heatmaps right on your live site and watch recent session recordings for any page you are on with our extension. <br><br> <ul> <li>GDPR & CCPA ready</li> <li>No sampling</li> <li>Built on open source</li> </ul> </div><div class='quoteAttribution'> &nbsp;&ndash; From <a href='https://clarity.microsoft.com/live-extension' rel='nofollow' target='_blank'>Microsoft Clarity</a></div> </div> <!-- endregion --> <!-- #region update 3 --> <h2 id="update3">Update 2023-10-04</h2> <p> I just stumbled across the <a href='https://learn.microsoft.com/en-us/clarity/setup-and-installation/clarity-api#customize-your-identifiers' target='_blank' rel="nofollow">Clarity Client API</a>. </p> <div class='quote'> <p> You can quickly get started with Clarity without coding but by interacting with the Clarity client API. This API can help you access advanced features as described in this reference. Add the following calls to Clarity APIs to the HTML or JavaScript of your webpage to access these features. </p> <h2>Note</h2> <p> Your Clarity ID serves as your API key. No other client API key is necessary, and there is no cost for using Clarity client APIs. </p> </div> <!-- endregion --> <!-- #region video --> <h2 id="video">Check out this video!</h2> <p> This is one of the first user sessions that Microsoft Clarity recorded for this website. </p> <style>.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }</style><div class='embed-container'> <iframe title="YouTube video player" width="640" height="390" src="//www.youtube.com/embed/N-Tip0Y4GMU" frameborder="0" allowfullscreen></iframe></div> <p> As you can see, Microsoft Clarity lets me watch movies of users clicking and scrolling through my website; spooky yet very informative. </p> <p> Clarity is open source. <a href='https://github.com/microsoft/clarity' target='_blank' rel="nofollow">Here</a> is the GitHub project. </p> <p> I have since learned that <a href='https://www.hotjar.com/' target='_blank' rel="nofollow">Hotjar</a> is similar to Microsoft Clarity. </p> <!-- endregion --> <!-- #region behavior --> <h2 id="behavior">Online Behavior Matches Real-World Behavior</h2> <p> The user in the above video read a bit about my experience as a software expert witness, then straightaway downloaded my resume. They were on the website for just over one minute. Although they did call me, their online behavior showed a lack of urgency, and their &lsquo;real-world&rsquo; behavior mirrored what I saw in the movie. </p> <p> A week later another party visited my site, and spent 80 minutes carefully reading 3 pages, among others. Their &lsquo;real-world&rsquo; behavior also matched their online behavior, in that they exhibited a sense of urgency towards engaging a software expert. </p> <!-- endregion --> <!-- #region downloads --> <h2 id="downloads">Tracking Downloads And Other Behavior</h2> <p> Microsoft Clarity does not consider a download as a click. It does not even notice downloads. For me, downloads are what I care about most. If you never download my resume, you probably are not a candidate for hiring me as a <a href='/softwareexpert/index.html'>software expert</a>. I want to track downloads, not clicks. Clicks are nice, but there is a direct relationship between resume downloads and signing contracts with legal firms who represent new clients. </p> <p> AWS <a href='https://aws.amazon.com/premiumsupport/knowledge-center/view-iam-history/' target='_blank' rel="nofollow">CloudTrail</a> and <a href='https://aws.amazon.com/aws-cost-management/aws-cost-optimization/monitor-track-and-analyze/' target='_blank' rel="nofollow">CloudWatch</a> can provide download details and much more. For example, user IP addresses and geographic location can be captured when a monitored event occurs. </p> <!-- endregion --> <!-- #region surveil --> <h2 id="others">Many Websites Perform Surveillance</h2> <p> Wired Magazine published an article on a similar type of surveillance (keyloggers) on May 11, 2022. I paraphrased two sentences from that article: </p> <div class='quote'> <ul> <li>1.8% of websites studied gathered an EU user's email address without their consent, and a staggering 2.95% logged a US user's email.</li> <li>For US users, 8.4% of sites may have been leaking data to Meta, Facebook&rsquo;s parent company, and 7.4% of sites may be impacted for EU users.</li> </ul> <span class='quoteAttribution'> &nbsp;&ndash; From <a href='https://www.wired.com/story/leaky-forms-keyloggers-meta-tiktok-pixel-study/' rel='nofollow' target='_blank'>Thousands of Popular Websites See What You Type—Before You Hit Submit</a></span> </div> <p> Yours truly does nothing of the sort. </p> <!-- endregion --> <!-- #region Popular Pages --> <h2 id="pp">Popular Pages</h2> <p> The data that Clarity manages can be downloaded with the <a href='https://learn.microsoft.com/en-us/clarity/setup-and-installation/clarity-data-export-api' target='_blank' rel="nofollow">Clarity Data Export API</a>. API calls are limited to 10 calls per project, per day; that is not a problem. The endpoint is <code>www.clarity.ms/export-data/api/v1/project-live-insights</code>. </p> <p> An access token is required to use the API. To get one, open the Clarity project and select <b>Settings</b> -> <b>Data Export</b> -> <b>Generate new API token</b>. I called my token <code>clarity_data_export</code> and stored it in an environment variable called <code>CLARITY_DATA</code> </p> <p> Here is how I retrieved the information about the most popular pages in the last 30 days using the bash command line: </p> <!-- #region pre --> <div class="jekyll_pre" > <div class='codeLabel unselectable' data-lt-active='false'>Shell</div> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id210734e3cedc'><button class='copyBtn' data-clipboard-target='#id210734e3cedc' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>DAYS=30<br> <span class='unselectable'>$ </span>DIM1="PopularPages"<br> <span class='unselectable'>$ </span>PARAMS="numOfDays=$DAYS&dimension1=$DIM1"<br> <span class='unselectable'>$ </span>URL="https://www.clarity.ms/export-data/api/v1/project-live-insights"<br> <span class='unselectable'>$ </span>curl \ --location "$URL?$PARAMS" \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer $CLARITY_DATA" <span class='unselectable'>$ </span></pre> </div> <!-- endregion --> <!-- endregion --> <!-- #region next --> <h2 id="next">Looking Ahead</h2> <p> I would very much like to receive a continuous data feed of the above AI-g