Mike Slinn's Blog 2021-01-13T22:31:53-05:00 https://mslinn.github.io/blog Mike Slinn mslinn@gmail.com Jekyll Plugin Template Collection 2020-12-30T00:00:00-05:00 https://mslinn.github.io/blog/2020/12/30/jekyll-plugin-template-collection <p> Here are templates for you to start writing your next <a href='https://jekyllrb.com/docs/plugins/' target='_blank'>Jekyll plugin</a> in Ruby. Templates are provided for custom Jekyll filters, generators, tags and block tags. These templates all: </p> <p> <ul> <li> Set up their own custom logger as described in my <a href='/blog/2020/12/28/custom-logging-in-jekyll-plugins.html'>previous blog post</a>. </li> <li> Provide the <a href='https://jekyllrb.com/docs/variables/' target='_blank'>Jekyll <code>site</code>, <code>page</code> and <code>mode</code> variables</a> in all the places you need them but don't have them available in scope. Note that generators are only invoked once for the entire site, when all the pages have been scanned and the site structure is available for processing, but unlike the other templates generators do not have access to the <code>page</code> variable because they are invoked on a per-site basis, not a per-page basis. It is common for generators to include code that loops through various collections of pages. </li> <li> Come with documentation boilerplate for processing with <a href='https://yardoc.org/' target='_blank'>yard</a>. </li> </ul> <p> You can <a href='/jekyll/doc/top-level-namespace.html' target='_blank'>view the rendered documentation</a>. </p> <h2 id="download">Code Download Options</h2> <p> You have options for how you might download these F/OSS Jekyll plugin templates. Pick an option and save to your <code>_plugins/</code> directory of your Jekyll-powered site. Your download options are: </p> <ol> <li> <a href='/mslinn_jekyll_plugins.zip' target='_blank' rel='nofollow'>Download a zip file</a> containing <a href='/blog/2020/10/03/jekyll-plugins.html'>all the F/OSS Jekyll plugins I publish</a>. </li> <li> Copy the following code to your clipboard by clicking on the clipboard icon at the top right of the code container, then save the code to <code>_plugins/</code> directory of your Jekyll-powered site. </li> </ol> <h2 id="filter" class="code">jekyll_filter_template.rb</h2> <p> This Jekyll filter template converts strings passed to it to upper case. <a href='/jekyll/doc/JekyllFilterTemplate.html' target='_blank'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/jekyll_filter_template.rb" download="jekyll_filter_template.rb" title="Click on the file name to download the file">jekyll_filter_template.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id71e2bc3629f0"><button class='copyBtn' data-clipboard-target='#id71e2bc3629f0' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 # Template for Jekyll filters. module JekyllFilterTemplate require_relative 'logger_factory' # include the source of logger_factory.rb into this program @log = LoggerFactory.new.create_logger('my_filter_template', Jekyll.configuration(&#123;&#125;), :warn, $stderr) # Accessor allows classes in this module to use the logger def self.log @log end # Describe the filter here. # @param input_strings [Array&lt;String>] State what this parameter is for. # @return [String] # @example Describe an example of how to use it. # &#123;&#123; 1234 | my_filter_template &#125;&#125; def my_filter_template(input_strings) JekyllFilterTemplate.log.info "input_strings = #&#123;input_strings&#125;, upcased = #&#123;input_strings.upcase&#125;".cyan input_strings.upcase end end Liquid::Template.register_filter(JekyllFilterTemplate) </pre> <h3 id="filter_out">Output</h3> <p> Given this markup in an HTML file: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'>{{ "Hello, world!" | my_filter_template }}</pre> <p> This is what is rendered to the web page after being passed through the filter: </p> HELLO, WORLD! <h2 id="generator" class="code">jekyll_generator_template.rb</h2> <p> <a href='/jekyll/doc/JekyllGeneratorTemplate.html' target='_blank'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/jekyll_generator_template.rb" download="jekyll_generator_template.rb" title="Click on the file name to download the file">jekyll_generator_template.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id2803eff6f079"><button class='copyBtn' data-clipboard-target='#id2803eff6f079' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 # Describe this Jekyll generator here. class JekyllGeneratorTemplate &lt; Jekyll::Generator require_relative 'logger_factory' def initialize(config) super(config) @log = LoggerFactory.new.create_logger('my_generator_template', config, :warn, $stderr) end # Method prescribed by the Jekyll plugin lifecycle. # @param site [Jekyll.Site] Automatically provided by Jekyll plugin mechanism # @return [void] def generate(site) @config = site.config @mode = @config['env']['JEKYLL_ENV'] @log.info "mode=#&#123;@mode&#125;".green end end </pre> <h3 id="generator_out">Output</h3> <p> Generators do not display anything on the generated web site. They can create files containing pages in a directory. Generators usually log information to the console whenever a problem occurs, or progress needs to be shown. </p> <h2 id="tag" class="code">jekyll_tag_template.rb</h2> <p> <a href='/jekyll/doc/JekyllTagTemplate.html' target='_blank'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/jekyll_tag_template.rb" download="jekyll_tag_template.rb" title="Click on the file name to download the file">jekyll_tag_template.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id6206acfe1e5a"><button class='copyBtn' data-clipboard-target='#id6206acfe1e5a' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 # # Module-level description goes here. # # @example Heading for this example # Describe what this example does # &#123;% my_tag_template "parameter" %&#125; # # @example Heading for this example # Describe what this example does # &#123;% my_tag_template "parameter" %&#125; module MyTagTemplate # Start of custom logger definition require_relative 'logger_factory' # include the source of logger_factory.rb into this program @log = LoggerFactory.new.create_logger('my_tag_template', Jekyll.configuration(&#123;&#125;), :warn, $stderr) # This accessor allows classes in this module to use the logger. def self.log @log end # End of custom logger definition # This class implements the Jekyll tag functionality class MyTag &lt; Liquid::Tag # Constructor. # @param tag_name [String] is the name of the tag, which we already know. # @param arguments [Hash, String, Liquid::Tag::Parser] the arguments from the tag. # @param tokens [Liquid::ParseContext] tokenized command line # @return [void] def initialize(tag_name, arguments, tokens) super(tag_name, arguments, tokens) MyTagTemplate.log.info "tag_name [#&#123;tag_name.class&#125;] = '#&#123;tag_name&#125;' [#&#123;tag_name.class&#125;]".green MyTagTemplate.log.info "arguments [#&#123;arguments.class&#125;] = '#&#123;arguments&#125;'".green # @site = context.registers[:site] # This variable is handy but not required # @config = @site.config # This variable is handy but not required # @mode = @config['env']['JEKYLL_ENV'] # This variable is handy but not required # MyTagTemplate.log.info "mode=#&#123;@mode&#125;".green @arguments = arguments @arguments = '' if @arguments.nil? || @arguments.empty? end # Method prescribed by the Jekyll plugin lifecycle. # @return [String] def render(context) @site = context.registers[:site] @config = @site.config @mode = @config['env']['JEKYLL_ENV'] MyTagTemplate.log.info "mode='#&#123;@mode&#125;'".green @page = context.registers[:page] MyTagTemplate.log.info "page.path='#&#123;@page.path&#125;'".green MyTagTemplate.log.info "page.url='#&#123;@page.url&#125;'".green &lt;&lt;~HEREDOC &lt;p style="color: green; background-color: yellow; padding: 1em; border: solid thin grey;"> #&#123;@arguments&#125; &lt;/p> HEREDOC end end private # Describe the function's purpose # This is a link &#123;https://domain.com with some text&#125;. # @param parameter [String] Describe this parameter's purpose # @return [String, nil] Describe the return value def my_private_function(parameter) log.info "my_private_function.parameter=#&#123;parameter&#125;" end # parse, or return the args # @note you can pass in parsed args # @return [Liquid::Tag::Parser] def parse_args(args) return args if args.is_a?(Liquid::Tag::Parser) || args.is_a?(Hash) Liquid::Tag::Parser.new( @args ) end end Liquid::Template.register_tag(MyTagTemplate.log.progname, MyTagTemplate::MyTag) </pre> <h3 id="tag_out">Output</h3> <p> Given this markup in an HTML file: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'>{% my_tag_template This is a little song I wrote, I hope you sing it note for note... %}</pre> <p> This is how it looks after the block tag is rendered to the page: </p> <p style="color: green; background-color: yellow; padding: 1em; border: solid thin grey;"> This is a little song I wrote, I hope you sing it note for note... </p> <h2 id="tag" class="code">jekyll_block_template.rb</h2> <p> <a href='/jekyll/doc/JekyllBlockTemplate.html' target='_blank'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/jekyll_block_template.rb" download="jekyll_block_template.rb" title="Click on the file name to download the file">jekyll_block_template.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id6cfada717bed"><button class='copyBtn' data-clipboard-target='#id6cfada717bed' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 # # Module-level description goes here. # # @example Heading for this example # Describe what this example does # &#123;% my_block_template "parameter" %&#125; # Hello, world! # &#123;% endmy_block_template %&#125; # # @example Heading for this example # Describe what this example does # &#123;% my_block_template "parameter" %&#125; # Hello, world! # &#123;% endmy_block_template %&#125; module MyBlockTagTemplate # Start of custom logger definition require_relative 'logger_factory' # include the source of logger_factory.rb into this program @log = LoggerFactory.new.create_logger('my_block_template', Jekyll.configuration(&#123;&#125;), :warn, $stderr) # This accessor allows classes in this module to use the logger. def self.log @log end # End of custom logger definition # This class implements the Jekyll tag functionality class MyBlock &lt; Liquid::Block # Constructor. # @param tag_name [String] the name of the tag, which we already know. # @param text [Hash, String, Liquid::Tag::Parser] the arguments from the tag. # @param tokens [Liquid::ParseContext] parsed and tokenized command line # @return [void] def initialize(tag_name, arguments, tokens) super(tag_name, arguments, tokens) MyTagTemplate.log.info "tag_name [#&#123;tag_name.class&#125;] = '#&#123;tag_name&#125;' [#&#123;tag_name.class&#125;]".green MyTagTemplate.log.info "arguments [#&#123;arguments.class&#125;] = '#&#123;arguments&#125;'".green @arguments = arguments @arguments = '' if @arguments.nil? || @arguments.empty? end # Method prescribed by the Jekyll plugin lifecycle. # @return [String] def render(context) content = super # This magically underdocumented assignment somehow returns the text within the block. @site = context.registers[:site] @config = @site.config @mode = @config['env']['JEKYLL_ENV'] MyBlockTagTemplate.log.info "mode='#&#123;@mode&#125;'".green @page = context.registers[:page] MyBlockTagTemplate.log.info "page.path='#&#123;@page.path&#125;'".green MyBlockTagTemplate.log.info "page.url='#&#123;@page.url&#125;'".green &lt;&lt;~HEREDOC &lt;p style="color: green; background-color: yellow; padding: 1em; border: solid thin grey;"> #&#123;content&#125; &lt;/p> HEREDOC end end end Liquid::Template.register_tag(MyBlockTagTemplate.log.progname, MyBlockTagTemplate::MyBlock) </pre> <h3 id="block_out">Output</h3> <p> Given this markup in an HTML file: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'>{% my_block_template %}Hello, world!{% endmy_block_template %}</pre> <p> This is how it looks after the block tag is rendered to the page: </p> <p style="color: green; background-color: yellow; padding: 1em; border: solid thin grey;"> Hello, world! </p> Custom Logging in Jekyll Plugins 2020-12-28T00:00:00-05:00 https://mslinn.github.io/blog/2020/12/28/custom-logging-in-jekyll-plugins <p> Debugging Jekyll plugins can be a challenge, but a configurable logging facility can be a big help. Writing <code>puts</code> statements to generate output does work, but they are tedious to control, because you must comment and uncomment lines that generate output as you work through issues. It is better to set up another standard Ruby logger, perhaps outputting to <code>STDERR</code>, and control the output by setting the log level for the plugin&rsquo;s logger. </p> <p> Loggers are cheap &ndash; and easy to set up. Be sure to set the log level to <code>warn</code> or <code>error</code> when you are not debugging, so site generation goes full speed. </p> <p> Use a new logger for each plugin that you write. At the end of this article I have provided the source code for <code>logger_factory.rb</code>, a Ruby library routine that I wrote which allows you to quickly make custom loggers for your plugins. </p> <h2 id="color">Colored Log Output</h2> <p> It can be difficult to find what you are looking for as you watch miles and miles of log output spew onto your console, hour after hour, while you work on a problem. Colored output can be a big help. Jekyll is configured with a <a href='https://github.com/octopress/colorator' target='_blank' rel='nofollow'>colorizer</a>, so you can use colors on any terminal output written by <code>puts</code>, or sent to a log that writes to <code>STDERR</code> or <code>STDOUT</code>. It is easy to do this, just append a suffix to a string that indicates the color you want that string to be displayed with. For example: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'>puts "This green text is written to STDOUT".green log.warn "This cyan text is written to STDERR because that is how the logger is configured".cyan</pre> <p style="color: green;">This green text is written to STDOUT</p> <p style="color: cyan;">This cyan text is written to STDERR because that is how the logger is configured</p> <p> Supported colors are: </p> <ul> <li><code>red</code></li> <li><code>black</code></li> <li><code>green</code></li> <li><code>yellow</code></li> <li><code>magenta</code></li> <li><code>white</code></li> <li><code>blue</code></li> <li><code>cyan</code></li> <li><code>bold</code></li> </ul> <h2 id="wrong">First, the Obvious But Wrong Way</h2> <p> You can send a message to <code>Jekyll.logger</code> from a plugin like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'> Jekyll.logger.debug "Hello, world" Jekyll.logger.info "Hello, world" Jekyll.logger.warn "Hello, world" Jekyll.logger.error "Hello, world" </pre> <p> However, the default <code>Jekyll.logger.log_level</code> is <code>:info</code>. This is a global setting that affects all of Jekyll because only one <code>Logger</code> is used throughout Jekyll. It outputs to <code>STDERR</code>. </p> <p> If you want to control the output by adjusting log levels you will quickly realize that attempting to adjust the <code>log_level</code> for Jekyll&rsquo;s logger singleton is problematic. This is because when you need to see verbose output from your plugin, verbose output from the rest of Jekyll will spew all over your terminal. </p> <h3 id="bad">Adjusting <span class="code">Jekyll.logger.log_level</span></h3> <p> <code>Jekyll.logger</code> is under-documented. This section is just provided to round out your understanding of how logging works in Jekyll. Please do not attempt to use Jekyll&rsquo;s logger in your plugins &mdash; instead, use the <code>logger_factory.rb</code> I provide later in this article. </p> <p> To set <code>Jekyll.logger.log_level</code> (globally), specify the <code>--verbose</code> / <code>-V</code> or <code>--quiet</code> / <code>-q</code> options when starting Jekyll: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>bundle exec jekyll build --verbose # Sets log_level to :debug <span class='unselectable'>$ </span>bundle exec jekyll build --quiet # Sets log_level to :error <span class='unselectable'>$ </span>JEKYLL_ENV=development bundle exec jekyll serve --verbose <span class='unselectable'>$ </span>JEKYLL_ENV=development bundle exec jekyll serve --quiet </pre> <p> You can also reset the Jekyll log level from within your Jekyll plugin like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'>Jekyll.logger.adjust_verbosity(:verbose => true)</pre> <p> I found that <code>_config.yml</code> has no effect on <code>Jekyll.logger.log_level</code>; the following entry does nothing: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'> quiet: true </pre> <h2 id="custom">Custom Ruby Logger To the Rescue!</h2> <p> Here is a working code example for a tag plugin that shows how to set up a custom logger in a plugin. While this code works, and is quite useful, I have packaged this code into a library routine that is more convenient to set up. You will read all about the library routine later in this article. But first, please read through this section so you understand how the library routine works. </p> <ul> <li> The logger, called <code>log</code>, is defined at the module scope. Output can be sent to <code>STDOUT</code> or <code>STDERR</code>. </li> <li> An accessor is defined called <code>self.log</code>, which is invoked as <code>MyPlugin.log</code> from within module classes. </li> <li>The logger output does not include a timestamp because that is generally not helpful.</li> <li> The program name that is passed to <code>@@log.progname</code> is also used to register the plugin with Jekyll. This ensures that the program name displayed in the log output corresponds to the registered name for the Jekyll plugin. </li> </ul> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'> require 'logger' module MyPlugin # See https://ruby-doc.org/stdlib-2.7.1/libdoc/logger/rdoc/Logger.html @@log = Logger.new(STDERR) # Could send output to STDOUT @@log.level = Logger::WARN @@log.progname = 'my_tag_name' @@log.formatter = proc { |severity, datetime, progname, msg| "#{severity} #{progname}: #{msg}\n" } def self.log @@log end class MyClass < Liquid::Tag # Write output to the custom logger like this: MyPlugin.log.info "Hello, world" MyPlugin.log.warn "You can colorize the log output".yellow end end Liquid::Template.register_tag(MyPlugin.log.progname, MyPlugin::MyClass) </pre> <p> The above works fine, but it is tedious to apply to every plugin. We need a logger factory for creating loggers in a standard way. </p> <h2 id="factory" class="code">logger_factory.rb</h2> <p> You have options for how you might download this F/OSS library plugin. Pick an option and save to a file in your Jekyll-powered site called <code>_plugins/logger_factory.rb</code>. Placing the file within the <code>_plugins/</code> directory will make this library class available to all your Jekyll plugins by following the instructions I will provide in a moment. Your options are: </p> <ol> <li> <a href='/mslinn_jekyll_plugins.zip' target='_blank' rel='nofollow'>Download a zip file</a> containing <a href='/blog/2020/10/03/jekyll-plugins.html'>all the F/OSS Jekyll plugins I publish</a>. </li> <li> Copy the following code to your clipboard by clicking on the clipboard icon at the top right of the code container, then save the code to a file in your Jekyll-powered site called <code>_plugins/logger_factory.rb</code>. </li> </ol> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/logger_factory.rb" download="logger_factory.rb" title="Click on the file name to download the file">logger_factory.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idf3bdf301e4f2"><button class='copyBtn' data-clipboard-target='#idf3bdf301e4f2' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 Michael Slinn # @license SPDX-License-Identifier: Apache-2.0 # Looks within _config.yml for a key corresponding to the plugin progname. # For example, if the plugin's progname has value "abc" then an entry called logger_factory.abc # will be read from the config file, if present. # If the entry exists, its value overrides the value specified when create_logger() was called. # If no such entry is found then the log_level value specified when create_logger() was called is used. # # @example Create a new logger using this code like this: # LoggerFactory.new.create_logger('my_tag_name', site.config, Logger::WARN, $stderr) # # For more information about the logging feature in the Ruby standard library, # @see https://ruby-doc.org/stdlib-2.7.1/libdoc/logger/rdoc/Logger.html class LoggerFactory require 'logger' # @param log_level [String, Symbol, Integer] can be specified as $stderr or $stdout, # or an integer from 0..3 (inclusive), # or as a case-insensitive string # (`debug`, `info`, `warn`, `error`, or `DEBUG`, `INFO`, `WARN`, `ERROR`), # or as a symbol (`:debug`, `:info`, `:warn`, `:error` ). # # @param config [YAML] is normally created by reading a YAML file such as Jekyll's `_config.yml`. # When invoking from a Jekyll plugin, provide `site.config`, # which is available from all types of Jekyll plugins as `Jekyll.configuration(&#123;&#125;)`. # # @example If `progname` has value `abc`, then the YAML to override the programmatically set log_level to `debug` is: # logger_factory: # abc: debug def create_logger(progname, config, log_level, stream_name) config_log_level = check_config_log_level(config: config, progname: progname) logger = Logger.new(stream_name) logger.level = config_log_level || log_level logger.progname = progname logger.formatter = proc &#123; |severity, _, prog_name, msg| "#&#123;severity&#125; #&#123;prog_name&#125;: #&#123;msg&#125;\n" &#125; logger end private # @param config [YAML] Configuration data that might contain a entry for `logger_factory` # @param progname [String] The name of the `config` subentry to look for underneath the `logger_factory` entry # @return [String, FalseClass] def check_config_log_level(config:, progname:) log_config = config['logger_factory'] return false if log_config.nil? progname_log_level = log_config[progname] return false if progname_log_level.nil? progname_log_level end end </pre> <p> The contents of <code>logger_factory.rb</code> defines a class called <code>LoggerFactory</code>. This class could be used to easily create custom loggers in any Ruby program. Following is a complete example of how you could use <code>LoggerFactory</code> in your Jekyll tag plugin. </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/jekyll_tag_template.rb" download="jekyll_tag_template.rb" title="Click on the file name to download the file">jekyll_tag_template.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id0a6ba0920466"><button class='copyBtn' data-clipboard-target='#id0a6ba0920466' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 # # Module-level description goes here. # # @example Heading for this example # Describe what this example does # &#123;% my_tag_template "parameter" %&#125; # # @example Heading for this example # Describe what this example does # &#123;% my_tag_template "parameter" %&#125; module MyTagTemplate # Start of custom logger definition require_relative 'logger_factory' # include the source of logger_factory.rb into this program @log = LoggerFactory.new.create_logger('my_tag_template', Jekyll.configuration(&#123;&#125;), :warn, $stderr) # This accessor allows classes in this module to use the logger. def self.log @log end # End of custom logger definition # This class implements the Jekyll tag functionality class MyTag &lt; Liquid::Tag # Constructor. # @param tag_name [String] is the name of the tag, which we already know. # @param arguments [Hash, String, Liquid::Tag::Parser] the arguments from the tag. # @param tokens [Liquid::ParseContext] tokenized command line # @return [void] def initialize(tag_name, arguments, tokens) super(tag_name, arguments, tokens) MyTagTemplate.log.info "tag_name [#&#123;tag_name.class&#125;] = '#&#123;tag_name&#125;' [#&#123;tag_name.class&#125;]".green MyTagTemplate.log.info "arguments [#&#123;arguments.class&#125;] = '#&#123;arguments&#125;'".green # @site = context.registers[:site] # This variable is handy but not required # @config = @site.config # This variable is handy but not required # @mode = @config['env']['JEKYLL_ENV'] # This variable is handy but not required # MyTagTemplate.log.info "mode=#&#123;@mode&#125;".green @arguments = arguments @arguments = '' if @arguments.nil? || @arguments.empty? end # Method prescribed by the Jekyll plugin lifecycle. # @return [String] def render(context) @site = context.registers[:site] @config = @site.config @mode = @config['env']['JEKYLL_ENV'] MyTagTemplate.log.info "mode='#&#123;@mode&#125;'".green @page = context.registers[:page] MyTagTemplate.log.info "page.path='#&#123;@page.path&#125;'".green MyTagTemplate.log.info "page.url='#&#123;@page.url&#125;'".green &lt;&lt;~HEREDOC &lt;p style="color: green; background-color: yellow; padding: 1em; border: solid thin grey;"> #&#123;@arguments&#125; &lt;/p> HEREDOC end end private # Describe the function's purpose # This is a link &#123;https://domain.com with some text&#125;. # @param parameter [String] Describe this parameter's purpose # @return [String, nil] Describe the return value def my_private_function(parameter) log.info "my_private_function.parameter=#&#123;parameter&#125;" end # parse, or return the args # @note you can pass in parsed args # @return [Liquid::Tag::Parser] def parse_args(args) return args if args.is_a?(Liquid::Tag::Parser) || args.is_a?(Hash) Liquid::Tag::Parser.new( @args ) end end Liquid::Template.register_tag(MyTagTemplate.log.progname, MyTagTemplate::MyTag) </pre> <h2 id="ack">Acknowledgements</h2> <p> Thanks to <a href='https://github.com/ashmaroli' target='_blank' rel='nofollow'>Ashwin Maroli (@ashmaroli)</a> for his kind assistance. </p> Propagating Git Template Changes Downstream 2020-11-30T00:00:00-05:00 https://mslinn.github.io/blog/2020/11/30/propagating-git-template-changes <p> I have developed a <a href='https://jekyllrb.com/' target='_blank' rel='nofollow'>Jekyll</a> template that I use as the starting point for most of my websites. Whenever I improve the template I can easily incorporate the changes into all the websites that are based on it. This article describes how I set that up, and the information applies to all templates in general &ndash; Jekyll is not required. Templates do not need to have any special characteristics, beyond being generally useful in some sense. </p> <p> <a href='https://docs.github.com/en/free-pro-team@latest/github/creating-cloning-and-archiving-repositories/creating-a-template-repository' target='_blank' rel='nofollow'><code>GitHub</code> has a template feature</a> but this article does not require <code>GitHub</code> and works with all <code>git</code> hosts. </p> <p> This diagram shows the local and hosted versions of a template repository and a project repository based on the template. </p> <div style=""> <picture> <source srcset="/blog/images/gitTemplates.webp" type="image/webp"> <source srcset="/blog/images/gitTemplates.png" type="image/png"> <img src="/blog/images/gitTemplates.png" title="Local and Hosted versions of a project and its template" class=" liImg2 rounded shadow" alt="Local and Hosted versions of a project and its template" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> Local and Hosted versions of a project and its template </figcaption> </figure> </div> <h2 id="template_clone">Copy from the Template Repository</h2> <p> To make a new local repository called <code>new_project</code> based on the repository called <code>template</code> from GitHub user with ID <code>mslinn</code>, type the following incantation. Please modify this command to suit your project, which might be hosted on AWS CodeCommit, Bitbucket, GitLab, etc. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>git clone git@github.com:mslinn/template.git new_project</pre> <p><i>I have no <code>git</code> repository called <code>template</code>, so the above is just for explanation purposes.</i></p> <p> <code>git</code> automatically sets up a remote origin from the local repository pointing to <code>template</code> for <code>git fetch</code> and <code>git push</code> commands, as we can see from the following: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>git remote -v <span class='unselectable'>origin git@github.com:mslinn/template.git (fetch) origin git@github.com:mslinn/template.git (push)</span> </pre> <h2 id="upstream_downstream">Define Upstream and Downstream Repositories</h2> <p> To separately obtain updates from the <code>new_project</code> repository tracked at <code>origin</code> and updates from the <code>upstream</code> template, we need to define remote URLs for both the <code>origin</code> and <code>template</code> repositories. </p> <h3 id="upstream">Define Upstream Repository</h3> <p> The template will be an upstream remote repository. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>git remote rename origin upstream</pre> <p> To ensure read-only status we should disable pushing from <code>new_project</code> to the <code>upstream</code> repository, like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>git remote set-url --push upstream no_push</pre> <h3 id="create_downstream">Create New Downstream Repository</h3> <p> We need to create a new hosted repository for <code>new_project</code>. All the git hosting sites provide a way to do this using a web browser. However, I much prefer to use command line interfaces (CLIs). </p> <h4 id="gitlab_create">AWS CodeCommit</h4> <p> <a href='https://docs.aws.amazon.com/codecommit/latest/userguide/how-to-create-repository.html#how-to-create-repository-cli' target='_blank' rel='nofollow'>The AWS CLI</a> incantation looks something like this: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>aws codecommit create-repository --repository-name new_project \ --repository-description "My downstream project"</pre> </p> <h4 id="bitbucket_create">Bitbucket</h4> <p> <a href='https://marketplace.atlassian.com/apps/1211193/bitbucket-command-line-interface-cli?hosting=server&tab=overview' target='_blank' rel='nofollow'>The Bitbucket CLI</a> incantation looks something like this: </p> <p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>acli bobswift9 --action createRepository \ --project new_project --repository new_project --name new_project</pre> </p> <h4 id="github_create">GitHub</h4> <p> The shiny new official <a href='https://cli.github.com' target='_blank' rel='nofollow'>GitHub CLI</a> unfortunately cannot do something that the tried-and-true <a href='https://hub.github.com/' target='_blank' rel='nofollow'>Hub <code>gh</code></a> does: decouple the creation of a local git repo from the creation of the remote repo on GitHub. That means we must use Hub. like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>cd new_project <span class='unselectable'>$ </span>hub create new_project <span class='unselectable'>Existing repository detected Updating origin Warning: No xauth data; using fake authentication data for X11 forwarding. X11 forwarding request failed on channel 0 https://github.com/mslinn/new_project</span></pre> <h4 id="gitlab_create">GitLab</h4> <p> All the GitLab CLIs I found had been abandoned. </p> <h3 id="git_all">All Git Host Sites</h3> <p> We can verify that the remotes for the downstream git project on your computer are now set up appropriately. The output shown shows that I used the GitHub CLI: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>git remote -v <span class='unselectable'>origin git@github.com:mslinn/new_project.git (fetch) origin git@github.com:mslinn/new_project.git (push) upstream git@github.com:mslinn/template.git (fetch) upstream no_push (push)</span></pre> <h2 id="template_update">Updating From the Downstream Repository</h2> <p> We can pull changes from the downstream <code>new_project</code> <code>origin</code> repository into the local copy of the downstream project like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>git pull <span class='unselectable'>From github.com:mslinn/template * branch master -> FETCH_HEAD Already up-to-date.</span></pre> <p> We could have typed this more verbose version, which accomplishes the same thing: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>git pull origin <span class='unselectable'>Everything up-to-date </span></pre> <h2 id="template_update">Updating From the Upstream Template</h2> <p> We can pull changes from the upstream <code>template</code> repository into the local copy of the downstream <code>new_project</code> repository like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>git pull upstream <span class='unselectable'>From github.com:mslinn/template * branch master -> FETCH_HEAD Already up-to-date.</span></pre> <h2 id="pushing">Pushing Changes</h2> <p> We can push changes from the local copy of the <code>new_project</code> repository to the hosted copy. From the top-level <code>new_project</code> directory, type: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>git add -A <span class='unselectable'>$ </span>git commit -m "Commit message goes here" <span class='unselectable'>$ </span>git push origin <span class='unselectable'>Everything up-to-date </span></pre> <p> However, if we try to push changes from <code>new_project</code> to <code>upstream</code>, we get the following error message. This is good because it means we cannot accidentally modify the upstream template when working on a project derived from the template. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>git push upstream <span class='unselectable'>fatal: 'no_push' does not appear to be a git repository fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.</span></pre> <h2 id="update_upstream">Updating the Upstream Template From a Downstream Repo</h2> <p> It is convenient to use two-way merge utilities to propagate selected changes in a downstream repository with the upstream repository. My favorite such utilities are: </p> <ul> <li><a href='https://meldmerge.org' target='_blank' rel='nofollow'>Meld</a> &ndash; Available for Linux and Windows. Also, available for Mac without support. </li> <li><a href='https://www.jetbrains.com/help/idea/command-line-merge-tool.html' target='_blank' rel='nofollow'>IntelliJ</a> &ndash; Available as a command-line utility for Windows, Mac, and Linux. <a href='https://www.jetbrains.com/help/idea/settings-tools-diff-and-merge.html' target='_blank' rel='nofollow'>Also integrated with the IDEA GUI.</a> </li> </ul> <p> Once you have propagated selected changes from the downstream project to the upstream template repo, commit the changes to the upstream repo. From the top-level template project directory, type: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>git add -A <span class='unselectable'>$ </span>git commit -m "Commit message goes here" <span class='unselectable'>$ </span>git push origin <span class='unselectable'># the word 'origin' is optional here</span> <span class='unselectable'>Everything up-to-date </span></pre> <!-- <p> The following makes it possible to push changes to <code>new_project</code> without affecting <code>template</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>git remote add origin git@github.com:mslinn/new_project.git <span class='unselectable'>$ </span>git push -u origin master</pre> --> <h2 id="2nd">Setting Up Another Computer</h2> <p> You might need to work on your project on another computer, and update from <code>upstream</code> the same way you set up the first computer. The process to do this is much the same as what was just described, but with fewer steps. It looks something like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>git clone git@github.com:mslinn/new_project.git <span class='unselectable'>Cloning into 'new_project'... Warning: No xauth data; using fake authentication data for X11 forwarding. X11 forwarding request failed on channel 0 remote: Enumerating objects: xxxx, done. remote: Total xxxx (delta 0), reused 0 (delta 0), pack-reused 1139 Receiving objects: 100% (xxxx/xxxx), xx.xx MiB | x.x MiB/s, done. Resolving deltas: 100% (xxx/xxx), done.</span> <span class='unselectable'>$ </span>cd new_project <span class='unselectable'>$ </span>git remote add upstream https://github.com/mslinn/template <span class='unselectable'>$ </span>git remote set-url --push upstream no_push </pre> <h2 id="script">GitHub Script</h2> <p> The following bash script is an example of how to automate the process of creating a project based on a template, using GitHub as the repository service. </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/cloneTemplate" download="cloneTemplate" title="Click on the file name to download the file">cloneTemplate</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id923a2b4334fe"><button class='copyBtn' data-clipboard-target='#id923a2b4334fe' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash # See https://www.mslinn.com/blog/2020/11/30/propagating-git-template-changes.html function help &#123; if [ "$1" ]; then printf "\nError: $1\n\n"; fi echo "Usage: $0 templateUrl newProjectName" exit 1 &#125; if [ -z "$(which git)" ]; then echo "Please install git and rerun this script" exit 2 fi if [ -z "$(which hub)" ]; then echo "Please install hub and rerun this script" exit 3 fi if [ -z "$1" ]; then help "No git project was specified as a template."; fi if [ -z "$2" ]; then help "Please provide the name of the new project based on the template"; fi git clone "$1" "$2" cd "$2" git remote rename origin upstream git remote set-url --push upstream no_push # Add the -p option to create a private repository hub create "$2" git branch -M master git push -u origin master </pre> <h2 id="acks">Acknowledgements</h2> <p> This posting was inspired by <a href='https://medium.com/@smrgrace/having-a-git-repo-that-is-a-template-for-new-projects-148079b7f178' target='_blank' rel='nofollow'>this article</a>. </p> Installing a New SSH Key on AWS EC2 with User Data 2020-10-27T00:00:00-04:00 https://mslinn.github.io/blog/2020/10/27/installing-a-new-ssh-key-on-awc-ec2-with-user-data <editor-fold Intro> <p> For some reason the <code>ssh</code> certificates that AWS generated for me 3 years ago are no longer recognized by Ubuntu 20.10. This article shows how to create new certificates and push them to an AWS server that was just upgraded to Ubuntu 20.01, and now cannot be logged into. I decided to use OpenSSH to generate the new keypairs instead of AWS to generate the keypairs because the current problem stems from <a href='https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html' target='_blank' rel='nofollow'>AWS-generated keys</a> gradually becoming incompatible with OpenSSH servers. </p> <p> This article describes the following: </p> <ol> <li>Tracking down the problem</li> <li>Create a new <code>ssh</code> certificate keypair.</li> <li>Even though the system cannot accept logins, the new <code>ssh</code> public key must be copied to the <code>ubuntu</code> user&rsquo;s <code>~/.ssh</code> directory on the problem server. This is done by defining a user data script on the server instance prior to booting it.</li> <li>Log into the problem server using the new certificates.</li> <li>Complete the upgrade to XUbuntu 20.10.</li> <li>Remove the user data script from the problem server instance.</li> </ol> </editor-fold Intro> <editor-fold Discovery> <h2 class="numbered" id="discovery">Discovery</h2> <p> I was able to log into another of my machines (<code>gojira</code>), so first I wanted to know if the problem machines (<code>va</code> and <code>france</code>) had OpenSSH configured differently. I used the <a href='https://linux.die.net/man/1/comm' target='_blank' rel='nofollow'>comm</a> Linux utility to perform the comparison. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>for x in cipher mac key kex; do comm -3 <(ssh -Q $x france|sort) <(ssh -Q $x gojira|sort) done <span class='unselectable'>$ </span>for x in cipher mac key kex; do comm -3 <(ssh -Q $x france|sort) <(ssh -Q $x gojira|sort) done </pre> <p> So the problem was not OpenSSH configuration per se. Next I wondered if the ssh connections to the problem machines were different somehow from the ssh connection to the working machine. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>comm -3 <(ssh -Gva|sort) <(ssh -G gojira|sort) hostname gojira hostname va identityfile ~/.ssh/id_rsa identityfile ~/.ssh/sslTest user mslinn user ubuntu </pre> <p>So the only differences were the hostnames and the keys offered.</p> <p> One of the problem machines, <code>france</code>, resided on <a href='https://scaleway.com' target='_blank' rel='nofollow'><code>scaleway</code></a>. I used the most recently available <a href='https://medium.com/@moul/scaleway-bootscript-simple-kernel-management-for-your-c1-server-de0c301de721' target='_blank' rel='nofollow'>bootscript</a> to launch the server and examined <code>/var/log/auth.log</code>. I found this: <code>sshd[27025]: Unable to negotiate with 205.185.123.173 port 40555: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1 [preauth]</code> </p> <p> This error message is produced by OpenSSH 7.0+. <a href='https://www.openssh.com/txt/release-7.0' target='_blank' rel='nofollow'>The release notes say</a> &ldquo;Support for the 1024-bit diffie-hellman-group1-sha1 key exchange is disabled by default at run-time. It may be re-enabled using the instructions at <code><a href='http://www.openssh.com/legacy.html' target='_blank' rel='nofollow'>http://www.openssh.com/legacy.html</a></code> &rdquo; </p> <p> So it seems that the version of OpenSSH installed with Ubuntu 20.10 rejects my old keys. The release notes for the new version of OpenSSH also indicate that OpenSSH 7.1 will be even stricter: </p> <ul> <li> &ldquo;This focus of this release is primarily to deprecate weak, legacy and/or unsafe cryptography&rdquo; </li> <li> &ldquo;Refusing all RSA keys smaller than 1024 bits (the current minimum is 768 bits)&rdquo; </li> <li> &ldquo;Several ciphers will be disabled by default: <code>blowfish-cbc</code>, <code>cast128-cbc</code>, all <code>arcfour</code> variants and the <code>rijndael-cbc</code> aliases for AES.&rdquo; </li> <li> &ldquo;MD5-based <code>HMAC</code> algorithms will be disabled by default.&rdquo; </li> </ul> <p> Clearly I need to generate better <code>ssh</code> keys. </p> <p> The version of OpenSSH installed by Ubuntu 20.10 is 8.3: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'> $ sshd -V <span class='unselectable'>unknown option -- V OpenSSH_8.3p1 Ubuntu-1, OpenSSL 1.1.1f 31 Mar 2020 usage: sshd [-46DdeiqTt] [-C connection_spec] [-c host_cert_file] [-E log_file] [-f config_file] [-g login_grace_time] [-h host_key_file] [-o option] [-p port] [-u len]</span> $ ssh -V <span class='unselectable'>OpenSSH_8.3p1 Ubuntu-1, OpenSSL 1.1.1f 31 Mar 2020 </span> </pre> </editor-fold Discovery> <editor-fold Setup> <h2 class="numbered" id="setup">Setup</h2> <h3 class="numbered" id="setupAwsCli">AWS CLI</h3> <p> I prefer to use the AWS CLI instead of the <a href='https://console.aws.amazon.com' target='_blank' rel='nofollow'>web console</a>. Installation instructions are <a href='https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html' target='_blank' rel='nofollow'>here</a>. This article uses the AWS CLI exclusively in favor of the AWS web console. </p> <h3 class="numbered" id="setupJq"><span class="code">jq</span></h3> <p> I also use <a href='https://stedolan.github.io/jq/' target='_blank' rel='nofollow'>jq</a> for parsing JSON in the bash shell. Install it on Debian-style Linux distros such as Ubuntu like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install jq</pre> <h3 class="numbered" id="setupRemember"><span class="code">remember</span> Script</h3> <p> This article defines several environment variables. To keep track of them, and to persist them, I use the <a href='/blog/2020/10/23/working-with-command-line-interps.html'><code>remember</code> script and the <code>mem</code> command alias</a> that I discussed in a previous blog post. If you are unfamiliar with my <code>remember</code> script and the <code>mem</code> command alias by which the <code>remember</code> script is normally invoked, please check out that article before trying to understand the command lines shown in this article. </p> </editor-fold Setup> <editor-fold nameKeys> <h2 class="numbered" id="nameKeys">Name the New Keypair</h2> <p> I wanted to make new <code>ecdsa</code> keys because this algorithm is the currently accepted best practice for commercial security concerns. <code>ecdsa</code> stands for <a href='https://en.bitcoin.it/wiki/Elliptic_Curve_Digital_Signature_Algorithm' target='_blank' rel='nofollow'>Elliptic Curve Digital Signature Algorithm</a>. </p> <p> Unfortunately, AWS EC2 only accepts <code>RSA</code> keys. The name of the new key pair will be of the form <code>~/.ssh/rsa-YYYY-MM-DD</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_KEY_PAIR_FILE "$HOME/.ssh/rsa-$( date '+%Y-%m-%d' )" <span class='unselectable'>AWS_KEY_PAIR_FILE=/home/mslinn/.ssh/rsa-2020-11-03</span> </pre> <p> The new public key will be called <code>~/.ssh/rsa-2020-11-03.pub</code> and the new private key will be called <code>~/.ssh/rsa-2020-11-03</code>. </p> </editor-fold nameKeys> <editor-fold createKeys> <h2 class="numbered" id="createKeys">Create a New Keypair</h2> <ol> <li type='a'>This is how I would have created a new <code>ECDSA</code> keypair. <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ssh-keygen -b 521 -C "mslinn@mslinn.com" -f "$AWS_KEY_PAIR_FILE" -P "" -t ecdsa <span class='unselectable'>Generating public/private ecdsa key pair. Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-03 Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-03.pub The key fingerprint is: SHA256:HEKjAA1GZxHbpwqjm85DXQpQEeIWrcjZ6fl84RHQaHE mslinn@mslinn.com The key's randomart image is: +---[RSA 2048]----+ |=O*Bo.*E | |+.=oo*.o | |+o+.+.o.. | |o= o .o+ . | | o+ +. S | |..o=. o | |o .o . o | |.+ o o | |+o. . | +----[SHA256]-----+ $ </span>chmod 400 $AWS_KEY_PAIR_FILE </pre> </li> <li type='a'> Instead, I created a new <code>RSA</code> keypair like this: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ssh-keygen -b 4096 -C "mslinn@mslinn.com" -f "$AWS_KEY_PAIR_FILE" -m PEM -P "" -t rsa <span class='unselectable'>Generating public/private rsa key pair. Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-03 Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-03.pub The key fingerprint is: SHA256:bQScX0UMn0xGDorxSvElMZzwMyyk7hgs2FNbshBNenA mslinn@mslinn.com The key's randomart image is: +---[RSA 4096]----+ | ooE .*++o+** | | =. ooXo=.B.. | | o + o +.X. = | | o = * . =.o | |. + = . S o | | o + . | | . . | | | | | +----[SHA256]-----+ $ </span>chmod 400 $AWS_KEY_PAIR_FILE </pre> </li> <li type='a'> I would have liked to copy the keypair to the problem system using <code>ssh-copy-id</code>, but that only works when login is possible. <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ssh-copy-id -i $AWS_KEY_PAIR_FILE ubuntu@$AWS_PROBLEM_IP </pre> </li> <li type='a'> Instead, I decided to paste the public key into an AWS user data script and execute that script on the problem server the next time it booted. The purpose of the script is to copy the new public key that was just made to <code>~/.ssh/</code> on the problem server. This is the user data script I wrote to install the new public key, called <code>rescue_ubuntu2010.sh</code>: <br /><br /> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/factor12/rescue_ubuntu2010.sh" download="rescue_ubuntu2010.sh" title="Click on the file name to download the file">rescue_ubuntu2010.sh</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id6b5dc2e7037c"><button class='copyBtn' data-clipboard-target='#id6b5dc2e7037c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash KEY_FILE_NAME=/home/ubuntu/.ssh/rsa-2020-11-03.pub cat > "$KEY_FILE_NAME" &lt;&lt;EOF ssh-rsa ABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFAABCDEFA== mslinn@mslinn.com EOF chown ubuntu: /home/ubuntu/.ssh/* chmod 400 "$KEY_FILE_NAME" cat "$KEY_FILE_NAME" >> /home/ubuntu/.ssh/authorized_keys </pre> The script runs on the problem server as <code>root</code> next time the system boots, and it reboots the server on the last line. </li> <li type='a'> The script need to be converted into base 64, in a file called <code>rescue_ubuntu2010.b64</code>. <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>base64 rescue_ubuntu2010.sh > rescue_ubuntu2010.b64</pre> </li> <li type='a'> The problem EC2 instance can be shut down like this: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 stop-instances --instance-id $AWS_PROBLEM_INSTANCE_ID <span class='unselectable'>$ </span>aws ec2 wait instance-stopped --instance-ids $AWS_PROBLEM_INSTANCE_ID</pre> </li> <li type='a'> With the problem EC2 instance stopped, its user data was set to the base64-encoded version of the rescue script. <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 modify-instance-attribute \ --instance-id $AWS_PROBLEM_INSTANCE_ID \ --attribute userData \ --value file://rescue_ubuntu2010.b64 </pre> </li> <li type='a'> Now the problem EC2 instance can be restarted. The script will add the new key to <code>/home/ubuntu/.ssh/authorized_keys</code> and login should be possible. <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 start-instances --instance-id $AWS_PROBLEM_INSTANCE_ID <span class='unselectable'>{ "StartingInstances": [ { "CurrentState": { "Code": 0, "Name": "pending" }, "InstanceId": "i-d3b03954", "PreviousState": { "Code": 80, "Name": "stopped" } } ] }</span> <span class='unselectable'>$ </span>aws ec2 wait instance-running --instance-ids $AWS_PROBLEM_INSTANCE_ID</pre> </li> </ol> </editor-fold createKeys> <editor-fold resetData> <h2 class="numbered" id="resetData">Reset User Data for Next Time</h2> <p> Next time the problem server is stopped, clear the user data so it is not provided the next time the server restarts. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 modify-instance-attribute \ --instance-id $AWS_PROBLEM_INSTANCE_ID \ --user-data Value= </pre> </editor-fold resetData> Rescuing a Catastrophic Upgrade to Ubuntu 20.10 2020-10-25T00:00:00-04:00 https://mslinn.github.io/blog/2020/10/25/rescuing-catastrophic-upgrades-to-ubuntu-20_10 <editor-fold Intro> <p> The upgrade from Ubuntu 20.04 to 20.10 has been especially problematic for each of the half-dozen XUbuntu systems that I manage. One important server that I run on <a href='https://www.scaleway.com' target='_blank' rel='nofollow'>Scaleway</a> became unresponsive and would not boot shortly after starting the installation, and another important server on <a href='https://aws.amazon.com' target='_blank' rel='nofollow'>AWS</a> ran fine, but did not allow logins. </p> <p> This blog post details what I did to recover the AWS server using a standard <a href='https://en.wikipedia.org/wiki/Unix-like' target='_blank' rel='nofollow'>*nix</a> procedure that any competent system administrator would be comfortable with: <a href='https://en.wikipedia.org/wiki/Chroot' target='_blank' rel='nofollow'><code>chroot</code></a>. Because the <code>chroot</code> environment will be set up in a way that it shares the rescue system&rsquo;s <code>/var/run</code> directory, the rescue system should have all upgrades in place and should be rebooted if <code>/var/run/reboot-required</code> exists. </p> <p> AWS also provides a tool called <a href='https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-ec2rescue.html' target='_blank' rel='nofollow'><code>EC2Rescue</code></a>, which does a complicated series of actions to accomplish something similar. <a href='https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html' target='_blank' rel='nofollow'>Here is additional documentation.</a> Because I find the AWS documentation is frequently obtuse, and the approach taken by most AWS products and tools is extremely general, I often find myself wasting a lot of time trying to get things to work. I don&rsquo;t subscribe to AWS support; if I had subscribed to expensive enterprise-level support, complete with an AWS expert to hold my hand while I attempted to resurrect the server, I might have tried using <code>EC2Rescue</code>. On the other hand, when pressed with an emergency, I prefer to lean on tried-and-true methods like <code>chroot</code>. </p> </editor-fold Intro> <editor-fold Setup> <h2 class="numbered" id="setup">Setup</h2> <h3 class="numbered" id="setupAwsCli">AWS CLI</h3> <p> I prefer to use the AWS CLI instead of the <a href='https://console.aws.amazon.com' target='_blank' rel='nofollow'>web console</a>. Installation instructions are <a href='https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html' target='_blank' rel='nofollow'>here</a>. This article uses the AWS CLI exclusively in favor of the AWS web console. </p> <h3 class="numbered" id="setupJq"><span class="code">jq</span></h3> <p> I also use <a href='https://stedolan.github.io/jq/' target='_blank' rel='nofollow'>jq</a> for parsing JSON in the bash shell. Install it on Debian-style Linux distros such as Ubuntu like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install jq</pre> <h3 class="numbered" id="setupRemember"><span class="code">remember</span> Script</h3> <p> This article defines several environment variables. To keep track of them, and to persist them, I use the <a href='/blog/2020/10/23/working-with-command-line-interps.html'><code>remember</code> script and the <code>mem</code> command alias</a> that I discussed in a previous blog post. If you are unfamiliar with my <code>remember</code> script and the <code>mem</code> command alias by which the <code>remember</code> script is normally invoked, please check out that article before trying to understand the command lines shown in this article. </p> </editor-fold Setup> <editor-fold info> <h2 class="numbered" id="info">Discover information about the Problem EC2 instance</h2> <h3 class="numbered" id="volumeId">Getting the AWS EC2 Instance Information</h3> <p> Because my problem EC2 instance has a tag called <code>Name</code> with <code>Value</code> <code>production</code>, I was able to easily obtain a JSON representation of all the information about it. I stored the JSON in an environment variable called <code>AWS_EC2_PRODUCTION</code>. </p> <p> The results are shown in unselectable text. This is so you can easily use this sample code yourself. You can copy the code to run into your clipboard. Just click on the little copy icon at the top right hand corner of the scrolling code display area. Because the prompt and the results and are unselectable, your clipboard will only pick up the code you need to paste in order to run the code example yourself. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_EC2_PRODUCTION "$( aws ec2 describe-instances | \ jq '.Reservations[].Instances[] | select((.Tags[].Key=="Name") and (.Tags[].Value=="production"))' )" <span class='unselectable'>AWS_EC2_PRODUCTION= '{ "AmiLaunchIndex": 0, "ImageId": "ami-e29b9388", "InstanceId": "i-825eb905", "InstanceType": "t2.small", "KeyName": "sslTest", "LaunchTime": "2017-10-12T16:24:14.000Z", "Monitoring": { "State": "disabled" }, "Placement": { "AvailabilityZone": "us-east-1c", "GroupName": "", "Tenancy": "default" }, "PrivateDnsName": "ip-10-0-0-201.ec2.internal", "PrivateIpAddress": "10.0.0.201", "ProductCodes": [], "PublicDnsName": "", "PublicIpAddress": "52.207.225.143", "State": { "Code": 16, "Name": "running" }, "StateTransitionReason": "", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "Architecture": "x86_64", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "AttachTime": "2016-04-05T19:07:17.000Z", "DeleteOnTermination": true, "Status": "attached", "VolumeId": "vol-1c8903b4" } } ], "ClientToken": "GykZz1459883236367", "EbsOptimized": false, "Hypervisor": "xen", "NetworkInterfaces": [ { "Association": { "IpOwnerId": "amazon", "PublicDnsName": "", "PublicIp": "52.207.225.143" }, "Attachment": { "AttachTime": "2016-04-05T19:07:16.000Z", "AttachmentId": "eni-attach-a58bd15f", "DeleteOnTermination": true, "DeviceIndex": 0, "Status": "attached" }, "Description": "Primary network interface", "Groups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "Ipv6Addresses": [], "MacAddress": "0a:a4:be:1b:8e:eb", "NetworkInterfaceId": "eni-fa4f65bb", "OwnerId": "031372724784", "PrivateIpAddress": "10.0.0.201", "PrivateIpAddresses": [ { "Association": { "IpOwnerId": "amazon", "PublicDnsName": "", "PublicIp": "52.207.225.143" }, "Primary": true, "PrivateIpAddress": "10.0.0.201" } ], "SourceDestCheck": true, "Status": "in-use", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "InterfaceType": "interface" } ], "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SecurityGroups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "SourceDestCheck": true, "Tags": [ { "Key": "Name", "Value": "production" } ], "VirtualizationType": "hvm", "CpuOptions": { "CoreCount": 1, "ThreadsPerCore": 1 }, "CapacityReservationSpecification": { "CapacityReservationPreference": "open" }, "HibernationOptions": { "Configured": false }, "MetadataOptions": { "State": "applied", "HttpTokens": "optional", "HttpPutResponseHopLimit": 1, "HttpEndpoint": "enabled" } }'</span> </pre> <h3 class="numbered" id="problemInstanceId">Getting the AWS EC2 Problem Instance Id</h3> <p> The instance ID for the problem EC2 instance can be extracted from the JSON returned by the preceding results easily: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_PROBLEM_INSTANCE_ID "$( jq -r .InstanceId <<< $AWS_EC2_PRODUCTION )" <span class='unselectable'>AWS_PROBLEM_INSTANCE_ID=i-825eb905</span> </pre> <h3 class="numbered" id="problemInstanceId">Getting the AWS EC2 Problem Instance IP Address</h3> <p> The IP address for the problem EC2 instance can be extracted from the JSON returned by the preceding results easily: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_PROBLEM_IP "$( jq -r .PublicIpAddress <<< $AWS_EC2_PRODUCTION )" <span class='unselectable'>AWS_PROBLEM_IP=52.207.225.143</span> </pre> <h3 class="numbered" id="problemVolumeId">Getting the AWS EC2 Problem Availability Zone</h3> <p> The AWS availability zone for the problem EC2 instance can be extracted from the JSON returned by the preceding results easily: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_AVAILABILITY_ZONE "$( jq -r .Placement.AvailabilityZone <<< $AWS_EC2_PRODUCTION )" <span class='unselectable'>AWS_AVAILABILITY_ZONE=us-east-1c</span> </pre> <h3 class="numbered" id="volumeId">Getting the AWS EC2 Problem Volume ID</h3> <p> The following command line extracts the volume id of the problem server&rsquo;s system drive into an environment variable called <code>$AWS_PROBLEM_VOLUME_ID</code>: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_PROBLEM_VOLUME_ID "$( jq -r '.BlockDeviceMappings[].Ebs.VolumeId' <<< "$AWS_EC2_PRODUCTION" )" <span class='unselectable'>AWS_PROBLEM_VOLUME_ID=vol-1c8903b4</span> </pre> </editor-fold info> <editor-fold snapshotProblem> <h2 class="numbered" id="snapshotProblem">Make a Snapshot of the Problem Server</h2> <p> One approach, which would be living dangerously, would be to mount the system volume of the problem server on another server, set up <code>chroot</code>, attempt to repair the drive image, remount the repaired drive on the problem server, and reboot the server. I am never that optimistic. Things invariably go wrong. Instead, we will take a snapshot of the problem drive, turn the snapshot into a volume, repair the volume, swap in the repaired volume on the problem system, and reboot that system. </p> <p> It is better to shut down the EC2 instance before making a snapshot, however a snapshot can be taken whenever the server is idling. We will need to shut down the server anyway, so that could be done now, or at the last minute. </p> <p> I made a snapshot with a tag called <code>Name</code> with the value like <code>production 2020-10-25</code> and saved the snapshot id in an environment variable called <code>AWS_PROBLEM_SNAPSHOT_ID</code>: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_PROBLEM_SNAPSHOT_ID "$( aws ec2 create-snapshot --volume-id $AWS_PROBLEM_VOLUME_ID \ --description "production `date '+%Y-%m-%d'`" \ --tag-specifications "ResourceType=snapshot,Tags=[{Key=Created, Value=`date '+%Y-%m-%d'`},{Key=Name, Value=\"Broken do-release-upgrade 20.{04,10\"}]" | \ jq -r .SnapshotId )" <span class='unselectable'>$ </span>aws ec2 wait volume-available --volume-id $AWS_RESCUE_VOLUME_ID <span class='unselectable'>AWS_PROBLEM_SNAPSHOT_ID=snap-0a856be1f58b8a856</span> </pre> <p> Snapshots only take a few minutes to complete. The <code>aws ec2 wait</code> command blocks until the specified operation finishes. </p> </editor-fold snapshotProblem> <editor-fold rescueVolume> <h2 class="numbered" id="rescueVolue">Create Rescue Volume From Snapshot</h2> <p> Once the snapshot process has completed, create a new volume from the snapshot. The default volume type is <a href='https://aws.amazon.com/ebs/features/#Amazon_EBS_volume_types' target='_blank' rel='nofollow'><code>gp2</code></a>. We&rsquo;ll refer to this volume as <code>$AWS_RESCUE_VOLUME_ID</code>. It is important to create the volume in the same availability zone as the problem EC2 instance so that it can easily be attached. This command applies a tag called <code>Name</code>, with the value <code>rescue</code>, for easy identification. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_RESCUE_VOLUME_ID "$( aws ec2 create-volume \ --availability-zone $AWS_AVAILABILITY_ZONE \ --snapshot-id $AWS_PROBLEM_SNAPSHOT_ID \ --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=rescue}]' | \ jq -r .VolumeId )" <span class='unselectable'>AWS_RESCUE_VOLUME_ID=vol-0e20fd22d2dc5a933</span> <span class='unselectable'>$ </span>aws ec2 wait volume-available --volume-id $AWS_RESCUE_VOLUME_ID </pre> </editor-fold rescueVolume> <editor-fold snapshotRescue> <h2 class="numbered" id="spot">Use an EC2 Spot Instance For the Rescue Server</h2> <p> Now that the rescue volume is <code>available</code>, we need to mount it on a server, which I&rsquo;ll call the rescue server. We&rsquo;ll refer to the server where the rescue volume is prepared via its instance id, saved as <code>AWS_EC2_RESCUE_ID</code>. You can either create a new EC2 instance for this purpose, or use an existing EC2 instance. </p> <p> The rescue server does not need to be anything special; a tiny virtual machine of any description will do fine. However, some rescue operations will be much easier if the type of operating system is the same as that on the problem drive. Volumes can be attached to running and stopped server instances. The load on the rescue server will likely be light and short-lived. An EC2 spot instance is ideal, and only costs two cents per hour! The spot instance will likely only be needed for 15 minutes. I specified my VPC id as <code>SubnetId</code>, the security group <code>sg-4cbc6f35</code> and the <code>AvailabilityZone</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_EC2_RESCUE aws ec2 run-instances \ --image-id ami-0dba2cb6798deb6d8 \ --instance-market-options '{ "MarketType": "spot" }' \ --instance-type t2.small \ --key-name rsa-2020-11-03.pub \ --network-interfaces '[ { "DeviceIndex": 0, "Groups": ["sg-4cbc6f35"], "SubnetId": "subnet-49de033f", "DeleteOnTermination": true, "AssociatePublicIpAddress": true } ]' \ --placement '{ "AvailabilityZone": "us-east-1c" }' <span class='unselectable'>{ "Groups": [], "Instances": [ { "AmiLaunchIndex": 0, "ImageId": "ami-0dba2cb6798deb6d8", "InstanceId": "i-012a54aefcd333de9", "InstanceType": "t2.small", "KeyName": "rsa-2020-11-03.pub", "LaunchTime": "2020-11-03T23:19:50.000Z", "Monitoring": { "State": "disabled" }, "Placement": { "AvailabilityZone": "us-east-1c", "GroupName": "", "Tenancy": "default" }, "PrivateDnsName": "ip-10-0-0-210.ec2.internal", "PrivateIpAddress": "10.0.0.210", "ProductCodes": [], "PublicDnsName": "", "State": { "Code": 0, "Name": "pending" }, "StateTransitionReason": "", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "Architecture": "x86_64", "BlockDeviceMappings": [], "ClientToken": "026583fb-c94e-4bca-bdd2-8dcdcaa3aae9", "EbsOptimized": false, "EnaSupport": true, "Hypervisor": "xen", "InstanceLifecycle": "spot", "NetworkInterfaces": [ { "Attachment": { "AttachTime": "2020-11-03T23:19:50.000Z", "AttachmentId": "eni-attach-04feb4d36cf5c6792", "DeleteOnTermination": true, "DeviceIndex": 0, "Status": "attaching" }, "Description": "", "Groups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "Ipv6Addresses": [], "MacAddress": "0a:6d:ba:c5:65:4b", "NetworkInterfaceId": "eni-09ef90920cfb29dd9", "OwnerId": "031372724784", "PrivateIpAddress": "10.0.0.210", "PrivateIpAddresses": [ { "Primary": true, "PrivateIpAddress": "10.0.0.210" } ], "SourceDestCheck": true, "Status": "in-use", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "InterfaceType": "interface" } ], "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SecurityGroups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "SourceDestCheck": true, "SpotInstanceRequestId": "sir-rrs9gm3j", "StateReason": { "Code": "pending", "Message": "pending" }, "VirtualizationType": "hvm", "CpuOptions": { "CoreCount": 1, "ThreadsPerCore": 1 }, "CapacityReservationSpecification": { "CapacityReservationPreference": "open" }, "MetadataOptions": { "State": "pending", "HttpTokens": "optional", "HttpPutResponseHopLimit": 1, "HttpEndpoint": "enabled" } } ], "OwnerId": "031372724784", "ReservationId": "r-0d45e1919e7bad5c9" }</span> </pre> <p> We need to retrieve the IP address of the newly created EC2 spot instance. This instance will disappear (terminate) once it shuts down, so do not reboot it. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'> <span class='unselectable'>$ </span>aws ec2 describe-instances --instance-ids i-012a54aefcd333de9 <span class='unselectable'>{ "Reservations": [ { "Groups": [], "Instances": [ { "AmiLaunchIndex": 0, "ImageId": "ami-0dba2cb6798deb6d8", "InstanceId": "i-012a54aefcd333de9", "InstanceType": "t2.small", "KeyName": "rsa-2020-11-03.pub", "LaunchTime": "2020-11-03T23:19:50.000Z", "Monitoring": { "State": "disabled" }, "Placement": { "AvailabilityZone": "us-east-1c", "GroupName": "", "Tenancy": "default" }, "PrivateDnsName": "ip-10-0-0-210.ec2.internal", "PrivateIpAddress": "10.0.0.210", "ProductCodes": [], "PublicDnsName": "", "PublicIpAddress": "54.242.88.254", "State": { "Code": 16, "Name": "running" }, "StateTransitionReason": "", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "Architecture": "x86_64", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "AttachTime": "2020-11-03T23:19:51.000Z", "DeleteOnTermination": true, "Status": "attached", "VolumeId": "vol-0c44c8c009d1fafda" } } ], "ClientToken": "026583fb-c94e-4bca-bdd2-8dcdcaa3aae9", "EbsOptimized": false, "EnaSupport": true, "Hypervisor": "xen", "InstanceLifecycle": "spot", "NetworkInterfaces": [ { "Association": { "IpOwnerId": "amazon", "PublicDnsName": "", "PublicIp": "54.242.88.254" }, "Attachment": { "AttachTime": "2020-11-03T23:19:50.000Z", "AttachmentId": "eni-attach-04feb4d36cf5c6792", "DeleteOnTermination": true, "DeviceIndex": 0, "Status": "attached" }, "Description": "", "Groups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "Ipv6Addresses": [], "MacAddress": "0a:6d:ba:c5:65:4b", "NetworkInterfaceId": "eni-09ef90920cfb29dd9", "OwnerId": "031372724784", "PrivateIpAddress": "10.0.0.210", "PrivateIpAddresses": [ { "Association": { "IpOwnerId": "amazon", "PublicDnsName": "", "PublicIp": "54.242.88.254" }, "Primary": true, "PrivateIpAddress": "10.0.0.210" } ], "SourceDestCheck": true, "Status": "in-use", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "InterfaceType": "interface" } ], "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SecurityGroups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "SourceDestCheck": true, "SpotInstanceRequestId": "sir-rrs9gm3j", "VirtualizationType": "hvm", "CpuOptions": { "CoreCount": 1, "ThreadsPerCore": 1 }, "CapacityReservationSpecification": { "CapacityReservationPreference": "open" }, "HibernationOptions": { "Configured": false }, "MetadataOptions": { "State": "applied", "HttpTokens": "optional", "HttpPutResponseHopLimit": 1, "HttpEndpoint": "enabled" } } ], "OwnerId": "031372724784", "ReservationId": "r-0d45e1919e7bad5c9" } ] }</span></pre> <!-- <p> It is possible that the work necessary to rescue the problem disk image might make changes to the rescue system. The rescue system should therefore have a snapshot taken before going any further. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_RESCUE_SNAPSHOT_ID "$( aws ec2 create-snapshot --volume-id $AWS_RESCUE_VOLUME_ID %} \ --description "production `date '+%Y-%m-%d'`" \ --tag-specifications "ResourceType=snapshot,Tags=[{Key=Created, Value=`date '+%Y-%m-%d'`},{Key=Name, Value=\"Broken do-release-upgrade 20.{04,10\"}]" | \ jq -r .SnapshotId )" <span class='unselectable'>AWS_RESCUE_SNAPSHOT_ID=snap-0a856be1f58b8359a</span> <span class='unselectable'>$ </span>aws ec2 wait snapshot-completed --snapshot-ids $AWS_RESCUE_SNAPSHOT_ID </pre> --> </editor-fold snapshotRescue> <editor-fold mount> <h2 class="numbered" id="mount">Mount the Rescue Volume On the Rescue Server</h2> <!--<p> We need to select a device name for the rescue disk. The name depends on what names are already in use on the rescue server. After logging into the rescue server, I ran the <code>lsblk</code> Linux command to see the available disk devices and their mount points. </p> <pre data-lt-active="false"> <span class="unselectable">$ </span>lsblk <span class="unselectable">NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop1 7:1 0 53.1M 1 loop /snap/lxd/10984 loop2 7:2 0 88.4M 1 loop /snap/core/7169 loop3 7:3 0 97.8M 1 loop /snap/core/10185 loop4 7:4 0 53.1M 1 loop /snap/lxd/11348 xvda 202:0 0 8G 0 disk └─xvda1 202:1 0 8G 0 part /</span> </pre> <p> The lsblk output does not show full device paths, instead, the <code>/dev/</code> prefix is omitted. With that in mind we can see that the only <code>disk</code> device on the rescue server is <code>/dev/xvda</code>, and its only partition called <code>/dev/xvda1</code> is mounted on the root directory. Because Linux drives are normally named sequentially, we should name the rescue disk <code>/dev/xvdb</code>. Let&rsquo;s define an environment variable called <code>AWS_RESCUE_DRIVE</code> to memorialize that decision. </p> <pre data-lt-active="false"> <span class="unselectable">$ </span>AWS_RESCUE_DRIVE=/dev/xvdb </pre> --> <p> The <code>aws ec2 attach-volume</code> command will attach the rescue volume to the rescue server. It automatically selects an appropriate device name for the rescue volume, which in the following example is <code>/dev/xvdb</code>: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_ATTACH_VOLUME "$( aws ec2 attach-volume \ --device $AWS_RESCUE_DRIVE \ --instance-id $AWS_EC2_RESCUE_ID \ --volume-id $AWS_RESCUE_VOLUME_ID )" <span class='unselectable'>AWS_ATTACH_VOLUME={ "AttachTime": "2020-10-26T14:34:55.222Z", "InstanceId": "i-d3b03954", "VolumeId": "vol-0e20fd22d2dc5a933", "State": "attaching", "Device": "/dev/xvdb" }</span> <span class='unselectable'>$ </span>aws ec2 wait volume-in-use --volume-id $AWS_RESCUE_VOLUME_ID </pre> <p> We need to use the rescue volume&rsquo;s device name later, so we&rsquo;ll save it in <code>AWS_RESCUE_DRIVE</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_RESCUE_DRIVE "$( jq -r .Device <<< $AWS_ATTACH_VOLUME )" <span class='unselectable'>AWS_RESCUE_DRIVE=/dev/xvdb</span> </pre> <p> The details of the mounted rescue drive are provided by <code>fdisk -l</code>: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo fdisk -l | sed -n -e '/xvdb/,$p' <span class='unselectable'>Disk /dev/xvdb: 12 GiB, 12884901888 bytes, 25165824 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x00000000 Device Boot Start End Sectors Size Id Type /dev/xvdb1 * 16065 25165790 25149726 12G 83 Linux</span> </pre> <p> Now it is time to mount the rescue drive on the rescue server. Ubuntu has a directory called <code>/mnt</code> whose purpose is to act as a mount point: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo mount /dev/xvdb1 /mnt </pre> <p> Let&rsquo;s confirm that the drive is mounted: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>df -h | grep '^/dev/' | grep -v '^/dev/loop' <span class='unselectable'>/dev/xvda1 7.8G 6.3G 1.1G 86% / /dev/xvdb1 12G 9.0G 2.2G 82% /mnt</span> </pre> <p> The last line shows that this drive is mounted on <code>/mnt</code> and it is 82% full. </p> </editor-fold mount> <editor-fold chroot> <h2 class="numbered" id="chroot">Set Up a <span class='code'>chroot</span> to Establish an Environment for Making Repairs</h2> <p> We need to mount some more file systems before we perform the <code>chroot</code>. The following mounts the rescue server&rsquo;s <code>/dev</code>, <code>/dev/shm</code>, <code>/sys</code>, and <code>/run</code> to the same paths within the rescue volume. Because programs like <code>do-release-upgrade</code> need a <code>tty</code>, I also mount <code>devtps</code> and <code>proc</code>. These mounts only last until the next server reboot. After all the mounts the <code>chroot</code> is issued. </p> <p class="warning"> <b>Warning</b> - mounting <code>/run</code> and then updating the system on the rescue disk from within a chroot may change the host system&rsquo;s <code>/run</code> contents; if the package managers (<code>apt</code> and <code>dpkg</code>) get out of sync with the actual state on the host system you won&rsquo;t be able to update the host system until you restore the host system&rsquo;s image from the snapshot that we made <a href="#snapshotRescue">earlier</a>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo mount -o bind /dev /mnt/dev <span class='unselectable'>$ </span>sudo mount -o bind /dev/shm /mnt/dev/shm <span class='unselectable'>$ </span>sudo mount -o bind /sys /mnt/sys <span class='unselectable'>$ </span>sudo mount -o bind /run /mnt/run <span class='unselectable'>$ </span>sudo mount -t proc proc /mnt/proc <span class='unselectable'>$ </span>sudo mount -t devpts devpts /mnt/dev/pts <span class='unselectable'>$ </span>sudo chroot /mnt <span class='unselectable'>root@ip-10-0-0-189:/#</span> </pre> <p> Notice how the prompt changed after the <code>chroot</code>. That is your clue that it is active. </p> <!-- <p> I edited <code>/etc/hosts</code> in the <code>chroot</code> to add the name of the system to the entry for <code>localhost</code> (<code>127.0.0.1</code>): </p> <pre data-lt-active="false"> 127.0.0.1 localhost ip-10-0-0-189 </pre> --> </editor-fold chroot> <editor-fold fix> <h2 class="numbered" id="fix">Correct the Problem</h2> <p> This step depends on whatever is wrong. I won&rsquo;t bore you with the problem I had. </p> </editor-fold fix> <editor-fold unmountRescue> <h2 class="numbered" id="unmount">Unmount the New Volume</h2> <p> Exit the <code>chroot</code> and unmount the rescue volume from the rescue server. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'># </span>exit <span class='unselectable'>$ </span>sudo umount /mnt/dev <span class='unselectable'>$ </span>sudo umount /mnt/dev/shm <span class='unselectable'>$ </span>sudo umount /mnt/sys <span class='unselectable'>$ </span>sudo umount /mnt/run <span class='unselectable'>$ </span>sudo umount /mnt/proc <span class='unselectable'>$ </span>sudo umount /mnt/dev/pts <span class='unselectable'>$ </span>sudo umount /mnt </pre> <p> Detach the rescue volume from the rescue server. This can be done from any machine that is configured with <code>aws cli</code> for use with your account credentials. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 detach-volume --volume-id $AWS_RESCUE_VOLUME_ID <span class='unselectable'>$ </span>aws ec2 wait volume-available --volume-id $AWS_RESCUE_VOLUME_ID </pre> </editor-fold unmountRescue> <editor-fold unmountProblem> <h2 class="numbered" id="unmount">Unmount the Problem Volume</h2> <p> The problem server must be shut down for this to work. Detach the problem volume from the problem server. This can be done from any machine that is configured with <code>aws cli</code> for use with your account credentials. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 stop-instances --instance-id $AWS_PROBLEM_INSTANCE_ID <span class='unselectable'>$ </span>aws ec2 wait instance-stopped --instance-ids $AWS_PROBLEM_INSTANCE_ID <span class='unselectable'>$ </span>aws ec2 detach-volume --volume-id $AWS_PROBLEM_VOLUME_ID <span class='unselectable'>$ </span>aws ec2 wait volume-available --volume-id $AWS_PROBLEM_VOLUME_ID </pre> </editor-fold unmountProblem> <editor-fold replace> <h2 class="numbered" id="replace">Replace the Problem Volume</h2> <p> Now it is time to replace the problem volume containing the problem boot drive on the problem system with the newly created volume. AWS EC2 always refers to boot drives as <code>/dev/sda1</code>, even when the device has a different name, such as <code>/dev/xvdb1</code>. </p> <h3 id="replaceSystemVolume"><span>replaceSystemVolume</span> Bash function</h3> <p> This Bash function detaches the volume containing the current boot drive of an EC2 instance and replaces it with another volume. If the EC2 instance is running then it is first stopped. </p> <p>Paste the above into</p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>replaceSystemVolume "$AWS_PROBLEM_INSTANCE_ID" "$AWS_RESCUE_VOLUME_ID"</pre> Preview 2 instance id is <code>AWS_EC2_RESCUE_ID</code>. Replace rescue volume on preview with preview's original volume: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>replaceSystemVolume "$AWS_EC2_RESCUE_ID" "$AWS_PREVIEW_VOLUME_ID"</pre> </editor-fold replace> <editor-fold boot> <h2 class="numbered" id="boot">Boot the problem system</h2> <p> Boot the problem system and verify the problem is solved. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 start-instances --instance-ids $AWS_PROBLEM_INSTANCE_ID <span class='unselectable'>$ </span>aws ec2 wait instance-started --instance-ids $AWS_PROBLEM_INSTANCE_ID </pre> </editor-fold boot> <editor-fold acknowledgements> <h2 id="acknowledgements">Acknowledgements</h2> <p> This article was inspired by <a href='https://www.rootusers.com/how-to-repair-an-aws-ec2-instance-without-console' target='_blank'>this excellent article</a>, which uses the AWS web console to achieve similar results. </p> </editor-fold acknowledgements> dotenv For Command Line Interpreters 2020-10-24T00:00:00-04:00 https://mslinn.github.io/blog/2020/10/24/working-with-command-line-interps <p> I&rsquo;ve built a shell that allows me to be extremely productive while writing technical documentation. The generated documentation displays code examples quickly and easily. This article discusses the shell, <code>dotenv-python</code>. </p> <h2 id="JSON">JSON Output</h2> <p> Many command line programs return JSON. Examples include: </p> <ol> <li><a href='https://docs.ansible.com/ansible/latest/collections/ansible/netcommon/cli_command_module.html' target='_blank' rel='nofollow'>ansible.netcommon.cli_command</a></li> <li><a href='https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html' target='_blank' rel='nofollow'>Amazon Web Services CLI</a></li> <li><a href='https://developers.cloudflare.com/workers/' target='_blank' rel='nofollow'>Cloudflare Workers CLI</a></li> <li><a href='https://docs.databricks.com/dev-tools/cli/index.html' target='_blank' rel='nofollow'>Databricks CLI</a></li> <li><a href='https://www.digitalocean.com/docs/apis-clis/doctl/' target='_blank' rel='nofollow'>Digital Ocean CLI</a></li> <li><a href='https://docs.docker.com/engine/reference/commandline/docker/' target='_blank' rel='nofollow'>Docker CLI</a></li> <li><a href='https://cloud.google.com/sdk/gcloud' target='_blank' rel='nofollow'>Google Cloud CLI</a></li> <li><a href='https://kubernetes.io/docs/reference/kubectl/' target='_blank' rel='nofollow'>Kubernetes kubectl CLI</a></li> <li><a href='https://www.linode.com/docs/guides/linode-cli/' target='_blank' rel='nofollow'>Linode CLI</a></li> <li><a href='https://docs.microsoft.com/en-us/cli/azure/' target='_blank' rel='nofollow'>Microsoft Azure CLI</a></li> <li><a href='https://docs.mulesoft.com/runtime-manager/anypoint-platform-cli-commands' target='_blank' rel='nofollow'>Mulesoft Anypoint Platform CLI</a></li> <li><a href='https://developer.rackspace.com/docs/rack-cli/' target='_blank' rel='nofollow'>Rackspace CLI</a></li> <li><a href='https://developer.salesforce.com/docs/atlas.en-us.sfdx_cli_reference.meta/sfdx_cli_reference/cli_reference.htm' target='_blank' rel='nofollow'>Salesforce CLI</a></li> <li><a href='https://github.com/scaleway/scaleway-cli' target='_blank' rel='nofollow'>Scaleway CLI</a></li> <li><a href='https://www.twilio.com/docs/twilio-cli/quickstart' target='_blank' rel='nofollow'>Twilio CLI</a></li> <li><a href='https://marketplace.zoom.us/docs/guides/zoom-rooms/zoom-rooms-cli' target='_blank' rel='nofollow'>Zoom Rooms</a></li> </ol> <p> I work with JSON for two reasons: </p> <ol> <li> To interact with a server, such as those listed above. This is repetitive, tedious and boring. </li> <li> To write documentation about how to work with servers. This is repetitive, tedious and boring. </li> </ol> <h3 id="more">Easy to Manage, Not Boring, Tedious or Repetitve</h3> <p> It is difficult to keep track of CLI output and manage it over time. To keep track of returned name/value pairs, and to persist them, I wrote a Python library called <code>mem</code>. This article shows how <code>mem</code> uses hidden files called <code>.env</code> in concert with environment variables to store and retrieve names and values. Files called <code>.env</code> are used in <i>Twelve-factor webapps</i>. </p> <h2 id='dotenv'>Twelve-Factor Apps and <span class='code'>dotenv-python</span></h2> <p> <a href='https://12factor.net/config' target='_blank' rel='nofollow'>Storing configuration in the environment</a> is one of the tenets of <a href='https://12factor.net' target='_blank' rel='nofollow'>twelve-factor web applications</a>. Anything that is likely to change between deployment environments, such as resource handles for databases or credentials for external services, should be extracted from the code into environment variables. However, it is not always practical to set environment variables on development machines or continuous integration servers where multiple projects are run. Instead, <code>dotenv-python</code> loads variables from an <code>.env</code> file into the environment when the application starts. </p> <h2 id="theCode">The GitHub Repository for <span class='code'>mem</span></h2> <p>The code lives in a <a href='https://github.com/mslinn/mem' target='_blank' rel='nofollow'>GitHub repository</a>.</p> <h2 id="installation">Installation</h2> <p> TODO write me. </p> <h2 id="usage">About <span class="code">mem</span></h2> <p> <code>mem</code> saves key/value pairs in a hidden file in the current directory called <code>.env</code>. This is useful when working with commands that generate JSON. Each of your projects can have a different <code>.env</code> file, which means that the key/value pairs defined within them are specific to the projects that they are located with. </p> <p> <code>mem</code> works two ways: </p> <ol> <li>By receiving the name of a new environment variable and its value, specified separately.</li> <li>By receiving the name of an existing environment variable that has been exported.</li> </ol> <p> Once a key/value pair is defined in <code>.env</code>, it becomes available after reading the file back in using the <code>source</code> command. The alias defined above reads the file back in automatically. The alias display the new or modified name/value pair that it just defined. </p> <h3 id="clipboard">System Clipboard Support</h3> <p> <code>mem</code> also works with the system clipboard. This greatly speeds up the process of writing documentation generated by <a href='https://jekyllrb.com/' target='_blank' rel='nofollow'>Jekyll</a>, which is used to create this website. </p> <ol> <li> It copies a block of HTML code to the system clipboard each time it runs. The HTML invokes Jekyll plugins I wrote called <a href='/blog/2020/10/03/jekyll-plugins.html#pre_noselect' target='_blank' rel='nofollow'><code>pre</code> and <code>noselect</code></a>. For example, if you run <code>mem</code> like this: <pre>mem NAME VALUE</pre> Then after the <code>mem</code> script completes the clipboard might contain something like this: <pre>{% pre copyButton %}{% noselect %}mem NAME "$( COMPUTATION )" {% noselect NAME=VALUE%} {% endpre %} </pre> Which renders as: <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem NAME "$( COMPUTATION )" <span class='unselectable'>NAME=VALUE</span></pre> </li> <li> If the system clipboard is not empty when <code>mem</code> runs then the <code>COMPUTATION</code> placeholder shown above will be set to the clipboard contents. Otherwise the value of the <code>COMPUTATION</code> placeholder will be <code>fill_in_the_computation</code>. </li> </ol> <h2 id="usage">Usage</h2> <p> Here is the help message for <code>mem</code>: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>$mem/src/main.py -h <span class='unselectable'>usage: main.py [-h] [-a DIALOG_FILE_NAME] [-b] [-c] [-d] [-e DOTENV_FILE_NAME] [-f {jekyll,html5,markdown,plain}] [-H] [-p] [-u {qt,cli}] [-v] [variable_name] [command_or_value] Persists environment variables (names and values) to a file. By default the file is called .env and is found in the current directory. The original command can also be captured when run as a shell. Environment variable substitution and subshells can be used for constructing values. mem copies a block of markup code to the system clipboard each time it runs, to speed up the process of writing documentation. Supported formats are Jekyll, Markdown, HTML5 and plain text. mem can either present a QT-based gui or a command-line interface. The CLI mode provides two sub-modes: 1) A name and value are provided on the command line that invokes mem. SYNTAX: mem [OPTIONS] environment_variable_name command_or_value This mode is automatically entered when environment_variable_name and command_or_value are both provided. The -u/--ui switch is ignored. EXAMPLE: $ mem DIR pwd DIR=/mnt/_/work/mem # The value of CPU is available to subsequent mem invocations as "$DIR" $ mem FILES ls $DIR FILES='bin/ data/ docs/ LICENSE __pycache__/ README.md src/ tests/' 2) No parameters are provided on the command line that invokes mem. In this mode mem loops continually, prompting for name / value pairs. SYNTAX: mem [OPTIONS] EXAMPLE: $ mem mem: name> DIR mem: new value for DIR> /etc DIR='/etc' # The value of DIR is available to subsequent mem invocations as "$DIR" $ source .env $ echo "$DIR" /etc mem: name> # Press Enter, ^D or ^C to exit. For more information about mem, see https://mslinn.com/blog/2020/10/24/working-with-command-line-interps.html positional arguments: variable_name Optional name of environment variable command_or_value Optional command or value of environment variable optional arguments: -H, --heading Prompt for heading. -a, --append DIALOG_FILE_NAME Append the cli dialog history to specified file -b, --clipboard Copy formatted example code to clipboard after every definition. -c, --comment Prompt for comment (paragraphs). -d, --debug Enable debug output -e, --dotenv DOTENV_FILE_NAME Dotenv file to read/write keys & values. -f, --format {jekyll,html5,markdown,plain} Format to save dialog in. Markdown uses GitHub syntax. Jekyll format requires Mike Slinn's plugins. -h, --help show this help message and exit -p, --append_previous Continue appending to the most recent markup file. -u, --ui {qt,cli} User interface style. -v, --version show program's version number and exit Copyright 2020 Michael Slinn. All rights reserved.</span></pre> <p> A very common usage pattern is to provide a computed value computed within a subshell, like this <code>"$()"</code>. For example, define a new environment variable called TODAY, with the value computed by <code>"$( date )"</code>: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem TODAY "$( date )" <span class='unselectable'>TODAY='Tue Oct 27 10:18:18 EDT 2020'</span></pre> <h2 id="json">Loading Name/Value Pairs</h2> <p> To restore the names and values in this shell, or another shell, make the directory current containing the <code>.env</code> files that define the name/value pairs for a project. Type: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>source .env</pre> <p> This means you can walk away and resume the session anywhere the <code>.env</code> file is available without losing context. </p> <h2 id="json">Working with JSON</h2> <p> I frequently use <a href='https://stedolan.github.io/jq/' target='_blank' rel='nofollow'>jq</a> for parsing JSON in the bash shell. Install it like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo apt install jq</pre> <h3 id="literalStrings">Single Quote Literal Strings</h3> <p> This example saves some JSON in <code>.env</code> and extracts a value from it later. JSON surrounds names and string values with double quotes. In the following example single quotes surround the value, which is specified as a literal string. This is so the JSON literal string will be stored as a single string in <code>.env</code>. If double quotes had been used to enclose the literal string of JSON then the double quotes inside the JSON literal string would have needed to be escaped. </p> <p> The <code><<<</code> in the following command history pipes the string defined to the right of the <code><<<</code> into the <code>jq</code> process as <code>stdin</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem JSON '{"name1": "value1", "name2": "value2"}' <span class='unselectable'>JSON='{"name1": "value1", "name2": "value2"}'</span> <span class='unselectable'>$ </span>jq -r .name1 <<< $JSON <span class='unselectable'>value1</span> <span class='unselectable'>$ </span>jq -r .name2 <<< $JSON <span class='unselectable'>value2</span> </pre> <h3 id="subshell">Double-Quote Subshell Output</h3> <p> Bash subshells are useful for computing values. They are defined by writing a dollar sign followed by parentheses, like this: <code>$()</code> If there is a possibility that the value might have whitespace in it, or special characters, the subshell should be enclosed in double quotes, like this: <code>"$()"</code>. </p> <p> The double quotes in the following code example ensure that the JSON returned by <code>curl</code> is stored as a single string in <code>.env</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem JOKE "$( curl -s http://api.icndb.com/jokes/random )" <span class='unselectable'>JOKE='{ "type": "success", "value": { "id": 327, "joke": "Chuck Norris once ordered a steak in a restaurant. The steak did what it was told.", "categories": [] } }'</span> <span class='unselectable'>$ </span>jq -r '.value.joke' <<< $JOKE <span class='unselectable'>Chuck Norris once ordered a steak in a restaurant. The steak did what it was told.</span> </pre> <p> You could also use backticks (<code>``</code>) instead of <code>"$()"</code>: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem JOKE `curl -s http://api.icndb.com/jokes/random` <span class='unselectable'>JOKE='{ "type": "success", "value": { "id": 327, "joke": "Chuck Norris once ordered a steak in a restaurant. The steak did what it was told.", "categories": [] } }'</span> </pre> <!-- < h2 id="yaml">Working with YAML</h2> <p> I frequently use <a href='https://github.com/mikefarah/yq' target='_blank' rel='nofollow'>yq</a> for parsing YAML in the bash shell. Install it on Ubuntu like this: </p> <pre data-lt-active="false"> <span class="unselectable">$ </span>sudo add-apt-repository ppa:rmescandon/yq <span class="unselectable">$ </span>sudo apt update <span class="unselectable">$ </span>yes | sudo apt install yq </pre> --> Working With EC2 Spot Instances From AWS CLI 2020-10-24T00:00:00-04:00 https://mslinn.github.io/blog/2020/10/24/ec2-spot-instances-cli <editor-fold Intro> <p> AWS EC2 <code>T2.medium</code> spot instances <a href='https://aws.amazon.com/ec2/spot/pricing/' target='_blank' rel='nofollow'>cost less than 2 cents per hour</a> for Linux and can be created very easily from the command line. They self-destruct once shut down. These powerful virtual machines can do an incredible amount of work in an hour for less than 2 cents! </p> <p> Permanent storage can either be on S3 or an EBS volume, which can easily be mounted. This article shows how all this can be done via the command line. I also provide an <a href="#createEc2Spot">interactive bash script</a> for automating this process. </p> <p> Once again this article uses <code>AWS CLI</code>, the 12-factor webapp utility I wrote called <a href='/blog/2020/10/23/working-with-command-line-interps.html'><code>mem</code></a> and <code>jq</code>. </p> </editor-fold Intro> <editor-fold nameKeys> <h2 class="numbered" id="nameKeys">Create and Import a New Keypair</h2> <p> I want to create a new temporary ssh keypair that will just be used for this spot instance. The name of the new key pair will be of the form <code>~/.ssh/rsa-YYYY-MM-DD-mm-ss</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_KEY_PAIR_NAME rsa-$( date '+%Y-%m-%d-%M-%S' ) <span class='unselectable'>AWS_KEY_PAIR_NAME=rsa-2020-11-04-43-54</span> <span class='unselectable'>$ </span>mem AWS_KEY_PAIR_FILE ~/.ssh/$AWS_KEY_PAIR_NAME <span class='unselectable'>AWS_KEY_PAIR_FILE=~/.ssh/rsa-2020-11-04-43-54</span> </pre> <p> Now we can make the keypair. AWS EC2 does not accept keys longer than 2048 bits. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ssh-keygen -b 2048 -f "$AWS_KEY_PAIR_FILE" -P "" -N "" -t rsa <span class='unselectable'>Generating public/private rsa key pair. Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-43-54 Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-43-54.pub The key fingerprint is: SHA256:bQScX0UMn0xGDorxSvElMZzwMyyk7hgs2FNbshBNenA mslinn@mslinn.com The key's randomart image is: +---[RSA 4096]----+ | ooE .*++o+** | | =. ooXo=.B.. | | o + o +.X. = | | o = * . =.o | |. + = . S o | | o + . | | . . | | | | | +----[SHA256]-----+</span> </pre> <p> The new public key will be stored in <code>~/.ssh/2020-11-04-43-54.pub</code> and the new private key will be stored in <code>~/.ssh/2020-11-04-43-54</code>. </p> <p> Now set the permissions for the key. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>chmod 400 $AWS_KEY_PAIR_FILE </pre> <p> Now we can import the key pair into AWS: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 import-key-pair \ --key-name $AWS_KEY_PAIR_NAME \ --public-key-material fileb://${AWS_KEY_PAIR_FILE}.pub <span class='unselectable'>{ "KeyFingerprint": "c7:76:90:53:17:d0:fc:ba:45:dd:93:d2:93:03:c2:19", "KeyName": "2020-11-04-43-54", "KeyPairId": "key-092a2306ec3f4aff6" }</span> </pre> </editor-fold nameKeys> <editor-fold AMI> <h2 class="numbered" id="ami">Select an AMI</h2> <p> New <a href='https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html' target='_blank' rel='nofollow'>AMIs</a> become available every day. You probably want your EC2 spot instance to be created from the most recent AMI that matches your needs. For most of my work I want an Ubuntu 64-bit Intel/AMD server distribution. <a href='https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/finding-an-ami.html' target='_blank' rel='nofollow'>AWS documentation</a> is helpful and gives us a head start in automating the AMI selection. </p> <p> The following incantation sets an environment variable called <code>AWS_AMI</code> to the details in JSON syntax of the AMI for the most recent 64-bit Ubuntu server release for Intel/AMD architecture. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_AMI "$( aws ec2 describe-images \ --owners 099720109477 \ --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-????????-????????-amd64-server-????????" \ "Name=state,Values=available" \ --query "reverse(sort_by(Images, &CreationDate))[:1]" | \ jq -r '.[0]' )" <span class='unselectable'>AWS_AMI={ "Architecture": "x86_64", "CreationDate": "2020-10-30T14:07:42.000Z", "ImageId": "ami-0c71ec98278087e60", "ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030", "ImageType": "machine", "Public": true, "OwnerId": "099720109477", "PlatformDetails": "Linux/UNIX", "UsageOperation": "RunInstances", "State": "available", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "DeleteOnTermination": true, "SnapshotId": "snap-00bf581086dd686e5", "VolumeSize": 8, "VolumeType": "gp2", "Encrypted": false } }, { "DeviceName": "/dev/sdb", "VirtualName": "ephemeral0" }, { "DeviceName": "/dev/sdc", "VirtualName": "ephemeral1" } ], "Description": "Canonical, Ubuntu, 20.10, amd64 groovy image build on 2020-10-30", "EnaSupport": true, "Hypervisor": "xen", "Name": "ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030", "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SriovNetSupport": "simple", "VirtualizationType": "hvm" }</span> </pre> <p> Now let's extract the ID of the AMI image and save it as <code>AWS_AMI_ID</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_AMI_ID "$( jq -r '.[0].ImageId' <<< "$AWS_AMI" )" <span class='unselectable'>AWS_AMI_ID=ami-0c71ec98278087e60</span> </pre> </editor-fold AMI> <editor-fold Spot> <h2 class="numbered" id="create">Create an EC2 Spot Instance</h2> <p> For my work I often want my spot instance to be created in the same VPC subnet as my other resources, with the same security group. That is why the following environment variables are defined for the <code>Groups</code> and <code>SubnetId</code> values within the <code>network-interfaces</code> option, as well as the AWS region. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_GROUP sg-4cbc6f35 <span class='unselectable'>AWS_GROUP=sg-4cbc6f35</span> <span class='unselectable'>$ </span>mem AWS_SUBNET subnet-49de033f <span class='unselectable'>AWS_SUBNET=subnet-49de033f</span> <span class='unselectable'>$ </span>mem AWS_ZONE us-east-1c <span class='unselectable'>AWS_ZONE=us-east-1c</span> <span class='unselectable'>$ </span>mem AWS_EC2_TYPE t2.medium <span class='unselectable'>AWS_EC2_TYPE t2.medium</span> </pre> <p> The following creates an AWS EC2 spot instance with a public IP address and runs it. Details about the newly created spot instance are stored as JSON in <code>AWS_SPOT_INSTANCE</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_SPOT_INSTANCE "$( aws ec2 run-instances \ --image-id $AWS_AMI_ID \ --instance-market-options '{ "MarketType": "spot" }' \ --instance-type $AWS_EC2_TYPE \ --key-name $AWS_KEY_PAIR_NAME \ --network-interfaces "[ { \"DeviceIndex\": 0, \"Groups\": [\"$AWS_GROUP\"], \"SubnetId\": \"$AWS_SUBNET\", \"DeleteOnTermination\": true, \"AssociatePublicIpAddress\": true } ]" \ --placement "{ \"AvailabilityZone\": \"$AWS_ZONE\" }" | \ jq -r .Instances[0] )" <span class='unselectable'>AWS_SPOT_INSTANCE={ "AmiLaunchIndex": 0, "ImageId": "ami-0dba2cb6798deb6d8", "InstanceId": "i-012a54aefcd333de9", "InstanceType": "t2.small", "KeyName": "rsa-2020-11-03.pub", "LaunchTime": "2020-11-03T23:19:50.000Z", "Monitoring": { "State": "disabled" }, "Placement": { "AvailabilityZone": "us-east-1c", "GroupName": "", "Tenancy": "default" }, "PrivateDnsName": "ip-10-0-0-210.ec2.internal", "PrivateIpAddress": "10.0.0.210", "ProductCodes": [], "PublicDnsName": "", "State": { "Code": 0, "Name": "pending" }, "StateTransitionReason": "", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "Architecture": "x86_64", "BlockDeviceMappings": [], "ClientToken": "026583fb-c94e-4bca-bdd2-8dcdcaa3aae9", "EbsOptimized": false, "EnaSupport": true, "Hypervisor": "xen", "InstanceLifecycle": "spot", "NetworkInterfaces": [ { "Attachment": { "AttachTime": "2020-11-03T23:19:50.000Z", "AttachmentId": "eni-attach-04feb4d36cf5c6792", "DeleteOnTermination": true, "DeviceIndex": 0, "Status": "attaching" }, "Description": "", "Groups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "Ipv6Addresses": [], "MacAddress": "0a:6d:ba:c5:65:4b", "NetworkInterfaceId": "eni-09ef90920cfb29dd9", "OwnerId": "031372724784", "PrivateIpAddress": "10.0.0.210", "PrivateIpAddresses": [ { "Primary": true, "PrivateIpAddress": "10.0.0.210" } ], "SourceDestCheck": true, "Status": "in-use", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "InterfaceType": "interface" } ], "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SecurityGroups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "SourceDestCheck": true, "SpotInstanceRequestId": "sir-rrs9gm3j", "StateReason": { "Code": "pending", "Message": "pending" }, "VirtualizationType": "hvm", "CpuOptions": { "CoreCount": 1, "ThreadsPerCore": 1 }, "CapacityReservationSpecification": { "CapacityReservationPreference": "open" }, "MetadataOptions": { "State": "pending", "HttpTokens": "optional", "HttpPutResponseHopLimit": 1, "HttpEndpoint": "enabled" } }</span> </pre> <p> Now extract the EC2 spot instance id and save it in <code>AWS_SPOT_ID</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_SPOT_ID "$( jq -r .InstanceId <<< "$AWS_SPOT_INSTANCE" )" <span class='unselectable'>i-012a54aefcd333de9</span> </pre> <p> Wait for the instance to start. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 wait instance-running --instance-ids $AWS_SPOT_ID</pre> </editor-fold Spot> <editor-fold Connect> <h2 class="numbered" id="connect">Connect to the Spot Instance</h2> <p> In order to <code>ssh</code> into the spot instance we first need to discover its IP address, which is saved in <code>AWS_SPOT_IP</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>mem AWS_SPOT_IP "$( aws ec2 describe-instances \ --instance-ids $AWS_SPOT_ID | \ jq -r .Reservations[0].Instances[0].PublicIpAddress )" <span class='unselectable'>54.242.88.254</span> </pre> <p> Now we can connect to the spot instance via <code>ssh</code>. The default userid for Ubuntu is <code>ubuntu</code>. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>ssh ubuntu@$AWS_SPOT_IP </pre> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span># Do your work on the spot instance now</pre> <p>We'll disconnect and clean up next.</p> </editor-fold Connect> <editor-fold Disconnect> <h2 class="numbered" id="connect">Disconnect from the Spot Instance and Clean Up</h2> <p> Once the spot instance stops it is automatically terminated. The instance will survive a <code>reboot</code>, but not a <code>halt</code>. From a prompt on the spot instance, type: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>sudo halt </pre> <p> Back in the shell that launched the spot instance, wait for the spot instance to stop before cleaning up. </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 wait instance-stopped --instance-ids $AWS_SPOT_ID</pre> <p> Delete the temporary <code>ssh</code> keypair we created. Copies exist on AWS and the local machine; we need to remove all of them, like this: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>aws ec2 delete-key-pair --key-name $AWS_KEY_PAIR_NAME <span class='unselectable'>$ </span>rm $AWS_KEY_PAIR_FILE $AWS_KEY_PAIR_FILE.pub </pre> </editor-fold Disconnect> <editor-fold Script> <h2 class="numbered" id="createEc2Spot">Bash Script <span class="code">createEc2Spot</span></h2> <h3 class="numbered" id="createEc2Spot_code">Source Code</h3> <p> This script does everything discussed above, plus it prompts the user with default values for parameters unique to each invocation. Click on the name of the script and save it this script to a directory on your <code>PATH</code>. </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/blog/factor12/createEc2Spot" download="createEc2Spot" title="Click on the file name to download the file">createEc2Spot</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id4c0fbdceaf4d"><button class='copyBtn' data-clipboard-target='#id4c0fbdceaf4d' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash set -e function readWithDefault &#123; >&2 printf "\n$1: " read -e -i "$2" VALUE echo "$VALUE" &#125; echo "Please answer a few questions so the AWS EC2 spot instance can be created." mem AWS_GROUP "$( readWithDefault "AWS security group" sg-4cbc6f35 )" mem AWS_SUBNET "$( readWithDefault "EC2 subnet" subnet-49de033f )" mem AWS_ZONE "$( readWithDefault "AWS availability zone" us-east-1c )" mem AWS_EC2_TYPE "$( readWithDefault "EC2 machine type" t2.medium )" mem AWS_KEY_PAIR_NAME rsa-$( date '+%Y-%m-%d-%M-%S' ) mem AWS_KEY_PAIR_FILE ~/.ssh/$AWS_KEY_PAIR_NAME ssh-keygen -b 4096 -f "$AWS_KEY_PAIR_FILE" -P "" -N "" -t rsa chmod 400 $AWS_KEY_PAIR_FILE aws ec2 import-key-pair \ --key-name $AWS_KEY_PAIR_NAME \ --public-key-material fileb://$AWS_KEY_PAIR_FILE echo "Searching for the latest 64-bit Intel/AMD Ubuntu AMI." mem AWS_AMI "$( aws ec2 describe-images \ --owners 099720109477 \ --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-????????-????????-amd64-server-????????" \ "Name=state,Values=available" \ --query "reverse(sort_by(Images, &CreationDate))[:1]" | \ jq -r '.[0]' )" source .env echo "Obtaining the AMI image ID." mem -d AWS_AMI_ID "$( jq -r '.ImageId' &lt;&lt;&lt; "$AWS_AMI" )" source .env echo "Creating the EC2 spot instance." mem AWS_SPOT_INSTANCE "$( aws ec2 run-instances \ --image-id $AWS_AMI_ID \ --instance-market-options '&#123; "MarketType": "spot" &#125;' \ --instance-type $AWS_EC2_TYPE \ --key-name $AWS_KEY_PAIR_NAME \ --network-interfaces "[ &#123; \"DeviceIndex\": 0, \"Groups\": [\"$AWS_GROUP\"], \"SubnetId\": \"$AWS_SUBNET\", \"DeleteOnTermination\": true, \"AssociatePublicIpAddress\": true &#125; ]" \ --placement "&#123; \"AvailabilityZone\": \"$AWS_ZONE\" &#125;" | \ jq -r .Instances[0] )" source .env echo "Obtaining the EC2 spot instance ID." mem AWS_SPOT_ID "$( jq -r .InstanceId &lt;&lt;&lt; "$AWS_SPOT_INSTANCE" )" source .env echo "Awaiting for the EC2 spot instance $AWS_SPOT_ID to enter the running state." aws ec2 wait instance-running --instance-ids $AWS_SPOT_ID echo "Obtaining the IP address of the new EC2 spot instance $AWS_SPOT_ID." mem AWS_SPOT_IP "$( aws ec2 describe-instances \ --instance-ids $AWS_SPOT_ID | \ jq -r .Reservations[0].Instances[0].PublicIpAddress )" source .env echo "About to ssh to the EC2 spot instance as ubuntu@$AWS_SPOT_IP. echo "When you are done, type: sudo halt." echo "The spot instance will then terminate and be gone forever." echo "Any predefined resources, such as volumes that you attach will be freed." ssh ubuntu@AWS_SPOT_IP echo "Waiting for the EC2 spot instance $AWS_SPOT_ID to enter the stopped state." aws ec2 wait instance-stopped --instance-ids $INSTANCE_ID echo "The spot instance is no longer available. Deleting its keypair." aws ec2 delete-key-pair --key-name AWS_KEY_PAIR_NAME rm $AWS_KEY_PAIR_FILE $AWS_KEY_PAIR_FILE.pub </pre> <p>Make the script executable.</p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>chmod a+x createEc2Spot</pre> <h3 class="numbered" id="createEc2Spot_usage">Sample Usage</h3> <p> The script is easy to use: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><span class='unselectable'>$ </span>createEc2Spot <span class='unselectable'>Just a few questions before the AWS EC2 spot instance can be created. AWS security group: </span>sg-4cbc6f35 <span class='unselectable'>AWS_GROUP='sg-4cbc6f35' EC2 subnet: </span>subnet-49de033f <span class='unselectable'>AWS_SUBNET='subnet-49de033f' AWS availability zone: </span>us-east-1c <span class='unselectable'>AWS_ZONE='us-east-1c' EC2 machine type: </span>t2.medium <span class='unselectable'>AWS_EC2_TYPE='t2.medium' AWS_KEY_PAIR_NAME='rsa-2020-11-04-46-00' AWS_KEY_PAIR_FILE='/home/mslinn/.ssh/rsa-2020-11-04-46-00' Generating public/private rsa key pair. Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-43-54 Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-43-54.pub The key fingerprint is: SHA256:bQScX0UMn0xGDorxSvElMZzwMyyk7hgs2FNbshBNenA mslinn@mslinn.com The key's randomart image is: +---[RSA 4096]----+ | ooE .*++o+** | | =. ooXo=.B.. | | o + o +.X. = | | o = * . =.o | |. + = . S o | | o + . | | . . | | | | | +----[SHA256]-----+ { "KeyName": "2020-11-04-43-54", "KeyFingerprint": "1f:51:ae:28:bf:89:e9:d8:1f:25:5d:37:2d:7d:b8:ca" } Searching for the latest 64-bit Intel/AMD Ubuntu AMI. AWS_AMI='{ "Architecture": "x86_64", "CreationDate": "2020-10-30T14:07:42.000Z", "ImageId": "ami-0c71ec98278087e60", "ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030", "ImageType": "machine", "Public": true, "OwnerId": "099720109477", "PlatformDetails": "Linux/UNIX", "UsageOperation": "RunInstances", "State": "available", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "DeleteOnTermination": true, "SnapshotId": "snap-00bf581086dd686e5", "VolumeSize": 8, "VolumeType": "gp2", "Encrypted": false } }, { "DeviceName": "/dev/sdb", "VirtualName": "ephemeral0" }, { "DeviceName": "/dev/sdc", "VirtualName": "ephemeral1" } ], "Description": "Canonical, Ubuntu, 20.10, amd64 groovy image build on 2020-10-30", "EnaSupport": true, "Hypervisor": "xen", "Name": "ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030", "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SriovNetSupport": "simple", "VirtualizationType": "hvm" }' Obtaining the AMI image ID. AWS_AMI_ID='ami-0c71ec98278087e60' Creating the EC2 spot instance. AWS_SPOT_INSTANCE={ "AmiLaunchIndex": 0, "ImageId": "ami-0dba2cb6798deb6d8", "InstanceId": "i-012a54aefcd333de9", "InstanceType": "t2.small", "KeyName": "rsa-2020-11-03.pub", "LaunchTime": "2020-11-03T23:19:50.000Z", "Monitoring": { "State": "disabled" }, "Placement": { "AvailabilityZone": "us-east-1c", "GroupName": "", "Tenancy": "default" }, "PrivateDnsName": "ip-10-0-0-210.ec2.internal", "PrivateIpAddress": "10.0.0.210", "ProductCodes": [], "PublicDnsName": "", "State": { "Code": 0, "Name": "pending" }, "StateTransitionReason": "", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "Architecture": "x86_64", "BlockDeviceMappings": [], "ClientToken": "026583fb-c94e-4bca-bdd2-8dcdcaa3aae9", "EbsOptimized": false, "EnaSupport": true, "Hypervisor": "xen", "InstanceLifecycle": "spot", "NetworkInterfaces": [ { "Attachment": { "AttachTime": "2020-11-03T23:19:50.000Z", "AttachmentId": "eni-attach-04feb4d36cf5c6792", "DeleteOnTermination": true, "DeviceIndex": 0, "Status": "attaching" }, "Description": "", "Groups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "Ipv6Addresses": [], "MacAddress": "0a:6d:ba:c5:65:4b", "NetworkInterfaceId": "eni-09ef90920cfb29dd9", "OwnerId": "031372724784", "PrivateIpAddress": "10.0.0.210", "PrivateIpAddresses": [ { "Primary": true, "PrivateIpAddress": "10.0.0.210" } ], "SourceDestCheck": true, "Status": "in-use", "SubnetId": "subnet-49de033f", "VpcId": "vpc-f16a0895", "InterfaceType": "interface" } ], "RootDeviceName": "/dev/sda1", "RootDeviceType": "ebs", "SecurityGroups": [ { "GroupName": "testSG", "GroupId": "sg-4cbc6f35" } ], "SourceDestCheck": true, "SpotInstanceRequestId": "sir-rrs9gm3j", "StateReason": { "Code": "pending", "Message": "pending" }, "VirtualizationType": "hvm", "CpuOptions": { "CoreCount": 1, "ThreadsPerCore": 1 }, "CapacityReservationSpecification": { "CapacityReservationPreference": "open" }, "MetadataOptions": { "State": "pending", "HttpTokens": "optional", "HttpPutResponseHopLimit": 1, "HttpEndpoint": "enabled" } } Obtaining the EC2 spot instance ID. i-012a54aefcd333de9 Awaiting for the EC2 spot instance i-012a54aefcd333de9 to enter the running state. Obtaining the IP address of the new EC2 spot instance i-012a54aefcd333de9D. 54.242.88.254 When you are done, type: sudo halt. The spot instance will then terminate and be gone forever. Any predefined resources, such as volumes that you attach will be freed. Waiting for the EC2 spot instance $AWS_SPOT_ID to enter the stopped state. The spot instance is no longer available. Deleting its keypair.</span> </pre> </editor-fold Script> Scala-Style Lambda Function Placeholder Syntax in Python 3 2020-10-22T00:00:00-04:00 https://mslinn.github.io/blog/2020/10/22/scala-style-functional-programming-in-python-3 <p> Pipes are the ultimate in functional programming. It is clear when reading code that uses pipes that data is not mutated. Piping data into and out of lambda functions (and regular functions) is a succinct way of elegantly expressing a computation. This article discusses how to do this using similar syntax in Python 3, Scala 2 and Scala 3. </p> <p> After programming in Scala for more than ten years I have grown to appreciate Scala's implementation of lambda functions, including the ability to use the underscore character as a placeholder for variables. Scala 2.13 introduced pipelining between functions, which is rather like <a href='https://en.wikipedia.org/wiki/Unix-like' target='_blank' rel='nofollow'>*nix</a> pipes between processes. </p> <p> Python 3 can also do something similar. This article demonstrates how to use <a href='https://github.com/sspipe/sspipe' target='_blank' rel='nofollow'><code>sspipe</code></a> and <a href='https://github.com/JulienPalard/Pipe' target='_blank' rel='nofollow'>JulienPalard&rsquo;s <code>pipe</code></a> with Scala's underscore placeholder for <a href='https://stackoverflow.com/questions/29767310/pythons-lambda-with-underscore-for-an-argument' target='_blank' rel='nofollow'>Python 3 lambda functions</a>. </p> <h2 id="pythonSetup">Python 3 Setup</h2> The key concept is to use this specific Python import: <pre data-lt-active="false"> from sspipe import p, px, px as _ </pre> <p> All the Python code examples that follow require this import. The Python code examples are modified versions of the <a href='https://github.com/sspipe/sspipe#examples' target='_blank' rel='nofollow'><code>sspipe</code> examples</a> to illustrate how to use underscores as placeholders. </p> <p> This import is unusual because it imports <code>px</code> twice: once as a normal import, and once aliased to <code>_</code>. I use the <code>_</code> alias to support Scala-like syntax, and I use <code>px</code> when I need to reference a parameter twice. Python is unlike Scala in that the Python compiler does not treat variables called <code>_</code> specially; those variables are merely called <code>_</code>. I could use <code>_</code> in Python code many times to refer to the same value, but a Scala programmer reading that code would expect that each reference to <code>_</code> would be another input parameter, not a regular variable reference. The examples that follow should make this clear. </p> <h3 id="installation">Python 3 Installation</h3> Install <code>sspipe</code> using <code>pip</code>: <pre data-lt-active="false"> <span class="unselectable">$ </span>pip install --upgrade sspipe </pre> <h2 id="scalaSetup">Scala 2.13+ Setup</h2> <p> Scala 2.13 introduced <a href='https://www.scala-lang.org/api/current/scala/util/ChainingOps.html' target='_blank' rel='nofollow'><code>ChainingOps</code></a>, which adds chaining methods <code>tap</code> and <code>pipe</code> to every type. The key concept is to import the following prior to attempting the code examples below: </p> <pre data-lt-active="false"> implicit class Smokin[A](val a: A) { import scala.util.chaining._ import scala.language.implicitConversions implicit def |>[B](f: (A) => B): B = a.pipe(f) } </pre> <h2 id="examples">Python and Scala Usage Examples</h2> <h3 id="2-3">One Lambda Function and 1 Pipe</h3> <p> This Python example employs one lambda function and 1 pipe to add 2 to the number 5: </p> <pre data-lt-active="false"> >>> 5 | _ + 2 7 </pre> <p> The Scala equivalent of the above is: </p> <pre data-lt-active="false"> <span class="unselectable">scala> </span>5 |> ((_: Int) + 2) val res0: Int = 7 </pre> <h3 id="2-3">Two Lambda Functions and 2 Pipes</h3> <p> This Python example employs two lambda functions and 2 pipes to multiply the previous result by 5 and then add the previous result. Recall that I said that in Python, an underscore when used this way is the name of a normal variable and that the compiler does not treat underscores as placeholders for lambda parameters. A Scala programmer would complain about the following code, because they would expect that the second lambda function would require 2 inputs: </p> <pre data-lt-active="false"> >>> 5 | _ + 2 | _ * 5 + _ 42 </pre> <p> A better way to write the above would be to use the special variable <code>px</code>, which was imported above. Now everyone either knows that <code>px</code> holds the piped value, or they complain about <code>px</code> being a magic variable. A possible solution to this complaint would be to alias <code>px</code> to a more descriptive name, such as <code>pipedValue</code> &mldr; which is still magical, but at least it is more descriptive. </p> <pre data-lt-active="false"> >>> 5 | _ + 2 | px * 5 + px 42 </pre> The Scala equivalent of the above is: <pre data-lt-active="false"> <span class="unselectable">scala> </span>5 |> ((_: Int) + 2) |> ((x: Int) => x * 5 + x) val res1: Int = 42 </pre> <h3 id="2-3">Two Lambda Functions and 3 Pipes</h3> <p> This Python example employs 2 lambda functions and 3 pipes to add 10 to the even numbers from 0 to 5, exclusive. </p> <pre data-lt-active="false"> >>> ( range(5) | p(filter, _ % 2 == 0) | p(map, _ + 10) | p(list) ) [10, 12, 14] </pre> <p> Scala has a better way of performing this type of computation that does not require pipes or computation. It is better because it is simpler to understand. </p> <pre data-lt-active="false"> <span class="unselectable">scala> </span>for { | x <- (0 until 5).toList if x % 2 == 0 | y = x + 10 | } yield y val res12: List[Int] = List(10, 12, 14) </pre> <h2 id="other">Other examples of placeholder syntax</h2> <p> <code>NumPy</code> expressions (NumPy is Python-specific): </p> <pre data-lt-active="false"> range(10) | np.sin(_)+1 | p(plt.plot) </pre> <p> Pandas expressions (Pandas is Python-specific): </p> <pre data-lt-active="false"> people_df | _.loc[_.age > 10, 'name'] </pre> <p> Solution for the <a href='https://projecteuler.net/problem=2' target='_blank' rel='nofollow'>2nd Project Euler exercise</a>: </p> <pre data-lt-active="false"> >>> def fib(): a, b = 0, 1 while True: yield a a, b = b, a + b >>> euler2 = ( fib() | p.where(_ % 2 == 0) | p.take_while(_ < 4000000) | p.add() ) >>> euler2 4613732 </pre> <h2 id="dotty">Looking Ahead to Scala 3 (Dotty)</h2> <p> The next major version of Scala, due out in a few months, will probably allow a Scala 3 extension method to define the vertical bar as a method for more readabile code: </p> <pre data-lt-active="false"> def [A,B](a: A) |(f: (A) => B): B = a.pipe(f) # Sample usage: val x = 5 | doSomething | doSomethingElse | doSomethingMore </pre> <h2 id="scalacourses">To Learn More</h2> <p> My <a href='https://www.scalacourses.com/showCourse/40' target='_blank'>Introduction to Scala</a> course on ScalaCourses.com teaches Scala lambda functions. </p> I've Been Writing Jekyll Plugins 2020-10-03T00:00:00-04:00 https://mslinn.github.io/blog/2020/10/03/jekyll-plugins <editor-fold Intro> <p> This is a Jekyll-powered web site. <a href='https://jekyllrb.com/' target='_blank' rel='nofollow'>Jekyll</a> is a free open-source preprocessor that generates static web sites. You can extend Jekyll by using the <a href='https://jekyllrb.com/docs/liquid/' target='_blank' rel='nofollow'>Liquid</a> language to write includes. Includes are just macros for Jekyll. </p> <p> I prefer to write and use Jekyll plugins instead of Jekyll includes. Non-trivial Jekyll includes require logic to be expressed in the Liquid language. Liquid is an interesting language, but it is quite verbose, syntax can be awkward, some expressions are impossible to formulate, and there are no debugging tools. </p> <p> In contrast, plugins are written in <a href='https://www.ruby-lang.org' target='_blank' rel='nofollow'>Ruby</a>. Plugin usage syntax is more flexible and require less typing for users. </p> <p> The argument against writing plugins is that the Ruby language is subtle and powerful, and could be overwhelming for novice programmers. However, just as the <a href='https://spark.apache.org/' target='_blank' rel='nofollow'>Apache Spark</a> framework allows novice Scala programmers to write in <a href='https://databricks.com/session/just-enough-scala-for-spark' target='_blank' rel='nofollow'>Just Enough Scala for Spark</a>, and the <a href='https://rubyonrails.org' target='_blank' rel='nofollow'>Ruby on Rails</a> framework allows novice Ruby programmers to write <a href='https://ivanoats.github.io/just_enough_ruby' target='_blank' rel='nofollow'>Just Enough Ruby for Rails</a>, writing plugins for the Jekyll framework generally does not require total mastery of Ruby. </p> <p> Here are some of my plugins. The source code for a plugin can be copied to the clipboard whenever you click on this icon at the top right corner of the code: <img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'>. </p> <p> <a href='../../../../mslinn_jekyll_plugins.zip'>A zip file containing all the plugins</a> is available. </p> <h2 id="update">Update 2020-12-28</h2> <p> ... but wait, there is more! I wrote a <code>LogFactory</code> Ruby library class after publishing this article. Most of these plugins have been retrofitted with <code>LogFactory</code> &ndash; calls to <code>LoggerFactory.new.create_logger</code> create a custom logger. </p> <p> You can use <code>LogFactory</code> in your Jekyll plugins for debugging. <a href='/blog/2020/12/28/custom-logging-in-jekyll-plugins.html'>I wrote it up separately</a> but <code>log_factory.rb</code> is included in the above zip file. </p> </editor-fold Intro> <editor-fold archive_display> <h2 id="archiveDisplay" class="spaceAbove"><span class="code">archive_display</span></h2> <p> Lists the names and contents of each file in a <code>tar</code> file. For each text file, the following HTML is emitted: </p> <pre data-lt-active="false" class="snippet" id="foo" style="position: relative">&lt;div class='codeLabel'>{tar_entry.full_name}&lt;/div> &lt;pre data-lt-active='false'>&lt;code>{tar_entry.file_contents}&lt;/pre> </pre> <p> Binary files are displayed like this: </p> <div class="codeLabel">usr/bin/ruby2.7 <span style="font-size: smaller">(application/x-sharedlib; charset=binary)</span></div> <pre data-lt-active="false"><i>Binary file</i></pre> <h3 id="archiveDisplaySyntax">Syntax</h3> <pre data-lt-active="false">{% archive_display filename.tar %}</pre> <p> Sample output is: </p> <h3 id="archiveDisplaySource">Source Code</h3> <p> <a href='/jekyll/doc/ArchiveDisplayTag.html' target='_blank'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/archive_display.rb" download="archive_display.rb" title="Click on the file name to download the file">archive_display.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id1058c66c3d75"><button class='copyBtn' data-clipboard-target='#id1058c66c3d75' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 # # Displays information about the contents of tar files # # Install dependencies: # - Ubuntu: `sudo apt install libmagic-dev` # - Mac: `brew install libmagic` module ArchiveDisplayTag @log = LoggerFactory.new.create_logger('my_tag', Jekyll.configuration(&#123;&#125;), :warn, $stderr) # accessor allows classes in this module to use the logger def self.log @log end class ArchiveDisplay &lt; Liquid::Tag # Constructor. # @param tag_name [String] is the name of the tag, which we already know. # @param archive_name [Hash, String, Liquid::Tag::Parser] the arguments from the web page. # @param tokens [Liquid::ParseContext] tokenized command line # @return [void] def initialize(tag_name, archive_name, tokens) super archive_name.strip! @archive_name = archive_name end # Method prescribed by the Jekyll plugin lifecycle. # @return [String] def render(context) source = context.registers[:site].config['source'] tar_name = "#&#123;source&#125;/#&#123;@archive_name&#125;" ArchiveDisplayTag.log.info "archive_display: tar_name=#&#123;tar_name&#125;" traverse_tar(tar_name) end private # Walks through a `tar` file. # # Modified from this &#123;https://gist.github.com/sinisterchipmunk/1335041/5be4e6039d899c9b8cca41869dc6861c8eb71f13 gist by sinisterchipmunk &#125;. # # @param tar_name [String] Name of tar file to examine. # @return [String] containing HTML describing the contents of the `tar`. def traverse_tar(tar_name) require 'rubygems/package' require 'ruby-filemagic' # sudo apt install libmagic-dev # brew install libmagic file_magic = FileMagic.new(FileMagic::MAGIC_MIME) File.open(tar_name, "rb") do |file| Gem::Package::TarReader.new(file) do |tar| return tar.each.map &#123; |entry| next if entry.file? content = entry.read fm_type = file_magic.buffer(content) &#123; name: entry.full_name, content: content.strip, is_text: (fm_type.start_with? "text"), fm_type: fm_type &#125; &#125;.compact.sort_by &#123; |entry| entry[:name] &#125;.map &#123; |entry| heading = "&lt;div class='codeLabel'>#&#123;entry[:name]&#125; &lt;span style='font-size: smaller'>(#&#123;entry[:fm_type]&#125;)&lt;/span>&lt;/div>" if entry[:is_text] "#&#123;heading&#125;\n&lt;pre data-lt-active='false'>#&#123;entry[:content]&#125;&lt;/pre>" else "#&#123;heading&#125;\n&lt;p>&lt;i>Binary file&lt;/i>&lt;/pre>" end &#125; end end end end end Liquid::Template.register_tag('archive_display', ArchiveDisplayTag::ArchiveDisplay) </pre> <h3 id="archiveDisplayInstall">Installation</h3> <ol> <li> Install <code>libmagic</code>. <br /> <div class='codeLabel' style="margin-top: 1.25em;">Ubuntu & WSL</div> <pre data-lt-active="false"><span class="unselectable">$ </span>sudo apt install libmagic-dev</pre> <div class='codeLabel'>Mac</div> <pre data-lt-active="false"><span class="unselectable">$ </span>brew install libmagic</pre> </li> <li> Add this line to <code>Gemfile</code> in your Jekyll site's top-level directory: <pre data-lt-active="false">gem 'ruby-filemagic'</pre> </li> <li> Install the <code>ruby-filemagic</code> gem. From your Jekyll site's top-level directory, type: <pre data-lt-active="false"><span class="unselectable">$ </span>bundle install</pre> </li> <li> Copy <code>archive_display.rb</code> and <code>logger_factory.rb</code> into the <code>_plugins/</code> directory of your Jekyll site. </li> <li> Restart Jekyll. </li> </ol> </editor-fold archive_display> <editor-fold basename> <h2 id="basename" class="spaceAbove"><span class="code">basename</span>, <span class="code">dirname</span> and <span class="code">basename_without_extension</span></h2> <p> These filters all return portions of a string. They are all defined in the same plugin. </p> <ul> <li><code>basename</code> &mdash; Filters a string containing a path, returning the filename and extension.</li> <li><code>dirname</code> &mdash; Filters a string containing a path, returning the portion before the filename and extension.</li> <li><code>basename_without_extension</code> &mdash; Filters a string containing a path, returning the filename without the extension.</li> </ul> <h3 id="basename_syntax">Syntax</h3> <pre data-lt-active="false"> {{ "blah/blah/filename.ext" | basename }} {{ "blah/blah/filename.ext" | dirname }} {{ "blah/blah/filename.ext" | basename_without_extension }} </pre> <h3 id="archiveDisplaySource">Source Code</h3> <p> <a href='/jekyll/doc/Basename.html' target='_blank'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/basename.rb" download="basename.rb" title="Click on the file name to download the file">basename.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id5c0af443f06c"><button class='copyBtn' data-clipboard-target='#id5c0af443f06c' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 Michael Slinn # @license SPDX-License-Identifier: Apache-2.0 # # Jekyll filters for working with paths. module Basename # Filters a string containing a path. # @return [String] the filename extracted from the path, including the filetype. # @example Extracts "filename.ext" from the path # &#123;&#123; "blah/blah/filename.ext" | basename &#125;&#125; def basename(filepath) File.basename(filepath) end # Filters a string containing a path. # @return [String] the portion of th path before the filename and extension. # @example Extracts "blah/blah" from the path. # &#123;&#123; "blah/blah/filename.ext" | dirname &#125;&#125; def dirname(filepath) File.dirname(filepath) end # Filters a string containing a path. # @return the filename without the extension. # @example Extracts "filename" from the path. # &#123;&#123; "blah/blah/filename.ext" | basename_without_extension &#125;&#125; def basename_without_extension(filepath) File.basename(filepath).split('.')[0...-1].join('.') end end Liquid::Template.register_filter(Basename) </pre> </editor-fold basename> <editor-fold flexible_include> <h2 id="flexibleInclude" class="spaceAbove"><span class="code">flexible_include</span></h2> <p> Jekyll&#39;s built-in <code>include</code> tag does not support including files outside of the <code>_includes</code> folder. Originally called <code>include_absolute</code>, this plugin name is now called <code>flexible_include</code> because it no longer just includes absolute file names. This plugin now supports 4 types of includes: </p> <ol> <li>Absolute filenames (first character is <code>/</code>).</li> <li>Filenames relative to the top-level directory of the Jekyll web site (unnecessary to preface with <code>./</code>).</li> <li>Filenames relative to the user home directory (first character is <code>~</code>).</li> <li>Executable filenames on the <code>PATH</code> (first character is <code>!</code>).</li> </ol> <p> In addtion, filenames that require environment expansion because they contain a <code>$</code> character are expanded according to the environment variables defined when <code>jekyll build</code> executes. </p> <h3 id="flexibleIncludeSyntax">Syntax</h3> <pre data-lt-active="false">{% flexible_include 'path' optionalParam1='yes' optionalParam2='green' %}</pre> <p> The optional parameters can have any name. The included file will have parameters substituted. </p> <h3 id="flexibleIncludeUsage">Usage Examples</h3> <ol> <li> Include files without parameters; all four types of includes are shown. <pre data-lt-active="false">{% flexible_include '../../folder/outside/jekyll/site/foo.html' %} {% flexible_include 'folder/within/jekyll/site/bar.js' %} {% flexible_include '/etc/passwd' %} {% flexible_include '~/.ssh/config' %} </pre> Here is another <code>flexible_include</code> invocation using environment variables: <pre data-lt-active="false"> {% flexible_include '$HOME/.bash_aliases' %} </pre> </li> <li> Include a file and pass parameters to it. <pre data-lt-active="false">{% flexible_include '~/folder/under/home/directory/foo.html' param1='yes' param2='green' %}</pre> </li> </ol> <h3 id="flexibleIncludeSource">Source Code</h3> <p> This code lives in a <a href="https://github.com/mslinn/jekyll-flexible-include-plugin" rel="nofollow" target="_blank">GitHub repository</a>. <a href='/jekyll/doc/FlexibleIncludeTag.html' target='_blank'>Yard docs are here.</a> </p> </editor-fold flexible_include> <editor-fold from_to_until> <h2 id="from_to_until" class="spaceAbove"><span class="code">from</span>, <span class="code">to</span> and <span class="code">until</span></h2> <p> These filters all return portions of a multiline string. They are all defined in the same plugin. A <a href='https://ruby-doc.org/core-2.5.7/Regexp.html' target='_blank' rel='nofollow'>regular expression</a> is used to specify the match; the simplest regular expression is a string. </p> <ul> <li><code>from</code> &mdash; returns the portion beginning with the line that satisfies a regular expression to the end of the multiline string.</li> <li><code>to</code> &mdash; returns the portion from the first line to the line that satisfies a regular expression, including the matched line.</li> <li><code>until</code> &mdash; returns the portion from the first line to the line that satisfies a regular expression, excluding the matched line.</li> </ul> <p> <a href='https://rubular.com/' target='_blank' rel='nofollow'>Rubular</a> is a handy online tool to try out regular expressions. </p> <h3 id="from_to_until_syntax">Syntax</h3> <p> The regular expression may be enclosed in single quotes, double quotes, or nothing. </p> <h4 id="from_syntax" class="code">from</h4> All of these examples perform identically. <pre data-lt-active="false"> {{ sourceOfLines | from: 'regex' }} {{ sourceOfLines | from: "regex" }} {{ sourceOfLines | from: regex }} </pre> <h4 id="to_syntax" class="code">to</h4> All of these examples perform identically. <pre data-lt-active="false"> {{ sourceOfLines | to: 'regex' }} {{ sourceOfLines | to: "regex" }} {{ sourceOfLines | to: regex }} </pre> <h4 id="until_syntax" class="code">until</h4> All of these examples perform identically. <pre data-lt-active="false"> {{ sourceOfLines | until: 'regex' }} {{ sourceOfLines | until: "regex" }} {{ sourceOfLines | until: regex }} </pre> <p> <b>Important:</b> the name of the filter must be followed by a colon (:). If you fail to do that an error will be generated and the Jekyll site building process will halt. The error message looks something like this: <code>Liquid Warning: Liquid syntax error (line 285): Expected end_of_string but found string in "{{ lines | from '2' | until: '4' | xml_escape }}" in /some_directory/some_files.html Liquid Exception: Liquid error (line 285): wrong number of arguments (given 1, expected 2) in /some_directory/some_file.html Error: Liquid error (line 285): wrong number of arguments (given 1, expected 2)</code> </p> <h3 id="from_to_until_examples">Usage Examples</h3> <p> Some of the following examples use a multiline string containing 5 lines, called <code>lines</code>, which was created this way: </p> <pre data-lt-active="false">{% capture lines %}line 1 line 2 line 3 line 4 line 5 {% endcapture %} </pre> <p> Other examples use a multiline string containing the contents of <code>.gitignore</code>, which looks like this: </p> <div class="codeLabel">.gitignore</div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idcbc1f7fc5071">*.gz *.sublime* *.swp *.out *.Identifier *.log .idea* *.iml *.tmp *~ .DS_Store .idea .jekyll-cache/ .jekyll-metadata .sass-cache/ .yardoc/ jekyll/doc/ out/ __pycache__/ __MACOSX _build/ _package/ _site/ ~* bin/*.class node_modules/ Notepad++/ package/ cloud9.tar cloud9.zip instances.json rescue_ubuntu2010 rescue_ubuntu2010.b64 landingPageShortName.md stderr test.html RUNNING_PID mslinn_jekyll_plugins.zip mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip </pre> <h4 id="until_example_line3">From the third line of string</h4> <p> These examples return the lines of the file from the beginning of the until a line with the string <code>"3"</code> is found, including the matched line. The only difference between the examples is the delimiter around the regular expression. </p> <pre data-lt-active="false">{{ lines | from: '3' }}</pre> <pre data-lt-active="false">{{ lines | from: "3" }}</pre> <pre data-lt-active="false">{{ lines | from: 3 }}</pre> <p> These all generate: </p> <pre data-lt-active="false">line 3 line 4 line 5 </pre> <h4 id="until_example_filec">From Line In a File Containing 'PID'</h4> <pre data-lt-active="false">{% capture gitignore %}{% flexible_include '.gitignore' %}{% endcapture %} {{ gitignore | from: 'PID' | xml_escape }}</pre> <p> This generates: </p> <pre data-lt-active="false">RUNNING_PID mslinn_jekyll_plugins.zip mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip </pre> <h4 id="to_example_line3">To the third line of string</h4> <p> These examples return the lines of the file from the first line until a line with the string <code>"3"</code> is found, including the matched line. The only difference between the examples is the delimiter around the regular expression. </p> <pre data-lt-active="false">{{ lines | to: '3' }}</pre> <pre data-lt-active="false">{{ lines | to: "3" }}</pre> <pre data-lt-active="false">{{ lines | to: 3 }}</pre> <p> These all generate: </p> <pre data-lt-active="false">line 1 line 2 line 3 </pre> <h4 id="until_example_file">To Line In a File Containing 'idea'</h4> <pre data-lt-active="false">{{ gitignore | to: 'idea' }}</pre> <p> This generates: </p> <pre data-lt-active="false">*.gz *.sublime* *.swp *.out *.Identifier *.log .idea* </pre> <h4 id="until_example_line3b">Until the third line of string</h4> <p> These examples return the lines of the file until a line with the string <code>"3"</code> is found, excluding the matched line. The only difference between the examples is the delimiter around the regular expression. </p> <pre data-lt-active="false">{{ lines | until: '3' }}</pre> <pre data-lt-active="false">{{ lines | until: "3" }}</pre> <pre data-lt-active="false">{{ lines | until: 3 }}</pre> <p> These all generate: </p> <pre data-lt-active="false">line 1 line 2 </pre> <h4 id="until_example_fileb">Until Line In a File Containing 'idea'</h4> <pre data-lt-active="false">{{ gitignore | until: 'idea' }}</pre> <p> This generates: </p> <pre data-lt-active="false">*.gz *.sublime* *.swp *.out *.Identifier *.log </pre> <h4 id="until_example_lines_2_4">From the string "2" until the string "4"</h4> <p> These examples return the lines of the file until a line with the string <code>"3"</code> is found, excluding the matched line. The only difference between the examples is the delimiter around the regular expression. </p> <pre data-lt-active="false">{{ lines | from: '2' | until: '4' }}</pre> <pre data-lt-active="false">{{ lines | from: "2" | until: "4" }}</pre> <pre data-lt-active="false">{{ lines | from: 2 | until: 4 }}</pre> <p> These all generate: </p> <pre data-lt-active="false">line 2 line 3 </pre> <h4 id="until_example_file_no_match">From Line In a File Containing 'idea' Until no match</h4> <p> The <code>.gitignore</code> file does not contain the string <code>xx</code>. If we attempt to match against that string the remainder of the file is returned for the <code>to</code> and <code>until</code> filter, and the empty string is returned for the <code>from</code> filter. </p> <pre data-lt-active="false">{{ gitignore | from: 'PID' | until: 'xx' }}</pre> <p> This generates: </p> <pre data-lt-active="false">RUNNING_PID mslinn_jekyll_plugins.zip mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip </pre> <h4 id="until_example_complex">More Complex Regular Expressions</h4> <p> The <code>from</code>, <code>to</code> and <code>until</code> filters can all accept more complex regular expressions. This regular expression matches lines that have either the string <code>sun</code> or <code>cloud</code> at the beginning of the line. </p> <pre data-lt-active="false">{{ gitignore | from: '^(cloud|sun)' }}</pre> <p> This generates: </p> <pre data-lt-active="false">cloud9.tar cloud9.zip instances.json rescue_ubuntu2010 rescue_ubuntu2010.b64 landingPageShortName.md stderr test.html RUNNING_PID mslinn_jekyll_plugins.zip mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip cloud9.tar mslinn_jekyll_plugins.zip </pre> <h3 id="includeSource">Source Code</h3> <p> <a href='/jekyll/doc/FromToUntil.html' target='_blank'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/from_to_until.rb" download="from_to_until.rb" title="Click on the file name to download the file">from_to_until.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id6d249f13e079"><button class='copyBtn' data-clipboard-target='#id6d249f13e079' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 Michael Slinn # @license SPDX-License-Identifier: Apache-2.0 # Jekyll filters for working with multiline strings. module FromToUntil # Filters a multiline string, returning the portion beginning with the line that satisfies a regex. # The regex could be enclosed in single quotes, double quotes, or nothing. # @param input_strings [String] The multi-line string to scan # @param regex [String] The regular expression to match against each line of `input_strings` until found # @return [String] The remaining multi-line string # @example Returns remaining lines starting with the line containing the word `module`. # &#123;&#123; flexible_include '/blog/2020/10/03/jekyll-plugins.html' | from 'module' &#125;&#125; def from(input_strings, regex) return '' unless check_parameters(input_strings, regex) regex = remove_quotations(regex.to_s.strip) matched = false result = '' input_strings.each_line do |line| matched = true if !matched && line =~ /#&#123;regex&#125;/ result += line if matched end result end # Filters a multiline string, returning the portion from the beginning until and including the line that satisfies a regex. # The regex could be enclosed in single quotes, double quotes, or nothing. # @example Returns lines up to and including the line containing the word `module`. # &#123;&#123; flexible_include '/blog/2020/10/03/jekyll-plugins.html' | to 'module' &#125;&#125; def to(input_strings, regex) return '' unless check_parameters(input_strings, regex) regex = remove_quotations(regex.to_s.strip) result = '' input_strings.each_line do |line| result += line return result if line =~ /#&#123;regex&#125;/ end result end # Filters a multiline string, returning the portion from the beginning until but not including the line that satisfies a regex. # The regex could be enclosed in single quotes, double quotes, or nothing. # @example Returns lines up to but not including the line containing the word `module`. # &#123;&#123; flexible_include '/blog/2020/10/03/jekyll-plugins.html' | until 'module' &#125;&#125; def until(input_strings, regex) return '' unless check_parameters(input_strings, regex) regex = remove_quotations(regex.to_s.strip) result = '' input_strings.each_line do |line| return result if line =~ /#&#123;regex&#125;/ result += line end result end private def check_parameters(input_strings, regex) if input_strings.nil? || input_strings.empty? then puts "Warning: Plugin 'from' received no input." return false end regex = regex.to_s if regex.nil? || regex.empty? then puts "Warning: Plugin 'from' received no regex." return false end true end def remove_quotations(str) str = str.slice(1..-2) if (str.start_with?('"') && str.end_with?('"')) || (str.start_with?("'") && str.end_with?("'")) str end end Liquid::Template.register_filter(FromToUntil) </pre> <h3 id="includeInstall">Installation</h3> <ol> <li> Copy <code>from_to_until.rb</code> and <code>logger_factory.rb</code> into the <code>_plugins/</code> directory of your Jekyll site. </li> <li> Restart Jekyll. </li> </ol> </editor-fold from_to_until> <editor-fold href> <h2 id="href" class="spaceAbove"><span class="code">href</span></h2> <p>Generates an <code>a href</code> tag with <code>target=&quot;_blank&quot;</code> and <code>rel=nofollow</code>.</p> <h3 id="hrefSyntax">Syntax</h3> <pre data-lt-active="false">{% href url text to display %}</pre> <p>The url should not be enclosed in quotes.</p> <h3 id="hrefExamples">Usage Examples</h3> <h4 id="hrefExample1">Default</h4> <pre data-lt-active="false">{% href https://www.mslinn.com The Awesome %}</pre> <p> This generates: </p> <pre data-lt-active="false">&lt;a href='https://www.mslinn.com' target='_blank' rel='nofollow'&gt;The Awesome&lt;/a&gt;</pre> <p> Which renders as: <a href='https://www.mslinn.com' target='_blank' rel='nofollow'>The Awesome</a> </p> <h4 id="hrefExample2"><span class='code'>follow</span></h4> <pre data-lt-active="false">{% href follow https://www.mslinn.com The Awesome %}</pre> <p> This generates: </p> <pre data-lt-active="false">&lt;a href='https://www.mslinn.com' target='_blank'&gt;The Awesome&lt;/a&gt;</pre> <h4 id="hrefExample3"><span class="code">notarget</span></h4> <pre data-lt-active="false">{% href notarget https://www.mslinn.com The Awesome %}</pre> <p> This generates: </p> <pre data-lt-active="false">&lt;a href='https://www.mslinn.com' rel='nofollow'&gt;The Awesome&lt;/a&gt;</pre> <h4 id="hrefExample4"><span class="code">follow notarget</span></h4> <pre data-lt-active="false">{% href follow notarget https://www.mslinn.com The Awesome %}</pre> <p> This generates: </p> <pre data-lt-active="false">&lt;a href='https://www.mslinn.com'&gt;The Awesome&lt;/a&gt;</pre> <h3 id="hrefSource">Source Code</h3> <p> <a href='/jekyll/doc/HrefTag.html' target='_blank'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/href.rb" download="href.rb" title="Click on the file name to download the file">href.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id99bf59d61847"><button class='copyBtn' data-clipboard-target='#id99bf59d61847' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># @author Copyright 2020 Michael Slinn # @license SPDX-License-Identifier: Apache-2.0 # # Generates an href. # Note that the url should not be enclosed in quotes. # By default the link includes `target='_blank'`, # which causes the link to open in a new tab or window. # By default the link also includes `rel=nofollow` for SEO purposes. # # To suppress the `nofollow` attribute, preface the link with the word `follow`. # To suppress the `target` attribute, preface the link with the word `notarget`. # # @example General form # &#123;% href [follow] [notarget] url text to display %&#125; # # @example Generates `nofollow` and `target` attributes. # &#123;% href https://mslinn.com The Awesome %&#125; # # @example Does not generate `nofollow` or `target` attributes. # &#123;% href follow notarget https://mslinn.com The Awesome %&#125; # # @example Does not generate `nofollow` attribute. # &#123;% href follow https://mslinn.com The Awesome %&#125; # # @example Does not generate `target` attribute. # &#123;% href notarget https://mslinn.com The Awesome %&#125; module HrefTag class ExternalHref &lt; Liquid::Tag # Constructor. # @param tag_name [String] is the name of the tag, which we already know. # @param command_line [Hash, String, Liquid::Tag::Parser] the arguments from the web page. # @param tokens [Liquid::ParseContext] tokenized command line # @return [void] def initialize(tag_name, command_line, tokens) super @follow = " rel='nofollow'" @target = " target='_blank'" tokens = command_line.strip.split(" ") followIndex = tokens.index("follow") if followIndex then tokens.delete_at(followIndex) @follow = "" end targetIndex = tokens.index("notarget") if targetIndex then tokens.delete_at(targetIndex) @target = "" end @link = tokens.shift @text = tokens.join(" ").strip @text = if @text.empty? then @link else @text end end # Method prescribed by the Jekyll plugin lifecycle. # @return [String] def render(_) "&lt;a href='#&#123;@link&#125;'#&#123;@target&#125;#&#123;@follow&#125;>#&#123;@text&#125;&lt;/a>" end end end Liquid::Template.register_tag('href', HrefTag::ExternalHref) </pre> <h3 id="hrefInstall">Installation</h3> <ol> <li> Copy <code>href.rb</code> and <code>logger_factory.rb</code> into the <code>_plugins/</code> directory of your Jekyll site. </li> <li> Restart Jekyll. </li> </ol> </editor-fold href> <editor-fold link> <h2 id="link" class="spaceAbove"><span class="code">link</span></h2> <p> This plugin generates a link to the given URI, which must be a file on the server. The file name can be absolute or relative to the top-level directory of the web site. </p> <h3 id="linkSyntax">Syntax</h3> <pre data-lt-active="false">{% link uri %}</pre> <h3 id="linkExample">Usage Example</h3> <pre data-lt-active="false">{% link cloud9.tar %}</pre> <p> Generates: </p> <pre data-lt-active="false">&lt;a href="/cloud9.tar">&lt;code>cloud9.tar&lt;/code>&lt;/a> (4.5 KB)</pre> <p> Which renders as: <a href="/cloud9.tar"><code>cloud9.tar</code></a> (4.5 KB) </p> <h3 id="linkSource">Source Code</h3> <p> <a href='/jekyll/doc/LinkTag.html' target='_blank'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/link.rb" download="link.rb" title="Click on the file name to download the file">link.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="ida3c9b62462c9"><button class='copyBtn' data-clipboard-target='#ida3c9b62462c9' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># @author Copyright 2020 Michael Slinn # @license SPDX-License-Identifier: Apache-2.0 module LinkTag # Generates an href to a file for the user to download from the site. # Also shows the file size in a human-readable format. class Linker &lt; Liquid::Tag # Constructor. # @param tag_name [String] is the name of the tag, which we already know. # Contains the name of the file, relative to the website top level directory # @param text [Hash, String, Liquid::Tag::Parser] the arguments from the web page. # @param tokens [Liquid::ParseContext] tokenized command line # @return [void] def initialize(tag_name, text, tokens) super(tag_name, text, tokens) @filename = text.delete('"').delete("'").strip end # Method prescribed by the Jekyll plugin lifecycle. # @return [String] def render(context) source = context.registers[:site].config['source'] file_fq = File.join(source, @filename) abort("Error: '#&#123;file_fq&#125;' not found. See the link tag in") unless File.exist?(file_fq) "&lt;a href='/#&#123;@filename&#125;'>&lt;code>#&#123;@filename&#125;&lt;/code>&lt;/a> (#&#123;as_size(File.size(file_fq))&#125;)" end def as_size(s) units = %w[B KB MB GB TB] size, unit = units.reduce(s.to_f) do |(fsize, _), utype| fsize > 512 ? [fsize / 1024, utype] : (break [fsize, utype]) end "#&#123;size > 9 || size.modulo(1) &lt; 0.1 ? '%d' : '%.1f'&#125; %s" % [size, unit] end end end Liquid::Template.register_tag('link', LinkTag::Linker) </pre> <h3 id="linkInstall">Installation</h3> <ol> <li> Copy <code>link.rb</code> and <code>logger_factory.rb</code> into the <code>_plugins/</code> directory of your Jekyll site. </li> <li> Restart Jekyll. </li> </ol> </editor-fold link> <editor-fold make_archive> <h2 id="make_archive" class="spaceAbove"><span class="code">make_archive</span></h2> <p> Creates <code>tar</code> and <code>zip</code> archives according to the <code>make_archive</code> entry in <code>_config.yml</code>. In <code>production</code> mode, the archives are built each time Jekyll generates the web site. In <code>development</code> mode, the archives are only built if they do not already exist, or if <code>delete: true</code> is set for that archive in <code>_config.yml</code>. Archives are placed in the top-level of the Jekyll project, and are copied to <code>_site</code> by Jekyll's normal build process. Entries are created in <code>.gitignore</code> for each of the generated archives. </p> <h3 id="make_archive_files">File Specifications</h3> <p>This plugin supports 4 types of file specifications:</p> <ol> <li>Absolute filenames (start with <code>/</code>).</li> <li>Filenames relative to the top-level directory of the Jekyll web site (Do not preface with <code>.</code> or <code>/</code>).</li> <li>Filenames relative to the user home directory (preface with <code>~</code>).</li> <li>Executable filenames on the <code>PATH</code> (preface with <code>!</code>).</li> </ol> <h3 id="make_archive_code"><span class="code">_config.yml</span> Syntax</h3> <p> Any number of archives can be specified. Each archive has 3 properties: <code>archive_name</code>, <code>delete</code> (defaults to <code>true</code>) and <code>files</code>. Take care that the dashes have exactly 2 spaces before them, and that the 2 lines following each dash have exactly 4 spaces in front. </p> <pre data-lt-active="false">make_archive: - archive_name: cloud9.zip delete: true # This is the default, and need not be specified. files: [ index.html, error.html, ~/.ssh/config, /etc/passwd, '!update' ] - archive_name: cloud9.tar delete: false # Do not overwrite the archive if it already exists files: [ index.html, error.html, ~/.ssh/config, /etc/passwd, '!update' ] </pre> <h3 id="make_archive_source">Source Code</h3> <p> <a href='/jekyll/doc/MakeArchive.html' target='_blank'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/make_archive.rb" download="make_archive.rb" title="Click on the file name to download the file">make_archive.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id24fd0688607b"><button class='copyBtn' data-clipboard-target='#id24fd0688607b' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 Michael Slinn # @license SPDX-License-Identifier: Apache-2.0 require 'fileutils' require 'ptools' require 'rubygems' require 'rubygems/package' require 'tmpdir' require 'zlib' # Makes tar or zip file based on _config.yml entry class MakeArchive &lt; Jekyll::Generator require_relative 'logger_factory' priority :high def initialize(config) super(config) @log = LoggerFactory.new.create_logger('make_archive', config, :warn, $stderr) end # Method prescribed by the Jekyll plugin lifecycle. # @param site [Jekyll.Site] Automatically provided by Jekyll plugin mechanism # @return [void] def generate(site) @live_reload = site.config['livereload'] archive_config = site.config['make_archive'] return if archive_config.nil? archive_config.each do |config| @archive_name = config['archive_name'] # Relative to _site abort 'Error: archive_name was not specified in _config.yml.' if @archive_name.nil? if @archive_name.end_with? '.zip' @archive_type = :zip elsif @archive_name.end_with? '.tar' @archive_type = :tar else abort "Error: archive must be zip or tar; #&#123;@archive_name&#125; is of an unknown archive type." end @archive_files = config['files'].compact abort 'Error: archive files were not specified in _config.yml.' if @archive_files.nil? delete_archive = config['delete'] @force_delete = delete_archive.nil? ? !@live_reload : delete_archive @log.info "@archive_name=#&#123;@archive_name&#125;; @live_reload=#&#123;@live_reload&#125;; @force_delete=#&#123;@force_delete&#125;; @archive_files=#&#123;@archive_files&#125;" doit site.source site.keep_files &lt;&lt; @archive_name end end private def doit(source) archive_name_full = "#&#123;source&#125;/#&#123;@archive_name&#125;" archive_exists = File.exist?(archive_name_full) return if archive_exists && @live_reload @log.info "#&#123;archive_name_full&#125; exists? #&#123;archive_exists&#125;" if archive_exists && @force_delete @log.info "Deleting old #&#123;archive_name_full&#125;" File.delete(archive_name_full) end if !archive_exists || @force_delete @log.info "Making #&#123;archive_name_full&#125;" case @archive_type when :tar make_tar(archive_name_full, source) when :zip make_zip(archive_name_full, source) end end return unless File.foreach('.gitignore').grep(/^#&#123;@archive_name&#125;/).any? @log.info "#&#123;@archive_name&#125; not found in .gitignore, adding entry." File.open('.gitignore', 'a') do |f| f.puts File.basename(@archive_name) end end def make_tar(tar_name, source) Dir.mktmpdir do |dirname| @archive_files.each do |filename| fn, filename_full = qualify_file_name(filename, source) @log.info "Copying #&#123;filename_full&#125; to temporary directory #&#123;dirname&#125;; filename=#&#123;filename&#125;; fn=#&#123;fn&#125;" FileUtils.copy(filename_full, dirname) end write_tar(tar_name, dirname) end end def write_tar(tar_name, dirname) # Modified from https://gist.github.com/sinisterchipmunk/1335041/5be4e6039d899c9b8cca41869dc6861c8eb71f13 File.open(tar_name, 'wb') do |tarfile| Gem::Package::TarWriter.new(tarfile) do |tar| Dir[File.join(dirname, '**/*')].each do |filename| write_tar_entry(tar, dirname, filename) end end end end def write_tar_entry(tar, dirname, filename) mode = File.stat(filename).mode relative_file = filename.sub(%r&#123;^#&#123;Regexp.escape dirname&#125;/?&#125;, '') if File.directory?(filename) tar.mkdir relative_file, mode else tar.add_file relative_file, mode do |tf| File.open(filename, 'rb') &#123; |f| tf.write f.read &#125; end end end def make_zip(zip_name, source) require 'zip' Zip.default_compression = Zlib::DEFAULT_COMPRESSION Zip::File.open(zip_name, Zip::File::CREATE) do |zipfile| @archive_files.each do |filename| filename_in_archive, filename_original = qualify_file_name(filename, source) @log.info "make_zip: adding #&#123;filename_original&#125; to #&#123;zip_name&#125; as #&#123;filename_in_archive&#125;" zipfile.add(filename_in_archive, filename_original) end end end # @return tuple of filename (without path) and fully qualified filename def qualify_file_name(path, source) case path[0] when '/' # Is the file absolute? @log.info "Absolute filename: #&#123;path&#125;" [File.basename(path), path] when '!' # Should the file be found on the PATH? clean_path = path[1..-1] filename_full = File.which(clean_path) abort "Error: #&#123;clean_path&#125; is not on the PATH." if filename_full.nil? @log.info "File on PATH: #&#123;clean_path&#125; -> #&#123;filename_full&#125;" [File.basename(clean_path), filename_full] when '~' # Is the file relative to user's home directory? clean_path = path[2..-1] filename_full = File.join(ENV['HOME'], clean_path) @log.info "File in home directory: #&#123;clean_path&#125; -> #&#123;filename_full&#125;" [File.basename(clean_path), filename_full] else # The file is relative to the Jekyll website top-level directory @log.info "Relative filename: #&#123;path&#125;" [File.basename(path), File.join(source, path)] # join yields the fully qualified path end end end </pre> <h3 id="make_archive_install">Installation</h3> <ol> <li> Copy <code>make_archive.rb</code> and <code>logger_factory.rb</code> into the <code>_plugins/</code> directory of your Jekyll site. </li> <li> Restart Jekyll. </li> </ol> </editor-fold make_archive> <editor-fold pre> <h2 id="pre_noselect" class="spaceAbove"><span class="code">pre</span> and <span class="code">noselect</span></h2> <p> This plugin provides 2 tags that frequently work together: </p> <ol> <li>A <code>pre</code> block tag that can optionally display a copy button.</li> <li>A <code>noselect</code> tag that can renders HTML content passed to it unselectable.</li> </ol> <h3 id="preSyntax">Syntax</h3> <pre data-lt-active="false">{% pre [copyButton] %} Contents of pre tag {% endpre %}</pre> <pre data-lt-active="false">{% pre [copyButton] %} {% noselect [text string]%}Contents of pre tag {% endpre %}</pre> <h3 id="preUse1">Usage Example 1</h3> <p> This example does not generate a copy button and does not demonstrate <code>noselect</code>. </p> <pre data-lt-active="false">{% pre %} Contents of pre tag {% endpre %}</pre> <p> Generates: </p> <pre data-lt-active="false">&lt;pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'&gt; Contents of pre tag &lt;/pre&gt;</pre> <p> Which renders as: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'> Contents of pre tag </pre> <h3 id="preUse2">Usage Example 2</h3> <p> This example generates a copy button and does not demonstrate <code>noselect</code>. </p> <pre data-lt-active="false">{% pre copyButton %}Contents of pre tag {% endpre %}</pre> <p> Generates: </p> <pre data-lt-active="false">&lt;pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'&gt;&lt;button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'&gt;&lt;img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'&gt;&lt;/button&gt;Contents of pre tag &lt;/pre&gt;</pre> <p> Which renders as: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>Contents of pre tag </pre> <h3 id="preUse3">Usage Example 3</h3> <p> This example generates a copy button and does demonstrates the default usage of <code>noselect</code>, which renders an unselectable dollar sign followd by a space. </p> <pre data-lt-active="false">{% pre copyButton %} {% noselect %}Contents of pre tag {% endpre %}</pre> <p> Generates: </p> <pre data-lt-active="false">&lt;pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'&gt;&lt;button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'&gt;&lt;img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'&gt;&lt;/button&gt;&lt;span class='unselectable'&gt;$ &lt;/span&gt;Contents of pre tag &lt;/pre&gt;</pre> <p> Which renders as: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>$ </span>Contents of pre tag </pre> <h3 id="preUse4">Usage Example 4</h3> <p> This example generates a copy button and does demonstrates the <code>noselect</code> being used twice: the first time to render an unselectable custom prompt, and the second time to render unselectable output. </p> <pre data-lt-active="false">{% pre copyButton %}{% noselect >>> %}Contents of pre tag {% noselect How now brown cow%} {% endpre %}</pre> <p> Generates: </p> <pre data-lt-active="false">&lt;pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'&gt;&lt;button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'&gt;&lt;img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'&gt;&lt;/button&gt;&lt;span class='unselectable'&gt;&gt;&gt;&gt; &lt;/span&gt;contents of pre tag &lt;span class='unselectable'&gt;How now brown cow&lt;/span&gt;&lt;/pre&gt;</pre> <p> Which renders as: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button><span class='unselectable'>>>> </span>contents of pre tag <span class='unselectable'>How now brown cow</span></pre> <h3 id="preCss">CSS</h3> <p> Here are the CSS declarations that I defined pertaining to the <code>pre</code> and <code>noselect</code> tags: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'><button class='copyBtn' data-clipboard-target='#id${SecureRandom.hex(6)}' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>.copyBtn &#123; -webkit-user-select: none; -moz-user-select: none; -ms-user-select: none; -webkit-appearance: none; background-color: #eee; background-image: linear-gradient(#fcfcfc, #eee); border: 1px solid #d5d5d5; border-radius: 3px; color: #333; cursor: pointer; display: inline-block; float: right; font-size: 13px; font-weight: 700; line-height: 20px; padding: 2px 2px 0 4px;; position: -webkit-sticky; position: sticky; right: 4px; top: 0; user-select: none; z-index: 1; &#125; .copyContainer &#123; position: relative; &#125; .maxOneScreenHigh &#123; max-height: 500px; &#125; .unselectable &#123; color: #7922f9; -moz-user-select: none; -khtml-user-select: none; user-select: none; &#125; </pre> <h3 id="fyi">Comprehensive Example</h3> <p> The code I wrote to generate the above CSS was a good example of how the plugins work together: </p> <pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='id${SecureRandom.hex(6)}'> {% capture css %}{% flexible_include '_sass/mystyle.scss' %}{% endcapture %} {% pre copyButton %}{{ css | from: '.copyBtn' | to: '^$' | strip }} {{ css | from: '.copyContainer' | to: '^$' | strip }} {{ css | from: '.maxOneScreenHigh' | to: '^$' | strip }} {{ css | from: '.unselectable' | to: '^$' | strip }} {% endpre %} </pre> <h3 id="preSource">Source Code</h3> <p> <a href='/jekyll/doc/PreTag.html' target='_blank'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/pre.rb" download="pre.rb" title="Click on the file name to download the file">pre.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id1582a10573c3"><button class='copyBtn' data-clipboard-target='#id1582a10573c3' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 require 'securerandom' module PreTag # """ # \\&#123;% pre %&#125; # Content here # \\&#123;% endpre %&#125; # # \\&#123;% pre copyButton %&#125; # Content here # \\&#123;% endpre %&#125;""" class PreTagBlock &lt; Liquid::Block @@prefix = "&lt;button class='copyBtn' data-clipboard-target=" @@suffix = " title='Copy to clipboard'>&lt;img src='/assets/images/clippy.svg' " \ "alt='Copy to clipboard' style='width: 13px'>&lt;/button>" def self.make_copy_button(pre_id) "#&#123;@@prefix&#125;'##&#123;pre_id&#125;'#&#123;@@suffix&#125;" end def self.make_pre(make_copy_button, content) pre_id = 'id$&#123;SecureRandom.hex(6)&#125;' copy_button = make_copy_button ? PreTagBlock.make_copy_button(pre_id) : '' "&lt;pre data-lt-active='false' class='maxOneScreenHigh copyContainer' id='#&#123;pre_id&#125;'>#&#123;copy_button&#125;#&#123;content&#125;&lt;/pre>" end # Constructor. # @param tag_name [String] is the name of the tag, which we already know. # @param text [Hash, String, Liquid::Tag::Parser] the arguments from the web page. # @param tokens [Liquid::ParseContext] tokenized command line # @return [void] def initialize(tag_name, text, tokens) super(tag_name, text, tokens) text = '' if text.nil? @make_copy_button = text.strip! == 'copyButton' end # Method prescribed by the Jekyll plugin lifecycle. # @return [String] def render(context) content = super PreTagBlock.make_pre(@make_copy_button, content) end end # """\\&#123;% noselect %&#125; or \\&#123;% noselect this all gets copied. # Also, space before the closing percent is signficant %&#125;""" class UnselectableTag &lt; Liquid::Tag def initialize(tag_name, text, tokens) super(tag_name, text, tokens) @content = text # puts "UnselectableTag: content1= '#&#123;@content&#125;'" @content = '$ ' if @content.nil? || @content.empty? # puts "UnselectableTag: content2= '#&#123;@content&#125;'" end def render(_) "&lt;span class='unselectable'>#&#123;@content&#125;&lt;/span>" end end end Liquid::Template.register_tag('pre', PreTag::PreTagBlock) Liquid::Template.register_tag('noselect', PreTag::UnselectableTag) </pre> <h3 id="preInstall">Installation</h3> <ol> <li> Copy <code>pre.rb</code> and <code>logger_factory.rb</code> into the <code>_plugins/</code> directory of your Jekyll site. </li> <li> Restart Jekyll. </li> </ol> </editor-fold pre> <editor-fold random_hex_string> <h2 id="random_hex_string" class="spaceAbove"><span class="code">random_hex_string</span></h2> <p> This Liquid filter generates a random hexadecimal string of any length. Each byte displays as two characters. You can specify the number of bytes in the hex string; if you do not, 6 random bytes (12 characters) will be generated. </p> <h3 id="random_hex_string_usage">Usage Example</h3> <p> This example generates a random hex string 6 bytes long and stores the result in a Liquid variable called <code>id</code>. Both of the following do the same thing: </p> <pre data-lt-active="false"> {% assign id = random_hex_string %} {% assign id = random_hex_string 6 %} </pre> <p> The generated 6 bytes (12 characters) might be: <code>4d1e0d1296bf</code>. </p> <h3 id="random_hex_string_source">Source Code</h3> <p> <a href='/jekyll/doc/RandomHex.html' target='_blank'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/random_hex.rb" download="random_hex.rb" title="Click on the file name to download the file">random_hex.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id588670e03042"><button class='copyBtn' data-clipboard-target='#id588670e03042' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 module RandomHex # Outputs a string of random hexadecimal characters of any length. # Defaults to a six-character string. # @example Generate 6 random characters. # &#123;&#123; random_hex_string &#125;&#125; # @example Generate 20 random characters. # &#123;&#123; random_hex_string 10 &#125;&#125; class RandomNumberTag &lt; Liquid::Tag # Called by Jekyll only once to register the module. # @param tag_name [String] Describe this parameter's purpose # @param text [String] Describe this parameter's purpose # @param context [String] Describe this parameter's purpose # @return [String, nil] Describe the return value def initialize(tag_name, text, context) super(tag_name, text, context) text.to_s.strip! if text.empty? @n = 6 else tokens = text.split(' ') abort "random_hex_string error - more than one token was provided: '#&#123;text&#125;'" if tokens.length > 1 not_integer = !Integer(text, exception: false) abort "random_hex_string error: '#&#123;text&#125;' is not a valid integer" if not_integer @n = text.to_i end end def render(_) require 'securerandom' SecureRandom.hex(@n) end end end Liquid::Template.register_tag('random_hex_string', RandomHex::RandomNumberTag) </pre> <h3 id="random_hex_string_install">Installation</h3> <ol> <li> Copy <code>random_hex.rb</code> and <code>logger_factory.rb</code> into the <code>_plugins/</code> directory of your Jekyll site. </li> <li> Restart Jekyll. </li> </ol> </editor-fold random_hex_string> <editor-fold site_inspector> <h2 id="site_inspector" class="spaceAbove"><span class="code">site_inspector</span></h2> <p> Dumps lots of information from <code>site</code> when enabled by the <code>site_inspector</code> setting in <code>_config.yml</code>. </p> <h3 id="site_inspector_syntax"><span class="code">_config.yml</span> Syntax</h3> <pre data-lt-active="false">site_inspector: true # Run in development mode</pre> <pre data-lt-active="false">site_inspector: force # Run in development and production modes</pre> <pre data-lt-active="false">site_inspector: false # The default is to not run</pre> <h3 id="site_inspector_output">Sample Output</h3> <pre data-lt-active="false" class="maxOneScreenHigh"> site is of type Jekyll::Site site.time = 2020-10-05 05:18:27 -0400 site.config['env']['JEKYLL_ENV'] = development site.collections.posts site.collections.expertArticles site.config.source = '/mnt/_/www/www.mslinn.com' site.config.destination = '/mnt/_/www/www.mslinn.com/_site' site.config.collections_dir = '' site.config.plugins_dir = '_plugins' site.config.layouts_dir = '_layouts' site.config.data_dir = '_data' site.config.includes_dir = '_includes' site.config.collections = '{"posts"=>{"output"=>true, "permalink"=>"/blog/:year/:month/:day/:title:output_ext"}, "expertArticles"=>{"output"=>true, "relative_directory"=>"_expertArticles", "sort_by"=>"order"}}' site.config.safe = 'false' site.config.include = '[".htaccess"]' site.config.exclude = '["_bin", ".ai", ".git", ".github", ".gitignore", "Gemfile", "Gemfile.lock", "script", ".jekyll-cache/assets"]' site.config.keep_files = '[".git", ".svn", "cloud9.tar"]' site.config.encoding = 'utf-8' site.config.markdown_ext = 'markdown,mkdown,mkdn,mkd,md' site.config.strict_front_matter = 'false' site.config.show_drafts = 'true' site.config.limit_posts = '0' site.config.future = 'true' site.config.unpublished = 'false' site.config.whitelist = '[]' site.config.plugins = '["classifier-reborn", "html-proofer", "jekyll", "jekyll-admin", "jekyll-assets", "jekyll-docs", "jekyll-environment-variables", "jekyll-feed", "jekyll-gist", "jekyll-sitemap", "kramdown"]' site.config.markdown = 'kramdown' site.config.lsi = 'false' site.config.excerpt_separator = ' ' site.config.incremental = 'true' site.config.detach = 'false' site.config.port = '4000' site.config.host = '127.0.0.1' site.config.baseurl = '' site.config.show_dir_listing = 'false' site.config.permalink = '/blog/:year/:month/:day/:title:output_ext' site.config.paginate_path = '/page:num' site.config.timezone = '' site.config.quiet = 'false' site.config.verbose = 'false' site.config.defaults = '[]' site.config.liquid = '{"error_mode"=>"warn", "strict_filters"=>false, "strict_variables"=>false}' site.config.rdiscount = '{"extensions"=>[]}' site.config.redcarpet = '{"extensions"=>[]}' site.config.kramdown = '{"auto_ids"=>true, "toc_levels"=>"1..6", "entity_output"=>"as_char", "smart_quotes"=>"lsquo,rsquo,ldquo,rdquo", "input"=>"GFM", "hard_wrap"=>false, "footnote_nr"=>1, "show_warnings"=>false}' site.config.author = 'Mike Slinn' site.config.compress_html = '{"blanklines"=>false, "clippings"=>"all", "comments"=>["<!-- ", " -->"], "endings"=>"all", "ignore"=>{"envs"=>["development"]}, "profile"=>false, "startings"=>["html", "head", "body"]}' site.config.email = 'mslinn@mslinn.com' site.config.feed = '{"categories"=>["AI", "Blockchain", "Scala", "Software-expert"]}' site.config.ignore_theme_config = 'true' site.config.site_inspector = 'false' site.config.make_archive = '[{"archive_name"=>"cloud9.tar", "delete"=>true, "files"=>["!killPortFwdLocal", "!killPortFwdOnJumper", "!tunnelToJumper"]}]' site.config.sass = '{"style"=>"compressed"}' site.config.title = 'Mike Slinn' site.config.twitter = '{"username"=>"mslinn", "card"=>"summary"}' site.config.url = 'http://localhost:4000' site.config.livereload = 'true' site.config.livereload_port = '35729' site.config.serving = 'true' site.config.watch = 'true' site.config.assets = '{}' site.config.tag_data = '[]' site.keep_files: [".git", ".svn", "cloud9.tar"] </pre> <h3 id="site_inspector_source">Source Code</h3> <p> <a href='/jekyll/doc/SiteInspector.html' target='_blank'>Yard docs are here.</a> </p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_plugins/site_inspector.rb" download="site_inspector.rb" title="Click on the file name to download the file">site_inspector.rb</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="idaeb291803e64"><button class='copyBtn' data-clipboard-target='#idaeb291803e64' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button># frozen_string_literal: true # @author Copyright 2020 &#123;https: https://www.mslinn.com Michael Slinn&#125; # @license SPDX-License-Identifier: Apache-2.0 # # Dumps lots of information from `site` if in `development` mode and `site_inspector: true` in `_config.yml`. class SiteInspector &lt; Jekyll::Generator require_relative 'logger_factory' def initialize(config) super(config) @log = LoggerFactory.new.create_logger('site_inspector', config, :warn, $stderr) end # Displays information about the Jekyll site # # @param site [Jekyll.Site] Automatically provided by Jekyll plugin mechanism # @return [void] def generate(site) mode = site.config['env']['JEKYLL_ENV'] config = site.config['site_inspector'] return if config.nil? inspector_enabled = config != false return unless inspector_enabled force = config == 'force' return unless force || mode == 'development' @log.info "site is of type #&#123;site.class&#125;" @log.info "site.time = #&#123;site.time&#125;" @log.info "site.config['env']['JEKYLL_ENV'] = #&#123;mode&#125;" site.collections.each do |key, _| puts "site.collections.#&#123;key&#125;" end # key env contains all environment variables, quite verbose so output is suppressed site.config.sort.each &#123; |key, value| @log.info "site.config.#&#123;key&#125; = '#&#123;value&#125;'" unless key == 'env' &#125; site.data.sort.each &#123; |key, value| @log.info "site.data.#&#123;key&#125; = '#&#123;value&#125;'" &#125; # site.documents.each &#123;|key, value| @log.info "site.documents.#&#123;key&#125;" &#125; # Generates too much output! @log.info "site.keep_files: #&#123;site.keep_files.sort&#125;" # site.pages.each &#123;|key, value| @log.info "site.pages.#&#123;key&#125;'" &#125; # Generates too much output! # site.posts.each &#123;|key, value| @log.info "site.posts.#&#123;key&#125;" &#125; # Generates too much output! site.tags.sort.each &#123; |key, value| @log.info "site.tags.#&#123;key&#125; = '#&#123;value&#125;'" &#125; end end </pre> <h3 id="site_inspector_install">Installation</h3> <ol> <li> Copy <code>site_inspector.rb</code> and <code>logger_factory.rb</code> into the <code>_plugins/</code> directory of your Jekyll site. </li> <li> Restart Jekyll. </li> </ol> </editor-fold site_inspector> Bash Script to Create a New Jekyll Post 2020-08-16T00:00:00-04:00 https://mslinn.github.io/blog/2020/08/16/new-jekyll-post <p> I use <a href="https://jekyllrb.com/" target="_blank" rel="nofollow">Jekyll</a> to build this website. Some material is published as articles, some as blog posts. I wrote a script called <code>newPost</code> that creates a new draft blog post with SEO considerations. SEO rankings are improved when the description and title tag are neither too long nor too short. </p> <h2 id="usage">Sample Usage</h2> <p> Here is an example of how I used it on 2020-08-16: </p> <pre data-lt-active='false'> <span class="unselectable">$ </span>_bin/newPost Post Title (30-60 characters): ______________________________123456789012345678901234567890 This is a test of newPost for the greater good 46 characters, excellent! Publication date: 2020-08-16 Post Description (30-60 characters): ____________________________________________________________123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890 I wish newPost used some sort of web service to generate an SEO-optimized description 85 characters, excellent! Post Categories (comma delimited): Post Tags (comma delimited): Post Keywords (comma delimited): Goofiness, Silliness Banner image (bg_ .jpg): </pre> <p> For the above example the generated file is called <code> _drafts/2020-08-16-this-is-a-test-of-newpost-for-the-greater-good.html</code>. When you are happy with the new posting, move it from <code>_drafts</code> to <code>_posts</code> like this: </p> <pre data-lt-active='false'> <span class="unselectable">$ </span>mv _drafts/2020-08-16-new-jekyll-post.html _posts/ </pre> <h3 id="generated">Generated Posting</h3> <p> Here is the generated file: </p> <pre data-lt-active='false'> --- categories: [] description: I wish newPost used some sort of web service to generate an SEO-optimized description image: keywords: [Goofiness, Silliness] last_modified_at: 2020-08-16 layout: blog title: This is a test of newPost for the greater good tags: [] --- </pre> <h2 id="error">Error Handling</h2> <p> The script checks the length of the title and the posting description for SEO purposes. If either of these are too long or too short, the script allows the user to edit their input over and over until they get it right. For example, here you can see that at first the user just types in <code>xx</code> for the title, then they provide a string that is too long, then they edit it until it has an acceptable length: </p> <pre data-lt-active='false'> <span class="unselectable">$ </span>_bin/newPost Post Title (30-60 characters): ______________________________123456789012345678901234567890 xx 28 characters too short, please edit Post Title (30-60 characters): ______________________________123456789012345678901234567890 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 14 characters too long, please edit Post Title (30-60 characters): ______________________________:123456789012345678901234567890 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 59 characters, excellent! </pre> <h2 id="code">Source Code</h2> <p>This is the source code for the <code>newPost</code> bash script.</p> <div class="codeLabel"><a href="data:text/plain;charset=UTF-8,/_bin/newPost" download="newPost" title="Click on the file name to download the file">newPost</a></div> <pre data-lt-active="false" class="maxOneScreenHigh copyContainer" id="id518e99672842"><button class='copyBtn' data-clipboard-target='#id518e99672842' title='Copy to clipboard'><img src='/assets/images/clippy.svg' alt='Copy to clipboard' style='width: 13px'></button>#!/bin/bash # This bash script will setup a new HTML Jekyll draft blog post and open it for editing in Notepad++ # # See https://www.mslinn.com/blog/2020/08/16/new-jekyll-post.html # # Copyright Original by Katie Harron - @pibby from https://gist.github.com/pibby/6911493 # Copyright 2020 Modified by Mike Slinn - mslinn@mslinn.com # - added length checks for title and description # - added reprompt for when length check fails # - launch notepad instead of Mac editor # - modified front matter # - Added optional date as command line parameter # - Added date modification dialog # # SPDX-License-Identifier: Apache-2.0 function checkLength &#123; # $1 - minimum length # $2 - maximum length # $3 - string to test length=$&#123;#3&#125; if (( length &lt; $1 )); then >&2 echo "$(($1 - $length)) characters too short, please edit" return 1 elif (( length > $2 )); then >&2 echo "$(( $length - $2 )) characters too long, please edit" return 2 else >&2 echo "$length characters, excellent!" return 0 fi &#125; function edit &#123; if [ `which notepad` ]; then notepad "$filename" # Invokes mslinn's notepad++ script elif [ `which gedit` ]; then `which gedit` "$filename" else echo "No editor defined, please edit '$filename' somehow" exit 1 fi &#125; function emit_array &#123; if [ "$2" ]; then echo "$1: [$2]\n" else echo -n "" fi &#125; function emit_scalar &#123; if [ "$2" ]; then echo "$1: $2\n" else echo -n "" fi &#125; function reprompt &#123; # $1 - name of front matter variable # $2 - minimum length of user-provided value # $3 - maximum length of user-provided value if [ -z "$1" ]; then 2> echo "Error: no front matter variable provided"; exit 1; fi if [ -z "$2" ]; then 2> echo "Error: no minimum length provided"; exit 1; fi if [ -z "$3" ]; then 2> echo "Error: no maximum length provided"; exit 1; fi NAME="$1" MIN="$2" MAX="$3" LEADIN="'_%0.s'" SPACES="$( eval $(echo printf "$LEADIN" &#123;1..$MIN&#125;) )" (( COUNT= (($MAX * 10) - ($MIN * 10) + 5) / 10 )) NUMBERS="$( eval $(echo printf '"0123456789%0.s"' &#123;1..$COUNT&#125;) )" (( CHARS= $MAX - $MIN )) VALUE="" #>&2 printf "SPACES='$SPACES'\n" #>&2 printf "NUMBERS='$NUMBERS'\n" while : do >&2 printf "Post $1 (30-60 characters):\n$SPACES$&#123;NUMBERS:1:$CHARS&#125;\n" read -e -i "$VALUE" VALUE if $(checkLength "$MIN" "$MAX" "$VALUE"); then break; fi done >&2 echo echo "$VALUE" &#125; # Set cwd to project root GIT_ROOT="$( git rev-parse --show-toplevel )" cd "$&#123;GIT_ROOT&#125;" if [ ! -f _bin/loadConfigEnvVars ]; then echo -e "Error: _bin/loadConfigEnvVars was not found.\ncd \$msp" exit 1 fi source _bin/loadConfigEnvVars title="$( reprompt Title 30 60 )" ptitle=$&#123;title// /-&#125; # convert spaces in title to hyphens # Convert title to lowercase and remove slashes plc="$( echo "$ptitle" | tr '[:upper:]' '[:lower:]' | tr -d '/' )" if [ "$1" ]; then pdate="$1" # TODO verify $1 is YYYY-mm-dd else pdate="$( date +%Y-%m-%d )" # create date as YYYY-mm-dd fi read -p 'Publication date: ' -e -i "$pdate" pdate filename=_drafts/$pdate-$plc.html # location to create the new file as year-month-day-title.md mkdir -p _drafts touch "$filename" # create the new blank post desc="$( reprompt Description 60 150 )" printf "Post Categories (comma delimited): "; read categories printf "Post Tags (comma delimited): "; read tags #printf "Post Keywords (comma delimited): "; read keyw printf "Banner image (.png & .webp): "; read img printf "Enable clipboard (y/N): "; read clipboard if [[ "$clipboard" =~ ^(y|Y).* ]]; then javascript="/assets/js/clipboard.min.js" javascriptInline="new ClipboardJS('.copyBtn');" fi contents="---\n" contents="$contents$( emit_array categories "$categories" )" contents="$contents$( emit_scalar description "$desc" )" contents="$contents$( emit_scalar image "$img" )" contents="$contents$( emit_scalar javascript "$javascript" )" contents="$contents$( emit_scalar javascriptInline "$javascriptInline" )" #contents="$contents$( emit_array keywords "$keyw" )" contents="$contents$( emit_scalar last_modified_at "$pdate" )" contents="$contents$( emit_scalar layout blog )" contents="$contents$( emit_array tags "$tags" )" contents="$contents$( emit_scalar title "$title" )" contents="$contents---\n" echo -e "$contents" | sed '/^$/d' > "$filename" # fill out YAML Front Matter and insert into the new file echo "Created '$filename'" #edit printf "Use mem to append code examples to this post (Y/n): "; read clipboard if [[ "$clipboard" =~ ^(y|Y|$).* ]]; then ps ax | grep '[j]ekyll' | awk -F ' ' '&#123;print $1&#125;' | xargs sudo kill -15 _bin/mem "$filename" & _bin/serve -c fi </pre> Converting All Images in a Website to webp Format 2020-08-15T00:00:00-04:00 https://mslinn.github.io/blog/2020/08/15/converting-all-images-to-webp-format <p> I first launched this website in 1996. Since then, it has been re-incarnated using many different technologies. Presently I use <a href="https://jekyllrb.com/" target="_blank" rel="nofollow">Jekyll</a> to assemble the site, then push the image to a web-enabled AWS S3 bucket that is edge-cached by an AWS CloudFront distribution. </p> <p> Until yesterday, the site contained images with a mixture of image formats. I decided to convert them all to the new <a href="https://developers.google.com/speed/webp" target="_blank" rel='nofollow'><code>webp</code></a> format. Because there are hundreds of images in over 120 web pages, I wrote a bash script called <code>toWebP</code> to do the work. This posting provides the <code>toWebP</code> script plus instructions on how you could use it for your website. </p> <p> The script converts image types <code>gif</code>, <code>jpg</code>, <code>jpeg</code>, <code>png</code>, <code>tif</code>, and <code>tiff</code>. It also modifies the HTML pages, CSS and SCSS that reference those images. </p> <p> The conversions are set for maximum fidelity (lossless where possible), and maximum compression. This means the images look great and load quickly. </p> <h3>Caveat</h3> <p> The script assumes that all images are local to your website, which makes sense because the converted images need to be stored, and local storage is the only sensible option. It renames all references to images in HTML, CSS and SCSS files to <code>webp</code> format. If the images are remote (for example, on a CDN), they are not converted, but the image file types in the HTML, CSS and SCSS are adjusted anyway. I suppose I could fix the script, but I don't need to do that for myself. If someone needs that feature, go ahead and enhance the script... and please provide me the enhanced script, so I can update this blog posting. </p> <h2 id="prerequisites">Prerequisites</h2> <p> You need to install the WebP package.<br> </p> <h3 id="mac">Mac</h3> <p> Use <a href="https://formulae.brew.sh/formula/webp" target="_blank" rel='nofollow'>Homebrew</a> or <a href="https://ports.macports.org/?search=webp&search_by=name" target="_blank" rel='nofollow'>Macports</a>. </p> <h3 id="ubuntu">Ubuntu (this is the default Linux distribution for Windows Subsystem for Linux)</h3> <p>At a shell prompt type:</p> <pre data-lt-active='false'><span class="unselectable">$ </span>yes | sudo apt install webp</pre> <h2 id="running">Running <span class="code">toWebp</span></h2> <p> The program may emit warnings when it runs. Those warnings can be safely ignored. </p> <p> Hopefully, your website is managed by git. I suggest that you commit your work before running the script. That way if something goes wrong you just have to type <code>git stash</code> to return your website to its previous state. </p> <h3 id="usage">Usage</h3> <p>The general form of the command to convert all images and modify the HTML pages that they are referenced from is:</p> <pre data-lt-active='false'><span class="unselectable">$ </span>toWebp &lt;directoryName></pre> <h3 id="examples">Examples</h3> <p>To convert the website (images, html, scss & css) rooted at the current directory, type:</p> <pre data-lt-active='false'><span class="unselectable">$ </span>toWebp .</pre> <p>To convert the website called <code>mySite</code> rooted under your home directory, type:</p> <pre data-lt-active='false'><span class="unselectable">$ </span>toWebp ~/mySite</pre> <p>To just convert 1 specific image to <code>webp</code>, type:</p> <pre data-lt-active='false'><span class="unselectable">$ </span>toWebp images/blah.jpg</pre> <h2 id="gist">Gist Containing the <code>toWebP</code> Bash Script.</h2> <p>Put this file in one of the directories on your <code>PATH</code>, for example <code>/usr/local/bin</code>: <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/d87f13c921456a21070c4d96366c6778.js"> </script> <h3 id="chmod">Make it Executable</h3> <p> Remember to make the <code>toWebp</code> script executable before trying to use it: </p> <pre data-lt-active='false'><span class="unselectable">$ </span>chmod a+x /usr/local/bin/toWebp</pre> Dotty (Scala 3 Preview) Presentation at Hopper, Montreal 2019-11-28T00:00:00-05:00 https://mslinn.github.io/blog/2019/11/28/dotty-scala-3-preview <div style=""> <picture> <source srcset="/blog/images/dottyLambda_690x388.webp" type="image/webp"> <source srcset="/blog/images/dottyLambda_690x388.png" type="image/png"> <img src="/blog/images/dottyLambda_690x388.png" title="Mike Slinn presents" class=" liImg " alt="Mike Slinn presents" /> </picture> </div> <div style="text-align: center"> <p> Yesterday I presented <a href="https://www.meetup.com/lambda-montreal/events/266306046/" target="_blank" rel="nofollow">Dotty (Scala 3 Preview)</a> to Lambda Montreal. </p> <p> The slides are <a href="https://www.slideshare.net/mslinn/dotty-scala-3-preview" target="_blank" rel="nofollow">here</a>. </p> <p> The code is <a href="https://github.com/mslinn/dotty-example-project/" target="_blank" rel="nofollow">here</a>. </p> <p> The video recording is <a href="https://www.youtube.com/watch?v=7S68TY0S2e0" target="_blank" rel="nofollow">here</a>. </p> </div> <div style="text-align: center;"> <picture> <source srcset="/blog/images/lambdaMontreal.webp" type="image/webp"> <source srcset="/blog/images/lambdaMontreal.png" type="image/png"> <img src="/blog/images/lambdaMontreal.png" title="Mike Slinn presents" class="center quartersize liImg2 rounded shadow" alt="Mike Slinn presents" /> </picture> </div> A Hybrid Machine Learning / Personality Simulation Platform 2019-10-24T00:00:00-04:00 https://mslinn.github.io/blog/2019/10/24/hybrid-ml-simulation <div style="text-align: center;"> <a href="https://www.meetup.com/MTL-Machine-Learning/events/265039754/" target="_blank" rel="nofollow"><picture> <source srcset="/blog/images/mtlMLconference.webp" type="image/webp"> <source srcset="/blog/images/mtlMLconference.png" type="image/png"> <img src="/blog/images/mtlMLconference.png" class="center liImg2 rounded shadow" /> </picture></a> </div> <p> Yesterday I presented <a href="https://www.meetup.com/MTL-Machine-Learning/events/265039754/" target="_blank" rel="nofollow"> &ldquo;EmpathyWorks: A Hybrid Machine Learning / Personality Simulation Platform&rdquo; </a> to the <a href="https://www.meetup.com/MTL-Machine-Learning/events/265039754/" target="_blank" rel="nofollow">Fall 2019 Montreal Machine Learning Mini-Conference.</a> </p> <p> The slides are <a href="https://www.slideshare.net/mslinn/empathyworks-towards-an-eventbased-simulationml-hybrid-platform" target="_blank" rel="nofollow">here</a>. The video recording is <a href="https://youtu.be/PiDsiyJIMmo" target="_blank" rel="nofollow">here</a>. </p> <div style="text-align: center"> <div style="display: inline-block; margin: 0.5em; vertical-align: top;"> <picture> <source srcset="/assets/images/robotCircle207x207.webp" type="image/webp"> <source srcset="/assets/images/robotCircle207x207.png" type="image/png"> <img src="/assets/images/robotCircle207x207.png" title="Mike Slinn presents" class=" liImg " alt="Mike Slinn presents" /> </picture> </div> <div style="display: inline-block; margin: 0.5em; vertical-align: top;"> <picture> <source srcset="/blog/images/montrealMachineLearningMeetup.webp" type="image/webp"> <source srcset="/blog/images/montrealMachineLearningMeetup.png" type="image/png"> <img src="/blog/images/montrealMachineLearningMeetup.png" title="Montreal Machine Learning Meetup" class=" quartersize liImg2 rounded shadow" style="margin-left: 3em; margin-top: 3em;" alt="Montreal Machine Learning Meetup" /> </picture> </div> </div> Decentralized Ponytails 2018-09-13T00:00:00-04:00 https://mslinn.github.io/blog/2018/09/13/decentralized-ponytails <p> I&rsquo;d like to point out the similarity of the early days of the open-source movement with today&rsquo;s decentralized blockchain movement. </p> <p> Open-source software was brought to mainstream attention during the last technology bubble at the end of the last millennium. The open-source software movement had a loyal cadre of zealots who believed that their cause would overcome any need for a business case. Sun Microsystems was the hardware company whose servers powered the Internet, and their software included the Java programming language, plus many other important networking-related products. Sun's slogan was &ldquo;The network is the computer&rdquo;. </p> <div style="text-align: center;"> <picture> <source srcset="/assets/images/Sun-Logo_225x99.webp" type="image/webp"> <source srcset="/assets/images/Sun-Logo_225x99.png" type="image/png"> <img src="/assets/images/Sun-Logo_225x99.png" title="Sun Microsystems logo" class="center quartersize liImg2 rounded shadow" style="padding: 1em" alt="Sun Microsystems logo" /> </picture> </div> <p> Jonathan Schwartz, the CEO of Sun Microsystems was one of the open-source zealots. He was famous for his ponytail. Unfortunately, zealotry and dogma is bad for business, and as a result Sun Microsystems is no longer with us. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/jonathanSchwartz.webp" type="image/webp"> <source srcset="/blog/images/jonathanSchwartz.png" type="image/png"> <img src="/blog/images/jonathanSchwartz.png" title="Jonathan Schwartz and his ponytail" class="center liImg rounded shadow" alt="Jonathan Schwartz and his ponytail" /> </picture> </div> <p> Eventually companies like <a href="https://redhat.com" target="_blank" rel='nofollow'>Red Hat</a> developed solid business models for open-source software, but that took years to develop. Today we see many companies attempting using decentralized blockchain technology to create cryptocurrencies, other token-based economies, and evangelizing decentralized dogma without a solid business case. Most of these ventures will die a horrible death, and the investors will get nothing. It will take years for solid business models based on decentralization to be proven. </p> <p> Mr. Schwartz's ponytail was the fashion statement that fueled the YouTube parody below. For background, <a href="https://en.wikipedia.org/wiki/Scott_McNealy" target="_blank" rel='nofollow'>Scott McNealy</a> was the previous CEO at Sun Microsystems. </p> <iframe width="690" height="388" src="https://www.youtube.com/embed/5r3JSciJf5M" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen class="rounded shadow liImg"></iframe> <p> Full disclosure: I also had a ponytail in 2008, and for a few years I had my Sun Spark 2 workstation at home. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/mikeclose3.webp" type="image/webp"> <source srcset="/blog/images/mikeclose3.png" type="image/png"> <img src="/blog/images/mikeclose3.png" title="Mike Slinn and his ponytail back in 2008" class="center quartersize liImg2 rounded shadow" alt="Mike Slinn and his ponytail back in 2008" /> </picture> </div> <p> Here is a transcription of the video, which I paraphrased for clarity:</p> </p> <blockquote> <p><b>Steve Gilmore:</b> Hi, this is Steve Gilmore and this is a video special edition of the Gilmore gang. I'm here with Jonathan Schwartz. It's a great pleasure - it's been a long long time coming - I haven't seen Jonathan for quite a while. Jonathan Schwartz, who is the president and CEO of Sun Microsystems, agreed to sit down for the first time in three or four years and talk about what's going on with Sun. I wanna start, Jonathan, by thanking you for joining us. </p> <p><b>Jonathan Schwartz:</b> Thank you for having me, Steve. It has been a long time, nice to see you. </p> <p><b>Steve Gilmore:</b> So, you know there's been a lot of turmoil on Wall Street as I know you know. </p> <p><b>Jonathan Schwartz:</b> Yes. </p> <p><b>Steve Gilmore:</b> What's your take on that? </p> <p><b>Jonathan Schwartz:</b> Well, I think that it's a cyclical thing as you know Sun was been very prepared for this because we took <a href="https://www.thestreet.com/story/10334514/1/pipe-deal-shines-up-sun.html" target="_blank" rel='nofollow'>three quarters of a billion dollars off of KKR</a> a few years ago so that's in our war chest and I think that we're in a very good position moving forward, Steve. </p> <p><b>Steve Gilmore:</b> Ahh, and specifically what are you doing? </p> <p><b>Jonathan Schwartz:</b> Well, what we're doing is as you know Sun has always been very proactive in the open-source movement. I know you're very familiar with that. What we want to do to help our customers during this very difficult time is keep up with that trend of open-source. So I'm actually very pleased to announce, Steve, that Sun is starting a new open-source initiative. We're going to be releasing the source code to my ponytail as open-source, Steve. </p> <p><b>Steve Gilmore:</b> How's that going to help the situation? </p> <p><b>Jonathan Schwartz:</b> It's open-source, Steve. </p> <p><b>Steve Gilmore:</b> Yeah. </p> <p><b>Jonathan Schwartz:</b> It's my ponytail, Steve. </p> <p><b>Steve Gilmore:</b> You know, we've had this conversation in the past. </p> <p><b>Jonathan Schwartz:</b> Yes. </p> <p><b>Steve Gilmore:</b> Something is open-source, fine, and you get a lot of adoption, you get a lot of exposure in the in the marketing arena... </p> <p><b>Jonathan Schwartz:</b> Yes. </p> <p><b>Steve Gilmore:</b> ... but how do you make money on your ponytail? </p> <p><b>Jonathan Schwartz:</b> Well, what we're going to be doing is releasing my ponytail as open-source. So what we're hoping is that our developers take my ponytail and develop some kind of revenue stream with my ponytail. Did I mention this is open-source, Steve? </p> <p><b>Steve Gilmore:</b> Yeah, so how do you open-source a ponytail? What does that mean? </p> <p><b>Jonathan Schwartz:</b> Well, basically what we do is, we will have some of our best and brightest engineers here at Sun go through my ponytail and find out the unique attributes about what makes my ponytail so successful in the valley. And what we've done Steve, we've open-sourced my ponytail, Steve. </p> <p><b>Steve Gilmore:</b> Jonathan, you're just repeating this over and over again; it doesn't necessarily arrive at a business model. </p> <p><b>Jonathan Schwartz:</b> Umm, Steve? </p> <p><b>Steve Gilmore:</b> Yeah. </p> <p><b>Jonathan Schwartz:</b> We're gonna take my ponytail right and make it open-source. Now I know that this is a big concept for you but I really think it's a game changer, Steve. </p> <p><b>Steve Gilmore:</b> So, who do you see as your competition in the open-source ponytail arena? </p> <p><b>Jonathan Schwartz:</b> I think that we pretty much have it locked up. I don't see anybody who can compete with Sun Microsystems, when it comes to the open-source ponytail market. I think that we're in very good shape, Steve. How much do you miss Scott McNealy? </p> <p><b>Steve Gilmore:</b> Right now, a lot. </p> <p><b>Jonathan Schwartz:</b> Not nearly as much as I do, Steve. </p> <p><b>Steve Gilmore:</b> Okay, so what are you gonna do about, uh, you've laid off a lot of people in the last few months. </p> <p><b>Jonathan Schwartz:</b> Yes, it's going well in fact we have another round of layoffs coming. Once the ponytail is released into the wild, we'll be releasing the team that open-sourced my ponytail, Steve. </p> <p><b>Steve Gilmore:</b> Well you know I have to say that I was hoping for something a little bit more visionary from you Jonathan. </p> <p><b>Jonathan Schwartz:</b> Well I think that this is quite visionary. I don't think that IBM will be releasing a ponytail, and if they did it certainly wouldn't be open-source, Steve. We are very, very excited about our open-source ponytail program if you want you can go to <code>sunmicrosystems.com/ponytail</code>. Any other questions, Steve? </p> <p><b>Steve Gilmore:</b> Yeah I got one that I hope will be a stumper for you, which is a do you see the relationship between your open-source ponytail strategy and micro-messaging as popularized by Twitter? </p> <p><b>Jonathan Schwartz:</b> I'd like to open the pipe to the ponytail. Except as you know, Twitter is not exactly handling XMPP correctly at this time. I don't see the correlation between an open-source ponytail, and a closed-off micro-blogging system. Steve, I think that the ponytail is much bigger than Twitter. </p> <p><b>Steve Gilmore:</b> And the business model again? </p> <p><b>Jonathan Schwartz:</b> Let me see this real slowly and clearly: OPEN. SOURCE. PONYTAIL. </p> <p><b>Steve Gilmore:</b> This has been Jonathan Schwartz along with me, Steve Gilmore. Good luck, Jon. </p> <p><b>Jonathan Schwartz:</b> Thank you Steve. God, I miss McNealy. Think it's going to work? </p> <p><b>Steve Gilmore:</b> No. </p> </blockquote> IBM Personality Insights 2018-08-29T00:00:00-04:00 https://mslinn.github.io/blog/2018/08/29/personality-assessment <p> <a href="https://www.empathyworks.ai" rel="nofollow">empathyworks.ai</a> describes the original research I've done on modeling personality and behavior of individuals and groups since 2007. For me, the work has been both therapeutic and insightful, and I now have a better basis for understanding myself and others as a result. </p> <p> Recently, an IBM employee pointed me to <a href="https://console.bluemix.net/docs/services/personality-insights" rel="nofollow">IBM Personality Insights</a>, and I eagerly visited the site. The <a href="https://console.bluemix.net/docs/services/personality-insights/science.html#science" rel="nofollow">scientific basis</a> for the results is interesting. I am greatly interested in the analysis that IBM Personality Insights provided of the blog posting I wrote on my birthday last year. I also submitted a <a href="/blog/2008/04/28/cult-of-software-god.html">short humorous posting</a> I wrote ten years ago for analysis. </p> <h2 id="salt">A Modern Horoscope?</h2> <div style=""> <picture> <source srcset="/blog/images/zodiac_690x690.webp" type="image/webp"> <source srcset="/blog/images/zodiac_690x690.png" type="image/png"> <img src="/blog/images/zodiac_690x690.png" title="Horoscope" class=" liImg2 rounded shadow" alt="Horoscope" /> </picture> </div> <p> I think the results from IBM Personality Insights are about as accurate as a horoscope. <a href="https://www.quora.com/How-accurate-is-IBMs-Watson-Personality-Insights-application/answer/Abhishek-Srivastava-198" rel="nofollow">Abhishek Srivastava&rsquo;s Quora posting</a> of November 18, 2016, expresses this well. </p> <blockquote> I think most answers here have missed the point of IBM Watson’s personality insight service. It is a NLP based approach to find scores of individuals on some well-established scales in psychology like Big 5, Basic Humans Values and Needs. So, when those numbers are arrived using those standard questionnaires or through text analysis done by Watson, they are pretty close statistically. In that respect Watson is definitely very accurate. As far as interpretation of those results are concerned, that’s beyond the purview of current scope of Watson and rather is a question for psychologist. If I was a team member at IBM Watson, I would have actually not given the interpretation and would rather just give the scores and let people interpret them. </blockquote> <p> Furthermore, until a peer review of IBM Personality Insights concludes that the results are well-founded, I would not be comfortable using the technology for decision-making. </p> <p> To be fair, <a href="https://www.news.vcu.edu/article/An_untested_foundation_A_VCU_study_finds_that_many_published" rel="nofollow">a recent examination</a> of nearly 350 published psychological experiments found that 42% failed to show that they were based on a valid foundation of empirical evidence, suggesting that a wide swath of psychological science is based on an untested foundation. With that in mind, let's see what this modern horoscope serves up! </p> <h2 id="highLevel">High Level Results</h2> <p> Results were fairly consistent between the two blog postings. When I concatenated the two posts, the longer post dominated. <span class='diff'>Differences between the two personality assessments are shown this way.</span> <span class="comment">My comments are shown this way.</span> </p> <table class="table table_striped table_cell_top table_cell_vspace table_cell_justify" width="100%"> <tr> <th width="50%">Birthday Posting Summary</th> <th width="50%"><a href="/blog/2008/04/28/cult-of-software-god.html">Humorous Posting</a> Summary</th> </tr> <tr> <td> The results in JSON format are <a href="/blog/factor12/ibmPersonality.json">here</a>. </td> <td> The results in JSON format are <a href="/blog/factor12/ibmPersonality2.json">here</a>. </td> </tr> <tr> <td> You are shrewd and <span class='diff'>skeptical</span>. </td> <td> You are shrewd<span class='diff'>, inner-directed and guarded</span>. </td> </tr> <tr> <td> You are philosophical: you are open to and intrigued by new ideas and love to explore them. You are independent: you have a strong desire to have time to yourself. <span class='diff'>And you are authority-challenging: you prefer to challenge authority and traditional values to help bring about positive changes.</span> <td> You are philosophical: you are open to and intrigued by new ideas and love to explore them. You are independent: you have a strong desire to have time to yourself. <span class='diff'>And you are solemn: you are generally serious and do not joke much.</span> <span class="comment">I guess IBM did not like my humor!</span> </td> </tr> <tr> <td> Your choices are driven by a desire for <span class='diff'>discovery</span> <span class='comment'>often true</span>. </td> <td> Your choices are driven by a desire for <span class='diff'>organization</span> <span class='comment'>often true</span>. </td> </tr> <tr> <td> You are relatively unconcerned with both <span class='diff'>tradition</span> and taking pleasure in life. <span class='diff'>You care more about making your own path than following what others have done. And you prefer activities with a purpose greater than just personal enjoyment.</span> </td> <td> You are relatively unconcerned with both <span class='diff'>achieving success</span> and taking pleasure in life. <span class='diff'>You prefer activities with a purpose greater than just personal enjoyment. And you make decisions with little regard for how they show off your talents.</span> </td> </tr> </table> <p> An old friend, who I've known since university, is a clinical psychologist with a PhD and works as a doctor in a mental hospital. His comments on the results were:</p> <blockquote> <p> I think that some aspects of the profile are pretty accurate; however, other aspects such as calling you shrewd and skeptical are a bit evaluate: I would instead say intelligent and not naive. The openness results would seem pretty accurate. </p> <p> I think you're probably a bit more extraverted than this analysis suggests. </p> <p> The ‘emotional range’ factor of the Big 5 is typically labeled Neuroticism. I kind of think of Agreeableness as a compliance/ conformity dimension, and I if I am not mistaken, it is not unusual for (male) entrepreneurs to score low on that dimension. </p> </blockquote> <h2>But Wait, There's More!</h2> <p> IBM's Personality Insights provides additional information that explains the above in more detail: </p> <table class="table table_striped table_cell_top table_cell_vspace table_cell_justify"> <tr> <th width="50%">Birthday Posting Summary</th> <th width="50%"><a href="/blog/2008/04/28/cult-of-software-god.html">Humorous Posting</a> Summary</th> </tr> <tr> <td> You are <b>likely</b> to: <ul> <li> <span class='diff'>like musical movies</span><br /> <span class="comment">Oops, that was way off!</span> </li> <li> be sensitive to ownership cost when buying automobiles </li> <li> have experience playing music <br /> <span class="comment">True: I am a multi-instrumentalist</span> </li> </ul> <td> You are <b>likely</b> to: <ul> <li> <span class='diff'>like historical movies</span> <br /> <span class="comment">True!</span></li> <li>be sensitive to ownership cost when buying automobiles </li> <li> have experience playing music</li> </ul> </tr> <tr> <td> You are <b>unlikely</b> to: <ul> <li> be influenced by social media during product purchases <span class="comment">The truth is more complex</span> </li> <li> prefer style when buying clothes <span class="comment">True, and to compensate I try to shop for clothes with carefully selected friends</span> </li> <li><span class='diff'>like country music</span> <span class="comment">Spot on!</span></li> </ul> </td> <td> <p style="bold"> You are <b>unlikely</b> to: </p> <ul> <li> be influenced by social media during product purchases <br> <br> </li> <li> prefer style when buying clothes <br> <br> <br> </li> <li> <span class='diff'>be influenced by brand name when making product purchases</span> </li> </ul> </td> </tr> </table> <p> I think that the above was a pretty good assessment of me, and the detailed breakdowns which follow are interesting. Before you look at that, however, you should know that the <a href="https://www.verywellmind.com/the-big-five-personality-dimensions-2795422" rel="nofollow">Big 5 Personality Model</a> defines the 5 traits using words with meanings that might seem different from the meanings you might expect. The 5 traits are: <a href="https://console.bluemix.net/docs/services/personality-insights/openness.html" rel="nofollow"><i>openness</i></a>, <a href="https://console.bluemix.net/docs/services/personality-insights/conscientiousness.html" rel="nofollow"><i>conscientiousness</i></a>, <a href="https://console.bluemix.net/docs/services/personality-insights/extroversion.html" rel="nofollow"><i>extroversion</i></a>, <a href="https://console.bluemix.net/docs/services/personality-insights/agreeableness.html" rel="nofollow"><i>agreeableness</i></a> and <a href="https://console.bluemix.net/docs/services/personality-insights/emotional-range.html" rel="nofollow"><i>emotional range</i></a>. </p> <h2 id="pnv">Personality, Needs and Values</h2> <p> Each of the 5 traits are broken down into various aspects. For example, openness consists of <i>adventurousness</i>, <i>artistic interests</i>, <i>emotionality</i>, <i>imagination</i>, <i>intellect</i>, and <i>authority-challenging</i>. Again, these terms are <a href="https://console.bluemix.net/docs/services/personality-insights/openness.html#dimensions" rel="nofollow">defined in specific ways</a> that might be different from the definitions that you might expect. </p> <table class="table table_striped table_cell_top table_cell_vspace table_cell_justify"> <tr> <th width="50%">Birthday Posting Summary</th> <th width="50%"><a href="/blog/2008/04/28/cult-of-software-god.html">Humorous Posting</a> Summary</th> </tr> <tr> <td> <p><br /><br /> <i>1724 words; decent analysis.</i> </p> <div style=""> <picture> <source srcset="/blog/images/ibmPersonalityInsightMslinnSliders.webp" type="image/webp"> <source srcset="/blog/images/ibmPersonalityInsightMslinnSliders.png" type="image/png"> <img src="/blog/images/ibmPersonalityInsightMslinnSliders.png" title="Birthday posting summary by IBM Personality Insights" class=" rounded shadow zoom" alt="Birthday posting summary by IBM Personality Insights" /> </picture> </div> </td> <td> <p> <i>516 words; we need a minimum of 600, preferably 1,200 or more, to compute statistically significant estimates.</i> </p> <div style=""> <picture> <source srcset="/blog/images/ibmPersonalityInsightMslinnSliders2.webp" type="image/webp"> <source srcset="/blog/images/ibmPersonalityInsightMslinnSliders2.png" type="image/png"> <img src="/blog/images/ibmPersonalityInsightMslinnSliders2.png" title="Humorous Posting summary by IBM Personality Insights" class=" rounded shadow zoom" alt="Humorous Posting summary by IBM Personality Insights" /> </picture> </div> </td> </tr> </table> <h2 id="detail">Detailed Breakdown</h2> <p> Here is my detailed breakdown, shown as sunburst charts: </p> <table class="table table_striped table_cell_top table_cell_vspace table_cell_justify" style="width: 100%"> <tr> <th width="50%"> Birthday Posting Summary </th> <th width="50%"> <a href="/blog/2008/04/28/cult-of-software-god.html">Humorous Posting</a> Summary </th> </tr> <tr> <td> <div style=""> <picture> <source srcset="/blog/images/ibmPersonalityInsightMslinnSunburst.webp" type="image/webp"> <source srcset="/blog/images/ibmPersonalityInsightMslinnSunburst.png" type="image/png"> <img src="/blog/images/ibmPersonalityInsightMslinnSunburst.png" title="Birthday posting summary by IBM Personality Insights" class=" rounded shadow zoom" alt="Birthday posting summary by IBM Personality Insights" /> </picture> </div> </td> <td> <div style=""> <picture> <source srcset="/blog/images/ibmPersonalityInsightMslinnSunburst2.webp" type="image/webp"> <source srcset="/blog/images/ibmPersonalityInsightMslinnSunburst2.png" type="image/png"> <img src="/blog/images/ibmPersonalityInsightMslinnSunburst2.png" title="Humorous posting summary by IBM Personality Insights" class=" rounded shadow zoom" alt="Humorous posting summary by IBM Personality Insights" /> </picture> </div> </td> </tr> </table> <p> The sunburst charts paint me as an unusual person. If this personality assessment is accurate, I am rather complex. However, further reading suggests that people with very high openness scores defy most structured evaluations, and I score in the 99<sup>th</sup> percentile for openness. </p> <table class="table table_striped table_cell_top table_cell_vspace table_cell_justify" style="width: 100%"> <tr> <th width="50%"> Birthday Posting Summary </th> <th width="50%"> <a href="/blog/2008/04/28/cult-of-software-god.html">Humorous Posting</a> Summary </th> </tr> <tr> <td> <b>Openness to experience</b>: high (99<sup>th</sup> percentile). This means I have an unusually high fluid intelligence (I am able to learn complex concepts and tasks very quickly), and I am likely to be eccentric. <br /> <span class="comment">I believe this to be true</span>. </td> <td> <b>Openness to experience</b>: high (99<sup>th</sup> percentile) &ndash; <span class="comment">Same</span> </td> </tr> <tr> <td> <b>Extraversion</b>: Low (15th percentile) &ndash; Cold, withdrawn, unfriendly. <span class="comment">That feels harsh.</span> </td> <td> <b>Extraversion</b>: Even lower (6<sup>th</sup> percentile). <span class="comment">Yikes!</span> </td> </tr> <tr> <td> <b>Agreeableness</b>: Low (5<sup>th</sup> percentile) &ndash; Independent, tough, dominant, possibly manipulative. <span class="comment">Yes, I make up my mind and I follow what I believe to be the appropriate course, regardless of what others might say or do.</span> <br /> Additionally, I have very strong sympathy (97<sup>th</sup> percentile) and I am rather uncompromising (82nd percentile), strongly cooperative (82<sup>nd</sup> percentile) with strong altruism (87<sup>th</sup> percentile). <br /> <span class="comment">Perhaps that means that I am a crusty individual with a warm heart.</span> <br /> According to the breakdown, I am quite trusting (82<sup>nd</sup> percentile), yet I am also very cautious (90<sup>th</sup> percentile), so for me I go with &ldquo;trust but verify&rdquo;. </td> <td> <b><a href="https://cloud.ibm.com/docs/services/personality-insights?topic=personality-insights-agreeableness" rel="nofollow">Agreeableness</a></b>: Even lower (0<sup>th</sup> percentile). <span class="comment">Yikes!</span> </td> </tr> <tr> <td> <b>Emotional range</b>: average (> 59<sup>th</sup> percentile). <span class="comment"><a href="https://cloud.ibm.com/docs/services/personality-insights?topic=personality-insights-emotionalRange" rel="nofollow">IBM defines this</a> as &ldquo;the extent to which a person's emotions are sensitive to the individual's environment&rdquo;</span> </td> <td> <b>Emotional range</b>: very high (> 91<sup>st</sup> percentile). When coupled with low agreeableness, IBM predicts: temperamental, irritable, quarrelsome, impatient, grumpy. When coupled with low conscientiousness, IBM predicts: compulsive, nosy, self-indulgent, forgetful, impulsive. When coupled with low extroversion, IBM predicts: guarded, fretful, insecure, pessimistic, secretive. When coupled with high openness, IBM predicts: excitable, passionate, sensual. <span class="comment">Hmm, quarrelsome, impatient, compulsive, self-indulgent, forgetful, insecure, pessimistic and yet passionate and sensual. What a combination!</span> </td> </tr> <tr> <td> <b>Low Conservation (1<sup>st</sup> percentile)</b> &ndash; <span class="comment">This is called Hedonism in the other graph.</span> </td> <td> <b>Hedonism</b>: very low (1<sup>st</sup> percentile) &ndash; <span class="comment">This is called Conservation in the other graph.</span> Within this category I scored low self-enhancement, low Hedonism, low Openness to change and low conservation. </td> </tr> <tr> <td> <b>Low Harmony (5<sup>th</sup> percentile)</b> &ndash; <span class="comment">This is called Closeness in the other graph.</span> </td> <td> <b>Closeness:</b> low (1<sup>th</sup> percentile) &ndash; <span class="comment">This is called Harmony in the other graph.</span> Within this category I scored low Excitement, Harmony, Ideal, Liberty, Love, Self-expression and Stability. I also scored high Curiosity (78<sub>th</sub> percentile) and high Structure (88<sup>th</sup> percentile). <span class="comment">Sounds like I'm pretty much a curious robot.</span> </td> </tr> </table> <p> These traits, if true, would make me a good expert witness, and a good evaluator for technical due diligence. This might also explain my propensity for constantly inventing new things. </p> <p> According to these results, I also registered some extreme <a href="https://console.bluemix.net/docs/services/personality-insights/needs.html#needs" rel="nofollow">needs</a> and <a href="https://console.bluemix.net/docs/services/personality-insights/values.html#values" rel="nofollow">values</a>. Because the results of analyzing both documents were almost the same, I show them together. </p> <p> First, let's look at the results that I identify with: </p> <table class="table table_striped table_cell_top table_cell_vspace table_cell_justify" style="width: 100%"> <tr> <th> Both articles </th> </tr> <tr> <td> An extremely liberal mindset (<b>conservation</b>: 1st percentile) </td> </tr> <tr> <td> Virtually no interest in comfort (<b>hedonism</b>: 2nd percentile) </td> </tr> <tr> <td> Almost no interest in social power, authority, wealth, success, capability, etc. (<a href="https://www.researchgate.net/publication/322627891_The_five_pillars_of_self-enhancement_and_self-protection" rel="nofollow"> <b>self-enhancement</b></a>: 4th percentile) </td> </tr> <tr> <td> Almost no interest in <b>harmony</b> (5th percentile) </td> </tr> <tr> <td> Strong <b>curiosity</b> (87th percentile) </td> </tr> <tr> <td> Low connection to family and setting up a home (<b>closeness</b>: 9%) </td> </tr> <tr> <td> Very little respect, commitment, and acceptance of the customs and ideas that culture and/or religion provides. </td> </tr> </table> <p> I think the following needs assessments are way off the mark. Perhaps I protest too much? </p> <table class="table table_striped table_cell_top table_cell_vspace table_cell_justify" style="width: 100%"> <tr> <td> Very little interest in just having fun for its own sake (<b>excitement</b>: 7th percentile). </td> </tr> <tr> <td> Low desire for perfection or a sense of community <span class="comment">The truth is complicated: I have spent decades building professional and musical communities, yet I am isolated in many ways</span> </td> </tr> <tr> <td> Low interest in discovering and asserting their identities (<b>self-expression</b>: 9%) <span class="comment">This seems way off the mark, for example consider the motivation for my spending thousands of hours on EmpathyWorks.</span> </td> </tr> </table> Evaluating Blockchain Companies 2018-08-29T00:00:00-04:00 https://mslinn.github.io/blog/2018/08/29/evaluating-blockchain-companies <div style=""> <picture> <source srcset="/blog/images/2018-10-19_10-51-13_690x511.webp" type="image/webp"> <source srcset="/blog/images/2018-10-19_10-51-13_690x511.png" type="image/png"> <img src="/blog/images/2018-10-19_10-51-13_690x511.png" title="Mike Slinn presents" class=" liImg " alt="Mike Slinn presents" /> </picture> </div> <div> <p> I have performed <a href="/evaluation/index.html">technical due diligence</a> for investors since the mid-1980s. In this 40-minute presentation I explain how I evaluate blockchain-related technology companies. </p> <p> This talk was presented at the 6th Annual Global Big Data Conference in Santa Clara, California on August 29, 2018. The promotional video recording is <a href="https://www.youtube.com/watch?v=o_9q2USRzzI" target="_blank" rel="nofollow">here</a>. </p> <p> <a href="https://www.infoq.com/presentations/evaluate-blockchain-companies/" target="_blank" rel="nofollow">InfoQ produced the presentation</a> in their unique format. </p> <p> The slides are <a href="https://www.slideshare.net/mslinn/evaluating-blockchain-companies/mslinn/evaluating-blockchain-companies" target="_blank" rel="nofollow">here</a>. </p> <p> The video recording is <a href="https://big.mslinn.com/video/18-aug-evaluatingbccompanies.mp4" target="_blank" rel="nofollow">here</a>. </p> </div> <div style="text-align: center;"> <picture> <source srcset="/blog/images/6thGlobalBlockchain.webp" type="image/webp"> <source srcset="/blog/images/6thGlobalBlockchain.png" type="image/png"> <img src="/blog/images/6thGlobalBlockchain.png" title="Mike Slinn presents" class="center liImg2 rounded shadow" alt="Mike Slinn presents" /> </picture> </div> Bob Summerwill 2018-08-28T00:00:00-04:00 https://mslinn.github.io/blog/2018/08/28/summerwill <p> This is a great photo of Bob Summerwill and me in San Francisco August 2018! </p> <div style=""> <picture> <source srcset="/assets/images/ethereum/meBobSummerwill_690x518.webp" type="image/webp"> <source srcset="/assets/images/ethereum/meBobSummerwill_690x518.png" type="image/png"> <img src="/assets/images/ethereum/meBobSummerwill_690x518.png" title="Mike Slinn and Bob Summerwill in San Francisco August 2018" class=" liImg2 rounded shadow" alt="Mike Slinn and Bob Summerwill in San Francisco August 2018" /> </picture> </div> Keynote Panel Discussion - The Future of Blockchain 2018-08-23T00:00:00-04:00 https://mslinn.github.io/blog/2018/08/23/blockchain-conference-2018-08-28 <div style="text-align: right;"> <picture> <source srcset="/blog/images/mikeBigData_500x431.webp" type="image/webp"> <source srcset="/blog/images/mikeBigData_500x431.png" type="image/png"> <img src="/blog/images/mikeBigData_500x431.png" title="Global Big Data Conference featuring Mike Slinn as a speaker" class="right liImg2 rounded shadow" style="width: 100%; height: auto" alt="Global Big Data Conference featuring Mike Slinn as a speaker" /> </picture> </div> <p> I will participate in a keynote panel discussion on the Future of Blockchain on August 23 from 4:50 PM to 6:00 PM at the <a href="https://sanjose.eventful.com/events/blockchain-new-infrastructure-ai-/E0-001-112130527-9" target="_blank" rel="nofollow">6th Annual Global Big Data Conference</a> in Santa Clara, California. </p> <p> Following are some ideas I hope to discuss with the other members of the panel. First, I would like to remind the reader that blockchain data is passive; without a program to access it blockchain data is inert. Now I'd like to give the definition that I will use for this discussion. </p> <p> The word <i>blockchain</i> is defined by <a href="https://en.wikipedia.org/wiki/Blockchain" target="_blank" rel='nofollow'>Wikipedia</a> as: &ldquo;a growing list of records, called blocks, which are linked using cryptography.&rdquo; The main theme I&rsquo;d like to personally bring forward during this discussion is that Blockchain does not imply or require decentralization, or even distributed systems. Instead, blockchain is a useful technology in its own right, even if it is not distributed / decentralized. I have nothing against decentralization, when used appropriately; the point I want to make is that decentralization is not necessary for blockchain to provide value. </p> <p> Here are the use cases I want to talk about: </p> <h2 id="dbs">#1: Secure Mobile and Embedded Databases</h2> <p> For me, <i>blockchain</i> is a file format that yields immutable data, highly resistant to modification. It is not a database, instead, it is a generic type of storage technology. Yes, an API could be provided that provides a structured interface to blockchain data, but there is no universally accepted standard in use for this in any blockchain implementation that I am aware of. Such a standard should exist, and I would be interested in learning about any standards work in this area. </p> <p> A database built using blockchain would not support the full <a href="https://en.wikipedia.org/wiki/Create,_read,_update_and_delete" target="_blank" rel='nofollow'>CRUD</a> API (create, read, update, and delete) functionality traditionally supported by SQL. Instead, only create and read functionality would be supported. </p> <p> This seems especially useful for mobile and embedded data recorders, for example flight recorders used in airplanes. I have not yet attempted to compute how feasible it might be to use blockchain to store video or audio streams from devices such as <a href="https://www.amazon.com/slp/police-body-camera/4r72vj5jucggbyv" target="_blank" rel='nofollow'>police body cameras</a>. </p> <h2 id="data">#2: Secure Data Distribution</h2> <p> Blockchain is a data storage technology that is known to be highly resistant to corruption or modification. Because data can only be created or read, but not modified or deleted, if you have a file of blockchain data that passes inspection you can be confident that you have all the data that was available at the time the file was last updated. </p> <p> Blockchain data could be stored in RFID tags. This is possible because RFID tags exist which can <a href="https://www.rfidjournal.com/faq/show?66" target="_blank" rel='nofollow'>store up to 66 KB of data</a>, and read/write RFID tags are available. Read-write RFID tags usually have a serial number that can&rsquo;t be written over. Additional blocks of data can be used to store additional information about the items the tag is attached to (these can usually be locked to prevent overwriting of data). </p> <p> This would allow blockchain to be used for <a href="https://www.theglobalist.com/smart-passports-making-travel-safer/" target="_blank" rel='nofollow'>smart passports</a> and smart ID cards. Secure, smart passports would provide extra security for passport/ID cardholders and would make counterfeiting extremely difficult, and would provide a digital record of countries visited, instead of relying on passport stamps, which are easy to counterfeit. </p> <p> Similarly, smart ID cards could hold mutable yet secure data. </p> <h2 id="builders">#3: Secure Software Build Systems</h2> <p> There have been many instances of corporate spies infiltrating software companies or their repositories. The perpetrators altered the source code, accompanying data or modified other build dependencies. When the next version of the software is published, all the customers who install updates will potentially be compromised. </p> <p> <a href="https://en.wikipedia.org/wiki/Git" target="_blank" rel='nofollow'><code>git</code></a> uses similar cryptography techniques as does blockchain. A blockchain-based build system would use a git tag to kick off build, and would need to also store and compare hashes for all dependencies to ensure that they had not changed. Because the software build toolchain is also a dependency, hashes for all the software used in the toolchain would need to be validated at build time. </p> <h2 id="dist">#4: Secure Software Distribution</h2> <p> Computer viruses have been a problem for decades. There have been many instances where programs have viruses embedded after they are published. This is generally true for any software downloaded from a torrent site. If software was packaged using blockchain technology, it could be retrieved for installation with confidence. Software installers that used blockchain in this way could be trusted. </p> <hr /> <h2 id="panel">The Panelists</h2> <p> After the panel, we panelists were photographed together. From left to right: Hayden Kirkpatrick, VP of Strategy at Esurance (moderator); Karen Hsu, CRO at BlockXypher; Steve Beauregard, CRO at Bloq; Mike Slinn, CTO at Micronautics Research (holding the microphone); Mark Javier, Account Executive at Ambisafe; and Camille Sanandaji, CEO at Foodstems. </p> <div style=""> <picture> <source srcset="/assets/images/ethereum/futureOfBlockchainPanelCrop2_690x192.webp" type="image/webp"> <source srcset="/assets/images/ethereum/futureOfBlockchainPanelCrop2_690x192.png" type="image/png"> <img src="/assets/images/ethereum/futureOfBlockchainPanelCrop2_690x192.png" title="Hayden Kirkpatrick (moderator), Karen Hsu, Steve Beauregard, Mike Slinn, and Camille Sanandaji" class=" liImg2 rounded shadow" alt="Hayden Kirkpatrick (moderator), Karen Hsu, Steve Beauregard, Mike Slinn, and Camille Sanandaji" /> </picture> </div> <div style=""> <picture> <source srcset="/assets/images/ethereum/blockchainFuturePanel_690x690.webp" type="image/webp"> <source srcset="/assets/images/ethereum/blockchainFuturePanel_690x690.png" type="image/png"> <img src="/assets/images/ethereum/blockchainFuturePanel_690x690.png" title="Mark Javier, Mike Slinn, Hayden Kirkpatrick (moderator), Steve Beauregard, Karen Hsu, and Camille Sanandaji" class=" liImg2 rounded shadow" alt="Mark Javier, Mike Slinn, Hayden Kirkpatrick (moderator), Steve Beauregard, Karen Hsu, and Camille Sanandaji" /> </picture> </div> Windows Subsystem for Linux Revisited 2018-08-20T00:00:00-04:00 https://mslinn.github.io/blog/2018/08/20/wsl-revisited <div style=""> <picture> <source srcset="/blog/images/wsl1_690x431.webp" type="image/webp"> <source srcset="/blog/images/wsl1_690x431.png" type="image/png"> <img src="/blog/images/wsl1_690x431.png" title="Windows Subsystem for Linux (WSL)" class=" liImg2 rounded shadow" alt="Windows Subsystem for Linux (WSL)" /> </picture> </div> <p> I&rsquo;ve been using <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10" target="_blank" rel='nofollow'>Windows Subsystem for Linux</a> (WSL) headless since it was first released with Windows 10 version 1607 in August 2016. The <a href="https://blogs.msdn.microsoft.com/commandline/2018/03/07/windows10v1803/" target="_blank" rel='nofollow'>April 2018 release</a> of Windows 10 (version 1803) significantly improved WSL. </p> <p> It is possible to work with Ubuntu graphically on a vanilla Windows machine. No special drivers are required. No special Linux or Ubuntu support is required from the computer vendor. </p> <p> This is my setup for running an X client like <a href="https://www.x.org/releases/X11R7.5/doc/man/man1/xeyes.1.html" target="_blank" rel='nofollow'><code>xeyes</code></a> or <a href="https://www.jetbrains.com/idea/" target="_blank" rel='nofollow'>IntelliJ IDEA</a> from WSL or WSL2, accessed via a Windows X server like <a href="https://sourceforge.net/projects/vcxsrv/" target="_blank" rel='nofollow'>VcXsrv</a>. These scripts assume <a href="https://www.microsoft.com/en-us/p/ubuntu-1804/9n9tngvndl3q" target="_blank" rel='nofollow'>Ubuntu 18.04 was installed</a> under WSL. </p> <div style=""> <picture> <source srcset="/blog/images/wsl.webp" type="image/webp"> <source srcset="/blog/images/wsl.png" type="image/png"> <img src="/blog/images/wsl.png" title="Desktop showing Windows Subsystem for Linux (WSL)" class=" liImg2 rounded shadow" alt="Desktop showing Windows Subsystem for Linux (WSL)" /> </picture> </div> <h2 id="essential">Essential Scripts</h2> <p>Here are bash scripts to install everything:</p> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_init"> </script> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_setup"> </script> <h2 id="optional">Optional Scripts</h2> <p>The remaining scripts are all optional.</p> <h3 id="vnc">VNC</h3> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_vnc"> </script> <p>These bash scripts allow VNC to connect to a remote machine or to WSL on the local machine.</p> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=vncRun"> </script> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl"> </script> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=vncSample"> </script> <h2 id="packages">Packages and Programming Environments</h2> <p>Installation of various packages and programming environments follow.</p> <h3 id="git">Git</h3> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_git"> </script> <h3 id="python">Python</h3> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_python"> </script> <h3 id="nodejs">NodeJS</h3> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_node"> </script> <h3 id="jvm">Java Virtual Machine</h3> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_jvm"> </script> <h3 id="misc">Miscellaneous</h3> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_misc"> </script> <h3 id="atom">Atom Editor</h3> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/7be077ebeb9c172572e211e354a8c5f8.js?file=wsl_atom"> </script> <h3 id="wine">Wine</h3> <p> The advent of WSL means that Wine is no longer required! </p> <h2 id="update">Update Aug 17, 2020</h2> <p> Here are my notes on upgrading WSL and WSL2 from Ubuntu 19.11 to Ubuntu 20.04. </p> <noscript><pre>400: Invalid request</pre></noscript><script src="https://gist.github.com/50a3a8d26cbab51c634ed8e2edf129ae.js"> </script> Ethereum Source Code Walkthrough 2018-06-13T00:00:00-04:00 https://mslinn.github.io/blog/2018/06/13/evm-source-walkthrough <div class="formalNotice rounded shadow" id="about"> <h2>April 27, 2020</h2> <p> <div style=""> <picture> <source srcset="/assets/images/ethereum/Ethereum_logo_2014.svg" type="image/webp"> <source srcset="/assets/images/ethereum/Ethereum_logo_2014.svg" type="image/png"> <img src="/assets/images/ethereum/Ethereum_logo_2014.svg" title="Ethereum logo" class=" liImg right" style="height: 100px;" alt="Ethereum logo" /> </picture> </div> This was a sample work in progress as part of a proposal to the Ethereum Foundation Grants committee. They declined to fund this activity, but gave no reason and no feedback as to what might be acceptable. The preparation of my proposal took weeks, and the preliminary feedback that I had received from many knowledgeable people was that it had great value. </p> <p> I felt that the evaluation process was broken, in fact the entire organization was broken, and there was little hope that Ethereum would ever become a professional organization. 2 years later, I believe history has proved me right. This was the last blockchain-related initiative I participated in. </p> <p> I no longer maintain <code><a href="/blog/2017/11/29/web3j-scala.html" target="_blank" rel='nofollow'>web3j-scala</a></code>, an Ethereum-related project for Scala programmers. I created that open source project and worked on it for free for 3 years. It has been forked and I&rsquo;ve been told it is or was used in production. However, I see no reason to continue working for free on it. Others can carry the project forward if they want. </p> </div> <p> This is a brief walkthrough of some of the core source files for smart contracts in the official <a href="https://golang.org/" target="_blank" rel='nofollow'>Go language</a> Ethereum implementation, which includes the <a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//core/vm" target="_blank" rel='nofollow'><code>geth</code></a> command-line Ethereum client program, along with many other programs. Ethereum clients include an implementation of the Ethereum Virtual Machine (EVM), which are able to parse and verify the Ethereum blockchain, including smart contracts, and provides interfaces to create transactions and mine blocks. </p> <p> I&rsquo;ve added some suggestions for how the source code might be improved. If there is general agreement that these suggestions make sense (tell me in the comments!) then I&rsquo;ll create a pull request. </p> <h2>License</h2> <p> <div style=""> <picture> <source srcset="/assets/images/lgplv3-147x51.webp" type="image/webp"> <source srcset="/assets/images/lgplv3-147x51.png" type="image/png"> <img src="/assets/images/lgplv3-147x51.png" title="LGPL logo" class=" right" alt="LGPL logo" /> </picture> </div> This Ethereum client project was released under the <a href="https://www.gnu.org/licenses/lgpl-3.0.en.html" target="_blank" rel='nofollow'>GNU Lesser General Public License, version 3</a> or later, which permits use of the code as a library in proprietary programs. </p> <h2>Source Files</h2> <p> The <a href="https://github.com/hhatto/gocloc" target="_blank" rel='nofollow'><code>gocloc</code></a> program counted the following source files and lines: </p> <table class="table table_striped table_right"> <tr> <th>Language</th> <th>Files</th> <th>Blank Lines</th> <th>Comment Lines</th> <th>Code Lines</th> </tr> <tr> <th>Go</th> <td>1824</td> <td>58,134</td> <td>81,861</td> <td>639,435</td> </tr> <tr> <th>C</th> <td>55</td> <td>17,257</td> <td>30,909</td> <td>84,719</td> </tr> <tr> <th>C Header</th> <td>97</td> <td>2,559</td> <td>6,318</td> <td>15,083</td> </tr> <tr> <th>Markdown</th> <td>88</td> <td>3,152</td> <td>0</td> <td>9,175</td> </tr> <tr> <th>JavaScript</th> <td>13</td> <td>1,845</td> <td>4,495</td> <td>7,986</td> </tr> <tr> <th>Assembly</th> <td>39</td> <td>557</td> <td>957</td> <td>3,783</td> </tr> <tr> <th>JSON</th> <td>17</td> <td>0</td> <td>0</td> <td>2,065</td> </tr> <tr> <th>Protocol Buffers</th> <td>2</td> <td>113</td> <td>40</td> <td>1,030</td> </tr> <tr> <th>Plain Text</th> <td>11</td> <td>217</td> <td>0</td> <td>954</td> </tr> <tr> <th>C++</th> <td>4</td> <td>132</td> <td>102</td> <td>937</td> </tr> <tr> <th>BASH</th> <td>10</td> <td>178</td> <td>315</td> <td>931</td> </tr> <tr> <th>Perl</th> <td>10</td> <td>268</td> <td>1,289</td> <td>879</td> </tr> <tr> <th>JSX</th> <td>11</td> <td>119</td> <td>245</td> <td>722</td> </tr> <tr> <th>XML</th> <td>9</td> <td>0</td> <td>0</td> <td>651</td> </tr> <tr> <th>M4</th> <td>4</td> <td>79</td> <td>99</td> <td>649</td> </tr> <tr> <th>YAML</th> <td>20</td> <td>77</td> <td>42</td> <td>581</td> </tr> <tr> <th>NSIS</th> <td>5</td> <td>86</td> <td>154</td> <td>446</td> </tr> <tr> <th>Java</th> <td>4</td> <td>143</td> <td>187</td> <td>438</td> </tr> <tr> <th>Makefile</th> <td>11</td> <td>101</td> <td>84</td> <td>381</td> </tr> <tr> <th>Python</th> <td>6</td> <td>154</td> <td>250</td> <td>339</td> </tr> <tr> <th>HTML</th> <td>3</td> <td>15</td> <td>9</td> <td>245</td> </tr> <tr> <th>Solidity</th> <td>7</td> <td>56</td> <td>171</td> <td>213</td> </tr> <tr> <th>Bourne Shell</th> <td>6</td> <td>23</td> <td>25</td> <td>119</td> </tr> <tr> <th>CMake</th> <td>1</td> <td>9</td> <td>0</td> <td>35</td> </tr> <tr> <th>Awk</th> <td>1</td> <td>4</td> <td>4</td> <td>17</td> </tr> <tr> <th>TOML</th> <td>1</td> <td>0</td> <td>0</td> <td>3</td> </tr> <tr> <th>Total</th> <th class="table_right">2,260</th> <th class="table_right">85,278</th> <th class="table_right">127,556</th> <th class="table_right">771,825</th> </tr> </table> <h2>Packages</h2> <p> I used the following incantation to discover that <code>geth</code> defines 244 packages: </p> <pre data-lt-active='false'><span class="unselectable">$ </span>grep -rh "^package" | grep -v "not installed" | \ tr -d ';' | sed 's^//.*^^' | awk '{$1=$1};1' | \ sort | uniq | wc -l</pre> <p> I won't list them all. The <a href="https://godoc.org/github.com/ethereum/go-ethereum#pkg-subdirectories" target="_blank" rel='nofollow'><code>godoc</code></a> for the project contains much of the <a href="https://godoc.org/github.com/ethereum/go-ethereum#pkg-subdirectories" target="_blank" rel='nofollow'>following documentation</a> for the top-level packages. I provided the rest of the information from disparate sources, including reading the source code: </p> <p> </p> <table class="table table_striped"> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//accounts" target="_blank" rel='nofollow'>accounts</a></th> <td>implements high-level Ethereum account management.</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/blob/master/trie/trie.go" target="_blank" rel='nofollow'>trie</a></th> <td>provides a binary Merkle tree implementation.</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//cmd" target="_blank" rel='nofollow'>cmd</a></th> <td>Contains the following command-line tools. Most tools support the <code>--help</code> option. <table class="table table_striped" style="margin-top: 0.5em"> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//cmd/abigen" target="_blank" rel='nofollow'>abigen</a></th> <td>source code generator to convert Ethereum contract definitions into easy to use, compile-time type-safe Go packages. It operates on plain <a href="https://github.com/ethereum/wiki/wiki/Ethereum-Contract-ABI" target="_blank" rel='nofollow'>Ethereum contract ABIs</a> with expanded functionality if the contract bytecode is also available. However it also accepts Solidity source files, making development much more streamlined. Please see the <a href="https://github.com/ethereum/go-ethereum/wiki/Native-DApps:-Go-bindings-to-Ethereum-contracts" target="_blank" rel='nofollow'>Native DApps wiki page</a> for details.</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//cmd/bootnode" target="_blank" rel='nofollow'>bootnode</a></th> <td>runs a bootstrap node for the Ethereum Discovery Protocol. This is a stripped-down version of geth that only takes part in the network node discovery protocol, and does not run any of the higher level application protocols. It can be used as a lightweight bootstrap node to aid in finding peers in private networks.</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//cmd/clef" target="_blank" rel='nofollow'>clef</a></th> <td>a standalone signer that manages keys across multiple Ethereum-aware apps such as Geth, Metamask, and cpp-ethereum. <i>Alpha quality, not released yet.</i></td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//cmd/ethkey" target="_blank" rel='nofollow'>ethkey</a></th> <td>a key/wallet management tool for Ethereum keys. Allows user to add, remove and change their keys, and supports cold wallet device-friendly transaction inspection and signing. <a href="https://github.com/ethereum/guide/blob/master/ethkey.md" target="_blank" rel='nofollow'>This documentation</a> was written for the C++ Ethereum client implementation, but it is probably suitable for the Go implementation as well. </td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//cmd/evm" target="_blank" rel='nofollow'>evm</a></th> <td>a version of the EVM (Ethereum Virtual Machine) for running bytecode snippets within a configurable environment and execution mode. Allows isolated, fine-grained debugging of EVM opcodes. Example usage: <pre data-lt-active='false'>evm --code 60ff60ff --debug</pre></td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//cmd/faucet" target="_blank" rel='nofollow'>faucet</a></th> <td>an Ether faucet backed by a light client.</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//cmd/geth" target="_blank" rel='nofollow'>geth</a></th> <td>official command-line client for Ethereum. It provides the entry point into the Ethereum network (main-, test- or private net), capable of running as a full node (default) archive node (retaining all historical state) or a light node (retrieving data live). It can be used by other processes as a gateway into the Ethereum network via JSON RPC endpoints exposed on top of HTTP, WebSocket and/or IPC transports. For more information see the <a href="https://github.com/ethereum/go-ethereum/wiki/Command-Line-Options" target="_blank" rel='nofollow'>CLI Wiki page</a>.</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//cmd/p2psim" target="_blank" rel='nofollow'>p2psim</a></th> <td>a simulation HTTP API. <a href="https://godoc.org/github.com/ethereum/go-ethereum/cmd/p2psim" target="_blank" rel='nofollow'>Docs are here</a>.</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//cmd/puppeth" target="_blank" rel='nofollow'>puppeth</a></th> <td>assembles and maintains private networks.</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//cmd/rlpdump" target="_blank" rel='nofollow'>rlpdump</a></th> <td>a pretty-printer for RLP data. RLP (Recursive Length Prefix) is the data encoding used by the Ethereum protocol. Sample usage: <pre data-lt-active='false'>rlpdump --hex CE0183FFFFFFC4C304050583616263</pre></td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/blob/master/swarm/README.md" target="_blank" rel='nofollow'>swarm</a></th> <td>provides the <code>bzzhash</code> command, which computes a swarm tree hash, and implements the swarm daemon and tools. See <a href="https://swarm-guide.readthedocs.io"> the swarm documentation</a> for more information.</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//cmd/wnode" target="_blank" rel='nofollow'>wnode</a></th> <td>simple Whisper node. It could be used as a stand-alone bootstrap node. Also could be used for different test and diagnostics purposes.</td> </tr> </table> </td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//common" target="_blank" rel='nofollow'>common</a></th> <td>contains various helper functions worth checking out</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//consensus" target="_blank" rel='nofollow'>consensus</a></th> <td>implements different Ethereum consensus engines (which must conform to the <a href="https://godoc.org/github.com/ethereum/go-ethereum/consensus#Engine" target="_blank" rel='nofollow'><code>Engine</code> interface</a>): <a href="https://godoc.org/github.com/ethereum/go-ethereum/consensus/clique" target="_blank" rel='nofollow'><code>clique</code></a> implements proof-of-authority consensus, and <a href="https://godoc.org/github.com/ethereum/go-ethereum/consensus/ethash" target="_blank" rel='nofollow'>ethash</a> implements proof-of-work consensus.</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//console" target="_blank" rel='nofollow'>console</a></th> <td> Ethereum implements a JavaScript runtime environment (JSRE) that can be used in either interactive (console) or non-interactive (script) mode. Ethereum's JavaScript console exposes the full web3 JavaScript Dapp API and the admin API. <a href="https://github.com/ethereum/go-ethereum/wiki/JavaScript-Console" target="_blank" rel='nofollow'>More documentation is here.</a> This package implements JSRE for the <code>geth console</code> and <code>geth console</code> subcommands. </td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8/containers" target="_blank" rel='nofollow'>containers</a></th> <td></td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//contracts" target="_blank" rel='nofollow'>contracts</a></th> <td></td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//core" target="_blank" rel='nofollow'>core</a></th> <td>implements the Ethereum consensus protocol, implements the Ethereum Virtual Machine, and other miscellaneous important bits</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//crypto" target="_blank" rel='nofollow'>crypto</a></th> <td>cryptographic implementations</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//dashboard" target="_blank" rel='nofollow'>dashboard</a></th> <td></td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//eth" target="_blank" rel='nofollow'>eth</a></th> <td>implements the Ethereum protocol</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//ethclient" target="_blank" rel='nofollow'>ethclient</a></th> <td>provides a client for the Ethereum RPC API</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//ethdb" target="_blank" rel='nofollow'>ethdb</a></th> <td></td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//ethstats" target="_blank" rel='nofollow'>ethstats</a></th> <td>implements the network stats reporting service</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//event" target="_blank" rel='nofollow'>event</a></th> <td>deals with subscriptions to real-time events</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//internal" target="_blank" rel='nofollow'>internal</a></th> <td>Debugging support, JavaScript dependencies, testing support</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//les" target="_blank" rel='nofollow'>les</a></th> <td>implements the Light Ethereum Subprotocol</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//light" target="_blank" rel='nofollow'>light</a></th> <td>implements on-demand retrieval capable state and chain objects for the Ethereum Light Client</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//log" target="_blank" rel='nofollow'>log</a></th> <td>provides an opinionated, simple toolkit for best-practice logging that is both human and machine readable</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//metrics" target="_blank" rel='nofollow'>metrics</a></th> <td>port of Coda Hale's Metrics library. Unclear why this was not implemented as a separate library, like <a href="https://github.com/rcrowley/go-metrics" target="_blank" rel='nofollow'>this one</a>.</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//miner" target="_blank" rel='nofollow'>miner</a></th> <td>implements Ethereum block creation and mining</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//mobile" target="_blank" rel='nofollow'>mobile</a></th> <td>contains the simplified mobile APIs to go-ethereum</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//node" target="_blank" rel='nofollow'>node</a></th> <td>sets up multi-protocol Ethereum nodes</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//p2p" target="_blank" rel='nofollow'>p2p</a></th> <td>implements the Ethereum p2p network protocols: Node Discovery Protocol, RLPx v5 Topic Discovery Protocol, Ethereum Node Records as defined in EIP-778, common network port mapping protocols, and p2p network simulation.</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//params" target="_blank" rel='nofollow'>params</a></th> <td></td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//rlp" target="_blank" rel='nofollow'>rlp</a></th> <td>implements the RLP serialization format</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//rpc" target="_blank" rel='nofollow'>rpc</a></th> <td>provides access to the exported methods of an object across a network or other I/O connection</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//signer" target="_blank" rel='nofollow'>signer</a></th> <td></td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8/swarm" target="_blank" rel='nofollow'>swarm</a></th> <td></td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//tests" target="_blank" rel='nofollow'>tests</a></th> <td>implements execution of Ethereum JSON tests</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//trie" target="_blank" rel='nofollow'>trie</a></th> <td>implements Merkle Patricia Tries</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//vendor" target="_blank" rel='nofollow'>vendor</a></th> <td>contains a minimal framework for creating and organizing command line Go applications, and a rich testing extension for Go's testing package</td> </tr> <tr> <th class="code table_left"><a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//whisper" target="_blank" rel='nofollow'>whisper</a></th> <td>implements the Whisper protocol</td> </tr> </table> <p> <i>I used the following incantation to list the package names:</i> </p> <pre data-lt-active='false'>find . -maxdepth 1 -type d | sed 's^\./^^' | sed '/\..*/d'</pre> <p> The <a href="https://github.com/ethereum/go-ethereum/tree/release/1.8//build" target="_blank" rel='nofollow'><code>build/</code></a> directory does not contain a Go source package; instead, it contains scripts and configurations for building the package in various environments. </p> <h2>Smart Contract Source Code</h2> <p> The <code>core/vm</code> directory contains the files that implement the EVM. These files are part of the <code>vm</code> package. Let's look at two of them: </p> <ul> <li> <code><a href="https://github.com/ethereum/go-ethereum/blob/master/core/vm/contract.go" target="_blank" rel='nofollow'>contract.go</a></code>, which defines smart contract behavior. </li> <li> <code><a href="https://github.com/ethereum/go-ethereum/blob/master/core/vm/contracts.go" target="_blank" rel='nofollow'>contracts.go</a></code>, responsible for executing smart contracts on the EVM. </li> </ul> <h3>Referenced Types</h3> <p> Two of the types used in the source files that we would like to understand are defined in <a href="https://github.com/ethereum/go-ethereum/blob/master/common/types.go" target="_blank" rel='nofollow'><code>common/types.go</code></a>. Let's look at them first. </p> <p> <a href="https://github.com/ethereum/go-ethereum/blob/master/common/types.go#L137-L138" target="_blank" rel='nofollow'><code>Address</code></a> is defined as an array of 20 <code>byte</code>s: </p> <pre data-lt-active='false'>const ( HashLength = 32 AddressLength = 20 ) // Address represents the 20 byte address of an Ethereum account. type Address [AddressLength]byte</pre> <p> <a href="https://github.com/ethereum/go-ethereum/blob/master/common/types.go#L43" target="_blank" rel='nofollow'><code>Hash</code></a> is defined as an array of 32 <code>byte</code>s: </p> <pre data-lt-active='false'>// Hash represents the 32 byte Keccak256 hash of arbitrary data. type Hash [HashLength]byte</pre> <p> The opcodes for each version of the EVM are defined in <a href="https://github.com/ethereum/go-ethereum/blob/master/core/vm/jump_table.go" target="_blank" rel='nofollow'><code>jump_table.go</code></a>. The <a href="https://github.com/ethereum/go-ethereum/blob/master/core/vm/jump_table.go#L35-L51" target="_blank" rel='nofollow'><code>operation</code></a> <code>struct</code> defines the properties: </p> <pre data-lt-active='false'> type operation struct { // execute is the operation function execute executionFunc // gasCost is the gas function and returns the gas required for execution gasCost gasFunc // validateStack validates the stack (size) for the operation validateStack stackValidationFunc // memorySize returns the memory size required for the operation memorySize memorySizeFunc halts bool // indicates whether the operation should halt further execution jumps bool // indicates whether the program counter should not increment writes bool // determines whether this a state modifying operation valid bool // indication whether the retrieved operation is valid and known reverts bool // determines whether the operation reverts state (implicitly halts) returns bool // determines whether the operations sets the return data content }</pre> <p> Notice the <code>jumps</code> property, a Boolean, which if set indicates that the program counter should not increment after executing any form of jump opcode. </p> <p> The <code>destinations</code> type maps the hash of a smart contract to a bit vector for each the smart contract's entry points. If a bit is set, that indicates the EMV's program counter should increment after executing the entry point. <a href="https://github.com/ethereum/go-ethereum/blob/master/core/vm/analysis.go#L25-L28" target="_blank" rel='nofollow'><code>analysis.go</code></a> defines the <code>destinations</code> type like this: </p> <pre data-lt-active='false'> // destinations stores one map per contract (keyed by hash of code). // The maps contain an entry for each location of a JUMPDEST instruction. type destinations map[common.Hash]bitvec</pre> <h3 class="code">contract.go</h3> <p> This file defines smart contract behavior. </p> <h4>Imports</h4> <p> This comment applies to all of the Go source files in the entire project. I think the following absolute import would have been better specified as a relative import: </p> <pre data-lt-active='false'>"github.com/ethereum/go-ethereum/common"</pre> <p> The relative import would look like this instead: </p> <pre data-lt-active='false'>"../../common"</pre> <p> If relative imports were used instead of absolute imports that point to the github repo, local changes to the project made by a developer would automatically be picked up. As currently written, absolute imports cause local changes to be ignored, in favor of the version on github. It might take a software developer a while to realize that the reason why their changes are ignored by most of the code base is because absoluate imports were used. It would then be painful to for the developer to modify the affected source files throughout the project such that they used relative imports. </p> <h4>Types</h4> <p> The publicly visible <a href="https://github.com/ethereum/go-ethereum/blob/master/core/vm/contract.go#L30-L40" target="_blank" rel='nofollow'><code>AccountRef</code></a> type is defined as: </p> <pre data-lt-active='false'> // Account references are used during EVM initialisation and // it's primary use is to fetch addresses. Removing this object // proves difficult because of the cached jump destinations which // are fetched from the parent contract (i.e. the caller), which // is a ContractRef. type AccountRef common.Address</pre> <p> The same file defines a type cast from <code>AccountRef</code> to <code>Address</code>: </p> <pre data-lt-active='false'> // Address casts AccountRef to a Address func (ar AccountRef) Address() common.Address { return (common.Address)(ar) }</pre> <p> The <a href="https://github.com/ethereum/go-ethereum/blob/master/core/vm/contract.go#L25-L28" target="_blank" rel='nofollow'><code>ContractRef</code></a> interface is used by the <code>Contract</code> <code>struct</code>, which we'll see in a moment. This <code>ContractRef</code> interface just consists of an <code>Address</code>. </p> <pre data-lt-active='false'> // ContractRef is a reference to the contract's backing object type ContractRef interface { Address() common.Address }</pre> <p> The <a href="https://github.com/ethereum/go-ethereum/blob/master/core/vm/contract.go#L42-L65" target="_blank" rel='nofollow'><code>Contract</code></a> struct defines the behavior of Ethereum smart contracts, and is central to the topic, so here it is in all its glory: </p><pre data-lt-active='false'> type Contract struct { CallerAddress common.Address caller ContractRef self ContractRef jumpdests destinations // result of JUMPDEST analysis. Code []byte CodeHash common.Hash CodeAddr *common.Address Input []byte Gas uint64 value *big.Int Args []byte DelegateCall bool }</pre> <p> <code>CallerAddress</code> is a publicly visible <code>Address</code> of the caller. <code>caller</code> and <code>self</code> are private <code>ContractRef</code>s, which as we know are really just <code>Address</code>es. </p> <p> <code>jumpdests</code>, a private field, has type <code>destinations</code>, which as we've already discussed defines if the entry point in the smart contract that need the program counter to be incremented after executing. </p> <p> <code>Code</code> is a a publicly visible <code>byte</code> slice. We don't yet know if this is the smart contract source code, compiled code, or something else. </p> <p> <code>CodeHash</code> is the publicly visible hash of the <code>Code</code>, while <code>CodeAddr</code> is a publicly visible pointer to the <code>Address</code> (of the code, presumably). </p> <p> <code>Gas</code> is the publicly visible amount of Ethereum gas allocated by the user for executing this smart contract, stored as an unsigned 64-bit integer. </p> <p> <code>Value</code> is a private pointer to a big integer. Possibly this might be the result of executing the contract? </p> <p> <code>Args</code> is a publicly visible <code>byte</code> slice, not sure what it is for. </p> <p> <code>DelegateCall</code> is a publicly visible Boolean value, unclear if this means the smart contract was invoked using <a href="https://solidity.readthedocs.io/en/v0.4.24/introduction-to-smart-contracts.html#delegatecall-callcode-and-libraries" target="_blank" rel='nofollow'><code>delegatecall</code></a>. From the documentation: "This means that a contract can dynamically load code from a different address at runtime. Storage, current address and balance still refer to the calling contract, only the code is taken from the called address. This makes it possible to implement the “library” feature in Solidity: Reusable library code that can be applied to a contract’s storage, e.g. in order to implement a complex data structure." </p> <h3 class="code">contracts.go</h3> <p> This file is responsible for executing smart contracts on the EVM. </p> <h4>Imports</h4> <p> The following imports are used: </p> <ul> <li> <a href="https://golang.org/pkg/crypto/sha256/" target="_blank" rel='nofollow'>Package <code>sha256</code></a> from the <code>crypto</code> project implements the SHA224 and SHA256 hash algorithms as defined in FIPS 180-4. </li> <li> <a href="https://godoc.org/github.com/pkg/errors" target="_blank" rel='nofollow'><code>errors</code></a>, the Go language simple error handling primitives, such as <code>error</code>. </li> <li> <a href="https://golang.org/pkg/math/big/" target="_blank" rel='nofollow'><code>math/big</code></a> implements arbitrary-precision arithmetic (big numbers). </li> <li> Other packages in this project (<code>go-ethereum</code>): <pre data-lt-active="false"> "github.com/ethereum/go-ethereum/common" "github.com/ethereum/go-ethereum/common/math" "github.com/ethereum/go-ethereum/crypto" "github.com/ethereum/go-ethereum/crypto/bn256" "github.com/ethereum/go-ethereum/params" </pre> <p> Again, I think the above imports would have been better specified as relative imports: </p> <pre data-lt-active="false"> "../../common" "../../common/math" "../../crypto" "../../crypto/bn256" "../../params"</pre> </li> <li> <a href="https://godoc.org/golang.org/x/crypto/ripemd160" target="_blank" rel='nofollow'><code>ripemd160</code></a> implements the <a href="https://homes.esat.kuleuven.be/~bosselae/ripemd160.html" target="_blank" rel='nofollow'>RIPEMD-160 hash algorithm</a>, a secure replacement for the MD4 and MD5 hash functions. These hashes are also termed RIPE message digests. </li> </ul> <h4>Type <span class="code">PrecompiledContract</span></h4> <p><a href="https://github.com/ethereum/go-ethereum/blob/master/core/vm/contracts.go#L32-L38" target="_blank" rel='nofollow'><code>PrecompiledContract</code></a> is the interface for native Go smart contracts. This interface is used by precompiled contracts, as we will see next. <code>Contract</code> is a <code>struct</code> defined in <code><a href="https://github.com/ethereum/go-ethereum/blob/master/core/vm/contract.go" target="_blank" rel='nofollow'>contract.go</a></code>. </p> <h4>Pre-Compiled Contract Maps</h4> <p> These maps specify various types of cryptographic hashes and utility functions, accessed via their address. </p> <p><a href="https://github.com/ethereum/go-ethereum/blob/master/core/vm/contracts.go#L40-L47" target="_blank" rel='nofollow'><code>PrecompiledContractsHomestead</code></a> contains the default set of pre-compiled contract addresses used in the Frontier and Homestead releases of Ethereum: <code>ecrecover</code>, <code>sha256hash</code>, <code>ripemd160hash</code> and <code>dataCopy</code>.</p> <p><a href="https://github.com/ethereum/go-ethereum/blob/master/core/vm/contracts.go#L49-L60" target="_blank" rel='nofollow'><code>PrecompiledContractsByzantium</code></a> contains the default set of pre-compiled contract addresses used in the Byzantium Ethereum release. All of the previously defined pre-compiled contract addresses are provided in Byzantium, plus: <code>bigModExp</code>, <code>bn256Add</code>, <code>bn256ScalarMul</code> and <code>bn256Pairing</code>.</p> <p> I'm not happy about the code duplication, whereby the contents of <code>PrecompiledContractsHomestead</code> are incorporated into <code>PrecompiledContractsByzantium</code> by listing the values again; this would be better expressed by referencing the values of <code>PrecompiledContractsHomestead</code> instead of duplicating them. </p> <h4>Contract Evaluator Function</h4> <p> The <code>RunPrecompiledContract</code> function runs and evaluates the output of a precompiled contract. It accepts three parameters: </p> <ul> <li> A <code>PrecompiledContract</code> instance. </li> <li> A byte array of input data. </li> <li> A reference to a <code>Contract</code>, defined in <code><a href="https://github.com/ethereum/go-ethereum/blob/master/core/vm/contract.go#L44-L65" target="_blank" rel='nofollow'>contract.go</a></code>, discussed above. </li> </ul> <p> The function returns: </p> <ul> <li> A byte array containing the output of the contract. </li> <li> An <code>error</code> value, which could be <code>nil</code>. </li> </ul> <h4>Other Functions</h4> <ul> <li> <code>RunPrecompiledContract</code> &ndash; runs and evaluates the output of a precompiled contract; returns the output as a byte array and an <code>error</code>. </li> <li> <code>RequiredGas</code> (overloaded) &ndash; Computes the gas required for input data, specified as a byte array and returns a <code>uint64</code>. </li> <li> <code>Run</code> (overloaded) &ndash; Computes the smart contract for input data, specified as a byte array and returns the result as a left-padded byte array and an <code>error</code>. </li> <li> <code>newCurvePoint</code> &ndash; Unmarshals a binary blob into a bn256 elliptic curve point. BN-curves are an elliptic curves suitable for cryptographic pairings that provide high security and efficiency cryptographic schemes. See the IETF paper on <a href="https://tools.ietf.org/id/draft-kasamatsu-bncurves-01.html" target="_blank" rel='nofollow'>Barreto-Naehrig Curves</a> for more information. </li> </ul> Smart Contracts That Learn 2018-04-03T00:00:00-04:00 https://mslinn.github.io/blog/2018/04/03/smart-contracts-that-learn <p> It seems natural that machine learning and smart contracts will commonly be integrated over the next few years. </p> <iframe width="853" height="480" src="https://www.youtube.com/embed/a9PjjkMXsi0?rel=0" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen class="rounded shadow liImg" style="width: 100%"></iframe> <p> I presented <b>Smart Contracts That Learn</b> at the Global Blockchain Conference on April 3, 2018, in Santa Clara. This was a 40-minute technical presentation. Slides are <a href="https://www.slideshare.net/mslinn/smart-contracts-that-learn" target="_blank" rel='nofollow'>here</a>. </p> <p> <a href="https://www.infoq.com/presentations/smart-contracts-learn" target="_blank" rel='nofollow'>InfoQ.com</a> published the presentation in their unique format, featuring synchronized video and transcripts. </p> <div style=""> <picture> <source srcset="/assets/images/ethereum/gbc2018MikeCrop.webp" type="image/webp"> <source srcset="/assets/images/ethereum/gbc2018MikeCrop.png" type="image/png"> <img src="/assets/images/ethereum/gbc2018MikeCrop.png" title="Mike Slinn presenting Smart Contracts That Learn" class=" liImg2 rounded shadow" alt="Mike Slinn presenting Smart Contracts That Learn" /> </picture> </div> <p> A full-length (70 minute) version of this presentation was presented at IBM Ottawa on May 28, 2018. <a href="https://www.slideshare.net/mslinn/fullsize-smart-contracts-that-learn" target="_blank" rel='nofollow'>Here are the slides for the full-length version</a>. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/ibmOttawa.webp" type="image/webp"> <source srcset="/blog/images/ibmOttawa.png" type="image/png"> <img src="/blog/images/ibmOttawa.png" class="center liImg2 rounded shadow" /> </picture> </div> Svief 2018-03-02T00:00:00-05:00 https://mslinn.github.io/blog/2018/03/02/svief <p> Images from the <a href="https://twitter.com/SVIEF1/status/978771845796134912" target="_blank" rel="nofollow">SVIEF Blockchain Conference</a> at Stanford University March 24, 2018. My presentation was entitled &ldquo;Smart Contracts that Learn&rdquo;. </p> <div style=""> <a href="https://us5.campaign-archive.com/?e=&id=29967a597f&u=21313a52472a2961f2cb58a34" target="_blank" ><picture> <source srcset="/assets/images/ethereum/bottos_svief_690x388.webp" type="image/webp"> <source srcset="/assets/images/ethereum/bottos_svief_690x388.png" type="image/png"> <img src="/assets/images/ethereum/bottos_svief_690x388.png" title="Chainovation with Michael Slinn and Sima Yazdani" class=" liImg2 rounded shadow" alt="Chainovation with Michael Slinn and Sima Yazdani" /> </picture></a> </div> <div style=""> <picture> <source srcset="/assets/images/ethereum/mike_690x460.webp" type="image/webp"> <source srcset="/assets/images/ethereum/mike_690x460.png" type="image/png"> <img src="/assets/images/ethereum/mike_690x460.png" title="Mike Slinn presenting at SVIEF Blockchain Conference" class=" liImg2 rounded shadow" alt="Mike Slinn presenting at SVIEF Blockchain Conference" /> </picture> </div> <div style=""> <picture> <source srcset="/assets/images/ethereum/mike2_690x460.webp" type="image/webp"> <source srcset="/assets/images/ethereum/mike2_690x460.png" type="image/png"> <img src="/assets/images/ethereum/mike2_690x460.png" title="Mike Slinn talking about fradulent events" class=" liImg2 rounded shadow" alt="Mike Slinn talking about fradulent events" /> </picture> </div> <div style=""> <picture> <source srcset="/assets/images/ethereum/mike5_690x460.webp" type="image/webp"> <source srcset="/assets/images/ethereum/mike5_690x460.png" type="image/png"> <img src="/assets/images/ethereum/mike5_690x460.png" title="Mike Slinn talking about self-optimizing contracts" class=" liImg2 rounded shadow" alt="Mike Slinn talking about self-optimizing contracts" /> </picture> </div> <div style=""> <picture> <source srcset="/assets/images/ethereum/mike6_690x460.webp" type="image/webp"> <source srcset="/assets/images/ethereum/mike6_690x460.png" type="image/png"> <img src="/assets/images/ethereum/mike6_690x460.png" title="Mike Slinn talking about the security / complexity tradeoff" class=" liImg2 rounded shadow" alt="Mike Slinn talking about the security / complexity tradeoff" /> </picture> </div> Smart Contracts For Enterprises 2018-01-18T00:00:00-05:00 https://mslinn.github.io/blog/2018/01/18/smart-contracts-for-enterprises <p> &lsquo;Polyglot&rsquo; means &ldquo;of many languages&rdquo;, and I believe that enterprises that integrate smart contracts into their infrastructure will require solutions that incorporate many simultaneous software languages, libraries and runtime environments, all working as one in a distributed environment. </p> <p> I anticipate significant need for system integration. I am interested in distributed consensus / blockchain / cryptocurrency architecture that support enterprise needs by providing useful features that are easy to use effectively. </p> <p text-align="left"> I presented &ldquo;Polyglot Ethereum Smart Contracts for the Enterprise&rdquo; at the <a href="https://wcef.co/" target="_blank" rel='nofollow'>World Crypto Economic Forum</a> in San Francisco, January 16, 2018. <a href="https://www.slideshare.net/mslinn/polyglot-ethereum-smart-contracts-for-the-enterprise" target="_blank" rel='nofollow'>Here is my slide deck.</a> </p> <div style=""> <picture> <source srcset="/assets/images/ethereum/WorldCryptoEconForum_690x439.webp" type="image/webp"> <source srcset="/assets/images/ethereum/WorldCryptoEconForum_690x439.png" type="image/png"> <img src="/assets/images/ethereum/WorldCryptoEconForum_690x439.png" title="Mike Slinn presents at the World Crypto Economic Forum" class=" liImg2 rounded shadow" style="width:100%" alt="Mike Slinn presents at the World Crypto Economic Forum" /> </picture> </div> <div style=""> <picture> <source srcset="/assets/images/ethereum/WCEF2_690x414.webp" type="image/webp"> <source srcset="/assets/images/ethereum/WCEF2_690x414.png" type="image/png"> <img src="/assets/images/ethereum/WCEF2_690x414.png" title="Mike Slinn presents at the World Crypto Economic Forum" class=" liImg2 rounded shadow" style="width:100%" alt="Mike Slinn presents at the World Crypto Economic Forum" /> </picture> </div> Tweet Stream Manager 2018-01-03T00:00:00-05:00 https://mslinn.github.io/blog/2018/01/03/tweet-stream-manager <p> I publish multiple tweet streams, so I wrote a Node.js web application using Express and Pug to help me manage them. This is what the user interface looks like. This is a silent video. Hopefully, it makes sense to everyone. </p> <iframe width="690px" height="388px" src="https://www.youtube.com/embed/r-47Jaa9Zek?rel=0" frameborder="1" class="liImg shadow" gesture="media" allow="encrypted-media" allowfullscreen></iframe> <p> For example, <a href="https://www.ScalaCourses.com" target="_blank" rel='nofollow'>ScalaCourses.com</a> currently has two tweet streams, each with their schedule (times shown are <code>HH:MM</code> for a 24-hour clock): </p> <ul> <li><b>ScalaCourses</b> &ndash; tweets are published at 6:45 every day</li> <li><b>SaleMore33</b> &ndash; This is a promotional campaign, starting on a certain day/time and ending on another day/time. This stream tweets at 3:24, 9:24, 13:24, 18:24, and 23:24.</li> </ul> <p><code>crontab</code>, running on a local XUbuntu machine, runs a Node.js command-line app that publishes the tweet streams:</p> <pre class="modest" style="font-size: small; padding: 0; margin-left: 0; margin-right:-20px"> NODE=/usr/bin/node TWEETER=/var/work/training/projects/tweeter 45 6 * * * $NODE $TWEETER/index.js $TWEETER/ScalaCourses 24 3,9,13,18,23 * * * $NODE $TWEETER/index.js $TWEETER/SaleMore33 2018-01-03 2018-01-15</pre> <p> I have been thinking about re-implementing this command-line program as an <a href="https://aws.amazon.com/lambda/" target="_blank" rel='nofollow'>AWS Lambda function</a> one day. Instead of using CSV files for persistence, I'll probably go with Dynamo. </p> The web3j-scala Ethereum Library 2017-11-29T00:00:00-05:00 https://mslinn.github.io/blog/2017/11/29/web3j-scala <div style="text-align: right;"> <picture> <source srcset="/assets/images/web3j.webp" type="image/webp"> <source srcset="/assets/images/web3j.png" type="image/png"> <img src="/assets/images/web3j.png" title="Web3j logo" class="right " style="margin-left: 1em; width: 15%" alt="Web3j logo" /> </picture> </div> <p> This video shows <a href="https://github.com/mslinn/web3j-scala" target="_blank" rel='nofollow'><code>web3j-scala</code></a>, an idiomatic Scala wrapper I wrote around <a href="https://web3j.io/" target="_blank" rel='nofollow'><code>web3j</code></a>, which is a Java 8 version of <a href="https://github.com/ethereum/web3.js" target="_blank" rel='nofollow'><code>web3.js</code></a>. These 3 libraries all leverage the <a href="https://github.com/ethereum/wiki/wiki/JSON-RPC" target="_blank" rel='nofollow'><code>json-rpc</code></a> protocol that all Ethereum clients support. <code>Web3j</code> is a lightweight, reactive, somewhat type-safe library for Java and Android that integrates with nodes on Ethereum blockchains. <code>Web3j-scala</code> provides type safety and enhanced scalability over its Java and JavaScript cousins, as well as the pleasure of writing solutions in Scala. </p> <iframe width="690" height="388" src="https://www.youtube.com/embed/8jZ7kICi4SE?rel=0" frameborder="0" allowfullscreen class="rounded shadow liImg"></iframe> <h2 id="transcript">Transcript</h2> <p> Much of the material for this presentation was taken from the <a href="https://github.com/mslinn/web3j-scala" target="_blank" rel='nofollow'><code>web3j-scala</code> GitHub page</a>. </p> <h2 id="poc">Proofs of Concept / Corporate Sponsors Wanted</h2> <p> The only way to provide value is to serve customers. I am actively seeking opportunities to develop blockchain-related prototypes and proofs of concept. Please let me know if you or your organization are interested in sponsoring opportunities for the open-source libraries I am working on. </p> How Much Do You Actually Program? 2017-08-07T00:00:00-04:00 https://mslinn.github.io/blog/2017/08/07/how-much-do-you-program <p> Last week someone asked me how much I programmed, or even if I actively programmed anymore. I recognized that the 20-something-year-old who asked me the question did not believe that people over 40 are capable of being skilled in relevant technology or productive programmers. Like all types of bigotry, ageism is ugliest when it is delivered with self-righteousness. </p> <h2 id="git-stats" style="font-family: Lucida Console, Courier, monospace">git-stats-scala</h2> <p> Following <a href="http://www.paulgraham.com/disagree.html" target="_blank" rel='nofollow'>Paul Graham's advice</a>, I decided to fight back with data. As the first step I wrote <a href="https://github.com/mslinn/git-stats-scala" target="_blank" rel='nofollow'><code>git-stats-scala</code></a>, a program that combs through an entire tree of git projects and summarizes the user's activity for a given time period. </p> <p>The resulting summary of commit statistics is intended to be placed near the top of one's resume.</p> <p> The output of this program merely answers the question: &ldquo;are you an active programmer?&rdquo; <code>Git-stats-scala</code> only reports textual additions and deletions, and includes the net change, which one hopes are indications of actual programming. Statistics are reported for each computer language found. The reader is free to impart any meaning they deem appropriate to this output. I make no claims regarding meaning. </p> <h2 id="myStats">My Statistics</h2> <p>Here is a summary of my git activity over the last 365 days:</p> <p style='font-weight: bold; margin-bottom: 0; padding-bottom: 0'>Subtotals By Language (lines changed across all projects)</p> <pre class="modest" style="font-size: smaller; padding: 0; margin: 0; line-height: 130%"> ┌───────────────────┬───────────────────┬───────────────────┬──────────────────┐ │Language │Lines added │Lines deleted │Net change │ ├───────────────────┼───────────────────┼───────────────────┼──────────────────┤ │Scala │+94,120 │-68,075 │+26,045 │ │SBT │+7,896 │-2,639 │+5,257 │ │Java │+23,955 │-20,597 │+3,358 │ │Markdown │+5,049 │-4,497 │+552 │ │Python │+371 │0 │+371 │ │HTML │+8,810 │-10,488 │-1,678 │ ├───────────────────┼───────────────────┼───────────────────┼──────────────────┤ │Total │+140,201 │-106,296 │+33,906 │ └───────────────────┴───────────────────┴───────────────────┴──────────────────┘ </pre> <p style="padding-top: 0"> Every project I was involved with had more features and better quality at the end of the time period compared to the state of the project at the beginning of the time period. You cannot value software by weighing it. </p> <div class="pullQuote"> You cannot value software by weighing it. </div> <p>I am curious to know how these numbers compare with those from other programmers.</p> <h2 id="cadenza" style="clear: both">The Cadenza Project</h2> <p> Here are stats over the same period for my contribution to the Cadenza core project. Cadenza is the web application that powers <a href="https://www.scalacourses.com" target="_blank" rel='nofollow'>ScalaCourses.com</a>. During this period I hived several F/OSS projects from the Cadenza code base, which is probably why it does not seem to be growing much even though I added many new features. </p> <pre class="modest" style="font-size: smaller; padding: 0; margin: 0; line-height: 130%"> ┌───────────────────┬───────────────────┬───────────────────┬──────────────────┐ │Language │Lines added │Lines deleted │Net change │ ├───────────────────┼───────────────────┼───────────────────┼──────────────────┤ │Scala │+42,687 │-38,237 │+4,450 │ │SBT │+233 │-120 │+113 │ │Java │+102 │-10 │+92 │ │Markdown │+519 │-2,617 │-2,098 │ │HTML │+1,134 │-5,345 │-4,211 │ ├───────────────────┼───────────────────┼───────────────────┼──────────────────┤ │Total │+44,675 │-46,329 │-1,654 │ └───────────────────┴───────────────────┴───────────────────┴──────────────────┘ </pre> Kafka Streams vs. Akka 2017-07-28T00:00:00-04:00 https://mslinn.github.io/blog/2017/07/28/kafka-vs-akka <div style="text-align: center; vertical-align: middle;"> <img src='/blog/images/kafka.webp' alt="Kafka logo" class="liImg" style="height: 150px" title="Kafka logo" /> <img src='/blog/images/akka_full_color.svg' alt="Akka logo" class="liImg" style="margin-left: 2em; height: 120px;" title="Akka logo" /> </div> <p> Context switches are expensive for CPUs to perform, and each incoming message in an <a href="https://akka.io/" target="_blank" rel="nofollow">Akka</a> system requires a context switch when fairness (minimal latency deviation) is important <sup style='font-size: smaller'><a href="#1">[1]</a></sup>. In contrast, <a href="https://kafka.apache.org/documentation/streams/" target="_blank" rel="nofollow">Apache Kafka Streams</a> offers consistent processing times without requiring context switches, so Kafka Streams produces significantly higher throughput than Akka can when contending with a high volume of small computations that must be applied fairly. </p> <h2 id="fairness">Fairness</h2> <p> &ldquo;Fairness&rdquo; in a streaming system is a measure of how evenly computing resources are applied to each incoming message. A fair system is characterized by consistent and predictable latency for the processing of each message. An emergent effect of fair systems is that messages are journaled, processed, and transformed for downstream computation in approximately the order received. The output of a fair system roughly matches input in real time; a perfectly fair system would provide a perfect correspondence between input messages and output messages. </p> <p> &ldquo;Fair enough&rdquo; systems have a some acceptable amount of jitter in the ordering of the input vs. output messages. </p> <p> You might expect that streaming systems are generally fair, but systems based on Akka rarely are because of the implementation details of Akka's multithreaded architecture. Instead, Akka-based systems enqueue incoming messages for each actor, and the Akka scheduler periodically initiates the processing of each actor's input message queue. This is actually a type of batch processing. The reason Akka does this is so the high cost of CPU context switches can be amortized over several messages, to keep system throughput at an acceptable rate. </p> <h2 id="contextSwitching">Context Switching</h2> <p> It is desirable for the primary type of context switching in Akka systems to be between tasks on a thread because other types of context switching are more expensive. However, <a href='https://stackoverflow.com/questions/20288379/analysis-performance-of-forkjoinpool' target="_blank" rel="nofollow">switching tasks on a thread</a> costs a surprisingly large amount of CPU cycles. To put this into context, the accompanying table shows various latencies compared to the amount of time necessary for a context switch on an <a href='https://ark.intel.com/products/27218/Intel-Xeon-Processor-5150-4M-Cache-2_66-GHz-1333-MHz-FSB' target="_blank" rel="nofollow">Intel Xeon 5150</a> CPU <a href='#2'><sup style='font-size:smaller'>[2]</sup></a>. </p> <div style=""> <picture> <source srcset="/blog/images/latency.webp" type="image/webp"> <source srcset="/blog/images/latency.png" type="image/png"> <img src="/blog/images/latency.png" title="Latency numbers every programmer should know" class=" liImg2 rounded shadow" alt="Latency numbers every programmer should know" /> </picture> </div> <p> As you can see, a lot of computing can be done during the time that a CPU performs a context switch. My <a href='https://scalacourses.com/showCourse/45' target="_blank" rel="nofollow">Intermediate Scala</a> course has several lectures that explore multithreading in great detail, with lots of code examples. </p> <h2 id="distributed">Building Distributed OSes is Expensive and Hard</h2> <p> Akka is rather low-level, and the actor model it is built around was originally conceived 45 years ago. Applications built using Akka require a lot of custom plumbing. Akka applications are rather like custom-built distributed operating systems spanning multiple <a href='https://en.wikipedia.org/wiki/OSI_model' target="_blank" rel="nofollow">layers of the OSI model</a>, where application-layer logic is intertwined with transport-layer, session-layer and presentation-layer logic. </p> <p> Debugging and tuning a distributed system is inherently difficult. Building an operating system that spans multiple network nodes and continues to operate properly in the face of network partitions is even harder. Because each Akka application is unique, customers find distributed systems built with Akka to be expensive to maintain. </p> <h2 id="summary">Kafka vs. Akka</h2> <p> For most software engineering projects, it is better to use vendor-supplied or open-source libraries for building distributed systems because the library maintainers can amortize the maintenance cost over many systems. This allows users to focus on their problem, instead of having to develop and maintain the complex and difficult-to-understand plumbing for their distributed application. </p> <p> Akka is a poor choice for apps that need to do small amounts of computation on high volumes of messages where latency must be minimal and roughly equal for all messages. <a href='https://docs.confluent.io/current/streams/index.html' target="_blank" rel='nofollow'>Kafka Streams</a> is a better choice for those types of applications because programmers can focus on the problem at hand, without having to build and maintain a custom distributed operating system. <a href='https://docs.confluent.io/current/streams/concepts.html#ktable' target="_blank" rel='nofollow'>KTable</a> is also a nice abstraction to work with. </p> <!--p> Kafka also provides a higher-level mechanism for partitioning work among many servers than <a href='https://doc.akka.io/docs/akka/current/scala/common/cluster.html' target="_blank">Akka clusters</a>, and it is no less efficient or fair. </p--> <div style=""> <picture> <source srcset="/assets/images/robotCircle400.webp" type="image/webp"> <source srcset="/assets/images/robotCircle400.png" type="image/png"> <img src="/assets/images/robotCircle400.png" title="EmpathyWorks™ – Artificial Personality For AI Systems" class=" right liImg" style="width: 150px" alt="EmpathyWorks™ – Artificial Personality For AI Systems" /> </picture> </div> <h2 id="motivation">Motivation</h2> <p> <a href='https://www.empathyWorks.ai' target="_blank" rel="nofollow">EmpathyWorks<sup>&trade;</sup></a> is a platform for modeling human emotions and network effects amongst groups of people as events impinge upon them. Each incoming event has the potential to cause a cascade of second-order events, which are spread throughout the network via people's relationships with one another. The actual processing required for each event is rather small, but given that the goal is to model everyone on earth (all 7.5 billion people), this is a huge computation to continuously maintain. For EmpathyWorks to function properly, incoming messages must be processed somewhat fairly. </p> <h2 id="ktables">KTables and Compacted Topics</h2> <p> Kafka Streams makes it easy to transform one or more immutable streams into another immutable stream. A KTable is an abstraction of a changelog stream, where each record represents an update. Instead of a using actors to track the state of multiple entities, a KTable provides an analogy to database tables, where the rows in a table contains the current state for each of the entities. Kafka is responsible for dealing with network partitions and data collisions, so the application layer does not become polluted with lower-layer concerns. </p> <p> Kafka&rsquo;s <a href='https://stackoverflow.com/questions/53390170/how-to-create-kafka-compacted-topic' target="_blank" rel="nofollow">compacted topics</a> are a unique type of stream that only maintains the most recent message for each key. This produces something like a <i>materialized</i> or table-like view of a stream, with up-to-date values (subject to eventual consistency) for all key/value pairs in the log. </p> <hr class="separator"> <p id="1"> [1] From the <a href='https://letitcrash.com/post/40755146949/tuning-dispatchers-in-akka-applications' target="_blank" rel="nofollow">Akka blog</a>: &ldquo;... pay close attention to your &lsquo;throughput&rsquo; setting on your dispatcher. This defines thread distribution &lsquo;fairness&rsquo; in your dispatcher &mdash; telling the actors how many messages to handle in their mailboxes before relinquishing the thread so that other actors do not starve. However, a context switch in CPU caches is likely each time actors are assigned threads, and warmed caches are one of your biggest friends for high performance. It may behoove you to be less fair so that you can handle quite a few messages consecutively before releasing it.&rdquo; </p> <p id="2">[2] Taken from <a href='https://gist.github.com/nelsnelson/3955759' target="_blank" rel="nofollow">Latency numbers every programmer should know.</a> The Intel Xeon 5150 was released in 2006; for a more up-to-date information about CPU context switch time, please see <a href='https://www.quora.com/Linux-Kernel-How-much-processor-time-does-a-process-switching-cost-to-the-process-scheduler#pphKmT' target="_blank" rel="nofollow">this Quora answer</a>. Although CPUs have gotten slightly faster since 2006, networks and SSDs have gotten much faster, so the relative latency penalty has actually increased over time. </p> Resume-Driven Development 2017-05-30T00:00:00-04:00 https://mslinn.github.io/blog/2017/05/30/resume-driven-development <div style="text-align: right;"> <a href="https://spark.apache.org/" target="_blank" rel="nofollow"><picture> <source srcset="/assets/images/spark.webp" type="image/webp"> <source srcset="/assets/images/spark.png" type="image/png"> <img src="/assets/images/spark.png" title="Apache Spark logo" class="right quartersize liImg2 rounded shadow" style="padding: 1em;" alt="Apache Spark logo" /> </picture></a> </div> <p>I had occasion to speak with a technical manager at a Very Large Corporation at length recently. He wanted advice on where to find Spark programmers. After my usual 20 questions I realized that his project did not have any requirement for the usual Spark capabilities:</p> <ul> <li>No need of machine learning or pattern recognition.</li> <li>He emphatically did not want to process streaming data.</li> <li>He had a rather small volume of data.</li> <li>He had no need of interactive analysis.</li> <li>There was no need for ETL.</li> <li>&ldquo;Data enrichment&rdquo; in his case was no more difficult than joining two SQL tables.</li> <li>There was no requirement for trigger event detection.</li> <li>... and there was no requirement for session analysis.</li> </ul> <p>This man just wanted to put &ldquo;I managed the development of an Apache Spark application&rdquo; on his resume.</p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/cart_before_the_horse1_450x313.webp" type="image/webp"> <source srcset="/blog/images/cart_before_the_horse1_450x313.png" type="image/png"> <img src="/blog/images/cart_before_the_horse1_450x313.png" title="The cart is before the horse" class="center halfsize liImg2 rounded shadow" alt="The cart is before the horse" /> </picture> </div> <h2 id="turd">Shine Up That Turd</h2> <p> Buffing up a resume with a nonsensical misapplication of the latest software fashion is nothing new. Few people involved with such a project have a positive experience, however. When complex technologies are misapplied they often fail to deliver value, or the project just fails, which contributes to the <a href='https://www.gartner.com/technology/research/methodologies/hype-cycle.jsp' target="_blank" rel="nofollow">trough of disillusionment</a> in the Gartner Hype Cycle. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/WCFields_450x576.webp" type="image/webp"> <source srcset="/blog/images/WCFields_450x576.png" type="image/png"> <img src="/blog/images/WCFields_450x576.png" title="W. C. Fields" class="center halfsize liImg2 rounded shadow" alt="W. C. Fields" /> </picture> </div> <p> Consider the cynical and selfish world view that such a person must have to engage in this sort of activity. Software professionals are rewarded for being fashionable, and part of the mystique is to say and do things that others don&rsquo;t fully comprehend. If one keeps moving fast enough, and learns some key phrases to use as put downs that sound impressive but have no meaning when taken out of context (&ldquo;your approach would likely cause a data race condition&rdquo;), others are kept off balance long enough for the perpetrator to achieve advantage. </p> <p> As <a href='https://en.wikipedia.org/wiki/W._C._Fields' target="_blank" rel='nofollow'>W.C. Fields</a> said 100 years ago: &ldquo;If you can&rsquo;t dazzle them with brilliance, baffle them with bullshit.&rdquo; He also made a movie entitled <a href="https://www.youtube.com/watch?v=-AdXkJbW1Tg" target="_blank" rel="nofollow">Never Give a Sucker an Even Break</a>... but I digress. </p> <h2 id="fashion">Fashion is the Enemy of Delivering Valuable Results on Time</h2> <div style="text-align: center;"> <picture> <source srcset="/assets/images/fashion_450x675.webp" type="image/webp"> <source srcset="/assets/images/fashion_450x675.png" type="image/png"> <img src="/assets/images/fashion_450x675.png" title="Fashion model" class="center halfsize liImg2 rounded shadow" alt="Fashion model" /> </picture> </div> <p> I often remark that the software business is just like the fashion business. The cycle goes like this: </p> <ol> <li>The latest thing is preannounced. No documentation is available. Big claims are made.</li> <li>Early code is provided, but it is slow, very buggy, there is no documentation, and the wild claims continue.</li> <li>Seminars pump up the hype. Arm-waving abounds. &lsquo;Experts&rsquo; associated with vendors are interviewed.</li> <li>A whole new vocabulary is invented to describe the features. However, since those terms are never defined, few people realize that the message is actually &ldquo;same old stuff with a new coat of paint.&rdquo; The cool kids learn the words and can parrot them back they way they heard them, but they don't actually know what they are saying. Even if you knew what the phrase &ldquo;a monad is a monoid in the category of endofunctors&rdquo; means, it would not help you add value to a business process... and for most programmers who work in the business world, that is what actually pays their bills.</li> <li>Gartner, ThoughtWorks and O&rsquo;Reilly Radar say nice things about the new tech.</li> <li>Marketing and sales activity shifts into high gear. Everyone&rsquo;s laptop has stickers on it that show the owner is hip.</li> <li>Some incomplete and mostly useless docs trickle out. Release candidates are eventually made available but it is not obvious how they provide value without significant magic being applied.</li> <li>Version 1 finally arrives, completely useless to everyone except those few customers who paid huge dollars to the vendor, so they could get the bragging rights for what is essentially a custom build.</li> <li>No useful benchmarks are available. Bugs crawl out from every crevasse. Documentation reads like it was badly translated from Chinese into Bantu, then into Navajo and then English.</li> <li>Version 1.1 comes out, to address stop-ship problems that somehow did not prevent version 1 from shipping.</li> <li>Version 2 is announced, completely incompatible with Version 1.x, and it is touted as revolutionary.</li> <li>The digital lemmings decide that this is the cliff they hold dearest in their hearts, and proceed to jump from it <i>en masse</i>.</li> <li>Gartner, ThoughtWorks and O&rsquo;Reilly Radar downgrade the new tech to &ldquo;meh&rdquo;</li> <li>... and the churn continues.</li> </ol> <p> Documentation is not written to be understood, they are written to encourage payments to the digital elite &ndash; who have no interest in providing value; they just want to make money from foolish lemmings. This encourages others to aspire to be elitists. The hype cycle only works because our society accepts bullshit, and software has become a cultural phenomenon. We have the software quality and utility that we deserve as a society. If we want better software, we as a society will have to modify our purchasing behavior accordingly. </p> <p> The software business has often demonstrated an insatiable need to be cool, at the expense of providing little or no value because evidencing value is either too hard for the many posers in the software business, or too boring for those who are capable but jaded. </p> Better Syntactic Sugar for Scala Futures 2017-05-25T00:00:00-04:00 https://mslinn.github.io/blog/2017/05/25/better-syntactic-sugar-for-scala-futures <div style="text-align: right;"> <picture> <source srcset="/blog/images/sugarCubes_225x169.webp" type="image/webp"> <source srcset="/blog/images/sugarCubes_225x169.png" type="image/png"> <img src="/blog/images/sugarCubes_225x169.png" title="Sugar cubes" class="right quartersize liImg2 rounded shadow" alt="Sugar cubes" /> </picture> </div> <p> Ever since Scala&rsquo;s <code>Future</code>s were initially provided as part of Akka 2.0, programmers have been confused by the non-intuitive syntax. That is why <a href='https://www.ScalaCourses.com' target="_blank" rel='nofollow'>ScalaCourses.com</a> dedicates an entire lecture to <a href='https://www.scalacourses.com/student/showLecture/172' target="_blank" rel='nofollow'>For-comprehensions With Futures</a>, as part of an 8 lecture series on Scala <code>Future</code>s within the <a href='https://www.scalacourses.com/showCourse/45'>Intermediate Scala</a> course. </p> <p> As Viktor Klang <a href='https://viktorklang.com/blog/Futures-in-Scala-protips-2.html' target="_blank" rel='nofollow'>points out</a> in his blog, the following code does not run 3 <code>Future</code>s in parallel: </p> <pre data-lt-active='false'> def doSomething(someParameter: SomeType) (implicit ec: ExecutionContext): Future[Something] = for { v1 &lt;- Future(someCalculation()) v2 &lt;- Future(someOtherCalculation()) v3 &lt;- Future(someDifferentCalculation()) } yield doSomethingWith(v1, v2, v3) </pre> <p> The compiler has no way of ascertaining the programmer&rsquo;s intent &ndash; perhaps it is desirable for some reason to run the 3 <code>Future</code>s one after the other. </p> <p>Viktor suggests this syntax to make futures run in parallel:</p> <pre data-lt-active='false'> def doSomething(someParameter: SomeType) (implicit ec: ExecutionContext): Future[Something] = for { f1 = Future(someCalculation()) f2 = Future(someOtherCalculation()) f3 = Future(someDifferentCalculation()) v1 &lt;- f1 v2 &lt;- f2 v3 &lt;- f3 } yield doSomethingWith(v1, v2, v3) </pre> <p> While Viktor&rsquo;s solution works, it is verbose. Worse, it silently fails to run the futures in parallel if the programmer accidentally writes even one of the expressions out of order: </p> <pre data-lt-active='false'> def doSomething(someParameter: SomeType) (implicit ec: ExecutionContext): Future[Something] = for { f1 = Future(someCalculation()) v1 &lt;- f1 f2 = Future(someOtherCalculation()) v2 &lt;- f2 f3 = Future(someDifferentCalculation()) v3 &lt;- f3 } yield doSomethingWith(v1, v2, v3) </pre> <p> Again, the compiler has no way of ascertaining the programmer's intent, so it should not generate an error or warning message. </p> <h2>We Need A Macro</h2> <p> A new right-associative operator, implemented as a Scala macro, would make the programmer's intent clear. Let&rsquo;s call this operator <code>&lt;=:</code> (parallel generator). The above code could be rewritten using the parallel generation operator like this: </p> <pre data-lt-active='false'> def doSomething(someParameter: SomeType) (implicit ec: ExecutionContext): Future[Something] = for { v1 &lt;=: Future(someCalculation()) v2 &lt;=: Future(someOtherCalculation()) v3 &lt;=: Future(someDifferentCalculation()) } yield doSomethingWith(v1, v2, v3) </pre> <p>There is no longer any doubt that the programmer intended for all 3 futures to run in parallel.</p> <p> The macro would examine all the for-expression&rsquo;s generators and expand consecutive expressions that use the <code>&lt;=:</code> operator to a series of variable declarations using the <code>=</code> operator followed by a series of assignments using the <code>&lt;-</code> operator, exactly as we saw earlier: </p> <pre data-lt-active='false'> def doSomething(someParameter: SomeType) (implicit ec: ExecutionContext): Future[Something] = for { f1 = Future(someCalculation()) f2 = Future(someOtherCalculation()) f3 = Future(someDifferentCalculation()) v1 &lt;- f1 v2 &lt;- f2 v3 &lt;- f3 } yield doSomethingWith(v1, v2, v3) </pre> <p> Because the compiler &lsquo;knows&rsquo; that the programmer&rsquo;s intention was to run the <code>Future</code>s in parallel, this sort of error could cause an error or warning message to be generated: </p> <pre data-lt-active='false'> def doSomething(someParameter: SomeType) (implicit ec: ExecutionContext): Future[Something] = for { v1 &lt;=: Future(someCalculation()) x <- List(1, 2, 3) v2 &lt;=: Future(someOtherCalculation()) y <- List("a", "b") v3 &lt;=: Future(someDifferentCalculation()) } yield doSomethingWith(v1, v2, v3) </pre> <p>... or the macro might reorder the generators and issue a warning that it did so.</p> <h2>Include the Macro in Scala 2.12.x</h2> <p> This macro should become part of the Scala language so long as <code>Future</code>s are part of the standard runtime. If and when <code>Future</code>s are hived out of the standard runtime, the macro should be packaged with <code>Future</code>s. </p> <hr /> <p> PS: <a href="https://twitter.com/flaviusbraz" target="_blank" rel='nofollow'>@flaviusbraz</a> tweeted on May 25, 2017: <a href='https://twitter.com/flaviusbraz/status/867868408267685888' target="_blank" rel='nofollow'>&ldquo;Easily doable with a macro transformation. In fact, I&rsquo;ve implemented this transformation at Twitter, but it&rsquo;s not open source.&rdquo;</a> </p> Exploratory Conversation With AIs 2017-04-29T00:00:00-04:00 https://mslinn.github.io/blog/2017/04/29/exploratory-conversations-with-ais <p> This is a high-level proposal, not documentation for any specific system that exists today. I would be happy to discuss possible implementation details with interested parties, including the creation of a prototype. </p> <h2 id="overview">Intents and Subgoals</h2> <p> Some of today&rsquo;s interactive voice response systems, such as AWS&rsquo;s <a href="https://aws.amazon.com/lex/" target="_blank" rel='nofollow'>Lex</a>, Amazon&rsquo;s <a href="https://aws.amazon.com/alexa/" target="_blank" rel='nofollow'>Alexa</a>, and Google&rsquo;s <a href="https://cloud.google.com/dialogflow/docs/" target="_blank" rel='nofollow'>Dialogflow</a>, use <i>intent</i>s as a means of gathering the required parameters to fulfill a voice command. The next big step in voice interfaces to computation would be the ability to have an exploratory conversation, where one or more subgoals may or may not emerge. </p> <p> Subgoals might be fulfilled by intents. Continuous tracking of viable subgoals could be accomplished by a <a href="https://en.wikipedia.org/wiki/Constraint_programming" target="_blank" rel='nofollow'>constraint-based solver</a> that uses rules to participate in an exploratory conversation by recognizing and prioritizing potential subgoals, thereby activating and deactivating intents during the conversation. Once a user confirms that a potential subgoal is desirable, the system might consider that goal to be the only goal worth pursuing, or it might continuously elicit more information from the user regarding other potential subgoals. User-supplied information might be applied to a model associated with a subgoal/intent, and/or it might be retained as a user preference. </p> <div style="text-align: center;"> <picture> <source srcset="/assets/images/anthropomorph.webp" type="image/webp"> <source srcset="/assets/images/anthropomorph.png" type="image/png"> <img src="/assets/images/anthropomorph.png" title="Do not anthropomorphize computers. They really hate that!" class="center halfsize liImg2 rounded shadow" alt="Do not anthropomorphize computers. They really hate that!" /> </picture> </div> <h2 id="anthropomorphism">Anthropomorphism</h2> <p>If an AI system is a good conversationalist, even if it has no physical presence, people will <a href="https://en.wikipedia.org/wiki/Anthropomorphism#In_computing" target="_blank" rel='nofollow'>anthropomorphically</a> attribute the system with human traits, emotions, and intentions. This should not be discouraged. Rather, AI systems should aspirationally model our higher selves. The movie <a href="https://en.wikipedia.org/wiki/Her_(film)" target="_blank" rel='nofollow'>Her</a> explored this in a charming way. </p> <h2 id="breakdown">Breaking It Down</h2> <p>An interactive voice response system recognizes an <i>intent</i> by a key word or phrase uttered by the user. For example, if the user says: &ldquo;Computer, order dinner&rdquo;, and the system was previously &lsquo;taught&rsquo; to understand the word &ldquo;order&rdquo; as a key word for launching the <code>OrderDinner</code> intent, the system would then elicit the kind of food the user wanted, how many people would be eating, etc, and then order the desired dinner.</p> <p style="font-style: italic">Anthropomorphically: if intents are the only programming technique employed, they present interactive voice systems as slaves.</p> <p>Intents are useful for recognizing commands and gathering enough associated data to carry out the command. Intents are not useful for open-ended conversations where there is no explicit goal, or potential subgoals emerge as the result of a dialog. Once a subgoal is identified and confirmed, however, processing an intent is an efficient mechanism for fulfilling the emergent subgoal. </p> <p>Designers of chatbots and video games are familiar with goal-seeking behavior. <i>Exploratory conversation</i> is how people interact with each other IRL (in real life). Each individual uses a variety of strategies for initiating exploratory conversation, and participating in it. For example, small talk at a gathering is a useful skill for getting to know other people, and a socially adept practitioner of small talk can innocuously gather a lot of information in this way. Another strategy is to make a controversial statement, and use the resulting banter to learn about the conversationalists and the possible subgoals that they might request. A wide variety of such strategies exist, and could be utilized by conversational systems. </p> <p style="font-style: italic">Anthropomorphically: with exploratory conversation capability, interactive voice systems can be presented socially as co-operative entities.</p> <h2 id="agenda">Agenda and Strategy</h2> <p>Unlike people, computers can only do a finite number of tasks, and voice recognition systems are programmed with a finite number of intents. I define the potential <i>agenda</i> for a chatbot or video game to be the entire scope of its pre-programmed intents. Agendas may be fully disclosed, or they might be obvious, or they might be unveiled over time. Ethical considerations might apply to the design and implementation of conversational AI systems. </p> <p>Users should be apprised of the agenda of every autonomous computer entity they encounter. To mitigate potential problems, an industrial code of conduct should be established. The Europe Union will likely be the first government to require published standards and possibly a certification process. </p> <p>I define <i>strategy</i> to mean the autonomous modification to goal-seeking behavior when a criterion is met. For example, an insurance chatbot might begin to solicit sensitive information only after it has reason to believe that a modicum of trust has been established with the prospect. How a strategy is executed is quite significant; by this I mean the degree of <i>diplomacy</i>. Some circumstances might sacrifice diplomacy to save lives, while under normal circumstances the AI entity should treat everyone with respect and kindness. Again, the Europe Union is likely to be the first government to require published standards and possibly a certification process. </p> <h2 id="2017-05-05">Update May 5, 2017</h2> <p>Today, a week after I first published this blog posting, I received the following email from <a href='https://cloud.google.com/dialogflow/docs/' target="_blank" rel='nofollow'>api.ai</a>. Two points worth mentioning:</p> <ol> <li>I have no apps running on api.ai that use their Small Talk feature.</li> <li>The Small Talk feature, even after the changes mentioned below, does not approach the capability I proposed above.</li> </ol> <p>Nonetheless, it was great to get the information.</p> <blockquote> <p>Hi Michael,</p> <p> We noticed you are using the <a href='https://dialogflow.cloud.google.com/#/agent/smalltalk' target="_blank" rel='nofollow'>Small Talk customization feature</a> in your agent, and that's why we are getting in touch. </p> <p>The Small Talk customization has been updated.</p> <p> The new version will be even faster and more accurate. It also includes some additional intents that you can customize. The update also includes changes in actions and parameters returned with the responses. </p> <p> If you use them in your business logic, please review the changes <a href='https://cloud.google.com/dialogflow/docs/' target="_blank" rel='nofollow'>here</a>. If you’re ready, you can apply change right now here. Otherwise, the changes will be applied automatically on May 29, 2017. </p> <p>If you’d like a more flexible way to implement small talk, then get your <a href='https://console.api.ai/api-client/#/agent/prebuiltAgents/Small-Talk' target="_blank" rel='nofollow'>source agent</a> and implement the use cases you care about in your timeline. </p> <p>Regards,</p> <p><a href='https://cloud.google.com/dialogflow' target="_blank">API.AI</a> Team</p> </blockquote> <h2>About the Author</h2> <p> Mike Slinn has been pursuing <a href="https://www.empathyworks.com" target="_blank" rel="nofollow">original AI research</a> (augmenting machine learning with simulations and event-driven architecture) since 2007. </p> UI Considerations for the Visually Impaired 2017-04-04T00:00:00-04:00 https://mslinn.github.io/blog/2017/04/04/ui-considerations-for-the-visually-impaired <p> If you look at a computer screen all day, and the screen has a lot of glare, the resulting eyestrain can temporarily degrade your vision to the same degree as a visually impaired person. </p> <h2 id="background">Background</h2> <p> According to the Farlex Free Medical Dictionary, <a href='https://medical-dictionary.thefreedictionary.com/Visual+Impairment' target="_blank" rel='nofollow'>visual impairment</a>, or low vision, is a severe reduction in vision that cannot be corrected with standard glasses or contact lenses and reduces a person's ability to function at certain or all tasks. The World Health Organization <a href='https://www.who.int/mediacentre/factsheets/fs282/en/' target="_blank" rel='nofollow'>estimates</a> that 285 million people are visually impaired worldwide, of which 246 million have low vision and 39 million are blind. People's vision generally deteriorates with age. </p> <p> In the US, cataracts are the most common form of visual impairment, according to the <a href='https://www.nei.nih.gov/learn-about-eye-health/resources-for-health-educators/eye-health-data-and-statistics' target="_blank" rel='nofollow'>National Eye Institute</a>. The NEI estimates that more than 24 million Americans over the age of 40 have cataracts. &ldquo;The risk of cataracts increases with each decade of life starting around age 40. By age 75, half of white Americans have cataracts. By age 80, 70 percent of whites have cataracts, compared with 53 percent of blacks and 61 percent of Hispanic Americans.&rdquo; Males are twice as likely as females to get cataracts. </p> <div style=""> <picture> <source srcset="/assets/images/1_2010_Prevalence_Rates_Age_Race_Cataract_v4.webp" type="image/webp"> <source srcset="/assets/images/1_2010_Prevalence_Rates_Age_Race_Cataract_v4.png" type="image/png"> <img src="/assets/images/1_2010_Prevalence_Rates_Age_Race_Cataract_v4.png" title="2010 US prevalence rates for cataracts by age and race" class=" liImg2 rounded shadow" style="width: 100%" alt="2010 US prevalence rates for cataracts by age and race" /> </picture> </div> <p> Cataracts start slowly. If you wear glasses, at first you think you can&lsquo;t quite clean them properly. Then you notice your sunglasses are also always dirty. Then you notice that white screens with black text are hard to read because of the glare from the white areas. </p> <p> The medical treatment is to replace the cloudy and .webp lens in the eye that has the cataract with a plastic lens. In <a href='https://ec.europa.eu/eurostat/statistics-explained/index.php/Surgical_operations_and_procedures_statistics' target="_blank" rel='nofollow'>many European countries</a>, cataract surgery is the most common operation performed in that country. Usually people delay the surgery until their quality of life and independence his diminished markedly. </p> <h2 id="coping">Coping With Visual Impairment</h2> <h3 id="saturation">Saturated Color and White Backgrounds</h3> <p> Dark screens with a minimum of saturated color helps. Increasing font size helps, and sanserif fonts, which have fewer spindly features, are easier to read. </p> <p> Some OSes offer a dark theme. You can enable the Windows 10 dark theme from <b>Settings</b> / <b>Personalization</b> / <b>Colors</b>. Scroll down and select &lsquo;Dark&rsquo; under &lsquo;Choose your app mode&rsquo;. The Universal Windows Platform applications will immediately darken. However, if a developer did not make the effort to support the dark theme, their program will continue with their original color theme. Unfortunately, Windows File Explorer is one of those applications. </p> <div style=""> <picture> <source srcset="/assets/images/img_57a8f5e080b37.webp" type="image/webp"> <source srcset="/assets/images/img_57a8f5e080b37.png" type="image/png"> <img src="/assets/images/img_57a8f5e080b37.png" title="Enabling the Windows 10 dark theme" class=" liImg2 rounded shadow" style="width: 100%" alt="Enabling the Windows 10 dark theme" /> </picture> </div> <p>Some software offers a nice dark theme that does not require any fussing, for example IntelliJ IDEA's Darkula, Adobe Premiere Pro and Adobe Audition.</p> <p> The Firefox web browser&rsquo;s <a href='https://addons.mozilla.org/en-US/firefox/addon/dark-background-light-text/' target="_blank" rel='nofollow'>Dark Background and Light Text</a> plugin is very helpful. By default, the plugin overrides the CSS styles for all web pages, so the background is black, and text is white. The plugin can be trained to leave certain pages untouched, for those pages that become unusable with the modified default stylesheet, for example, <a href="https://calendar.google.com" target="_blank" rel='nofollow'>Google Calendar</a>. Other sites already offer a nice dark theme, so they do not need the modified theme thrust upon them, for example <a href='https://tweetdeck.twitter.com/' target="_blank" rel='nofollow'><code>tweetdeck.twitter.com</code></a>. </p> <p> Some software offers a dark theme which requires further modification before a good result is a achieved. For example, the Thunderbird email client with the <a href='https://addons.thunderbird.net/en-US/thunderbird/addon/tt-deepdark/' target="_blank" rel='nofollow'>TT DeepDark</a> addon gets you partway there, then <b>Tools</b> / <b>Options</b> / <b>Display</b> / <b>Fonts & Colors</b> / <b>Colors</b> shows the following dialog, where you can set the colors as shown: </p> <div style="text-align: center;"> <picture> <source srcset="/assets/images/ttDeepDarkColors.webp" type="image/webp"> <source srcset="/assets/images/ttDeepDarkColors.png" type="image/png"> <img src="/assets/images/ttDeepDarkColors.png" title="Enabling the Windows 10 dark theme" class="center halfsize liImg2 rounded shadow" alt="Enabling the Windows 10 dark theme" /> </picture> </div> <p> As another example, Adobe Reader has a dark theme, which can be enabled via <b>Edit</b> / <b>Preferences</b> / <b>Accessibility</b>; enable <b>Replace Document Colors</b> and click on <b>Custom Color</b>; change the <b>Page Background</b> to black and <b>Document Text</b> to light gray. Disable <b>Only change the color of black text or line art</b>, and ensure that <b>Change the color of line art as well as text</b> is enabled. </p> <div style=""> <picture> <source srcset="/assets/images/adobeReaderColors_690x525.webp" type="image/webp"> <source srcset="/assets/images/adobeReaderColors_690x525.png" type="image/png"> <img src="/assets/images/adobeReaderColors_690x525.png" title="Enabling the Adobe Reader dark theme" class=" liImg2 rounded shadow" alt="Enabling the Adobe Reader dark theme" /> </picture> </div> <h3 id="hard">Small Text and Low Contrast Colors</h3> <p> Users who know CSS can enforce a custom stylesheet on a per-site or a per-page basis for hard-to-read web pages. The <a href='https://chrome.google.com/webstore/detail/my-style/ljdhjpmbnkbengahefamnhmegbdifhlb?hl=en' rel='nofollow'>My Style</a> plugin for the Chrome browser activates with Ctrl-M, and users can enter CSS to suit. As any web developer knows, <kbd>Ctrl</kbd>-<kbd>Shift</kbd>-<kbd>C</kbd> starts HTML inspection, so the HTML in question can be viewed, and the CSS experimented with before immortalizing the changes with My Style. </p> <h2 id="design">Design Considerations</h2> <p> If your web page has images in it, try to use transparency instead of a white background. It would help visually impaired readers to only inflict strong primary colors, or a white background, when necessary. For example, this entry in a stylesheet would only show a light gray background when the user mouses over the image, otherwise, the background would remain transparent: </p> <pre data-lt-active='false'>img:hover { background-color: #666; }</pre> Are My Hands-Free Devices Always Listening? 2017-01-15T00:00:00-05:00 https://mslinn.github.io/blog/2017/01/15/are-my-handsfree-devices-always-listening <p> The short answer: probably not. </p> <p> A longer answer: Depends on what you mean by &lsquo;listening&rsquo;. If you mean &ldquo;is Alexa storing or sending every sound or voice utterance to a server&rdquo;, the answer is again &ldquo;probably not&rdquo;. </p> <p> A more complete answer: unless all the source code for a device is made available for scrutiny, and the build process is similarly examined, the only way to be sure that device does not have operating modes that are contrary to expectations would be to connect the device to a network monitor that only allows communication with specific servers at designated times. Normal computer security concerns also pose risks; for example, viruses and other malware could be injected into a device could cause it to behave differently. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/echoDevices.webp" type="image/webp"> <source srcset="/blog/images/echoDevices.png" type="image/png"> <img src="/blog/images/echoDevices.png" title="Amazon Echo device locations in homes" class="center liImg2 rounded shadow" alt="Amazon Echo device locations in homes" /> </picture> </div> <h2 id="update2021-01-12">At Least Echo&rsquo;s Mute Button Works</h2> <p> <b>Update 2021-01-12</b> &ndash; Amazon Echo&rsquo;s mute button was <a href='https://electronupdate.blogspot.com/2021/01/amazon-echo-flex-microphone-mute-real.html' target='_blank' rel='nofollow'>reverse engineered</a>. </p> <p class='quote'> The mute button appears to be very real and functional. When the button glows red the power is removed from the microphones. </p> <h2 id="paranoia" style='clear: both'>Some Paranoia is Healthy</h2> <div style="text-align: right;"> <picture> <source srcset="/blog/images/aristotle_300x344.webp" type="image/webp"> <source srcset="/blog/images/aristotle_300x344.png" type="image/png"> <img src="/blog/images/aristotle_300x344.png" title="Mattel Aristotle" class="right liImg" style="width: 300px" alt="Mattel Aristotle" /> </picture> </div> <p> <a href='https://www.engadget.com/2017/01/03/mattel-aristotle-echo-speaker-kids/' target="_blank" rel='nofollow'> Mattel&rsquo;s Aristotle</a> is targeted at children. It uses Microsoft Bing for searching, Microsoft Cortana for voice processing and streams video to the cloud. It can read bedtime stories and play soothing sounds if your child wakes up at the night. It can recognize your children&rsquo;s imperfect speech, and apparently adapts as they get older and become curious about the world. Aristotle is an AI to help raise your child. Did they do a terrific job or a terrible job? It depends on the factors you take into consideration. </p> <p> Aristotle can respond to adults differently than to children. Its logging capability is profound. It tracks things like wet diapers and feedings, and can order supplies when asked by an adult. </p> <p> Aristotle can use object recognition to identify flashcards, or co-opt a toy without electronics and thereby enhance it with sound effects or even another personality. </p> <p> Mattel has about 500 partners, and they have been invited to build connected toys and apps. The hardware uses 256-bit encryption for all transmissions to Aristotle&rsquo;s servers, and data is handled internally in compliance with COPAA and HIPAA. </p> <p class='pullQuote'>It is not difficult to write code that detects when a child is alone</p> <p> How hard would it be to write code that detects when the child is alone? If a malicious entity wanted to, they could embed their program in the device, use IP geolocation and other characteristics to identify specific families, and cause the device to behave differently and/or tell the child things when no adults were paying attention. Who knows what malicious software might do? </p> <h2 id="open">Open Source To The Rescue</h2> <div style="text-align: right;"> <picture> <source srcset="/assets/images/louis.webp" type="image/webp"> <source srcset="/assets/images/louis.png" type="image/png"> <img src="/assets/images/louis.png" title="Justice Louis Brandeis" class="right liImg2 rounded shadow" alt="Justice Louis Brandeis" /> </picture> </div> <p> Closed source systems cannot be adequately vetted for public usage. It is conceivable that selected children might become secretly radicalized by their toy. In 1913 <a href='https://www.brandeis.edu/legacyfund/bio.html' target="_blank" rel='nofollow'>Justice Louis Brandeis</a> said &ldquo;Sunlight is the best disinfectant&rdquo;. Open-source applications, with updates that can be vetted by any interested party, are the only way to ensure these devices are truly safe. </p> Hands-Free Voice as a User Interface 2017-01-10T00:00:00-05:00 https://mslinn.github.io/blog/2017/01/10/voice-activated-apps <p> Amazon&rsquo;s Alexa is a runaway success. Already, 5% of US households have a hands-free voice operated device. 50% of Echo owners keep one in the kitchen. <a href='https://www.gartner.com/doc/3021226/market-trends-voice-ui-consumer' target="_blank" rel='nofollow'>Gartner predicts</a> that by 2018, 30% of our interactions with technology will be via voice conversations. The near future will have many hands-free, voice-driven, Internet-aware applications, and some will also provide a web interface in order to satisfy use cases that require a text or graphic display. </p> <h2 id="near">The Near Future</h2> <p> The near future will have many hands-free, voice-driven, Internet-aware applications, and some will also provide a web interface to satisfy use cases that require a text or graphic display. Potential usages include hands-free voice control of applications running on computers, tablets and phones, hands-free voice control of home devices and vehicles, dictaphones, and games that provide anthropomorphic characters with the ability to generate and understand speech. </p> <div> <div style=""> <picture> <source srcset="/blog/images/google-home_384x500.webp" type="image/webp"> <source srcset="/blog/images/google-home_384x500.png" type="image/png"> <img src="/blog/images/google-home_384x500.png" title="Google Home smart speaker" class=" liImg right rounded shadow" style="height: 500px" alt="Google Home smart speaker" /> </picture> </div> <div style=""> <picture> <source srcset="/blog/images/echo_205x500.webp" type="image/webp"> <source srcset="/blog/images/echo_205x500.png" type="image/png"> <img src="/blog/images/echo_205x500.png" title="Amazon Echo smart speaker" class=" liImg left" style="height: 500px" alt="Amazon Echo smart speaker" /> </picture> </div> </div> <p style="clear: both"> Services that developers and integrators can use for hands-free, voice-operated applications include <a href='https://www.amazon.com/gp/help/customer/display.html?nodeId=201602040' target="_blank" rel='nofollow'>Amazon's Alexa</a>, Google's Assist and Voice, Apple's Siri, and Microsoft's Cortana. </p> <h2 id="present">The Present</h2> <p> Only recently has it become feasible to use hands-free voice as a user interface. The best hands-free, voice applications are innately distributed &mdash; that is, they perform some computation on a local device, and they also require a connection to a server to do the heavy computation necessary for a high-quality experience. Most companies developing applications that use hands-free voice as a user interface require and will continue to require services provided by third parties. However, from the point of view of the application developer's company, the data shared with these third parties can represent a significant security risk. From the user's point of view, this data can represent a potentially significant privacy breach. I'm going to discuss why this is so in this article, and what can be done about it. </p> <p> Privacy and security concerns for synthesizing speech are minimal because quality speech can be synthesized without context specific to an individual. This means that user data need not be associated with the words or phrases being synthesized, so anonymous phrases can be and should be sent to the remote service that generates the audio files produced by the speech synthesis. </p> <p> The best value for high-quality voice generation is achieved with a distributed solution. In this scenario, voice generation is simple to initiate: a text string is sent to a remote service, and an audio clip containing synthesized speech is returned. Most voice generators support embedded markup to control inflection. For example, <a href='https://docs.aws.amazon.com/polly/latest/dg/ssml.html' target="_blank" rel='nofollow'>Amazon Polly</a> and <a href='https://api.ai' target="_blank" rel='nofollow'>Google's Assistant</a> (formerly <code>api.ai</code>) both use <a href='https://developers.google.com/actions/reference/ssml' target="_blank" rel='nofollow'>SSML</a>; Apple uses a <a href='https://eclecticlight.co/2015/12/09/opening-access-text-speech/' target="_blank" rel='nofollow'>variety of techniques</a> across its products, and Microsoft uses <a href='https://en.wikipedia.org/wiki/Microsoft_Speech_API' target="_blank" rel='nofollow'>SAPI</a>. </p> <p> In contrast, the heavy lifting necessary to recognize unconstrained vocabulary requires lots of compute resource, and raises privacy and ethical issues. The main issues are: </p> <ol> <li><a href='#trigger'>Recognizing</a> a trigger word or phrase</li> <li><a href='#training'>Training</a> a voice recognition engine</li> <li>Determining the appropriate <a href='#privacy'>privacy/effectiveness</a> trade-off for your application</li> <li><a href='#integrating'>Integrating</a> with third-party or proprietary services</li> </ol> <h2 id="components">What Are These Things Made From?</h2> <p> Hands-free voice operated devices have most of the same components as a tablet, but often do not have a screen. An ARM A8 to A11 processor is typically embedded in a chip die that also contains a powerful digital signal processor. In other words, they can do a lot of computing; they are more powerful than any mobile phone, and they are more powerful than most tablets. </p> <div class="centered"> <div style=""> <picture> <source srcset="/blog/images/echoParts.webp" type="image/webp"> <source srcset="/blog/images/echoParts.png" type="image/png"> <img src="/blog/images/echoParts.png" title="Amazon Echo parts" class=" liImg" style="width: 100%; max-height: 45%" alt="Amazon Echo parts" /> </picture> </div> <div style=""> <picture> <source srcset="/blog/images/echoBoard.webp" type="image/webp"> <source srcset="/blog/images/echoBoard.png" type="image/png"> <img src="/blog/images/echoBoard.png" title="Amazon Echo PC board" class=" liImg2 rounded shadow" style="width: 100%; max-height: 45%" alt="Amazon Echo PC board" /> </picture> </div> <div style=""> <picture> <source srcset="/blog/images/tiDM37x.webp" type="image/webp"> <source srcset="/blog/images/tiDM37x.png" type="image/png"> <img src="/blog/images/tiDM37x.png" title="TI DM37x processors" class=" liImg2 rounded shadow" style="width: 500px; max-height: 45%" alt="TI DM37x processors" /> </picture> </div> </div> <p style="clear: both"> The open source ecosystem that has grown up around these DSP/CPU chips includes a variety of operatoring systems, including real-time and various Linux distributions, and many applications. The OSes are: Android, DSP/BIOS, Neutrino, Integrity, Windows Embedded CE, Linux, and VxWorks. It is quick and easy to design a complete device similar to Alexa and bring it to market. </p> <h2 id="trigger" style="clear: both">Recognizing a trigger word or phrase</h2> <p> One of the first major challenges one enounters when designing a device that responds to voice, is how to ignore silence, or recognizing noise that should be ignored. This requires some level of signal processing. In contrast, requiring the user to push a button makes for a a much simpler user interface, but this requirement would greatly restrict the types of applications possible. </p> <div style=""> <picture> <source srcset="/blog/images/Icom-IC-706MKIIG_300x180.webp" type="image/webp"> <source srcset="/blog/images/Icom-IC-706MKIIG_300x180.png" type="image/png"> <img src="/blog/images/Icom-IC-706MKIIG_300x180.png" title="Icom 706 Mark II g" class=" right" style="width: 300px" alt="Icom 706 Mark II g" /> </picture> </div> <p> I am a ham radio operator, and I use the well-known protocol for addressing a specific individual when using a broadcast medium. If I want to talk to another ham operator, I address them by their call sign three times and await their acknowledgement before giving my message. &ldquo;KG6LBG, KG6LBG, KG6LBG, this is KG6LDE, do you read me?&rdquo; If that person hears their call sign, they reply &ldquo;KG6LDE, this is KG6LBG, go ahead.&rdquo; &ldquo;KG6LBG, I just called to say hello, over.&rdquo; &ldquo;KG6LDE, Hello yourself, over and out.&rdquo; </p> <p> Hands-free voice applications also need to recognize a trigger word or phrase (also known as a hotword) that prefaces an audio stream which will be processed for voice recognition. Without such a trigger, either the user would need to press a button to start recording their speech, or a continuous audio stream would have to be processed. I'm interested in hands-free voice recognition. Since the voice recognition processing must be done on a remote service, a lot of bandwidth and CPU power would be wasted processing silence or irrelevant sound. Because it is undesirable to a continuous audio stream to a server to accurately recognize trigger words, the methods used to recognize them are less accurate. </p> <h3 id="alternatives">Proprietary Alternatives</h3> <p> Both of these products can be configured to perform the trigger word recognition without requiring any bandwidth between the CPU running the recognition program and a server. Neither of them provide a JavaScript implementation, which means that unless the trigger word recognition program in installed on the local machine, it must be installed on a connected server and bandwidth will be used at all times. </p> <ul> <li> <div style="text-align: right;"> <picture> <source srcset="/blog/images/THF-VC-72dpi.webp" type="image/webp"> <source srcset="/blog/images/THF-VC-72dpi.png" type="image/png"> <img src="/blog/images/THF-VC-72dpi.png" title="Truly Handsfree voice control" class="right " style="width: 250px" alt="Truly Handsfree voice control" /> </picture> </div> Sensory, Inc's <a href='https://www.sensory.com/products/technologies/trulyhandsfree/' target="_blank" rel="nofollow">TrulyHandsfree library</a> (<a href='https://github.com/Sensory/alexa-rpi/blob/master/LICENSE.txt' target="_blank" rel="nofollow">license</a>), based in Santa Clara, CA. &ldquo;We do not have any low cost or free license or library. We are unable to support any student or individual.&rdquo; </li> <li style='clear: both'> <div style="text-align: right;"> <picture> <source srcset="/blog/images/snowboyLogo_250x56.webp" type="image/webp"> <source srcset="/blog/images/snowboyLogo_250x56.png" type="image/png"> <img src="/blog/images/snowboyLogo_250x56.png" title="Snowboy Logo" class="right " style="width: 250px; margin-top: 1em" alt="Snowboy Logo" /> </picture> </div> kitt.ai's <a href='https://github.com/Kitt-AI/snowboy' target="_blank" rel="nofollow">Snowboy</a>, based in Seattle, WA and <a href='https://www.geekwire.com/2016/backed-amazon-paul-allen-kitt-ai-launches-first-hotword-detection-software-toolkit/' target="_blank" rel="nofollow">partially funded by Amazon</a>. </li> </ul> <h3 id="sphinx" style='clear: both'>CMU Sphinx</h3> <div style="text-align: right;"> <picture> <source srcset="/blog/images/CMUSphinx_300x72.webp" type="image/webp"> <source srcset="/blog/images/CMUSphinx_300x72.png" type="image/png"> <img src="/blog/images/CMUSphinx_300x72.png" title="CMU Sphinx Logo" class="right " style="width: 300px" alt="CMU Sphinx Logo" /> </picture> </div> <p> Designed for low-resource platforms, implementations of <a href='http://cmusphinx.sourceforge.net/' target="_blank" rel="nofollow">CMU's Sphinx</a> exist for C (which supports Python) and Java. <a href='http://cmusphinx.sourceforge.net/wiki/faq#qhow_to_implement_hot_word_listening' target="_blank" rel="nofollow">Hotword</a> spotting is supported. <a href='https://en.wikipedia.org/wiki/CMU_Sphinx' target="_blank" rel="nofollow">Several versions</a> of Sphinx exist, with varying free and commercial licenses. Reports suggest that Sphinx works reasonably well but I have not tested yet. Sphinx powers <a href='https://jasperproject.github.io/documentation/faq/' target="_blank" rel="nofollow">Jasper</a>. </p> <h3 id="js">Javascript alternatives</h3> <ul> <li> <a href='https://github.com/TalAter/annyang' target="_blank" rel="nofollow">Annyang</a> works well, but requires Google's web browsers and servers, so it is probably not reasonable to use it just for hotword detection. </li> <li> <a href='https://github.com/jimmybyrum/voice-commands.js' target="_blank" rel="nofollow">Voice-commands.js</a> also requires Google's web browsers and servers. </li> <li> <a href='https://github.com/evancohen/sonus' target="_blank" rel="nofollow">Sonus</a> is a Node framework which will be able to be configured to use a variety of back ends one day. It does hotword detection by using SnowBoy; the authors obviously ignored Snowboy's license terms. </li> <li> <a href='https://github.com/zzmp/juliusjs' target="_blank" rel="nofollow">JuliusJS</a> is a JavaScript port of the &ldquo;Large Vocabulary Continuous Speech Recognition Engine Julius&rdquo;. It does not call any servers and runs in most browsers. Recognition is weak and it requires a lot of CPU. </li> <li <a href='https://justbuildsomething.com/cross-browser-voice-recognition-with-pocketsphinx-js/' target="_blank" rel="nofollow">PocketSphinx.js</a> is a free browser-based alternative, unfortunately it is horrible. </li> </ul> <h2 id="training">Training a voice recognition engine</h2> <p>Voice recognition engines need to be trained on a large dataset for the desired languages. Recognition effectiveness is less than linearly proportional to the size of the dataset, and high-quality datasets are important. Truly large amounts of data are required. Amazon, Apple, Google and Microsoft have commercial products that were trained using enormous proprietary datasets. This is a substantial investment, so only well-capitalized organizations will be able to offer their own voice recognition engines for unconstrained vocabularies. </p> <h2 id="privacy">Determining Your Application's Privacy / Effectiveness Tradeoff</h2> <p> A voice recognition's effectiveness increases for specific users if their voice streams are recorded and stored, then used for further training. However, this means that <a href='https://www.reddit.com/r/technology/comments/2wzmmr/everything_youve_ever_said_to_siricortana_has/' target="_blank" rel="nofollow">privacy</a> and security are traded off for effectiveness. This service provider's tradeoff might not be optimal for your use case. Apple's Siri only associates the stored voice recordings with you for <a href='https://www.wired.com/2013/04/siri-two-years/' target="_blank" rel="nofollow">6 months</a>. Google's Assist and Voice, and <a href='https://www.amazon.com/gp/help/customer/display.html?nodeId=201602040' target="_blank" rel="nofollow">Amazon's Alexa</a> store all your voice recordings forever, unless you explicitly delete them. I could not discover how long Microsoft's Cortana and Skype store voice recordings, or how to delete them. <p> <h2 id="integrating">Integrating With Third-party or Proprietary Services</h2> <p> Because voice recognition returns a structured document like JSON or XML, and voice generation is simple to initiate, integration is well understood and many options exist. </p> Setting Up Jekyll with Ubuntu or WSL 2017-01-08T00:00:00-05:00 https://mslinn.github.io/blog/2017/01/08/setting-up-github-pages <p> Here is a script I wrote in April 2020 to set up a local development environment for this website. The script uses <a href="https://jekyllrb.com/" target="_blank" rel="nofollow">Jekyll</a> to assemble the website that you are currently reading. </p> <pre data-lt-active='false'> #!/bin/bash # Installs the right version of Jekyll and all dependencies function installGem { sudo -H gem install $1 -v $2 --source 'https://rubygems.org/' } yes | sudo apt install bundler make ruby ruby-dev software-properties-common zlib1g-dev sudo -H gem update --system 3.0.6 sudo -H gem uninstall i18n jekyll jekyll-docs jekyll-sass-converter public_suffix --all installGem jekyll-timeago 0.13.1 installGem backports 3.17.1 installGem jekyll 3.3.0 #installGem jekyll-docs 3.3.0 sudo -H gem install bundler classifier-reborn github-pages jekyll-admin jekyll-assets jekyll-docs jekyll-gist jekyll-tagging jekyll-theme-architect html-proofer libz-dev sprockets installGem jekyll-sass-converter 1.5.2 installGem i18n 0.9.5 installGem nokogiri 1.10.9 bundle install </pre> <p> This remainder of this article is obsolete. <a href="https://jekyllrb.com/docs/" target="_blank" rel="nofollow">Follow these instructions instead.</a> </p> <hr /> <p> These are my notes for setting up GitHub pages using Ubuntu or Windows Subsystem for Linux. I updated these notes Jan 3, 2018 to include instructions on <code>jekyll-admin</code>. Since then, Jekyll has continued to evolve and these instructions should no longer be followed. I leave this page merely for posterity's sake. </p> <p> Read the docs on the <a href='https://jekyllrb.com/docs/github-pages/#use-the-github-pages-gem' target="_blank" rel="nofollow"><code>github-pages</code> gem</a>. </p> <p>Make a <code>Gemfile</code> with the following contents:</p> <pre data-lt-active='false'> source "https://rubygems.org" gem "classifier-reborn" gem "github-pages", group: :jekyll_plugins gem "html-proofer" gem "jekyll" gem 'jekyll-admin', group: :jekyll_plugins gem "jekyll-assets" gem "jekyll-docs" gem "jekyll-gist" gem "jekyll-theme-architect" gem "sprockets" </pre> <p> Create <code>_config.yml</code> with the following contents: </p> <pre data-lt-active='false'> exclude: [vendor] jekyll_admin: hidden_links: # - posts # - pages # - staticfiles # - datafiles # - configuration markdown: kramdown name: Mike Slinn, Connoisseur of Technology permalink: /blog/:year/:month/:day/:title plugins: [classifier-reborn, html-proofer, jekyll, jekyll-admin, jekyll-assets, jekyll-docs, jekyll-gist, jekyll-theme-cayman] title: Mike Slinn's Blog </pre> <p> Ruby 2.3+ is required, but Ubuntu defaults to an older version. I set up Ruby 2.3, with the option of installing other versions and making them default. For more background, see <a href='https://www.brightbox.com/docs/ruby/ubuntu/#Addingtherepository' target="_blank" rel="nofollow">Ruby packages for Ubuntu</a>. I also installed various gems necessary to provide the Jekyll functionality I desired. </p> <pre data-lt-active='false'> sudo apt-get install make software-properties-common sudo apt-add-repository ppa:brightbox/ruby-ng sudo apt-get update sudo apt install ruby2.3 ruby2.3-dev ruby-switch zlib1g-dev ruby-bundler ruby-switch --list sudo ruby-switch --set ruby2.3 sudo gem update --system sudo gem install bundler classifier-reborn jekyll github-pages jekyll-assets jekyll-gist \ jekyll-docs jekyll-theme-cayman html-proofer classifier-reborn jekyll-admin sprockets sudo bundle clean --force bundle install </pre> <p> Create repo <code>userId.github.io</code> (where <code>userId</code> is your GitHub user id) and clone it. </p> <pre data-lt-active='false'> <span class="unselectable">$ </span>git clone git@github.com:userId/userId.github.io.git <span class="unselectable">$ </span>cd userId.github.io/</pre> <h2>Running Jekyll</h2> <p>Read the docs.</p> <pre data-lt-active='false'> <span class="unselectable">$ </span>bundle exec jekyll docs</pre> <p> <a href='https://github.com/Microsoft/BashOnWindows/issues/216' target="_blank" rel="nofollow">Read</a> about how Bash on Windows does not yet support watched directories properly. </p> <pre data-lt-active='false'> <span class="unselectable">$ </span>bundle exec jekyll serve --force_polling </pre> <p>For other OSes:</p><pre data-lt-active='false'> <span class="unselectable">$ </span>bundle exec jekyll serve</pre> <p> Use the <code>--drafts</code> option to preview draft blog posts in the <code>_drafts</code> directory. For Bash on Windows: </p> <pre data-lt-active='false'> <span class="unselectable">$ </span>bundle exec jekyll serve --force_polling --drafts</pre> <p>For other OSes:</p> <pre data-lt-active='false'> <span class="unselectable">$ </span>bundle exec jekyll serve --drafts</pre> <h2>Visual Blog Editor</h2> <p><a href="https://github.com/planetjekyll/awesome-jekyll-editors" target="_blank" rel="nofollow">Many awesome Jekyll editors</a> exist. The above instructions installed <a href="https://github.com/jekyll/jekyll-admin" target="_blank" rel="nofollow">jekyll-admin</a>. Run Jekyll as described above and navigate to <a href="http://localhost:4000/admin" target="_blank" rel="nofollow">http://localhost:4000/admin</a> to access the administrative interface. </p> <p> Unfortunately, <code>jekyll-admin</code> does not provide an WYSIWYG editor like that provided by CKEditor. I <a href="https://github.com/jekyll/jekyll-admin/issues/437#issuecomment-355209137" target="_blank" rel="nofollow">suggested this new feature</a>. </p> I Updated the Apache Spark Reference Applications 2016-11-15T00:00:00-05:00 https://mslinn.github.io/blog/2016/11/15/lessons-from-updating-the-twitter-classifier-apache-spark-reference-application <h2 id="intro">Overview</h2> <p> The Apache Spark committers just accepted my pull request that updated the official <a href='https://github.com/databricks/reference-apps/tree/master/twitter_classifier' target="_blank" rel="nofollow">Twitter Classifier Reference Application</a> from Spark 1.4 / Scala 2.10 to Spark 2 / Scala 2.11. This post discusses some things I did in the pull request from the point of view of a Scala programmer. A primary goal was to rewrite the reference application using idiomatic and functional-style Scala. This post briefly discusses two unique aspects that I addressed: command-line parsing and DRYing up the code by importing scopes. I did several other things to improve the reference application, such as modularizing the code and providing run scripts, but this post does not address them because those techniques are generally well understood. </p> <p> I did not upgrade the reference application to Scala 2.12, which was released a couple of weeks ago because Spark does not yet support Scala 2.12. Josh Rosen of Databricks wrote me and said: </p> <blockquote> &ldquo;Some of Spark’s dependencies and by Scala-version-specific code changes necessary to work around method overloads became ambiguous in 2.12. The umbrella ticket tracking 2.12 support can be found at <a href='https://issues.apache.org/jira/browse/SPARK-14220' target="_blank" rel="nofollow"><code>issues.apache.org/jira/browse/SPARK-14220</code></a>. One of the hardest pieces will be <a href='https://issues.apache.org/jira/browse/SPARK-14643' target="_blank" rel="nofollow"><code>issues.apache.org/jira/browse/SPARK-14643</code></a> (see the linked design document on that issue). Lack of 2.12 support for Breeze and its dependencies is likely to be another serious blocker, but that might be avoidable by only publishing a subset of the projects with 2.12 to begin with (e.g. only Spark core / SQL at first).&rdquo; </blockquote> <h2 id="cli">Command Line Parsing</h2> <p>I modified the reference applications’ command line parsing to use a Scala library that supported idiomatic Scala (<a href='https://github.com/acrisci/commander-scala' target="_blank" rel="nofollow">Commander Scala</a>), instead of <a href='https://commons.apache.org/proper/commons-cli/' target="_blank" rel="nofollow">Apache Commons CLI</a>, which is the Java library that was previously used. The result was simple, clean and very terse code that is intuitive to understand and easy to maintain. Commander Scala automatically generates the help message. Take a look at the <a href='https://github.com/databricks/reference-apps/blob/1793e3dc2335696e98a335130673b58b35086c26/twitter_classifier/scala/src/main/scala/com/databricks/apps/twitterClassifier/CollectOptions.scala' target="_blank" rel="nofollow"><code>collect</code> command&rsquo;s parsing</a>. You’ll notice that it uses some common code for parsing Twitter authentication parameters. This code is much shorter than the previous code, easier to understand and modify, and is more flexible.</p> <pre data-lt-active='false'>import com.github.acrisci.commander.Program import java.io.File abstract sealed case class CollectOptions( twitterOptions: TwitterOptions, overWrite: Boolean = false, tweetDirectory: File = new File(System.getProperty("user.home"), "/sparkTwitter/tweets/"), numTweetsToCollect: Int = 100, intervalInSecs: Int = 1, partitionsEachInterval: Int = 1 ) object CollectOptions extends TwitterOptionParser { override val _program = super._program .option(flags="-w, --overWrite", description="Overwrite all data files from a previous run") .usage("Collect [options] &lt;tweetDirectory> &lt;numTweetsToCollect> &lt;intervalInSeconds> &lt;partitionsEachInterval>") def parse(args: Array[String]): CollectOptions = { val program: Program = _program.parse(args) if (program.args.length!=program.usage.split(" ").length-2) program.help new CollectOptions( twitterOptions = super.apply(args), overWrite = program.overWrite, tweetDirectory = new File(program.args.head.replaceAll("^~", System.getProperty("user.home"))), numTweetsToCollect = program.args(1).toInt, intervalInSecs = program.args(2).toInt, partitionsEachInterval = program.args(3).toInt ){} } }</pre> <p> Here is how to sidestep the Spark help message and display the help message for the collect entry point: </p> <pre data-lt-active='false'>$ <b>spark-shell \ -class com.databricks.apps.twitterClassifier.Collect \ -jars target/scala-2.11/spark-twitter-lang-classifier-assembly-2.0.0.jar \ -- -help</b> Usage: Collect [options] &lt;tweetDirectory> &lt;numTweetsToCollect> &lt;intervalInSeconds> &lt;partitionsEachInterval> Options: -h, — help output usage information -V, — version output the version number -w, — overWrite Overwrite all data files from a previous run -v, — accessTokenSecret [type] Twitter OAuth Access Token Secret -t, — accessToken [type] Twitter OAuth Access Token -s, — consumerSecret [type] Twitter OAuth Consumer Secret -c, — consumerKey [type] Twitter OAuth Consumer Key</pre> <h2 id="import">Importing Inner Scope Into Another Object</h2> <p> Apache Spark is unusual in that you cannot encapsulate a Spark streaming context in a type instance. A memory overflow occurs when you try to instantiate a Scala trait or class that creates a Spark context. The solution is to use a unique Scala feature: the ability to import inner scope from an object into another scope. This meant that the code was made DRY (common code was not repeated), without using classes or traits. </p> <p> Here is how I took advantage of this little-known Scala technique: first I defined the <a href='https://github.com/databricks/reference-apps/blob/1793e3dc2335696e98a335130673b58b35086c26/twitter_classifier/scala/src/main/scala/com/databricks/apps/twitterClassifier/package.scala#L7-L19' target="_blank" rel="nofollow"><code>SparkObject</code> object</a> within a package object so it was easily found: </p> <pre data-lt-active='false'>object SparkSetup { val spark = SparkSession .builder .appName(getClass.getSimpleName.replace("$", "")) .getOrCreate() val sqlContext = spark.sqlContext val sc: SparkContext = spark.sparkContext sc.setLogLevel("ERROR") }</pre> <p> Next I imported all the variables defined in <code>SparkSetup</code> into the <code>Collect</code> object’s scope, including <code>sc</code>, which was used twice, <a href='https://github.com/databricks/reference-apps/blob/1793e3dc2335696e98a335130673b58b35086c26/twitter_classifier/scala/src/main/scala/com/databricks/apps/twitterClassifier/Collect.scala' target="_blank" rel="nofollow">like this</a>: </p> <pre data-lt-active='false'>object Collect extends App { val options = CollectOptions.parse(args) import SparkSetup._ val ssc = new StreamingContext(sc, Seconds(options.intervalInSecs)) Collector.doIt(options, sc, ssc) }</pre> <p> Want to learn more practical Scala techniques? Head over to <a href='https://www.ScalaCourses.com' target="_blank" rel="nofollow"><code>ScalaCourses.com</code></a> and enroll! The combination of the Introduction to Scala and Intermediate Scala courses will teach you everything you need to know to start your journey with Apache Spark. </p> <p> Mike Slinn is the lead Scala instructor at <a href="https://www.ScalaCourses.com" target="_blank" rel="nofollow">ScalaCourses.com</a>. </p> Publishing Maven Artifacts to AWS S3 2013-07-07T00:00:00-04:00 https://mslinn.github.io/blog/2013/07/07/publishing-maven-artifacts-to-aws-s3 <p> I wanted a simple, flexible and cheap way of publishing proprietary Maven artifacts created by <code>sbt</code> projects such that they could be securely retrieved by authorized individuals. I liked the idea of versioned artifacts, but did not want to use GitHub or BitBucket to host the artifacts because of the hassle of maintaining ever-larger git repos. Instead, I opted for S3's optional versioning mechanism. </p> <p> The technique described here relies on the fact that publishing to a local file (using <code>sbt publish</code>) generates all the necessary artifacts, which merely need to be copied to the right directory on the Artifactory server. The server need not be anything special: a normal web server works fine, as does <code>webdav</code>. A variety of other protocols are also supported by <code>sbt</code>. </p> <p> I created two test projects on GitHub: <code><a href="https://github.com/mslinn/testPublishLib" target="_blank" rel="nofollow">testPublishLib</a></code> and <code><a href="https://github.com/mslinn/testPublishApp" target="_blank" rel="nofollow">testPublishApp</a></code>. You could clone them if you would like to try this. Each of them has a <code>README</code> that explains what they do.</p> I used <code>s3cmd</code> to manage the AWS S3 buckets that hold the repository. You can <a href="https://s3tools.org/s3cmd" target="_blank" rel="nofollow">obtain <code>s3cmd</code></a> for most OSes. I found I needed to install <code>python-magic</code>: </p> <pre data-lt-active='false'><span class="unselectable">$ </span>sudo pip install python-magic</pre> <ol> <li>Let <code>s3cmd</code> know your s3 keys: <pre data-lt-active='false'><span class="unselectable">$ </span>s3cmd --configure</pre> </li> <li>Create the S3 bucket, which must be unique. If you want to repository to be publicly visible, be sure that the bucket name starts with <code>www.</code> If you already have the S3 bucket then just omit this step. <pre data-lt-active='false'><span class="unselectable">$ </span>s3cmd mb s3://www.mymavenrepo</pre> </li> <li>If you want to repository to be publicly visible, you need to enable the S3 bucket web site option: <pre data-lt-active='false'><span class="unselectable">$ </span>s3cmd ws-create s3://www.mymavenrepo</pre></li> <li>Publish your library to a local repository. <a href="https://github.com/mslinn/testPublishLib" target="_blank" rel="nofollow"><code>testPublishLib</code></a> is an example of how to set up a project properly. <pre data-lt-active='false'><span class="unselectable">$ </span>sbt publish</pre></li> <li>Copy your locally published artifacts to S3. You can either type it out longhand: <pre data-lt-active='false'><span class="unselectable">$ </span>s3cmd -P sync ~/.ivy2/local/com/micronautics/test_publish_lib \ s3://www.mymavenrepo/snapshots/com/micronautics/test_publish_lib</pre> ... (note that the <code>-P</code> option makes the files publicly visible), or you can use <code>s3publish</code>: <pre data-lt-active='false'><span class="unselectable">$ </span>s3publish com/micronautics/test_publish_lib</pre> </li> <li>If your repository is not public, you will need to provide authentication in a file which I called <tt>~/.sbt/awsCreds.sbt</tt> for convenience: <pre data-lt-active='false'>credentials += Credentials("AWS Realm", "www.mavenrepo.s3.amazonaws.com", "myUserId", "myPassword")</pre> </li> <li>Use the published artifact from your sbt project by including a resolver of the form: <pre data-lt-active='false'>"AWS Snapshots" at "https://www.mavenrepo.s3.amazonaws.com/snapshots"</pre> The <code>TestPublishApp</code> project is a working example of how to do that.</li> </ol> <p> Following is the script I wrote to upload to S3, which I called <code>s3publish</code>. It assumes you are always publishing a snapshot; and that the files should be public. I leave it to you to extend this script to handle releases and private content if you have the need. </p> <noscript><pre>#!/bin/bash if [ $# -eq 0 ]; then echo &quot;Usage: `basename $0` pubpath/name [repo]&quot; echo &quot; Where pubpath might be something like com/micronautics&quot; echo &quot; name is name of artifact to publish&quot; echo &quot; repo is the optional name of the bucket to publish to&quot; echo &quot; Example: `basename $0` com/micronautics/test_publish_lib&quot; exit 1 fi REPO=www.mymavenrepo OPTIONS=-P if [ $# -eq 2 ]; REPO=$2; fi s3cmd $OPTIONS sync ~/.ivy2/local/$1 s3://$REPO/snapshots/$1</pre></noscript><script src="https://gist.github.com/mslinn/5945115.js"> </script> Load Testing ScalaCourses.com 2013-06-01T00:00:00-04:00 https://mslinn.github.io/blog/2013/06/01/load-testing-scalacoursescom <div style="text-align: center;"> <picture> <source srcset="/assets/images/ScalaCoursesEclipse.webp" type="image/webp"> <source srcset="/assets/images/ScalaCoursesEclipse.png" type="image/png"> <img src="/assets/images/ScalaCoursesEclipse.png" title="ScalaCourses logo" class="center liImg2 rounded shadow" alt="ScalaCourses logo" /> </picture> </div> <p> <a href="https://scalacourses.com/" target="_blank" rel="nofollow">ScalaCourses.com</a>, which will be announced next week, is built using the entire Typesafe stack: Scala 2.10, Play 2.1, Slick 1.0 and Akka 2.1. It runs on Heroku. </p> <p> I ran a load test on the app running on only one Heroku dyno. I configured JMeter to use 300 threads, hammering at full speed (no pauses between hits). Testing was done from my desktop. The test generated 16.5Mb/s inbound and 4.1Mb/s outbound according to <code>iptraf</code>. The test did not download page assets because they are served directly from AWS S3. </p> <p> 50% of all responses were received in under 110ms, and 95% were received in under 165ms. The distance from my workstation in Half Moon Bay, CA, USA to the Heroku app server, running from an AWS server in Ashburn, Virginia, USA is about 3000 miles or 4,828 km away. Considering that average ping time is ~100ms, that is amazingly good! Ping measures round-trim time for a test packet, and <code>mtr</code> showed the average ping time as ~95ms with a standard deviation of 18. </p> <div style=""> <picture> <source srcset="/blog/images/jmeter_690x448.webp" type="image/webp"> <source srcset="/blog/images/jmeter_690x448.png" type="image/png"> <img src="/blog/images/jmeter_690x448.png" title="50% of all responses were received in under 110ms, and 95% were received in under 165ms" class=" liImg2 rounded shadow" alt="50% of all responses were received in under 110ms, and 95% were received in under 165ms" /> </picture> </div> <p> Yes, I did put a lot of care into the design of the app so that it would scale well, but I had not expected such fantastic results. Kudos to each of the Typesafe product teams, and to Heroku! </p> Cleaning the Heroku Cache 2013-03-18T00:00:00-04:00 https://mslinn.github.io/blog/2013/03/18/cleaning-heroku-cache <p> In an <a href="/blog/2013/02/27/command-line-sbt-on-heroku-dyno.html">earlier post</a>, I talked about using a bash console to experiment with a Heroku app. I mentioned that you should not run <code>sbt</code>. Of course, that is exactly what I did, and I discovered the hard way that the cache can't be cleared from the Heroku bash shell. The build cache is held outside the dyno and cannot be accessed from inside a running dyno. </p> <p> Cleaning the cache is accomplished with a special cache cleaner buildpack. Set the buildpack in your app like this: </p> <pre data-lt-active='false'>heroku config:add BUILDPACK_URL=https://github.com/heroku/heroku-buildpack-scala.git#cleancache --app myapp</pre> <p>Push your code:</p> <pre data-lt-active='false'><span class="unselectable">$ </span>git push heroku master</pre> <p>Remove the cache cleaner buildpack:</p> <pre data-lt-active='false'><span class="unselectable">$ </span>heroku config:remove BUILDPACK_URL --app myapp</pre> <p>Change a file, commit and push again:</p> <pre data-lt-active='false'><span class="unselectable">$ </span>git push heroku master</pre> <p>All better!</p> Using Scala’s String Interpolation to Access a Map 2013-03-15T00:00:00-04:00 https://mslinn.github.io/blog/2013/03/15/using-scala-210s-string-interpolation <p> Given a map, would it not be nice to have a shorthand way of looking up a value from a key, and to provide a default value? I&rsquo;ve wanted to be able to do this for a long time. Scala 2.10 makes this really easy! </p> <p> The <code>MapLookup</code> class below contains a <code>Map[String, Int]</code> that can be looked up by using a dollar sign ($) from code that contains a reference to the implicit class. If the lookup key is not defined by the map, a zero is returned. </p> <pre data-lt-active='false'>implicit class MapLookup(val sc: StringContext) { val map = Map(("a", 1), ("b", 2), ("c", 3)).withDefaultValue(0) def $(args: Any*): Int = { val orig = sc.s (args : _*) map.get(orig) } }</pre> <p> Assuming that the above is stored in a file called <code>strInterp.scala</code>, here are some examples of usage: </p> <pre data-lt-active='false'><span class="unselectable">$ </span>scala -i strInterp.scala Loading strInterp.scala... defined class MapLookup Welcome to Scala version 2.10.1 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_17). Type in expressions to have them evaluated. Type :help for more information. scala&gt; $"a" res2: Int = 1 scala&gt; $"z" res3: Int = 0</pre> <p>Short and sweet!</p> Listing of all AWS Elastic Transcoder Presets 2013-03-15T00:00:00-04:00 https://mslinn.github.io/blog/2013/03/15/listing-of-all-aws-elastic-transcoder <p> Here is a listing of all <a href="https://console.aws.amazon.com/elastictranscoder/home?region=us-east-1#presets:" target="_blank" rel="nofollow">AWS Elastic Transcoder presets</a>, provided &lsquo;out of the box&rsquo; as system-wide presets. </p> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-000001</code><br> <b>Name:</b> <code>System preset: Generic 1080p</code><br> <b>Description:</b> <code>System preset generic 1080p</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=4},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 5400,FrameRate: 29.97,AspectRatio: 16:9,MaxWidth: 1920,MaxHeight: 1080,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-000010</code><br> <b>Name:</b> <code>System preset: Generic 720p</code><br> <b>Description:</b> <code>System preset generic 720p</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 2400,FrameRate: 29.97,MaxWidth: 1280,MaxHeight: 720,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-000020</code><br> <b>Name:</b> <code>System preset: Generic 480p 16:9</code><br> <b>Description:</b> <code>System preset generic 480p 16:9</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 1200,FrameRate: 29.97,MaxWidth: 854,MaxHeight: 480,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-000030</code><br> <b>Name:</b> <code>System preset: Generic 480p 4:3</code><br> <b>Description:</b> <code>System preset generic 480p 4:3</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 900,FrameRate: 29.97,MaxWidth: 640,MaxHeight: 480,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-000040</code><br> <b>Name:</b> <code>System preset: Generic 360p 16:9</code><br> <b>Description:</b> <code>System preset generic 360p 16:9</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 720,FrameRate: 29.97,MaxWidth: 640,MaxHeight: 360,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-000050</code><br> <b>Name:</b> <code>System preset: Generic 360p 4:3</code><br> <b>Description:</b> <code>System preset generic 360p 4:3</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 600,FrameRate: 29.97,MaxWidth: 480,MaxHeight: 360,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-000060</code><br> <b>Name:</b> <code>System preset: Generic 320x240</code><br> <b>Description:</b> <code>System preset generic 320x240</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 22050,BitRate: 64,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=1.3},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 300,FrameRate: 15,MaxWidth: 320,MaxHeight: 240,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-100010</code><br> <b>Name:</b> <code>System preset: iPhone4</code><br> <b>Description:</b> <code>System preset: iPod touch 5G, 4G, iPad 1G, 2G</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=3.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 2200,FrameRate: 30,MaxWidth: 1280,MaxHeight: 720,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-100020</code><br> <b>Name:</b> <code>System preset: iPhone4S</code><br> <b>Description:</b> <code>System preset: iPhone 5, iPad 3G, 4G, iPad mini, Samsung Galaxy S2/S3/Tab 2</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=high, Level=4.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 5000,FrameRate: 30,MaxWidth: 1920,MaxHeight: 1080,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-100030</code><br> <b>Name:</b> <code>System preset: iPhone3GS</code><br> <b>Description:</b> <code>System preset: iPhone 3GS</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 600,FrameRate: 30,MaxWidth: 640,MaxHeight: 480,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-100040</code><br> <b>Name:</b> <code>System preset: iPod Touch</code><br> <b>Description:</b> <code>System preset: iPhone 1, 3, iPod classic</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 1500,FrameRate: 30,MaxWidth: 640,MaxHeight: 480,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-100050</code><br> <b>Name:</b> <code>System preset: Apple TV 2G</code><br> <b>Description:</b> <code>System preset: Apple TV 2G</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=3.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 5000,FrameRate: 30,MaxWidth: 1280,MaxHeight: 720,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-100060</code><br> <b>Name:</b> <code>System preset: Apple TV 3G</code><br> <b>Description:</b> <code>System preset: Apple TV 3G, Roku HD/2 XD</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=high, Level=4},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 5000,FrameRate: 30,MaxWidth: 1920,MaxHeight: 1080,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-100070</code><br> <b>Name:</b> <code>System preset: Web</code><br> <b>Description:</b> <code>System preset: Facebook, SmugMug, Vimeo, YouTube</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=3.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 2200,FrameRate: 30,MaxWidth: 1280,MaxHeight: 720,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-100080</code><br> <b>Name:</b> <code>System preset: KindleFireHD</code><br> <b>Description:</b> <code>System preset: Kindle Fire HD</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=4},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 2200,FrameRate: 30,MaxWidth: 1280,MaxHeight: 720,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-100090</code><br> <b>Name:</b> <code>System preset: KindleFireHD8.9</code><br> <b>Description:</b> <code>System preset: Kindle Fire HD 8.9</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=4},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 5400,FrameRate: 30,MaxWidth: 1920,MaxHeight: 1080,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-100100</code><br> <b>Name:</b> <code>System preset: KindleFire</code><br> <b>Description:</b> <code>System preset: Kindle Fire</code><br> <b>Container:</b> <code>mp4</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 48000,BitRate: 160,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=3.1},KeyframesMaxDist: 90,FixedGOP: false,BitRate: 1600,FrameRate: 30,MaxWidth: 1024,MaxHeight: 576,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-200010</code><br> <b>Name:</b> <code>System preset: HLS 2M</code><br> <b>Description:</b> <code>System preset: HLS 2M</code><br> <b>Container:</b> <code>ts</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=3.1},KeyframesMaxDist: 90,FixedGOP: true,BitRate: 1872,FrameRate: auto,MaxWidth: 1024,MaxHeight: 768,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-200020</code><br> <b>Name:</b> <code>System preset: HLS 1.5M</code><br> <b>Description:</b> <code>System preset: HLS 1.5M</code><br> <b>Container:</b> <code>ts</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=3.1},KeyframesMaxDist: 90,FixedGOP: true,BitRate: 1372,FrameRate: auto,MaxWidth: 960,MaxHeight: 640,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-200030</code><br> <b>Name:</b> <code>System preset: HLS 1M</code><br> <b>Description:</b> <code>System preset: HLS 1M</code><br> <b>Container:</b> <code>ts</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=main, Level=3.1},KeyframesMaxDist: 90,FixedGOP: true,BitRate: 872,FrameRate: auto,MaxWidth: 640,MaxHeight: 432,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-200040</code><br> <b>Name:</b> <code>System preset: HLS 600k</code><br> <b>Description:</b> <code>System preset: HLS 600k</code><br> <b>Container:</b> <code>ts</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=3, Profile=baseline, Level=3.0},KeyframesMaxDist: 90,FixedGOP: true,BitRate: 472,FrameRate: auto,MaxWidth: 480,MaxHeight: 320,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> <div style="margin-bottom: 8pt;"> <b>Id:</b> <code>1351620000001-200050</code><br> <b>Name:</b> <code>System preset: HLS 400k</code><br> <b>Description:</b> <code>System preset: HLS 400k</code><br> <b>Container:</b> <code>ts</code><br> <b>Audio:</b> <code>{Codec: AAC,SampleRate: 44100,BitRate: 128,Channels: 2}</code><br> <b>Video:</b> <code>{Codec: H.264,CodecOptions: {MaxReferenceFrames=1, Profile=baseline, Level=3.0},KeyframesMaxDist: 90,FixedGOP: true,BitRate: 272,FrameRate: auto,MaxWidth: 400,MaxHeight: 288,DisplayAspectRatio: auto,SizingPolicy: ShrinkToFit,PaddingPolicy: NoPad}</code> </div> Bash shell on a Heroku Dyno 2013-02-27T00:00:00-05:00 https://mslinn.github.io/blog/2013/02/27/command-line-sbt-on-heroku-dyno <p> Up to now most of my work with Heroku has been via the <code>heroku</code> command line client, and pushing builds to app instances to have them compiled and run. Recently, I've been messing around with the remote <code>bash</code> shell of my Heroku apps. To enter a remote shell, just type the following from the root directory of a git project that declares your Heroku app as a remote repository. This command uses the <code>heroku</code> client provided by the Heroku toolbelt: </p> <pre data-lt-active='false'><span class="unselectable">$ </span>heroku run bash</pre> <p> Upon login, you are placed into the root directory of your deployed app. Unfortunately, you do not have access to the table of mounted file systems and you cannot run <code>sudo</code>. </p> <pre data-lt-active='false'><span class="unselectable">~ $ </span>pwd /app <span class="unselectable">~ $ </span>ls -alF total 64 drwx------ 10 u37570 37570 4096 Dec&nbsp; 1 01:07 ./ drwxr-xr-x 15 root&nbsp;&nbsp; root&nbsp; 4096 Oct 31&nbsp; 2011 ../ -rwx------&nbsp; 1 u37570 37570&nbsp; 183 Dec&nbsp; 1 01:05 .gitignore* drwx------&nbsp; 3 u37570 37570 4096 Dec&nbsp; 1 01:05 .ivy2/ drwxrwxr-x&nbsp; 6 u37570 37570 4096 Dec&nbsp; 1 01:05 .jdk/ drwx------&nbsp; 4 u37570 37570 4096 Dec&nbsp; 1 01:08 .sbt_home/ -rw-------&nbsp; 1 u37570 37570&nbsp;&nbsp; 98 Dec&nbsp; 1 01:05 Procfile -rw-------&nbsp; 1 u37570 37570 3833 Dec&nbsp; 1 01:05 README.md drwx------&nbsp; 5 u37570 37570 4096 Dec&nbsp; 1 01:05 app/ drwx------&nbsp; 2 u37570 37570 4096 Dec&nbsp; 1 01:05 conf/ drwx------&nbsp; 5 u37570 37570 4096 Dec&nbsp; 1 01:06 project/ drwx------&nbsp; 5 u37570 37570 4096 Dec&nbsp; 1 01:05 public/ -rwx------&nbsp; 1 u37570 37570&nbsp;&nbsp; 27 Dec&nbsp; 1 01:05 system.properties* drwx------&nbsp; 6 u37570 37570 4096 Dec&nbsp; 1 01:08 target/</pre> <p>Discover total used file space (my slug uses 1.7 GB):</p> <pre data-lt-active='false'><span class="unselectable">~ $ </span>du -sh / du: cannot read directory `/proc/tty/driver': Permission denied du: cannot read directory `/proc/1/task/1/fd': Permission denied du: cannot read directory `/proc/1/task/1/fdinfo': Permission denied du: cannot read directory `/proc/1/fd': Permission denied du: cannot read directory `/proc/1/fdinfo': Permission denied du: cannot access `/proc/11/task/11/fd/4': No such file or directory du: cannot access `/proc/11/task/11/fdinfo/4': No such file or directory du: cannot access `/proc/11/fd/4': No such file or directory du: cannot access `/proc/11/fdinfo/4': No such file or directory du: cannot read directory `/lost+found': Permission denied du: cannot read directory `/etc/ssl/private': Permission denied 1.7G /</pre> <p>All your environment variables are available:</p> <pre data-lt-active='false'><span class="unselectable">~ $ </span>set # Miles of output</pre> <p><code>ifconfig</code> is not available, but the IP address of your dyno can be discovered this way:</p> <pre data-lt-active='false'><span class="unselectable">~ $ </span>tail -n 1 /etc/hosts 10.92.81.76 e3d315e6-4392-4f3a-b7ae-75438c469697</pre> <p> You can run <code>sbt</code>, but this is a bad idea because the Ivy cache is held outside the dyno, and there is no way for you to clean it without using some voodoo. </p> <pre data-lt-active='false'><span class="unselectable">~ $ </span>sbt update [info] Loading global plugins from /app/.sbt_home/.sbt/plugins [info] Loading project definition from /app/project [info] Set current project to blahblah (in build file:/app/) [info] Updating {file:/app/}blahblah... ... miles of output ... [info] Done updating. [success] Total time: 6 s, completed Feb 28, 2013 4:40:21 AM</pre> AWS S3 websites and Naked HTTP Redirects 2012-11-14T00:00:00-05:00 https://mslinn.github.io/blog/2012/11/14/aws-s3-web-sites-and-naked-http <p> Sites hosted directly off AWS S3 only respond to the <tt>www</tt> subdomain. In other words, if you tried to navigate to <tt>https://mysite.com</tt> for a site that was hosted on AWS S3, a 404 status would result, however <tt>https://www.mysite.com</tt> would work. Some registrars, like Namecheap and GoDaddy have URL Redirect, while others, like GKG.net, do not. </p> <p> Here is how to set up Namecheap to redirect requests like <tt>https://mysite.com/blah</tt> to <tt>https://www.mysite.com/blah</tt> so AWS S3 will respond. I also show how to set up the DNS for email with Rackspace. </p> <ol> <li>Originally AWS buckets could only be used for serving websites if they started with <tt>www</tt>, for example: <code>www.artforhealingenvironments</code>. This restriction no longer exists. </li> <li>Go to the <a href="https://www.namecheap.com/products/freedns.aspx" target="_blank" rel="nofollow">Namecheap FreeDNS service</a>. This will set up your domain for a smooth transfer to Namecheap, for uninterrupted web presence during and after the transfer. </li> <li>Enter your domain name, for example <code>artforhealingenvironments.com</code>, and click on <b>Get DNS</b></li> <li>On the next page, click on <b>Add DNS Service For the Selected Domains</b></li> <li>Fill in the Namecheap <a href="https://manage.www.namecheap.com/myaccount/hosteddomainslist.aspx" target="_blank" rel="nofollow">Hosted Domains page</a>. It will look something like this when you are done, for the <tt>artforhealingenvironments.com</tt> domain. </li> </ol> <div style=""> <picture> <source srcset="/blog/images/dns.webp" type="image/webp"> <source srcset="/blog/images/dns.png" type="image/png"> <img src="/blog/images/dns.png" title="Namecheap Hosted Domains page" class=" liImg2 rounded shadow" style="width: 100%" alt="Namecheap Hosted Domains page" /> </picture> </div> <ol> <ol> <li>In the first IP ADDRESS/URL, put the fully resolved HTTP URL, with a trailing slash; set the record type to <b>URL Redirect</b>: <code>https://www.artforhealingenvironments.com/</code> </li> <li>In the second IP ADDRESS/URL, put the AWS bucket name, followed by the S3 site's domain; Namecheap will automatically add a period after, so you do not have to. Set the record type to <b>CNAME</b>:<code>www.artforhealingenvironments.com.s3-website-us-east-1.amazonaws.com</code> </li> <li>Namecheap's minimum TTL is 60 minutes.</li> <li>Namecheap's website automatically adds a period after each domain name.</li> </ol> </ol> <ol> <li>The MX records for email are shown at the bottom; they won't appear until you press the Save Changes button once (not shown). The MAILSERVER HOST NAME values are <code>mx1.emailsrvr.com</code> and <code>mx2.emailsrvr.com</code>. </li> <li>After the transfer completes, the FreeDNS entry will automatically be removed, and you will manage the domain on the Namecheap <a href="https://manage.www.namecheap.com/myaccount/domain-list.asp" target="_blank" rel="nofollow">Manage Domains</a> page. </li> </ol> Debugging JVM Programs on Heroku 2012-09-28T00:00:00-04:00 https://mslinn.github.io/blog/2012/09/28/debugging-jvm-programs-on-heroku <h2 id="java">Debugging Java Applications</h2> <p> Sometimes there is no substitute for debugging a remote application. The Java virtual machine provides the <a href="https://docs.oracle.com/javase/6/docs/technotes/guides/jpda/index.html" target="_blank" rel="nofollow">JPDA</a> facility for this. JPDA is flexible, and can be configured in a variety of ways. Two attachment mechanisms are supported for debugging remote applications: inbound connections, whereby a debugging process on your machine <i>attaches</i> to a remote process via designated port at a specific IP address, and outbound connections, whereby a debugging process on your local machine <i>listens</i> to a designated port for a remote process to attach to it. JPDA can be used by all JVM-based languages, such as Java, Scala, Groovy and Clojure. </p> <p> Heroku only allows one incoming port, so because an incoming port is used by Heroku to connect to the hosted app, debug connections originating from your IDE will not succeed. Instead, you must set up your Heroku application to initiate an outbound connection for debugging. This cannot be done if your app uses more than one dyno, so you must scale your Heroku back to one dyno before you can remotely debug it, like this: </p> <pre data-lt-active='false'><span class="unselectable">$ </span>heroku scale web=1</pre> <p> A Heroku app can initiate an outbound connection for debugging with or without a proxy server. The examples below use the Java debugger (<tt>jdb</tt>) and IntelliJ IDEA, but you could equally well set up a debug configuration for Eclipse in a similar manner. The settings below were used with a Play 2 application with Java and Scala controller classes. </p> <p> <b>Tip</b>: <tt>heroku restart</tt> will restart your Heroku app using your existing slug. This accomplishes the same thing as: </p> <pre data-lt-active='false'><span class="unselectable">$ </span>heroku scale web=0 <span class="unselectable">$ </span>heroku scale web=1</pre> <p> The other way you can restart your app is by building a new slug, which then automatically starts. You can do this by checking in a bogus file: </p> <pre data-lt-active='false'><span class="unselectable">$ </span>date &gt; ignoreme.txt; git add ignoreme.txt; git push heroku</pre> <p> <b>Warning</b>: Your Heroku app will crash if there is no listening process. Use the <tt>heroku logs</tt> command to check for a crash. </p> <pre data-lt-active='false'><span class="unselectable">$ </span>heroku logs 2012-09-29T18:20:53+00:00 heroku[web.1]: Starting process with command `target/start -Dhttp.port=${PORT} ${JAVA_OPTS}` 2012-09-29T18:20:55+00:00 app[web.1]: ERROR: transport error 202: connect failed: Connection refused 2012-09-29T18:20:55+00:00 app[web.1]: ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510) 2012-09-29T18:20:55+00:00 app[web.1]: JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized [../../../src/share/back/debugInit.c:741] 2012-09-29T18:20:55+00:00 app[web.1]: FATAL ERROR in native method: JDWP No transports initialized, jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197) 2012-09-29T18:20:56+00:00 heroku[web.1]: Process exited with status 134 2012-09-29T18:20:56+00:00 heroku[web.1]: State changed from starting to crashed</pre> <h2 id="remote">Remote Debugging Without a Proxy Server</h2> <p> If your local machine is accessible from the Internet at an IP address (a domain such as <code>blah.no-ip.info</code> would work equally well): </p> <pre data-lt-active='false'><span class="unselectable">$ </span>jdb -listen 9999&amp; # Do not type this line if you are using an IDE # If using an IDE, start debugging now <span class="unselectable">$ </span><span class="unselectable">$ </span># Assumes that eth0 accesses the Internet; only works if you are not behind NAT <span class="unselectable">$ </span><span class="unselectable">$ </span>IPADDR=`ifconfig eth0|grep "inet addr"|awk -F: '{print $2}'|awk '{print $1}'` <span class="unselectable">$ </span><span class="unselectable">$ </span># If you are behind NAT, you will need to use a dynamic DNS and set IPADDR to the machine name instead: <span class="unselectable">$ </span><span class="unselectable">$ </span>IPADDR=mycomputer.no-ip.info # modify to suit <span class="unselectable">$ </span><span class="unselectable">$ </span># This is a really long line. I wrapped it, but you should not: <span class="unselectable">$ </span><span class="unselectable">$ </span>heroku config:add JAVA_OPTS='-Xdebug -Xrunjdwp:transport=dt_socket,address=$IPADDR:9999 -Xms512M -Xmx1024M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M -XX:+UseCompressedOops' <span class="unselectable">$ </span>heroku restart</pre> <p> Here is what the run configuration for IntelliJ IDEA looks like. Note that debugger mode is set to <code>listen</code>, which implies <code>server="n"</code>. You need to launch this run configuration before restarting your Heroku app. </p> <div style=""> <picture> <source srcset="/blog/images/ideaListen.webp" type="image/webp"> <source srcset="/blog/images/ideaListen.png" type="image/png"> <img src="/blog/images/ideaListen.png" title="IntelliJ IDEA listening as a client" class=" liImg2 rounded shadow" style="width: 100%" alt="IntelliJ IDEA listening as a client" /> </picture> </div> <h2 id="impl">Bash Script Implementation</h2> <p>I put the bash scripts in a gist:</p> <noscript><pre>#!/bin/bash set -e export JAVA_OPTS=&quot;&quot; export JAVA_OPTS=&quot;$JAVA_OPTS -Xmx2048m -Xss512m -Xverify:none&quot; # Java 8 no longer supports MaxPermSize: # export JAVA_OPTS=&quot;$JAVA_OPTS -XX:MaxPermSize=384m&quot; </pre></noscript><script src="https://gist.github.com/mslinn/7937391.js"> </script> <h2 id="proxy">Remote Debugging With a Proxy Server</h2> <p> If your local machine is hidden from the Internet by a proxy server at <tt>blah.domain.com</tt>, open a tunnel to it from your local machine, and listen to it: </p> <pre data-lt-active='false'><span class="unselectable">$ </span># Assumes that eth0 accesses the Internet; only works if you are not behind NAT <span class="unselectable">$ </span>IPADDR=`ifconfig eth0|grep "inet addr"|awk -F: '{print $2}'|awk '{print $1}'` <span class="unselectable">$ </span># If you are behind NAT, you will need to use a dynamic DNS and set IPADDR to the machine name instead: <span class="unselectable">$ </span>IPADDR=mycomputer.no-ip.info # modify to suit <span class="unselectable">$ </span>ssh -NR *:9999:localhost:9999 $IPADDR&amp; <span class="unselectable">$ </span>jdb -listen 9999&amp; # Do not type this line if you are using an IDE</pre> <p>If using an IDE, start debugging now.</p> <pre data-lt-active='false'><span class="unselectable">$ </span># This is a really long line. I wrapped it, but you should not: <span class="unselectable">$ </span>heroku config:add JAVA_OPTS='-Xdebug -Xrunjdwp:transport=dt_socket,address=$IPADDR:9999 -Xms512M -Xmx1024M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M -XX:+UseCompressedOops' <span class="unselectable">$ </span>heroku restart</pre> <h2 id="disablingRemote">Disabling Remote Debugging</h2> <p> The <tt>JAVA_OPTS</tt> value set earlier will remain in effect across multiple restarts of your Heroku app until you change it. To disable remote debugging, redefine the environment variable without the <tt>-Xdebug</tt> and <tt>-Xrunjdwp</tt> options: </p> <pre data-lt-active='false'><span class="unselectable">$ </span># This is a really long line. I wrapped it, but you should not: <span class="unselectable">$ </span>heroku config:add JAVA_OPTS='-Xms512M -Xmx1024M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M -XX:+UseCompressedOops' <span class="unselectable">$ </span>heroku restart</pre> <h2 id="saving">Saving the Run Configuration</h2> <p> If you enable the <b>Share</b> checkbox in the IntelliJ IDEA run configuration dialog box, the definition will be written to <code>.idea/runConfiguration/</code>. Normally you would not check in the contents of <code>.idea/</code> to your source code repository, but this subdirectory is an exception, and because this run configuration has no local dependencies it can be shared without modification amongst all of your developer team. </p> <pre data-lt-active='false'><span class="unselectable">$ </span><b>cat .idea/runConfigurations/Heroku_remote.xml </b> &lt;component name="ProjectRunConfigurationManager"&gt; &lt;configuration default="false" name="Heroku remote" type="Remote" factoryName="Remote"&gt; &lt;option name="USE_SOCKET_TRANSPORT" value="true" /&gt; &lt;option name="SERVER_MODE" value="true" /&gt; &lt;option name="SHMEM_ADDRESS" value="javadebug" /&gt; &lt;option name="HOST" value="localhost" /&gt; &lt;option name="PORT" value="9999" /&gt; &lt;method /&gt; &lt;/configuration&gt; &lt;/component&gt;</pre> Composable Futures with Akka 2.0 2012-08-09T00:00:00-04:00 https://mslinn.github.io/blog/2012/08/09/composable-futures-with-akka-20 <div style="text-align: right;"> <picture> <source srcset="/blog/images/futuresCover226x296.webp" type="image/webp"> <source srcset="/blog/images/futuresCover226x296.png" type="image/png"> <img src="/blog/images/futuresCover226x296.png" title="Composable Futures with Akka 2.0 cover" class="right shadow rounded" alt="Composable Futures with Akka 2.0 cover" /> </picture> </div> <p> My latest book is finally complete! <a href="https://www.slinnbooks.com/books/futures/index.html" target="_blank" rel="nofollow">Composable Futures with Akka 2.0</a> features Java and Scala code examples. The book is available in PDF format. </p> <p> Akka is hot, and Akka futures are important for creating responsive applications that can scale. This book is intended for &lsquo;the rest of us&rsquo; &mdash; Java and Scala programmers who would like to quickly learn how design and integrate applications using composable futures. <p> <p> I <a href="https://www.meetup.com/sv-jug/events/50402942/" target="_blank" rel="nofollow">presented a preview of the book</a> at Googleplex April 18, 2012 to the Silicon Valley Java User Group. </p> Proposal – Mandatory Countervailing Tip 2012-08-07T00:00:00-04:00 https://mslinn.github.io/blog/2012/08/07/proposal-mandatory-countervailing-tip <p> Inequity between countries and ethnicities exploits individuals who are the primary producers and harms the individuals and societies that contain the primary consumers; it only benefits international traders. Over 99% of the world is exploited to serve less than the top 1%. This imbalance threatens world stability, as well as national, regional and local stability. </p> <p> This proposal is designed to create jobs in every country where the trade balance is negative, and is intended to reverse the exploitation of individuals in producing countries. I propose a Mandatory Countervailing Tip (MCT). It’s like a tip that you might offer someone who provides service, and whose wages are not sufficient to justify their employment. Waiters, waitresses, bellhops and musicians are paid tips for this reason. This proposal is unlike a countervailing tax because the consuming country does not keep the surcharge; instead, it would provide money directly to individuals in producing regions. The exact mechanism(s) for disbursing funds locally is not addressed in this proposal. </p> <p> The MCT would be levied at the point of purchase, like a sales tax, and would be disbursed directly to residents at the point(s) of origin of the goods who created the product. This would serve to improve the lives of all parties involved in the transaction, except the international trader. </p> <p> The MCT would be calculated such that exploited individuals themselves in the producing nation would directly receive enough money to raise their standard of living to within 25% of the standard of living for the middle 80% of the consuming region in the producing country. The money would be collected by the same government departments that collect sales tax and would be remitted every 30 days to a new agency in the UN. The entire staff of the UN agency would be required to be replaced every 2 years, and working committees and task forces in the agency would be comprised of residents of primary producers and consumers, as well as enough individuals from neutral countries to ensure impartiality and honesty. The states collecting the sales tax would be entitled to 1% of the funds collected for administrative overhead. The UN agency would be entitled to an additional 2%. The UN would be responsible for computing the MCT for every category of product sold, for every producing / consuming country. </p> <p> The sums involved would be enormous, so the potential for fraud is also enormous. To counteract this, unprecedented transparency must be implemented; this is possible because of modern computer and communications technology. Access to the data should be made public to the world at large. No sign-in should be required to download data concerning anonymized transactions and aggregated statistics. No data about individuals in producers or consumers must be available without security clearance. </p> <p> Transactions and queries requiring authorization must be audited, and auditors must be selected from presumably hostile countries and regions. For example, given that India and China are fierce competitors, they should provide the personnel to audit each other’s transactions. Those auditors would be paid from 0.002% of the total receipts. Auditors must also be reassigned every year, and may not ever be reassigned to the same region. </p> <p> Implementation need not require world-wide agreement; only one country need to decide that it wants to go ahead, and rough estimates of economic disparity would suffice to start. The first country to implement will be the first country to enjoy the benefits of a local economic resurgence. This can be done before the UN is ready, by setting up a temporary agency. </p> <p> This proposal is intended to greatly reduce the disparity between rich and poor throughout the world. It is fair because it is based on consumption, and the degree of equalization applied is proportional to the disparity between the standards of living between producer and consumer. This should greatly decrease the suicide rate of the indentured slaves who build iPads in China, while providing an opportunity for manufacturing to resurge in the US. Think of it as a regulated fair trade for world stability, prosperity and peace. </p> <p> Colonialism is based on exploitation, but capitalism does not require it. One might argue that a certain amount of exploitation is healthy because without some measure of economic disparity, undeveloped regions would not have a cost advantage and therefore never develop. This is probably true, so the question then becomes: how much economic disparity is healthy? I don’t know the answer, but a legislated policy on whatever that figure needs to be is preferable to uncontrolled exploitation. I doubt that the figure would need to be exceed 50%, and probably should be less. Current figures are in the range of 1% to 5%. </p> <p> The MCT is an example of how local, regional, national and world government can provide for the greater good. The only parties who have reason to oppose this type of proposal are those who benefit in some way from exploitation. Think of the goodwill that the recipients will feel towards their end users, and the countries that they live in. </p> Scala Existential Types and Salat 2012-08-06T00:00:00-04:00 https://mslinn.github.io/blog/2012/08/06/scala-existential-types <p> For Scala programmers, the term 'existential types' does not refer to <a href="https://en.wikipedia.org/wiki/Existentialism" target="_blank" rel="nofollow">philosophers</a>, or even authentic people. Ironically, most of the discussions of Scala's existential types that I found are too abstract to be useful to me. I learn by doing; rarely do I learn from reading an abstract treatise. I guess this means that I could be labelled an existentialist. </p> <p> Section 31.3 of <a href="https://www.artima.com/shop/programming_in_scala_2ed" target="_blank" rel="nofollow">Programming in Scala</a> has some good information on Scala's existential types. Existential types is an abstraction of Java types, and abstraction is the antithesis of existentialism... aaaanyway, here is a sentence I stole from the book:</p> <p class="rounded shadow liImg" style="padding: 1em;"><tt>Iterator[_]</tt> means the same thing as <tt>Iterator[T] forSome { type T }</tt></p> <p> <tt>forSome</tt> is the keyword which tells the compiler that an existential type is being defined. This becomes useful if upper and/or lower bounds are used to define the type. For example, you can use a lower bound to specify that the type must be a subclass of <tt>PubSubAction</tt> with the following existential type: </p> <pre data-lt-active='false'>T forSome { type T &lt;: PubSubAction }</pre> <p>That is a lot of characters, so let's give this type a name:</p> <pre data-lt-active='false'>type PubSubActionSubclass = T forSome { type T &lt;: PubSubAction }</pre> <p>We can also define a type to help us with <tt>SalatDAO</tt>:</p> <pre data-lt-active='false'>type SalatObject = T forSome { type T &lt;: AnyRef }</pre> <p>Now lets use <tt>SalatObject</tt> as the parametric type for an invocation of <tt>Manifest.classType()</tt>:</p> <pre data-lt-active='false'>val mot = Manifest.classType[SalatObject](msg.getClass)</pre> <p> This is useful because the manifest can be passed to a <a href="https://github.com/novus/salat/wiki/SalatDAO" target="_blank" rel="nofollow"><tt>SalatDAO</tt></a> constructor, which will create a DAO object for whatever type is supplied. In the following code, <tt>PubSubActionClass</tt> is just used to guarantee that the <tt>msg</tt> parameter is of the correct type; <tt>SalatDAO</tt>'s constructor is defined to accept all subclasses of <tt>AnyRef</tt>. </p> <pre data-lt-active='false'>import com.novus.salat.global.ctx abstract class PubSubAction object PubSubAction { def makeDAO[PubSubActionSubclass](msg: PubSubActionSubclass) (implicit coll: MongoCollection) = { val mot = Manifest.classType[SalatObject](msg.getClass) val mid = Manifest.classType[Int](classOf[Int]) new SalatDAO(coll)(mot, mid, ctx){} } }</pre> <p> Now you know that you do not have to hard-code the creation of a DAO for every <tt>PubSubAction</tt> subclass. You can examine the <a href="https://github.com/novus/salat/blob/master/salat-core/src/main/scala/com/novus/salat/dao/SalatDAO.scala" target="_blank" rel="nofollow">source code for <tt>SalatDAO</tt></a> if you would like to learn more. </p> <p> Assuming that <tt>psaMsg</tt> is an instance of a <tt>PubSubActionSubclass</tt>, you could call <tt>insert()</tt> as follows to do a generic insert. Because Salat has multiple methods called <tt>insert()</tt>, there is not enough type information for Scala to disambiguate the method reference unless the returned value from the single insert is stored in a variable, and that variable's type is provided: </p> <pre data-lt-active='false'>val ignoredIndex: Option[Int] = makeDAO.insert(psaMsg)</pre> <p> There is only one variant of <tt>insert()</tt> which accepts a collection, so the reference is not ambiguous and the return value can be ignored: </p> <pre data-lt-active='false'>dao.insert(Seq(psaMsg, psaMsg2, psaMsg3), WriteConcern.Normal)</pre> Pushing Notifications to Nagios from Java and Scala 2012-08-04T00:00:00-04:00 https://mslinn.github.io/blog/2012/08/04/pushing-notifications-to-nagios-from <p> I was investigating how to push notifications from JVM-based programs when I found the <a href="https://sourceforge.net/projects/nagiosappender" target="_blank" rel="nofollow">NagiosAppender</a> project. Push notifications are termed &lsquo;passive checks&rsquo; because Nagios does not poll for results. For the curious, see the Nagios Plugins &mdash; Passive Service Check section of <a href="https://www.novell.com/communities/node/4131/application-monitoring-made-easy-java-applications-using-nagios" target="_blank" rel="nofollow">Application Monitoring Made Easy for Java Applications Using Nagios</a>. </p> <p> NagiosAppender integrates Log4j or Logback with Nagios&rsquo; optional NSCA server. The only &lsquo;programming&rsquo; required is setting up configuration files for Nagios client and Nagios server, adding a new dependency, and writing appropriate log messages for forwarding to Nagios by the plugin. Unfortunately, NagiosAppender is not compatible with Akka because <a href="https://logback.qos.ch/manual/mdc.html" target="_blank" rel="nofollow">it uses MDC</a>, which uses <a href="https://docs.oracle.com/javase/7/docs/api/java/lang/ThreadLocal.html" target="_blank" rel="nofollow"><tt>ThreadLocal</tt></a> variables, which should not be used with Akka. I took the NagiosAppender project, slimmed it down, removed the Log4j interface and the MDC code, and created the <a href="https://github.com/mslinn/PushToNagios" target="_blank" rel="nofollow">PushToNagios</a> project. </p> <p> NB: The document mentions MDC without defining it. From the <a href="https://logging.apache.org/log4j/1.2/" target="_blank" rel="nofollow">Apache log4j</a> docs: &ldquo;A Mapped Diagnostic Context, or MDC, is an instrument for distinguishing interleaved log output from different sources. Log output is typically interleaved when a server handles multiple clients near-simultaneously. The MDC is managed on a per-thread basis. A child thread automatically inherits a copy of the mapped diagnostic context of its parent.&rdquo; The Logback documentation has a <a href="https://logback.qos.ch/manual/mdc.html" target="_blank" rel="nofollow">whole chapter on MDC</a>. </p> <h3 id="nsca">NSCA</h3> <p> NSCA is a Nagios add-on that allows you to send <a href="http://nagios.sourceforge.net/docs/nagioscore/3/en/passivechecks.html" target="_blank" rel="nofollow">passive check</a> results from remote hosts to the Nagios daemon running on the monitoring server. This is very useful in <a href="http://nagios.sourceforge.net/docs/nagioscore/3/en/distributed.html" target="_blank" rel="nofollow">distributed</a> and <a href="http://nagios.sourceforge.net/docs/nagioscore/3/en/redundancy.html" target="_blank" rel="nofollow">redundant/failover</a> monitoring setups. The NSCA addon can be found on <a href="http://exchange.nagios.org/directory/Addons/Passive-Checks/NSCA--2D-Nagios-Service-Check-Acceptor/details" target="_blank" rel="nofollow">Nagios Exchange</a>. For more information, see <a href="http://exchange.nagios.org/directory/Addons/Passive-Checks/NSCA--2D-Nagios-Service-Check-Acceptor/details" target="_blank" rel="nofollow">Addon &mdash; Nagios Passive Checks with NSCA</a>. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/nsca.webp" type="image/webp"> <source srcset="/blog/images/nsca.png" type="image/png"> <img src="/blog/images/nsca.png" title="Nagios addon: Passive Checks with NSCA" class="center liImg2 rounded shadow" alt="Nagios addon: Passive Checks with NSCA" /> </picture> </div> <h3 id="installation">Installation</h3> <p>For Ubuntu:</p> <pre data-lt-active='false'><span class="unselectable">$ </span>sudo apt-get install nagios3 nsca</pre> <p> Installs Nagios Core 3.2.3, which is outdated but compatible, and <tt>nsca 2.7.2+nmu2</tt>, which is current. The current version of Nagios Core is 3.4.1, released on 2012-05-14. Nagios starts automatically after installation, but NSCA needs to be started manually (don't do that yet, keep reading). Navigate your web browser to <a href="http://localhost/nagios3/" target="_blank" rel="nofollow">http://localhost/nagios3</a> and specify userid <tt>nagiosadmin</tt>. </p> <p>FYI, <tt>/etc/init.d/nagios3</tt> contains:</p> <pre data-lt-active='false'>DAEMON=/usr/sbin/nagios3 NAGIOSCFG="/etc/nagios3/nagios.cfg" CGICFG="/etc/nagios3/cgi.cfg"</pre> <p><tt>/etc/nagios3/nagios.cfg</tt> contains:</p> <pre data-lt-active='false'>log_file=/var/log/nagios3/nagios.log cfg_file=/etc/nagios3/commands.cfg cfg_dir=/etc/nagios-plugins/config</pre> <p><tt>/etc/nagios3/resource.cfg</tt> contains:</p> <pre data-lt-active='false'># Sets $USER1$ to be the path to the plugins $USER1$=/usr/lib/nagios/plugins</pre> <p><tt>/etc/init.d/nsca</tt> contains:</p> <pre data-lt-active='false'>DAEMON=/usr/sbin/nsca CONF=/etc/nsca.cfg OPTS="--daemon -c $CONF" PIDFILE="/var/run/nsca.pid"</pre> <p> We saw above that plugins are in <tt>/usr/lib/nagios/plugins/</tt>. I added one called <tt>check_domain_bus</tt> with permissions set to 755, and owned by <tt>nagios:nagios</tt>: </p> <pre data-lt-active='false'>#!/bin/sh echo "All OK: $1" exit 0 Configuration</pre> <p>Edit <tt>/etc/nagios3/nagios.cfg</tt> and enable external commands on line 145 so the entry looks like this:</p> <pre data-lt-active='false'>check_external_commands=1</pre> <p>Edit the last line of <tt>/etc/nsca.cfg</tt> to disable encryption:</p> <pre data-lt-active='false'>decryption_method=0</pre> <p>Define a new Nagios command called <tt>check_domain_bus</tt> in <tt>/etc/nagios3/commands.cfg</tt> by adding the following anywhere in that file:</p> <pre data-lt-active='false'>define command { command_name check_domain_bus command_line $USER1$/check_domain_bus $ARG1$ }</pre> <p> Define a template for passive services, and an instance of a passive service called <tt>domainBus</tt> that responds to the <tt>check_domain_bus</tt> command by adding the following to <tt>/etc/nagios3/conf.d/services_nagios2.cfg</tt>: </p> <pre data-lt-active='false'>define service { &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; name&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; passive-service &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; use&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; generic-service &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; check_freshness&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; passive_checks_enabled&nbsp; 1 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; active_checks_enabled&nbsp;&nbsp; 0 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; is_volatile&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; flap_detection_enabled&nbsp; 0 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; notification_options&nbsp;&nbsp;&nbsp; w,u,c,s &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; freshness_threshold&nbsp;&nbsp;&nbsp;&nbsp; 57600&nbsp;&nbsp;&nbsp;&nbsp; ;12hr } define service { &nbsp;&nbsp;&nbsp; use&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; passive-service&nbsp; &nbsp;&nbsp;&nbsp; host_name&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; localhost &nbsp;&nbsp;&nbsp; service_description&nbsp;&nbsp;&nbsp;&nbsp; domainBus &nbsp;&nbsp;&nbsp; check_command&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; check_domain_bus!0 }</pre> <h3 id="usage">Usage</h3> <p>Start NSCA server, then restart Nagios:</p> <pre data-lt-active='false'><span class="unselectable">$ </span>sudo service nsca start <span class="unselectable">$ </span>sudo service nagios3 restart</pre> <p> The custom service, called <tt>domainBus</tt>, should be viewable as a Nagios service, shown in the red rectangle below: </p> <div style=""> <picture> <source srcset="/blog/images/nagiosOK.webp" type="image/webp"> <source srcset="/blog/images/nagiosOK.png" type="image/png"> <img src="/blog/images/nagiosOK.png" title="Nagios domainBus service" class=" liImg2 rounded shadow" style="width: 100%" alt="Nagios domainBus service" /> </picture> </div> <p> Nagios will need to be restarted each time a service definition is modified. New services are shown as <tt>PENDING</tt> until they receive their first result. Passive services have no scheduled updates. </p> <h3 id="testing">Testing with the PushToNagios Java Client</h3> <p>See the <a href="https://github.com/mslinn/PushToNagios" target="_blank" rel="nofollow">PushToNagios</a> documentation.</p> <h3 id="test2">Testing With the Compiled C NSCA Client</h3> <p> Let&rsquo;s send a message and have the result displayed on the web interface. <code>send_nsca</code> is a <a href="https://sourceforge.net/projects/nagios/files/nsca-2.x/nsca-2.7.2/nsca-2.7.2.tar.gz/download" target="_blank" rel="nofollow">compiled C nsca client</a> that can be used to send a test message. Unpack <tt>nsca-2.7.2.tar.gz</tt> into a directory, and compile it: </p> <pre data-lt-active='false'><span class="unselectable">$ </span>./configure <span class="unselectable">$ </span>make install</pre> <p>Again, edit the last line of <tt>sample-config/send_nsca.cfg</tt> and change it to read:</p> <pre data-lt-active='false'>decryption_method=0</pre> <p>Create a test message in the root of the unpacked NSCA project. The format for a service check packet using NSCA contains tab characters and ends in a newline, like this:</p> <pre data-lt-active='false'>&lt;hostname&gt;[tab]&lt;svc_description&gt;[tab]&lt;return_code&gt;[tab]&lt;plugin_output&gt;</pre> <p>I am unsure if <tt>&lt;hostname&gt;</tt> refers to the Nagios host or the sending host. The allowable values for <tt>&lt;return_code&gt;</tt> are:</p> <blockquote>0 - OK state<br> 1 - Warning state<br> 2 - Error state<br> 3 - Unknown state</blockquote> <p><tt>&lt;plugin_output&gt;</tt> can be up to 512 bytes long.</p> <p>Create a text message called <tt>testCritical</tt>, with embedded tabs, that NSCA uses as a field delimiter.</p> <pre data-lt-active='false'>localhost&nbsp;&nbsp; domainBus&nbsp;&nbsp;&nbsp; 2&nbsp;&nbsp; This is a Test Error</pre> <p>Watch the Nagios log and syslog in one console:</p> <pre data-lt-active='false'><span class="unselectable">$ </span>tail -f /var/log/nagios3/nagios.log /var/log/syslog</pre> <p>Send the test message like this in another console; let's call this the command console:</p> <pre data-lt-active='false'><span class="unselectable">$ </span>src/send_nsca localhost -c sample-config/send_nsca.cfg &lt; testCritical</pre> <p>Notice the log output in the console with the log output:</p> <pre data-lt-active='false'>==&gt; /var/log/nagios3/nagios.log &lt;== [1343251589] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;localhost;domainBus;2;This is a Test Error ==&gt; /var/log/syslog &lt;== Jul 25 14:26:29 natty nagios3: EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;localhost;domainBus;2;This is a Test Error ==&gt; /var/log/nagios3/nagios.log &lt;== [1343251590] PASSIVE SERVICE CHECK: localhost;domainBus;2;This is a Test Error ==&gt; /var/log/syslog &lt;== Jul 25 14:26:30 natty nagios3: PASSIVE SERVICE CHECK: localhost;domainBus;2;This is a Test Error ==&gt; /var/log/nagios3/nagios.log &lt;== [1343251590] SERVICE ALERT: localhost;domainBus;CRITICAL;SOFT;1;This is a Test Error ==&gt; /var/log/syslog &lt;== Jul 25 14:26:30 natty nagios3: SERVICE ALERT: localhost;domainBus;CRITICAL;SOFT;1;This is a Test Error</pre> <p> In the web browser, click on <b>Services</b> again and notice that the status of the domainBus service is now <tt>CRITICAL</tt>, and <b>Status Information</b> now reads: </p> <pre data-lt-active='false'>This is a Test Error</pre> <div style=""> <picture> <source srcset="/blog/images/nagiosCritical.webp" type="image/webp"> <source srcset="/blog/images/nagiosCritical.png" type="image/png"> <img src="/blog/images/nagiosCritical.png" title="Status of the domainBus service is now 'CRITICAL'" class=" liImg2 rounded shadow" style="width: 100%" alt="Status of the domainBus service is now 'CRITICAL'" /> </picture> </div> <p>Create a text message called <tt>testClear</tt>, and do not forget the embedded tabs:</p> <pre data-lt-active='false'>localhost&nbsp;&nbsp; domainBus&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; Mischief Managed</pre> <p>Send this new test message in the command console:</p> <pre data-lt-active='false'><span class="unselectable">$ </span>src/send_nsca localhost -c sample-config/send_nsca.cfg &lt; testClear</pre> <p>The log output console should show something like this:</p> <pre data-lt-active='false'>==&gt; /var/log/nagios3/nagios.log &lt;== [1343252049] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;localhost;domainBus;0;Mischief Managed ==&gt; /var/log/syslog &lt;== Jul 25 14:34:09 natty nagios3: EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;localhost;domainBus;0;Mischief Managed ==&gt; /var/log/nagios3/nagios.log &lt;== [1343252050] PASSIVE SERVICE CHECK: localhost;domainBus;0;Mischief Managed [1343252050] SERVICE ALERT: localhost;domainBus;OK;SOFT;2;Mischief Managed ==&gt; /var/log/syslog &lt;== Jul 25 14:34:10 natty nagios3: PASSIVE SERVICE CHECK: localhost;domainBus;0;Mischief Managed</pre> <p> In the web browser, the <b>Services</b> information should automatically update after a pause of up to 90 seconds (by default), or you can click on Services to immediately see the new status. Notice that the status of the <tt>domainBus</tt> service is now <tt>OK</tt>, and <b>Status Information</b> now reads <tt>Mischief Managed</tt>. </p> <div style=""> <picture> <source srcset="/blog/images/nagiosOK.webp" type="image/webp"> <source srcset="/blog/images/nagiosOK.png" type="image/png"> <img src="/blog/images/nagiosOK.png" title="Status Information now reads 'Mischief Managed'" class=" liImg2 rounded shadow" alt="Status Information now reads 'Mischief Managed'" /> </picture> </div> <p> The third possible message status is <tt>warning</tt>. Create a text message called <tt>testWarning</tt> and do not forget the embedded tabs: </p> <pre data-lt-active='false'>localhost&nbsp;&nbsp; domainBus&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; Do you know where your chocolate is?</pre> <p>Send the test message like this in the command console:</p> <pre data-lt-active='false'><span class="unselectable">$ </span>src/send_nsca localhost -c sample-config/send_nsca.cfg &lt; testWarning</pre> <p> In the web browser, click on <b>Services</b> to immediately see the new status. Notice that the status of the <tt>domainBus</tt> service is now <tt>OK</tt>, and <b>Status Information</b> now reads <tt>Do you know where your chocolate is?</tt> </p> <div style=""> <picture> <source srcset="/blog/images/nagiosWarning.webp" type="image/webp"> <source srcset="/blog/images/nagiosWarning.png" type="image/png"> <img src="/blog/images/nagiosWarning.png" title="Status Information now reads 'Do you know where your chocolate is?''" class=" liImg2 rounded shadow" style="width: 100%" alt="Status Information now reads 'Do you know where your chocolate is?''" /> </picture> </div> Scala Type Parameters, Implicit Manifests and Salat 2012-08-02T00:00:00-04:00 https://mslinn.github.io/blog/2012/08/02/scala-type-parameters-implicit <p>Type parameters provide types for constructing any arguments that are implicit manifests.</p> <p>This helps when working with <a href="https://github.com/novus/salat/" target="_blank" rel="nofollow">Salat</a>. In the following code, <tt>dbo</tt> is a <tt>com.mongodb.DBObject</tt>, perhaps retrieved as the result of a <tt>find()</tt>. Salat uses <tt>_typeHint</tt> to store the fully qualified name of the persisted class; this property is returned as one of the properties of <tt>dbo</tt>. The call to Salat's <tt>grater()</tt> method converts <tt>dbo</tt> into the desired type. The type is constrained using a lower bound to be a subclass of <tt>PubSubAction</tt>.</p> <pre data-lt-active='false'>def findWithSeqNum[T &lt;: PubSubAction](seqNum: Long) (implicit manifest: Manifest[T]): Option[T] = { val klass = manifest.erasure.asInstanceOf[Class[T]] val item = myCollection.findOne( MongoDBObject("seqNum" -&gt; seqNum, "_typeHint" -&gt; klass.getName)) item match { case Some(dbo: DBObject) =&gt; Some(grater(ctx, Manifest.classType(klass)).asObject(dbo)) case None =&gt; None } }</pre> <p>When called as follows, the type parameter need not be provided because there is an explicit manifest argument:</p> <pre data-lt-active='false'>findWithSeqNum(11)(Manifest.classType(classOf("com.micronautics.Blah"))</pre> <p>When called as follows, the type parameter provides the value for constructing the implicit manifest argument:</p> <pre data-lt-active='false'>findWithSeqNum[Blah](11)</pre> <p>Of course, you can also explicitly define an implicit manifest at any scope that you desire:</p> <pre data-lt-active='false'>implicit val manifest: ClassManifest[Blah] findWithSeqNum(11)</pre> <p>If the type of the manifest is dynamic (only known at runtime), you can define manifest this way:</p> <pre data-lt-active='false'>val mtype = "com.micronautics.Blah" implicit val manifest = Manifest.classType(Class.forName(mtype)) findWithSeqNum(11)</pre> Pigs Can Fly 2011-09-30T00:00:00-04:00 https://mslinn.github.io/blog/2011/09/30/pigs-can-fly <div style="text-align: center;"> <picture> <source srcset="/assets/images/flyingPig.webp" type="image/webp"> <source srcset="/assets/images/flyingPig.png" type="image/png"> <img src="/assets/images/flyingPig.png" title="Flying pig" class="center liImg2 rounded shadow" alt="Flying pig" /> </picture> </div> <p> Details matter. If all you read are headlines and subtitles then it is likely that you are horribly misinformed. Let’s imagine that the text of an article was something like this: </p> <div class="informalNotice shadow rounded liImg"> <h2>Pigs Can Fly</h2> <p> Outfitted with 3D goggles and haptic feedback mechanisms customized for a pig&rsquo;s extremities, porcine subjects showed that they quickly adapted to visual stimulus that suggested to them that their movements caused them to fly. With training, pigs demonstrated that they could control the flight of simulated cargo planes, stunt triplanes, jet fighters and paper planes. </p> </div> <p> So, does the passage say that pigs can actually fly unassisted? Does it say that pigs <i>might think</i> they can fly? Does it discuss the possibility that pigs might be able to be trained as pilots? You would have to read the passage carefully to be able to answer those questions. If you just skim headlines then you would have no idea what the passage was about, or even if it was just humor. </p> <p> Details matter a lot when you are working with technologists. If you are easily put off by even the slightest techno-speak, then you have no business managing technologists. Specialized vocabulary summarizes complex thoughts succinctly; learn it, don&rsquo;t ask for the baby-talk version. </p> <ul> <li>You can&rsquo;t</li> <li>Summarize complex interactions</li> <li>Into a few bullet points</li> <li>Without getting it wrong</li> </ul> <p> This is true for reading and writing. If the best that a manager can grunt via a keyboard is &ldquo;you know what I mean&rdquo; then it is probable that the recipient of the email has at best only a vague idea of what the lazy, inarticulate writer meant. In truth, the writer has probably not thought through their ideas properly, and they are not clear what they meant either. </p> <p>Pick your information sources and sinks with discretion, and strive to communicate in complete thoughts.</p> <p> Here is some <a href="https://www.newscientist.com/article/dn24541-monkey-controls-two-virtual-arms-with-thoughts-alone/" target="_blank" rel="nofollow">related science</a>. </p> AMF over HTTP in another Multiverse 2011-09-21T00:00:00-04:00 https://mslinn.github.io/blog/2011/09/21/amf-over-http-in-another-multiverse <p> This silly story is taken from my book <a href="https://www.slinnbooks.com/books/serverSide/index.html" target="_blank" rel="nofollow">Flex Data Services, Hibernate and Eclipse</a>. No other part of that book is deliberately silly. I thought it would be fun to post this here. In the story, produce and food are allegories for data and a cellar is an allegory for a database. As for Klay and Adobe, well, you get the idea. </p> <p> Once upon a time, in an alternate multiverse, there might or might not have been a world called Klay. Klay was a world very like Earth, except that the peaceful inhabitants were obsessed with food and food technology. This world had perfected the art and science of growing and cooking food. Each region had its specialty, however because transportation was not mechanized. Only seasoned travelers had ever experienced the wondrous tastes and textures of the exquisite culinary creations in far-off lands. Because growing conditions varied widely, food could only be made from local produce, and thus could not be replicated outside where the food was grown. Instead, the pudgy townsfolk would gather around wandering minstrels as they sang of the nutritional marvels that they had experienced in far-off lands. </p> <p> Food was grown, prepared, cooked and eaten nearby the same fields that it was grown. The Klaylings valued freshness and texture very highly, but they found that some produce actually improved when it was aged. This produce was stored in cellars next to the fields where the produce was grown. When a chef prepared a meal, he/she would gather the ingredients and cook it to order. </p> <p> A group of rich merchants in a prosperous region realized that there was profit to be made by selling prepared food to neighboring regions. Their first attempt was to have a chef prepare a dish that resembled a souffl&eacute;, and have it delivered by horseback. The rider, holding the dish in front of him, ended up with egg all over himself. Clearly, a means of packaging the food was required. </p> <p> The merchants also had another problem. Each country had different customs regarding the shape of the containers from which they ate. It was inconceivable that a person accustomed to eating from a square bowl might eat instead from a round or oval bowl. This complication was considered serious because as any chef will tell you, presentation is very important. The food had to be prepared for the specific containers that it was served in, and once prepared could not be transferred to another container without ruining the presentation, thereby severely reducing the selling price. </p> <p> The merchants called upon the local wise woman, who meditated and went into a trance. An acolyte wrote down the muttered sentences that she uttered while communing with the Fat God. Without knowing what the words meant, she wrote down the words and presented them to the merchants after the wise woman fell asleep. The words were: </p> <p>Awesome Mouthwatering Food over Highspeed Trusty Transport Provider.</p> <p> Another acolyte, skilled in numerology, tried turning the words into sacred acronyms: AMF over HTTP. Try as he might, he could not understand any significance in those words. </p> <p>When the wise woman awoke, she drew this diagram:</p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/allegory.webp" type="image/webp"> <source srcset="/blog/images/allegory.png" type="image/png"> <img src="/blog/images/allegory.png" title="Wise Woman’s Food Delivery Service diagram" class="center halfsize liImg2 rounded shadow" style="padding: 1em" alt="Wise Woman’s Food Delivery Service diagram" /> </picture> </div> <p> The merchants gathered around the wise woman as she explained the diagram. &ldquo;The food from the fields and the cellars are brought into the kitchen as required. The meal is cooked to order, then deconstituted and packaged. A magic carpet uses AMF over HTTP to deliver the deconstituted food to the client’s kitchen, where it is reconstituted by one of the local chefs. The food is served into the appropriate containers for the patrons, and the carpet returns with the chef and payment. </p> <p> &ldquo;The key is AMF over HTTP. This magical process, yet to be invented, would consist of a tesseract, larger on the inside than it is on the outside. Within the tesseract, AMF would absorb the raw essence of the prepared food. The magic carpet would transport the tesseract almost instantaneously to the far-off land where the client lives. The magical AMF process would then reconstitute the food into the client’s desired containers.&rdquo; </p> <p> The merchants were incredulous. &ldquo;But we have never heard of AMF over HTTP. Where can we find this magical process, and the magic carpet?&rdquo; </p> <p> The old woman replied that she did not know. &ldquo;It was only a dream,&rdquo; she replied despondently. </p> <p> Fortunately, in this multiverse, AMF over HTTP does exist, and it transports data between client and servers just as the wise old woman described &mdash; and a lot more. Under the covers, AMF is used internally by Flex for many purposes. The next few chapters discuss how that can be done, and more. The Internet is our magic carpet. </p> Working With Bug Fix and Feature Branches in Git 2011-08-18T00:00:00-04:00 https://mslinn.github.io/blog/2011/08/18/working-with-bug-fix-and-feature <p> The project I am currently leading is following <a href="https://nvie.com/posts/a-successful-git-branching-model/" target="_blank" title="A Successful git Branching Model" rel="nofollow">A Successful git Branching Model</a> with only a few modifications. </p> <h3 id="create">Create a Bug Fix / New Feature Branch</h3> <p> When a developer goes to work on a new task, bug or feature they should make new branch from the <code>develop</code> branch to start their work. The new branch will be named after the Jira task ID, which in the following example is <code>EDOC-1234</code>. </p> <pre data-lt-active='false'><span class="unselectable">$ </span>git checkout develop // make develop branch current <span class="unselectable">$ </span>git pull // get latest commits <span class="unselectable">$ </span>git checkout -b EDOC-1234 develop // make new EDOC-1234 branch</pre> <p> That creates a new branch for you in your local repository named <code>EDOC-1234</code> based on the <code>develop</code> branch where everyone’s latest work is checked in. From there you can work on your new feature. For example: </p> <pre data-lt-active='false'><span class="unselectable">$ </span>echo "42"&gt;&gt;theAnswer.txt <span class="unselectable">$ </span>add theAnswer.txt <span class="unselectable">$ </span>commit -m "The answer to life, the universe and everything" <span class="unselectable">$ </span>echo "43"&gt;&gt;yetMore.txt <span class="unselectable">$ </span>add yetMore.txt <span class="unselectable">$ </span>commit -m "A present for the god who has everything: one more thing beyond 42"</pre> <h3 id="develop">Update your Bug Fix / Feature Branch with the Latest From the <code>develop</code> Branch</h3> <p> If someone commits to the <code>develop</code> branch while you are working on your feature you can get their work by (re)merging the <code>develop</code> branch to your <code>EDOC-1234</code> branch: </p> <pre data-lt-active='false'><span class="unselectable">$ </span>git checkout develop // switch to the develop branch <span class="unselectable">$ </span>git pull origin develop // pull the latest changes from the develop branch at origin <span class="unselectable">$ </span>git checkout EDOC-1234 // switch back to your feature branch <span class="unselectable">$ </span>git merge develop // add all the changes from the develop branch to your feature branch</pre> <h3 id="merge">Merge your Bug Fix / Feature Branch Back to the <code>develop</code> Branch</h3> <p> When you are done with your feature, merge it back to the <code>develop</code> branch. </p> <pre data-lt-active='false'><span class="unselectable">$ </span>git checkout develop // switch the the develop branch <span class="unselectable">$ </span>git pull origin develop // pull the latest commits to the develop branch on origin <span class="unselectable">$ </span>git merge --no-ff EDOC-1234 // merge the updates from the develop branch on origin <span class="unselectable">$ </span>git push origin develop // push your merged work to origin</pre> <h3 id="delete">Deleting a Bug Fix / Feature Branch</h3> <p> This will be done after the branch has had a code review and Q/A is complete. </p> <pre data-lt-active='false'><span class="unselectable">$ </span>git branch -d EDOC-1234 // remove your feature branch</pre> Tracking Down a Mismatched JAR 2011-08-04T00:00:00-04:00 https://mslinn.github.io/blog/2011/08/04/tracking-down-mismatched-jar <p> For debugging mismatched jars, first build a list of the jars in question, then run <a href="https://github.com/rh1247/jwhich/blob/master/releases/jwhich-1.01.zip" target="_blank"><code>jwhich</code></a> to discover the jar that provides a specific Java class. Put the jar and the script provided in the download in a directory on your <code>PATH</code> such as <tt>/usr/local/bin</tt>. </p> <p>For *nix and Mac, build the list of jars under the current directory like this:</p> <pre data-lt-active='false'>export CLASSPATH=$(find . -name \*.jar -printf %h/%p:)</pre> <p>For Cygwin, build the list of jars under the current directory like this:</p> <pre data-lt-active='false'>export CLASSPATH=$(find . -name \*.jar -printf '%h/%p;')</pre> <p>Run the script like this:</p> <pre data-lt-active='false'><span class="unselectable">$ </span>jwhich classNameToFindHere</pre> Debugging Spring's SEVERE Error ListenerStart 2011-08-04T00:00:00-04:00 https://mslinn.github.io/blog/2011/08/04/debugging-severe-error-listenerstart <p> This Spring startup error can be mystifying if you don&rdquo;t know how to tackle it. <code data-lt-active='false'>listenerStart()</code> configures and invokes application event listeners for a <code>Context</code>. Most Spring applications have several listeners, and they should be invoked in the following order. Each of them causes debug output, so if you do not see any, the first listener died: </p> <ol data-lt-active='false'> <li><code>org.springframework.web.context.ContextLoaderListener</code></li> <li><code>org.springframework.web.context.request.RequestContextListener</code></li> <li><code>flex.messaging.HttpFlexSession</code> (for Spring/Flex integration)</li> </ol> <p> If the error message appeared, one of the above did not initialize correctly. Put breakpoints in each of the listener classes&rsquo; <code>contextInitialized()</code> or <code>requestInitialized()</code> methods. </p> <p> To <a href="https://dev-answers.blogspot.com/2010/03/enable-debugtrace-level-logging-for.html" target="_blank" rel="nofollow">view debug log messages</a> for <code>StandardContext</code>, add: </p> <pre data-lt-active='false'>1catalina.org.apache.juli.FileHandler.bufferSize = -1</pre> <p> <code>SEVERE: Error listenerStart</code> messages can be debugged by setting a breakpoint at <code data-lt-active='false'>org.springframework.web.context.ContextLoaderListener</code>, line 47. </p> <pre data-lt-active='false'>contextLoader.initWebApplicationContext(event.getServletContext());</pre> <p> Step return once and wait for the container to load everything. If there is an error you will now be at <code data-lt-active='false'>standardcontext.listenerstart</code> and you will see the error in the variable window under the <code data-lt-active='false'>t</code> variable. </p> <p>For debug output, add this to <code>web.xml</code>:</p> <pre data-lt-active='false'>&lt;listener&gt; &lt;listener-class&gt; org.springframework.web.util.Log4jConfigListener &lt;/listener-class&gt; &lt;/listener&gt;</pre> Assessing Sample Code from Job Applicants 2011-05-24T00:00:00-04:00 https://mslinn.github.io/blog/2011/05/24/assessing-sample-code-from-job <p> Once again I am reviewing programmer resumes for a client. This client needs to hire an entire team, including Flex and Java programmers. Because programmers write software, I always request sample code from candidates. I do not ask stupid hypothetical questions or play mind games &mdash; I just want to assess the quality of their work. </p> <p> I request the code along with a resume. Code is truth. I do not interview candidates who do not provide sample code that passes inspection. The requirement for sample code is published in the job description. Surprisingly, most applicants do not provide sample code. I do not respond to those applications because clearly those people do not follow simple instructions. </p> <p> This is worth repeating: <b>If a job applicant wants to be considered as a candidate, they need to provide sample code along with their resume.</b> </p> <div style=""> <picture> <source srcset="/blog/images/samples_690x518.webp" type="image/webp"> <source srcset="/blog/images/samples_690x518.png" type="image/png"> <img src="/blog/images/samples_690x518.png" title="Samples of produce" class=" liImg2 rounded shadow" alt="Samples of produce" /> </picture> </div> <p> I assume that the programs that candidates show me are examples of their best work, or at least work that they are proud of. Most of my clients need programmers to develop software products or in-house applications, so efficiency and maintainability are important. </p> <p> I ask for sample code in at least two languages. It is up to the candidate to decide how much to show me, and what they show me. The more sample code they show me, the more sure I can be about my assessment. </p> <p>Here are some issues that I look for in submitted code:</p> <ol> <li>Is it difficult or expensive to maintain?</li> <li>Was the code written with the next programmer on this project in mind?</li> <li>Is logic used instead of data structures?</li> <li>Is the code neat, and does it follow a consistent pattern? I don’t care if particular formatting conventions are followed, I am looking for an ordered mind and a conscientious worker.</li> <li>Is the code testable? Were unit tests and/or integration tests provided?</li> <li>Does the code contain magic values instead of declared constants?</li> <li>Is the code inefficient (are there lots of conversions between string and int, for example)?</li> <li>Does the programmer use idioms specific to the language that it is written in?</li> <li>Are there appropriate comments, especially Javadoc / asdoc style comments?</li> <li>Are default actions or values present in the code? Defaults are tacitly understood by competent programmers, and providing them just adds noise. Code and documentation should be written with the assumption that readers will be technically competent.</li> <li>Is encapsulation violated because most methods are public?</li> <li>Are there syntax errors?</li> </ol> <p>For Adobe Flex programmers I also look for:</p> <ol> <li>Is there proper usage of the Flex component life cycle?</li> <li>Is there a heavy dependence on binding?</li> <li>Are events used properly?</li> <li>Are heavyweight Halo containers used, when Spark Groups would suffice?</li> <li>Is all logic in a controller, or is it shoved into views (or worse, item renderers)?</li> <li>Are style sheets used, or is style information explicitly coded?</li> </ol> <p> I also provide a reading and comprehension test, and ask them to critique the code I provide. Some code I provide is good, some bad. </p> <p> Many enticing resumes are simply not supported by good sample code. Would you hire a chef without tasting a sample of their food first? </p> <div style="text-align: center;"> <picture> <source srcset="/assets/images/chef.webp" type="image/webp"> <source srcset="/assets/images/chef.png" type="image/png"> <img src="/assets/images/chef.png" title="Happy chef offering a sample of her food" class="center halfsize liImg2 rounded shadow" alt="Happy chef offering a sample of her food" /> </picture> </div> Mobile and Desktop Technology Trends and Issues 2011-05-13T00:00:00-04:00 https://mslinn.github.io/blog/2011/05/13/mobile-and-desktop-technology-trends <p> As principal of Micronautics Research I develop or oversee development of desktop and mobile client/server applications. We use whatever technology is appropriate for a client project. </p> <p> I make a distinction between applications (apps) and web pages (HTML and HTML5). Over time this distinction will blur. This posting deals with current reality for apps that need to be developed and deployed using today’s technology. </p> <div style="text-align: center;"> <picture> <source srcset="/assets/images/HTML5.webp" type="image/webp"> <source srcset="/assets/images/HTML5.png" type="image/png"> <img src="/assets/images/HTML5.png" title="HTML5 logo" class="center liImg2 rounded shadow" style="height: auto; width: 300px; padding: 1em" alt="HTML5 logo" /> </picture> </div> <p> Mobile and desktop apps can be described in terms of usability, performance, development cost, and maintainability; the latter is an indicator of the time, cost and risk of modifying an existing application. Whereas desktop environments are quite stable, new mobile devices and operating system versions now appear daily. Mobile apps are therefore have a much more difficult time staying relevant than do desktop apps. Mobile app environments also vary a lot more than desktop app environments. </p> <p> There is no perfect one-size-fits-all technology for every app, and this is especially true for mobile apps. Some mobile apps must integrate tightly with the operating system; for mobile apps this currently means writing in Java for Android or Objective C for Apple iOS. The majority of mobile apps, however, must be deployed to any device that the user might happen to own; the time and cost required to target every possible device explodes unless a cross-platform technology is used. </p> <p> I’d like to share a lesson learned from migrating several desktop apps to mobile apps. To reuse the original logic and data structures from the desktop app in various mobile apps, the logic and data structures must be cleanly separated from presentation to the greatest extent possible. We have yet to see a project where rework was not required in this regard before we could begin the task of making mobile versions. </p> <p> Presentation logic for adaptable views is not much of an issue for desktop apps, but it is important when targeting devices whose resolutions and pixel densities vary greatly. Implementing the logic for adaptable views is not difficult, but designers today are generally unfamiliar with the issues of how to create <a href="https://mobile-patterns.com/" target="_blank" rel="nofollow">adaptable designs</a> for such a wide variety of platforms. We find that our interaction with designers has deepened when working on mobile projects. </p> <p> Processing power varies greatly between mobile devices. Tablets have quite a bit more power than phones, and desktops have yet more power. However, mobile devices have specialized hardware not present in most desktops, such as GPS and various forms of telephony communications; many Android devices also have a built-in bar code scanner. When adapting a desktop app to run on mobile apps, we strongly suggest using built-in hardware to the greatest extent possible. Mobile apps should obtain data from hardware or remote services instead of asking the user to type, click or swipe. </p> <p> We also enhance data structures as much as possible so less processing is required; although this is generally a good practice, today’s desktops are so powerful that careless programming often goes unnoticed. When an app is ported from a desktop environment to a mobile device, the reduced CPU and memory resources cause the app to run unacceptably slowly. </p> <p> For applications that must continue to work on desktops after they are ported to mobile devices, we recommend identifying and optimizing the application core such that it is present without changes for all environments. Sometimes clients are surprised by the results of this partitioning process because their assumptions of what is core do not align with actual commonalities between target environments. </p> Mounting compressed folders / Looping back zip files 2010-02-19T00:00:00-05:00 https://mslinn.github.io/blog/2010/02/19/mounting-compressed-folders-looping <p> Big file systems make computers run more slowly. This problem is most noticeable for laptops. On a project that I am currently working on, I am using multiple versions of three frameworks: The Flex SDK (source and ASDoc), the MicroStrategy Flex SDK and the Blaze DS source tree. The result is thousands of files that takes hours to copy and slows down disk access even when I’m not programming. </p> <h2 id="readonly">Read-only Files</h2> <p> Because these files are only read and never written, the solution is to not unzip them. Instead, the files should only be accessed from within the zip files that contains them. This capability is called a &ldquo;local loopback filesystem&rdquo; on Linux, and &ldquo;accessing compressed folders&rdquo; under Windows. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/loopback450x338.webp" type="image/webp"> <source srcset="/blog/images/loopback450x338.png" type="image/png"> <img src="/blog/images/loopback450x338.png" title="Hardware loop back (Ethernet)" class="center halfsize liImg2 rounded shadow" alt="Hardware loop back (Ethernet)" /> </picture> </div> <p> Results are dramatic. Usage is simple. What’s not to love? </p> <h2 id="windows">Windows</h2> <p> On Windows, I use the free <a href="https://www.pismotechnic.com/pfm/ap/" target="_blank" rel="nofollow">Pismo File Mount Audit Package</a>. It provides the ability for zip files to be mounted as virtual drives. </p> <h2 id="linux">Linux</h2> <p> On Linux, use <a href="https://github.com/refi64/fuse-zip" class="code" target="_blank" rel="nofollow">fuse-zip</a>. It provides the ability for zip files to be mounted on any directory mount point. </p> Shutting down Zamples.com 2009-11-23T00:00:00-05:00 https://mslinn.github.io/blog/2009/11/23/shutting-down-zamplescom <p> I&rsquo;ve decided that it is long past the time that zamples.com should be shut off. The site got steady traffic, but it generated no revenue and demanded a lot of my time. Now that the server needs to be replaced, it is time to say goodbye. </p> <p> I first created the predecessor to Zamples in 1994 for The Internet Factory, manufacturers of the first programmable web server. Eleven years later, it is abundantly clear that no-one really cares about live online code examples, at least not enough to spend money for the privilege of using the facility. </p> <p> <a href="https://www.w3schools.com/charsets/tryit.asp?deci=128521" target="_blank" rel="nofollow">w3schools</a> built their own client-side only version recently. </p> <p> It’s rather sad to turn the switch off, but I won’t miss spending any more time and money on Zamples. </p> <div style="text-align: center;"> <picture> <source srcset="/assets/images/zamples.webp" type="image/webp"> <source srcset="/assets/images/zamples.png" type="image/png"> <img src="/assets/images/zamples.png" title="Zamples.com logo" class="center liImg2 rounded shadow" alt="Zamples.com logo" /> </picture> </div> 2009 Open Source Community Leadership Summit 2009-07-20T00:00:00-04:00 https://mslinn.github.io/blog/2009/07/20/community-leadership-summit-09 <div style=""> <picture> <source srcset="/blog/images/community-leadership-summit.webp" type="image/webp"> <source srcset="/blog/images/community-leadership-summit.png" type="image/png"> <img src="/blog/images/community-leadership-summit.png" title="Open Source Community Leadership Summit" class=" fullsize liImg2 rounded shadow" alt="Open Source Community Leadership Summit" /> </picture> </div> <p> The first <a href='https://www.communityleadershipsummit.com' target='_blank' rel='nofollow'>Open Source Community Leadership Summit</a> happened this weekend, and I was lucky enough to be able to attend. Jono Bacon, Ubuntu Community Manager, and the other volunteers did a terrific job of putting this together. I had a chance to get to know and renew acquaintances with some fascinating people from Canonical, Cisco, Collab.net, the Linux Foundation, the Linux Fund, Novell, the Open Voting Consortium, O&rsquo;Reilly Media, Red Hat, the Software Freedom Law Center, Sun and two of the the &lsquo;old men&rsquo; of the open source world, Bruce Perens and Larry Rosen. </p> <div style="text-align: right;"> <a href="https://www.rosenlaw.com/rosen.htm" target="_blank" rel="nofollow"><picture> <source srcset="/assets/images/larryRosen.webp" type="image/webp"> <source srcset="/assets/images/larryRosen.png" type="image/png"> <img src="/assets/images/larryRosen.png" title="Larry Rosen" class="right quartersize liImg2 rounded shadow" alt="Larry Rosen" /> </picture></a> </div> <div style="text-align: left;"> <a href="https://perens.com" target="_blank" rel="nofollow"><picture> <source srcset="/assets/images/brucePerens.webp" type="image/webp"> <source srcset="/assets/images/brucePerens.png" type="image/png"> <img src="/assets/images/brucePerens.png" title="Bruce Perens" class="left quartersize liImg2 rounded shadow" alt="Bruce Perens" /> </picture></a> </div> Commonwealth Club Performance 2009-06-27T00:00:00-04:00 https://mslinn.github.io/blog/2009/06/27/commonwealth-club-performance <p> Cindie Steinmetz and I performed at the San Francisco Commonwealth Club two evenings ago. We had been invited to preview a jingle I wrote entitled &ldquo;I Love Chocolate&rdquo; for the &ldquo;Women and Chocolate&rdquo; event. I played the rest of the evening, while enjoying excellent wine and chocolate. </p> <div style=""> <picture> <source srcset="/blog/images/commonwealthChocolate_5_690x518.webp" type="image/webp"> <source srcset="/blog/images/commonwealthChocolate_5_690x518.png" type="image/png"> <img src="/blog/images/commonwealthChocolate_5_690x518.png" title="Mike Slinn and Cindie Steinmetz performing at the San Francisco Commonwealth Club" class=" liImg2 rounded shadow" style="width: 100%" alt="Mike Slinn and Cindie Steinmetz performing at the San Francisco Commonwealth Club" /> </picture> </div> Presenting EmpathyWorks at a Private Equity Roundtable 2009-02-22T00:00:00-05:00 https://mslinn.github.io/blog/2009/02/22/presenting-empathyworks-at-private <p> The private equity round table vSIG is focused on applying virtual worlds to solve pressing business issues. The group meets once a month on Sandhill Road in Palo Alto, CA. Members include software entrepreneurs, VCs, academia and researchers. </p> <p class="formalNotice"> 2020 Update: the vSIG has morphed into <a href="https://www.vcr-sv.com/" target="_blank" rel="nofollow">Venture Capital Roundtable Silicon Valley</a>. </p> <p> This will be part two of two. During the last meeting I introduced the topics of modeling relationships and behavior and presented background concepts. </p> <p>The main points I made were:</p> <ul> <li>Relationships are far more than just directory entries, they are the foundation of expression and provide context for interpreting behavior</li> <li>The ability to recognize another individual is necessary in order to form a relationship with them</li> <li>Personality is perceived, not an absolute or fixed quality</li> <li>Behavior is defined by responses and is contextually dependent</li> <li>Today’s artificial beings suffer from Alzheimer’s</li> <li>It is more useful to model an entire community than a single individual</li> <li>Personas represent an individual in various contexts</li> </ul> <p> I then introduced <a href="https://www.empathyworks.ai" target="_blank" rel="nofollow">EmpathyWorks</a>, a generalized relationship and behavior modeler that I have been working on for a few years. </p> <div style="text-align: center;"> <a href="https://www.empathyworks.ai" target="_blank" rel="nofollow"><picture> <source srcset="/assets/images/robotCircle400.webp" type="image/webp"> <source srcset="/assets/images/robotCircle400.png" type="image/png"> <img src="/assets/images/robotCircle400.png" title="The EmpathyWorks Mascot" class="center halfsize liImg2 rounded shadow" alt="The EmpathyWorks Mascot" /> </picture></a> </div> Good Code is Beautiful 2008-10-15T00:00:00-04:00 https://mslinn.github.io/blog/2008/10/15/good-code-is-beautiful <p> First, I'd like to point out a few practical reasons why orderliness is important. Can you spot the bug in the following Flex code? This custom component is not as wide as expected: </p> <pre data-lt-active='false'>&lt;mx:TextArea width="100" height="100%" contextMenu="{cm}" xmlns:components="components.*" creationComplete="init()" width="100%" xmlns:mx="https://www.adobe.com/2006/mxml"&gt; ... &lt;/mx:TextArea&gt;</pre> <p> The bug occurs because two width attributes were provided, one with the value 100 pixels and one with the value 100%. Jamming a long list of attributes into an MXML tag in no particular order is an invitation to chaos. No error message is generated if an attribute is specified twice. </p> <p> Long ago I adopted the convention of ordering all attributes in alphabetical order, one per line. I would write the above like this:</p> <pre data-lt-active='false'>&lt;mx:TextArea contextMenu="{cm}" creationComplete="init()" height="100%" width="100%" xmlns:components="components.*" xmlns:mx="https://www.adobe.com/2006/mxml"&gt; ... &lt;/mx:TextArea&gt;</pre> <p> Other people write certain attributes first, like id, width and height. Whatever convention you adopt, stick to it. </p> <p> Here are the <a href="https://web.archive.org/web/20111006191913/https://opensource.adobe.com/wiki/display/flexsdk/Coding+Conventions" target="_blank" rel="nofollow">official ActionScript coding conventions</a>, as published by Adobe back in 2011, prior to canceling the Flex product. I take some of it the way the characters of &ldquo;Pirates of the Caribbean&rdquo; consider the Pirate&rsquo;s Code: &ldquo;Argh, it be more like a set of guidelines.&rdquo; </p> <div style=""> <picture> <source srcset="/blog/images/pirateCodex_690x460.webp" type="image/webp"> <source srcset="/blog/images/pirateCodex_690x460.png" type="image/png"> <img src="/blog/images/pirateCodex_690x460.png" title="The Pirate's Codex, from the Pirates of the Caribbean movie" class=" fullsize liImg2 rounded shadow" alt="The Pirate's Codex, from the Pirates of the Caribbean movie" /> </picture> </div> <h3 id="compressed">ANSI/Unix vs. Vertically Compressed Styles</h3> <p>I realize that this topic might make me flame-bait, but my experience has shown that the ANSI/Unix style of vertically spacing parenthesis increases the maintenance costs of a program.</p> <p> Functions/methods should fit on a single screen; so should data structures. Because programs written in bygone days had to be editable on a green screen, 80 columns was the widest that a program could be written. In the &lsquo;bad old days&rsquo;, it was better to let a program run to more lines in order to avoid folding code. Here is an example: </p> <pre data-lt-active='false'>private function GridContextEventHandler(event:GridContextEvent):void { if (event.menuItem=='Edit') { EditGridItem(event.selectedIndex); } else if (event.menuItem=='Remove') { RemoveGridItem(event.selectedIndex); } }</pre> <p> A vertically compressed style is much more easily read. When the Python programming language was first introduced it was initially controversial in part because it used indentation to define control structure. Python programs do not need braces for scoping as a result. Why not write ActionScript in the same manner, so parenthesis are virtually redundant? The Sun/Java convention is a good example of this formatting convention. The above program fragment would be written like this using a vertically compressed style: </p> <pre data-lt-active='false'>private function GridContextEventHandler(event:GridContextEvent):void { if (event.menuItem=='Edit') { EditGridItem(event.selectedIndex); } else if (event.menuItem=='Remove') { RemoveGridItem(event.selectedIndex); } }</pre> <h2 id="enforcing">Enforcing Style</h2> <p> Coding style conventions can be automatically applied by installing the Flex Pretty Printer plugin for Eclipse / Flash Builder. If you like, you can importing <a href="https://www.slinnbooks.com/books/FlexFormatter.properties" target="_blank" rel="nofollow">my formatting rules</a>, which enforce the vertically compressed style above. </p> <h3 id="disregard">Complete Disregard for Style</h3> <p> Recently I examined a programming candidate&rsquo;s programming style. He didn&rsquo;t get the job. Here is how he wrote the above code: </p> <pre data-lt-active='false'>private function GridContextEventHandler(event:GridContextEvent):void{ if(event.menuItem=='Edit') EditGridItem(event.selectedIndex); else if(event.menuItem=='Remove') RemoveGridItem(event.selectedIndex); }</pre> <p> The rest of his code did not work very well. Seems like from the way he wrote it that he didn't really care. No surprise that someone else got the job. </p> Breathless Delirium 2008-09-24T00:00:00-04:00 https://mslinn.github.io/blog/2008/09/24/breathless-delerium <p> In his keynote presentation to the Agile 2008 conference in Toronto entitled &ldquo;<a href="https://medium.com/@MrAlanCooper/how-far-have-we-come-792a80625c94" target="_blank" rel="nofollow">The Wisdom of Experience</a>&rdquo;, Alan Cooper had a great sentence that described the &lsquo;first to market&rsquo; goal behind some products: </p> <p> <i>&ldquo;There is no large group of people out there waiting in a breathless delirium to purchase your lousy product sooner rather than later.&rdquo;</i> </p> <p> Sure, it is terrific to be first. But the product still has to be good. </p> <p> Alan also makes some good points about design, engineering and construction. He took the long way around, because he doesn't get into his main topic, interactive design, until slide 75. Requirements are discussed starting at slide 87. In the slide entitled "Requirements are not design", Alan says: </p> <ul> <li>Giving people what they say they desire does not result in success.</li> <li>Your customers are not the same as your users.</li> <li>Neither your customers nor your users know what they want or even what they do.</li> <li>What people tell you has little bearing on the truth.</li> <li>Good user experience is not dependent on features</li> <li>Radically different products can have identical features.</li> <li>A list of features is not the same as the design of behavior.</li> <li>Expertise in a subject does not correlate to expertise in designing software behavior.</li> </ul> <p> Philosophy starts on slide 92. I like philosophy, it is one of the foundations of architecture. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/geeklove.webp" type="image/webp"> <source srcset="/blog/images/geeklove.png" type="image/png"> <img src="/blog/images/geeklove.png" title="Geek love" class="center liImg2 rounded shadow" alt="Geek love" /> </picture> </div> Cult of the Software God 2008-04-28T00:00:00-04:00 https://mslinn.github.io/blog/2008/04/28/cult-of-software-god <p> I invented a &lsquo;religion&rsquo; in the mid-80s. Admittedly, it was done in jest, but I find that I make frequent reference to it and so it deserves a mention lest people I converse with find me just babbling. Have no fear, other whacked-out religions exist with high-profile devotees (you know some, I&rsquo;m sure) so I&rsquo;m in good company. </p> <p> The religion is called &ldquo;The Cult of the Software God&rdquo;, and this god oversees technical and financial success for software entrepreneurs and IT workers. Like many ancient religions of days gone by, the Software God requires sacrifices, which amounts to the destruction of software media. At one time he/she/it was content when I would hold a 5.25&rdquo; floppy disk over a garbage can and cut it up, while intoning praises to &ldquo;the great Software God&rdquo; and praying that the bits stored on the media be returned to their maker, but this is a God of the Geeks, and he/she/it has kept up with advances in storage media. </p> <p> When 3.5" diskettes became available, the Software God soon lost interest in clunky old floppy disks and demanded sacrifices of the smaller rigid diskettes instead. I, as High Priest of the Cult, interpreted the wishes of the Software God and used a Sacred Hammer stored on a bench near our diskette duplicating station to smash up diskettes that had write errors during the duplication process. &ldquo;Oh great Software God&rdquo;, I would say. &ldquo;We humbly beseech you to accept this offering and pray that you bring abundance and a measurable quality improvement to our next release.&rdquo; </p> <p> Time moved on, I moved to other ventures, and yet I never abandoned my role as High Priest of the Cult of the Software God. Occasionally I would snap CDs in half in the name of the Software God when a write error occurred, and moved on to DVDs. I even used a blowtorch on a RLL-encoded 5.25&rdquo; hard drive and ground my heel on a flash drive, while singing praises to the Software God. </p> <p> You might think I had too much time on my hands. Certainly I am at least eccentric. Yet there is more! </p> <p> As High Priest of the Cult of the Software God, I became aware that he/she/it is also fed by <a href="https://askubuntu.com/a/12110/58760" target="_blank"><code>/dev/null</code></a>. Where do you think bits piped to <code>/dev/null</code> go, anyway? Why, they nourish the Software God, of course! You can turn this otherwise wasted gift to the Software God into a sacrificial offering simply by making a prayer as the bits are piped. Of course, since the Software God is a god for the geeks and by the geeks, automation counts. That is why I wrote a program in the late 90s (since lost) that would print out any prayer you desired on the screen, over and over... rather like how prayer wheels work. </p> <p> Next time you really need your computer to not crash while running a semi-stable program, or have a serious virus infestation, remember the Cult of the Software God. He/she/it is virtually there for you. &#x1F609; </p> AAAI Symposium 2008-03-26T00:00:00-04:00 https://mslinn.github.io/blog/2008/03/26/aaai-symposium-for-emotion-personality <p>This week I am attending the <b>AAAI Symposium for Emotion, Personality and Social Behavior</b> at Stanford. The symposium provides a forum for an interdisciplinary discussion of modeling affect and personality in social behavior. Attendees include researchers studying social computing, virtual reality, game design, robotics, believable agents, affective computing, psychotherapy and the arts. Panel discussions discussed successful interactions between artificial beings and humans.</p> <p>Some of the discussion topics that I am interested in include:</p> <ul> <li>How can compelling artificial characters be designed?</li> <li>How to facilitate social interaction between humans, and / or artificial beings?</li> <li>What are the components in the software toolbox?</li> <li>What are the emerging standards in affective artificial characters, robots and systems?</li> <li>What architectural designs work best?</li> <li>What knowledge base designs work best?</li> </ul> <h2 id="scheutz">Believability of Robot Affect</h2> <p> Matthias Scheutz, Associate Professor of Informatics at Indiana University, presented a paper entitled &ldquo;<b>Empirical Investigations into the Believability of Robot Affect</b>&rdquo;. During the discussion afterwards he suggested that people are more critical of <a href="https://www.google.com/url?sa=t&amp;ct=res&amp;cd=2&amp;url=http%3A%2F%2Fwww.animatronics.org%2F&amp;ei=PLPrR_LFFJjgpgTfopx8&amp;usg=AFQjCNHUBENaXpnG15KYWrIuyDqpF-Eh_g&amp;sig2=kOGppoXBnh9Zuu8l3bcG4w" target="_blank" rel="nofollow">animatronic</a> (physical) entities, compared to characters that are merely imaged. Seems that the bar for believable behavior for robots is higher than for characters which are merely rendered on a computer screen because people don't expect a rendered character to be real anyway. </p> <h2 id="tapus">Socially Assistive Robots</h2> <p>Adriana Tapus&rsquo; presentation, &ldquo;<b>Socially Assistive Robots: The Link between Personality, Empathy, Physiological Signals and Task Performance</b>&rdquo; provided me two pieces of interesting common-sense information:</p> <ul> <li>A robot that challenges the user during therapy rather than offering praise will be preferred by extroverts;</li> <li>A robot that offers nurturing praise rather than challenging-based motivation by introverts.</li> </ul> <p> In the discussion that followed Adriana&rsquo;s talk, mention was made of (uncited) research which found that extroverts are more sensitive to robot personalities because they seek outside stimulus; introverts are self-stimulated. It seems logical that when a robot first encounters a person about which nothing is known regarding their personality, the robot should display mildly extroverted behavior, and then adapt according to social cues. </p> <h2 id="olsen">Emotions for Strategic Real-time Systems</h2> <p><a href="http://www.cs.loyola.edu/~olsen/" target="_blank" rel="nofollow">Megan Olsen</a> discussed how adding an emotion framework to game engines improved performance in her talk entitled &ldquo;<b>Emotions for Strategic Real-time Systems</b>&rdquo;. The emotional framework models fear and frustration in the form of emotional maps, which are overlaid on a physical map of terrain. Emotions diffuse out and are picked up by co-operating agents nearby. Emotions have an intensity value which linearly decays over time. She showed an interesting video which illustrated the aggregated emotions of many agents over time. She also showed that emotions can cause groups of synthetic individuals to be more successful in combat. Different game engines responded differently to each emotion; some responded better to the addition of the ability to model fear, which others responded better to the addition of frustration, and <a href="https://globulation2.org/wiki/AINicowar" target="_blank" rel="nofollow">NicoWar</a> was able to benefit from the addition of both emotions. </p> <h2 id="hudlicka">Emotion Modeling 101</h2> <p> <a href="https://works.bepress.com/eva_hudlicka/" target="_blank" rel="nofollow">Dr. Eva Hudlicka</a> of Psychometric Associates spoke on &ldquo;<b>Emotion Modeling 101</b>&rdquo;. Emotion modeling incorporates emotion expression, recognition, generation, and the effect on agent behavior. One could model feelings, moods, emotions, affective states and personality traits; some people mean all of the above when discussing modeling emotion. As well, attitudes and preferences can be modeled. Interpersonal roles include: social coordination, rapid communication of intent; intrapsychic roles include: motivation, homeostatis, and adaptive behavior. Emotions are manifested across multiple interacting modalities: somatic / physiological, cognitive / interpretive, behavioral / motivational and experiential / subjective. Unfortunately, the literature on emotional models is inconsistent and terms are unclear. Eva suggested we view emotion models in terms of emotion generation and effects; that emotion modeling building blocks be identified. Emotion is generated from stimuli and appraisal by the individual; Eva briefly discussed both topics and the need to standardize the process of mapping stimulus to emotion. </p> <h2 id="camurri">Coffee With Dr. Antonio Camurri</h2> <p>During a coffee break, I chatted with <a href="http://www.infomus.org/people/person.php?name=acamurri" target="_blank" rel="nofollow">Dr. Antonio Camurri</a>, Associate Professor at the University of Genoa, Italy. Seems we both share a passion for boats. Dr. Camurri mentioned EyesWeb, an open software platform project that he leads, that enables the development of real-time multimodal distributed interactive applications. </p> <h2 id="wrapup">Wrapup</h2> <p>In the wrap-up session, I drew attention to one of the issues mentioned in the preface to the Technical Report that the Symposium was to intended to address: &ldquo;What are the emerging standards in affective artificial characters, robots and systems?&rdquo; One member of the audience (Rosamaria Barone?) mentioned some standards work for characters (unfortunately, I didn't write it down), then an awkward silence followed. I believe that standards will be key to artificial personality and emotion moving from the lab to mainstream society. </p> IKVM.NET – Java applications on .NET 2008-01-16T00:00:00-05:00 https://mslinn.github.io/blog/2008/01/16/ikvmnet-java-applications-on-net <p> I am building the first implementation of <a href="https://www.empathyworks.ai" target="_blank" rel="nofollow">EmpathyWorks&trade;</a> as a Java library packaged as a JAR. After looking around for a while to understand how EmpathyWorks might run on .NET, I found <a href='https://www.ikvm.net/' target="_blank" rel="nofollow">IKVM.NET</a>. </p> <h2 id="ikvm">IKVM Works Well</h2> <p> IKVM is an open source project that converts Java JARs into a .NET assembly. I was quite skeptical, but after downloading IKVM I was able to convert EmpathyWorks into an EXE that ran on Windows XP in about ten minutes. </p> <p> IKVM also has a mechanism for converting .NET stubs into JARs, important when developing .NET code in an IDE so that Intellisense tooltips appear for code completion. I am next going to try converting the Microsoft Robotics Studio runtime to a JAR so I can try writing a sample MSR application that co-operates with EmpathyWorks. </p> <p> There is a brief note in the IKVM documentation that says debugging from Java in this circumstance does not work. Hmm, what about the other way... can one debug the .NET application from Visual Studio, and reach right into the converted Java code? I suppose no source code would be available. </p> <hr /> <h2 id="update1">Update Jan 18/08</h2> <p>I built a DLL from Java using IKVM's <code>-target:library</code> option. Visual C# 2008's object browser happily showed all the public objects and methods in the DLL. Once I discovered the &ldquo;Add to References&rdquo; button in the object browser my test C# program was able to call into the DLL. Most cool!</p> <hr /> <h2 id="update2">The End of IKVM.NET </h2> <p> Jeroen Frijters, the principal author of IKVM.NET <a href="https://weblog.ikvm.net/CommentView.aspx?guid=33ea525f-a291-418a-bd6a-abdf22d0662b" target="_blank" rel="nofollow">posted the following</a> Apr 21/17. </p> <div class="shadow rounded liImg" style="padding: 1em;"> After almost fifteen years I have decided to quit working on IKVM.NET. The decision has been a long time coming. Those of you that saw yesterday’s Twitter spat, please don’t assume that was the cause. It rather shared an underlying cause. I’ve slowly been losing faith in .NET. Looking back, I guess this process started with the release of .NET 3.5. On the Java side things don’t look much better. The Java 9 module system reminds me too much of the generics erasure debacle. <br /><br /> I hope someone will fork IKVM.NET and continue working on it. Although, I’d appreciate it if they’d pick another name. I’ve gotten so much criticism for the name over the years, that I’d like to hang on to it 😊 <br /><br /> I’d like to thank the following people for helping me make this journey or making the journey so much fun: Brian Goetz, Chris Brumme, Chris Laffra, Dawid Weiss, Erik Meijer, Jb Evain, John Rose, Mads Torgersen, Mark Reinhold, Volker Berlin, Wayne Kovsky, The GNU Classpath Community, The Mono Community. <br /><br /> And I want to especially thank my friend Miguel de Icaza for his guidance, support, inspiration and tireless efforts to promote IKVM. <br /><br /> Thank you all and goodbye. </div> I'm a Dragon on a Business Plan Panel 2007-11-21T00:00:00-05:00 https://mslinn.github.io/blog/2007/11/21/im-dragon-on-business-plan-panel <p> A few months ago the Canadian Consulate in Silicon Valley asked me if I would be interested in working with Canadian entrepreneurs. I was happy to oblige. Since then the Canadian National Research Council has been using me to provide guidance to young entrepreneurs. </p> <p> I was asked by <a href="https://www.linkedin.com/in/eanjackson/" target="_blank" rel="nofollow">Ean Jackson</a>, an instructor at <a href="https://www.sfu.ca/" target="_blank" rel="nofollow">Simon Fraser University</a> in Burnaby, BC to sit on a <a href="https://bus477.com/class-videos/" target="_blank" rel="nofollow">&ldquo;Dragon Panel&rdquo;</a> to evaluate student projects. The 4th year undergraduate business students enrolled in the New Ventures Planning Program share the course with 4th year Engineering students at SFU. </p> <p> One of the &ldquo;Suits&rdquo; is teamed up with three &ldquo;Geeks&rdquo; enrolled in the Engineering program. The Geeks come up with a technology idea and the Suits and Geeks work together to create a business plan. That business is entered into the Ken Spencer Business plan competition at SFU where cash prizes are offered. </p> <p> Today (November 21, 2007) is the final class. The 14 Suits will pitch their business ideas to a panel of &ldquo;investors&rdquo;. I am acting in the role of a prospective investor; each of the &ldquo;investors&rdquo; will grade the pitch. Sounds like fun! I'm looking forward to hearing the presentations this afternoon. </p> Digital Mentat 2007-09-23T00:00:00-04:00 https://mslinn.github.io/blog/2007/09/23/digital-mentat <div class="shadow rounded liImg"> <p class="notice">It is by caffeine alone I set my mind in motion.<p> <p class="notice">It is by the Beans of Java that thoughts acquire speed, the hands acquire shaking, the shaking becomes a warning.<p> <p class="notice">It is by caffeine alone I set my mind in motion.<p> </div> <p style="margin-left: 1em;">&mdash; Stolen from source unknown, in the SDM<br /> &mdash; After the Mentat Mantra of Dune by Brian Herbert</p> <p> It's just possible that I might have been spending too much time programming <a href='https://www.empathyWorks.ai' target="_blank" rel="nofollow">EmpathyWorks</a> lately! </p> Singularity Summit 2007-09-09T00:00:00-04:00 https://mslinn.github.io/blog/2007/09/09/singularity-summit <p> I'm at the Singularity Summit today, after spending two months on my sailboat in B.C. Quite a change in venue! </p> <p> As you can tell by reading this blog, I'm quite interested in AI. The theme of 'friendly intelligence' strikes a chord in me. I'm currently working on an artificial personality simulator (<a href='https://www.empathyworks.ai/' rel="nofollow" target="_blank">EmpathyWorks</a>), a project that I've been noodling on for over thirty years. Hopefully I'll get a chance to chat with various people here at the conference about integrating artificial personality into a whole artificial organism. </p> 2nd Annual Silicon Valley Ruby Conference 2007-04-25T00:00:00-04:00 https://mslinn.github.io/blog/2007/04/25/2nd-annual-silicon-valley-ruby <p> Whew, it is a wrap! The Second Annual Silicon Valley Ruby Conference is done, and I'm just about finished doing post-production on the videos of the talks. I'll hand a DVD to SDForum to make arrangements for attendees of the event to view the videos. </p> <p> Slides for most of the presentations are available at <a href="https://www.slideshare.net" target="_blank" rel="nofollow">slideshare.net</a>. </p> <p> 2007 seems to be shaping up as the Year of the Ruby/RoR IDE. Later this year commercial-quality IDEs from Sun and CodeGear will join Aptana's RadRails IDE. All three of these software publishers presented at this year's event. </p> <p> I ran a quick survey by a show of hands, and asked 24 questions. Here are a few results: </p> <table border="1" cellpadding="5" cellspacing="0" class="liImg"><tbody> <tr> <td style="text-align: left; padding: 5px;">Which of you use RoR to make web sites but don't care about Ruby?</td> <td align="right" style="padding: 5px;">25%</td> </tr> <tr> <td style="text-align: left; padding: 5px;">Which of you use Dreamweaver, Illustrator or Photoshop?</td> <td align="right" style="padding: 5px;">17%</td> </tr> <tr> <td style="text-align: left; padding: 5px;">Which of you use a Mac?</td> <td align="right" style="padding: 5px;">45%</td> </tr> <tr> <td style="text-align: left; padding: 5px;">Which of you use Linux?</td> <td align="right" style="padding: 5px;">35%</td> </tr> <tr> <td style="text-align: left; padding: 5px;">Which of you use Windows?</td> <td align="right" style="padding: 5px;">50%</td> </tr> </tbody></table> <p> It's been fun leading the organization of these first two years of this event, but this year will be my last. </p> Announcing Micronautics Research 2006-10-17T00:00:00-04:00 https://mslinn.github.io/blog/2006/10/17/micronautics-research-is-now-officially <div style="text-align: left;"> <a href="https://www.EmpathyWorks.ai" target="_blank" rel="nofollow"><picture> <source srcset="/blog/images/robotCircle.webp" type="image/webp"> <source srcset="/blog/images/robotCircle.png" type="image/png"> <img src="/blog/images/robotCircle.png" title=" Cool robot mascot, huh? I'm going to use it for EmpathyWorks&trade;, which at this point is still just a twinkle in my eye. " class="left liImg" alt=" Cool robot mascot, huh? I'm going to use it for EmpathyWorks&trade;, which at this point is still just a twinkle in my eye. " /> </picture></a> <figcaption class="" style="width: 100%; text-align: center;"> <a href="https://www.EmpathyWorks.ai" target="_blank" rel="nofollow"> Cool robot mascot, huh? I'm going to use it for EmpathyWorks&trade;, which at this point is still just a twinkle in my eye. </a> </figcaption> </figure> </div> A physical metaphor for IT innovation 2006-06-23T00:00:00-04:00 https://mslinn.github.io/blog/2006/06/23/a-physical-metaphor-for-it-innovation <p>I got an email today from a friend, an innovative engineer:</p > <div class='liImg rounded shadow' style="padding: 1em"> <p>Hi Mike, <p>The other day, I had a reasonably good idea for a startup. And I ran it by a friend, who said "Why would the enterprise buy that?"</p> <p>I replied "Much better than existing solutions."</p> <p>He said "Unless it's a 10x, nope. And, while it is better, it's not a 10x." </p> <p>I thought "That's gotta be wrong."</p> <p>But then it occurred to me. In spite of having worked as a consultant to enterprises, and in spite of having run professional services for an enterprise software company, I still don't have a very good model of enterprise software adoption. To wit:</p> <ul> <li>What makes a new piece of software or IT infrastructure compelling for the enterprise?</li> <li>When do enterprises decide to upgrade or change infrastructure?</li> </ul> <p>Do you know of any references or models?</p> </div> <p>Here is what I told him:</p> <p>A 10x return on investment applies to investors (although 7x works too). If the sales pitch is predicated on cost reduction (optimizing operations), then 5-10x would apply and is most interesting when the client's business is relatively mature.</p> <p>Increasing client sales (optimizing business development) needs to be presented along the client's strategic direction, which might be a moving target. 2x or 3x return would suffice if the timing between a vendor's presentation and a client's view of their strategic direction aligned. This would be difficult for the vendor to guess without incredible intelligence on specific targets.</p> <p>CIOs are risk averse, so solutions that do not require vetting new servers and client-side software are easier. When selling into a datacenter, a 10x return might not be enough if security issues are not mitigated. On the other hand, repurposing existing investments are much easier to sell: a 2x or 3x return might be sufficient.</p> <p>In other words: inertia is a dominant force of nature that also applies in IT. If you consider trends to be acceleration (positive or negative), then the net force driving the inertia is composed of external and internal factors. Those factors might be political, economic, technical, etc. Newton's <code>F=ma</code> can be restated in entrepreneurial terms as:</p> <div style="padding-left: 3em;"> net force = investment times rate of adoption</div> <p>or, more to the point:</p> <div style="padding-left: 3em;"> rate of adoption = net force / investment</div> <p>The investment (mass) a client has made in developing and deploying their intellectual property; training personnel; capital investment in infrastructure; and intimate knowledge of the current system. ("The devil you know".)</p> <p>For entrepreneurs that want to convince clients that a new technology that is worth adopting, continuing our physics metaphor may be illuminating. Let's equate profitability with speed, and technology platform with direction. Thus velocity is the cost-effectiveness of a given technical platform. It follows that in order to change a company's direction, and increase their profitability (speed), the client will need to experience a strong enough net force over a long enough period of time to overcome inertia and justify the risk and the investment. In addtion, the prospect will only understand new information in their current context, which might not use the same dictionary that an entrepreneur uses, since they tend to live in the future.</p> Success! Silicon Valley Ruby Conference 2006-04-24T00:00:00-04:00 https://mslinn.github.io/blog/2006/04/24/success-silicon-valley-ruby-conference <p> After months of planning, the big weekend arrived and a great time was had by all. Thank you, presenters! </p> <p>I organized the event and chaired the second day of the conference.</p> <div style=""> <picture> <source srcset="/blog/images/mikeSlinnRubyConf_690x524.webp" type="image/webp"> <source srcset="/blog/images/mikeSlinnRubyConf_690x524.png" type="image/png"> <img src="/blog/images/mikeSlinnRubyConf_690x524.png" title="Yours truly, addressing the 2006 Silicon Valley Ruby Conference" class=" liImg2 rounded shadow" alt="Yours truly, addressing the 2006 Silicon Valley Ruby Conference" /> </picture> <figcaption class="" style="width: 100%; text-align: center;"> Yours truly, addressing the 2006 Silicon Valley Ruby Conference </figcaption> </figure> </div> <p>See you next year...</p> Silicon Valley Ruby Conference 2006-02-21T00:00:00-05:00 https://mslinn.github.io/blog/2006/02/21/silicon-valley-ruby-conference <p> SDForum just announced the <a href="https://archive.upcoming.org/event/silicon-valley-ruby-conference-59161" target="_blank" rel="nofollow">Silicon Valley Ruby Conference</a>. I am one of the two co-chairmen. </p> <p> We expect the event to sell out very quickly. Some great speakers have been lined up, it's quite inexpensive and only 210 people can attend. </p> <p>See you there!</p> <div style=""> <picture> <source srcset="/blog/images/sdforum_ruby_690x565.webp" type="image/webp"> <source srcset="/blog/images/sdforum_ruby_690x565.png" type="image/png"> <img src="/blog/images/sdforum_ruby_690x565.png" class=" liImg2 rounded shadow" /> </picture> </div> <p> Update: Andrew Burke wrote up some good notes for <a href="http://www.andrewburke.me/blogposts/15" target="_blank" rel="nofollow">day 1</a> and <a href="http://www.andrewburke.me/blogposts/16" target="_blank" rel="nofollow">day 2</a> of the conference. </p> Instantiating Java Inner Classes 2005-12-30T00:00:00-05:00 https://mslinn.github.io/blog/2005/12/30/instantiating-java-inner-classes <p> I've been writing Java code for years. Today I learned something new. I have been adding a new feature to Zamples that was best expressed as a doubly nested inner class. The hierarchy looks like this: </p> <pre data-lt-active='false'>public class CodeRange { public CodeRange(String str) { /* ... */ } public class RangeSpec { public RangeSpec(int i, int j) { /* ... */ } public RangeSpec(RangeItem start, RangeItem end) { /* ... */ } public class RangeItem { public RangeItem(String str, int i) { /* ... */ } } } }</pre> <p>When it came time to write JUnit tests, I needed to instantiate the hierarchy. The syntax surprised me:</p> <pre data-lt-active='false'>CodeRange.RangeSpec.RangeItem range = new CodeRange("test").new RangeSpec(1, 1).new RangeItem("middle", 3);</pre> <p>Not exactly intuitive, eh? It gets even more interesting when trying to write unit tests for the second RangeSpec constructor, the one that accepts two <code>RangeItems</code>. I found I had to create a protected no-args constructor for <code>RangeSpec</code> with an empty body, plus a method within <code>RangeSpec</code> to create <code>RangeItems</code> on demand:</p> <pre data-lt-active='false'>/** For JUnit only */ protected RangeSpec() {} /** For JUnit only */ protected RangeItem newRangeItem(String searchString, int offset) { return new RangeItem(searchString, offset); }</pre> <p>Now I could write my test case setup code:</p> <pre data-lt-active='false'>codeRange = new CodeRange(code, ""); CodeRange.RangeSpec.RangeItem rangeItemStart = codeRange.new RangeSpec().newRangeItem("middle", 3); CodeRange.RangeSpec.RangeItem rangeItemEnd = codeRange.new RangeSpec().newRangeItem("back", 0); CodeRange.RangeSpec rangeSpec = codeRange.new RangeSpec(rangeItemStart, rangeItemEnd); /* ... */</pre> <p> Everything I read about doubly nested classes amounted to a warning to the effect that they should be avoided. I never had occasion to need this type of solution before, however the particular problem I am solving yields very nicely to this approach. It is simple, elegant and efficient. Don't believe everything you read (except this blog!) &#128521;. </p> Zamplized Ruby User’s Guide 2005-12-16T00:00:00-05:00 https://mslinn.github.io/blog/2005/12/16/zamplized-ruby-users-guide <p> Santa's elves have been cheering us on at Zamples! Just in time for the holidays, Zamples v2.6 is now live. </p> <p> We've &ldquo;Zamplized&rdquo; the Ruby User&rsquo;s Guide and made it available for free. Now you can curl up with a glass of eggnog and sit by the fire with your laptop, while learning Ruby, the hottest new programming language today. </p> <p> Zamples&rsquo; live code examples let you &ldquo;learn by doing&rdquo; and give you a hands-on and foolproof experience. There is no faster, easier way to gain experience with a language or programming interface than by interacting with a Zamplized document. </p> <p> Best of the season! </p> <div style="text-align: center;"> <picture> <source srcset="/assets/images/zamples.webp" type="image/webp"> <source srcset="/assets/images/zamples.png" type="image/png"> <img src="/assets/images/zamples.png" title="Zamples logo" class="center liImg2 rounded shadow" alt="Zamples logo" /> </picture> </div> Programmatic IM Generation 2005-12-16T00:00:00-05:00 https://mslinn.github.io/blog/2005/12/16/programmatic-im-generation <p> I thought it would be fun to create a live code example using Zamples that would generate an IM (instant message). </p> <p> The following Java code example uses Jive Software&rsquo;s Smack library to send a message to a Jabber user with a Gaim client. You will need to put real values in for <tt>from</tt>, <tt>password</tt> and <tt>to</tt> before a message will be sent. BTW, if the userid and password are wrong, you&rsquo;ll get an odd error message. Since the program fragment automatically compiles and runs after you click on the <b>Try It!</b> button, the error message will appear on the right. </p> <p> Once you provide the <tt>to</tt> and <tt>from</tt> user ids and <tt>password</tt> and rerun the code fragment, a message will be sent. I tested this by sending myself a message and found that <tt>from</tt> and <tt>to</tt> were the same. </p> <table> <tbody> <tr> <td> <form action="" class="zform" method="post" target="TomcatMain"> <input name="cmd" type="hidden" value="XMPPConnection connection = new XMPPConnection(&quot;jabber.org&quot;); connection.login(&quot;from&quot;, &quot;password&quot;); connection.createChat(&quot;to@jabber.org/Gaim&quot;).sendMessage(&quot;Howdy!&quot;);"> <input name="imports" type="hidden" value="import org.jivesoftware.smack.*;"> <input name="applet" type="hidden" value=""> <input name="format" type="hidden" value="2"> <input name="pre" type="hidden" value="false"> <input name="/stylesheet" type="hidden" value="StyleSheet.css"> <input class="zform" type="submit" value="Try It!"></form> </td> <td> <pre style="text-align: left; padding: 0.5em;">XMPPConnection connection = new XMPPConnection("jabber.org"); connection.login("from", "password"); connection.createChat("to@jabber.org/Gaim").sendMessage("Howdy!");</pre> </td> </tr> </tbody> </table> <br /> <div style="text-align: center;"> <picture> <source srcset="/assets/images/zamples.webp" type="image/webp"> <source srcset="/assets/images/zamples.png" type="image/png"> <img src="/assets/images/zamples.png" title="Zamples logo" class="center liImg2 rounded shadow" alt="Zamples logo" /> </picture> </div> Zamples REST Interface 2005-12-15T00:00:00-05:00 https://mslinn.github.io/blog/2005/12/15/zamples-rest-interface <p> Wouldn&rsquo;t it be great to be able to build in a facility for live code examples into your favorite software tools? We thought so, and are proud to announce the Zamples REST Interface. Now you can use any programming language you choose to embed Zamples&rsquo; functionality as an IDE plugin or online learning environment. </p> <div style="text-align: center;"> <picture> <source srcset="/assets/images/zamples.webp" type="image/webp"> <source srcset="/assets/images/zamples.png" type="image/png"> <img src="/assets/images/zamples.png" title="Zamples logo" class="center liImg2 rounded shadow" alt="Zamples logo" /> </picture> </div> I Want to Build a Girl 2005-07-28T00:00:00-04:00 https://mslinn.github.io/blog/2005/07/28/i-want-to-build-girl <div style="text-align: center;"> <picture> <source srcset="/blog/images/cyborg-woman_450x246.webp" type="image/webp"> <source srcset="/blog/images/cyborg-woman_450x246.png" type="image/png"> <img src="/blog/images/cyborg-woman_450x246.png" title="Cyborg woman" class="center halfsize liImg2 rounded shadow" alt="Cyborg woman" /> </picture> </div> <p> I said that as an undergraduate engineering student in 1975. Now a Japanese researcher has made <a href="https://news.bbc.co.uk/1/hi/sci/tech/4714135.stm" target="_blank" rel="nofollow">real progress</a> towards a synthetic companion. The artificial skin doesn't seem to have a high density sensory grid that other researchers are experimenting with (<a href="https://news.bbc.co.uk/1/hi/sci/tech/4154366.stm" target="_blank" rel="nofollow">future artificial skins</a> might incorporate sensors not only for pressure and temperature, but also for light, humidity, strain or sound), but I"m sure it won't be long before a similar android with tactile response and <a href="http://www.ai.mit.edu/projects/sociable/videos.html" target="_blank" rel="nofollow">expression ability</a> graduates from the lab. </p> <p> I still want to be involved in bringing that girl... and a boy... and a whole range of socially acceptable companions to market. Some day soon... </p> <p> Update: <a href="https://en.wikipedia.org/wiki/QRIO" target="_blank" rel="nofollow">Sony&rsquo;s Qrio</a>, a humanoid robot under development is exactly along the lines I&rsquo;ve been dreaming of! </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/sonyQrioRobot_225x300.webp" type="image/webp"> <source srcset="/blog/images/sonyQrioRobot_225x300.png" type="image/png"> <img src="/blog/images/sonyQrioRobot_225x300.png" title="Sony Qrio robot" class="center quartersize liImg2 rounded shadow" alt="Sony Qrio robot" /> </picture> </div> <p> Update: from PC Magazine, &ldquo;<a href="https://www.wearable.technology/" target="_blank" rel="nofollow">Eleksen</a>, a small UK-based firm is introducing electronic fabric, essentially carbon-embedded nylon sandwiched between layers of nylon mesh that, when a milliamps charge is passed through it, can recognize touch, pressure and even the direction and path of a stroke. This thin, flexible, and washable fabric connects to a small 8-bit processor, which then can be connected to a standard electronic device like an iPod. Eleksen company executives said the <a href="https://www.sciencedirect.com/topics/engineering/smart-fabric" target="_blank" rel="nofollow">washable fabric</a> can also withstand extreme pressure; they&rsquo;ve rolled a car over it without any ill effects.&rdquo; Seems we are moving towards artificial skin! </p> <p> Update: from The Economist, <a href="https://economist.com/world/asia/displaystory.cfm?story_id=5323427&amp;no_na_tran=1" target="_blank" rel="nofollow">Japan&rsquo;s humanoid robots &ndash; Better than people</a>:<br /> &ldquo;What seems to set Japan apart from other countries is that few Japanese are all that worried about the effects that hordes of robots might have on its citizens. Nobody seems prepared to ask awkward questions about how it might turn out. If this bold social experiment produces lots of isolated people, there will of course be an outlet for their loneliness: they can confide in their robot pets and partners. Only in Japan could this be thought less risky than having a compassionate Filipina drop by for a chat.&rdquo; </p> Self-Fullfilling Prophecy 2005-05-15T00:00:00-04:00 https://mslinn.github.io/blog/2005/05/15/self-fullfilling-prophecy <h2 id="heisenberg">Heisenberg&rsquo;s Uncertainty Principle</h2> <p>The Greek philosopher Democritus believed that the natural world was immutable. In 1927 <a href="https://www.aip.org/history/heisenberg/p01.htm" target="_blank" rel="nofollow">Werner Heisenberg</a> published his &ldquo;Uncertainty Principle&rdquo; which implied, amongst other things, that observing a quantum phenomenon also significantly altered the phenomenon. For many people, such an idea was counter-intuitive. </p> <p> What is true in the physical realm is even more true in the realms of interpersonal relationships and business. Our beliefs and belief systems affect outcomes more than we perhaps realize. Consider the diagram below. </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/prophecy1.webp" type="image/webp"> <source srcset="/blog/images/prophecy1.png" type="image/png"> <img src="/blog/images/prophecy1.png" title="First prophecy" class="center halfsize liImg2 rounded shadow" alt="First prophecy" /> </picture> </div> <h2 id="trends">Describing the Trend</h2> <p> Without knowing what the diagram represents, can you postulate the trend for the next cycle or two? The answer is, of course, no. You need to have an internal model of the behavior of the system and the entity&rsquo;s behavior within that system represented in the graph before you can make a prediction. Without any additional information, you might anticipate the trend like this: </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/prophecy2.webp" type="image/webp"> <source srcset="/blog/images/prophecy2.png" type="image/png"> <img src="/blog/images/prophecy2.png" title="Second prophecy" class="center halfsize liImg2 rounded shadow" alt="Second prophecy" /> </picture> </div> <p> Your model is based on incomplete information, and is necessarily a simplification of the actual system. If you are told that the graph depicts rainfall, and if you believe in a rain god, then your natural reaction might be to find a virgin to sacrifice. If you instead had the belief that rainfall had diminished due to clear-cutting of a vast forest which no longer acted as a natural storage mechanism for rainwater, your reaction might be to start lobbying for an intensive tree planting effort. Both belief systems have at their core a central belief that action can affect the outcome to some degree, so your prediction as to how much rain will fall in subsequent years will rely on the vigor with which countermeasures are applied, and your belief in their efficacy. Probably your belief would be that rainfall levels will tend to vary around historical norms, like this: </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/prophecy3.webp" type="image/webp"> <source srcset="/blog/images/prophecy3.png" type="image/png"> <img src="/blog/images/prophecy3.png" title="Third prophecy" class="center halfsize liImg2 rounded shadow" alt="Third prophecy" /> </picture> </div> <p> What if you were told instead that the first graph represented sales of a hybrid car model, and that &lsquo;experts&rsquo; believed that increasing SUV sales and the economy were believed to be responsible for declining sales? A marketing campaign that addressed the rising cost of gas and promoted decreasing the reliance of the US economy on foreign oil as being truly patriotic might gain traction. One might attribute the initial sales spike as being due to purchases by early adopters. The tapering off of sales might be the result of tax incentives that artificially promoted the sales of vehicles over 6,000 pounds. </p> <p> It is clear that China&rsquo;s consumption of gas will soon outpace even the US&rsquo;s rate of consumption, so gas prices will continue to rise, and remain at higher prices than any we have experienced to date. The US Department of Energy produced the following <a href="https://www.eia.doe.gov/emeu/cabs/china.html" target="_blank" rel="nofollow">projection</a> of Chinese gas consumption in June 2004. </p> <p> In 2003, BP showed <a href="http://www.gasandoil.com/goc/features/fex32825.htm" target="_blank" rel="nofollow">China's new influence on global oil supply</a>: &ldquo;Global oil demand, meanwhile, was broadly flat, increasing 290,000 bpd to 75.7 mm bpd from 75.5 mm bpd. All of the increase is attributable to China where oil consumption increased 5.8 % or 332,000 bpd.&rdquo; As demand for a commodity increases, prices increase. As China continues to dramatically increase its demand for oil, prices can only continue higher. In a <a href="http://www.gasandoil.com/goc/features/fex51961.htm" target="_blank" rel="nofollow">feature article</a> on <a href="http://www.gasandoil.com/goc/features/" target="_blank" rel="nofollow">Alexander&rsquo;s Gas &amp; Oil Connections</a>, John Vidal writes &ldquo;China's oil consumption, which accounted for a third of extra global demand last year, grew 17% and is expected to double over 15 years to more than 10 mm bpd &ndash; half the US's present demand. India's consumption is expected to rise by nearly 30% in the next five years.&rdquo; Jim Meyer, an expert at London's Oil Depletion Analysis Centre was quoted in <a href="http://www.gasandoil.com/goc/features/fex51962.htm" target="_blank" rel="nofollow">an article last month</a> as saying &ldquo;After 2007, we can&rsquo;t see enough new supplies to meet almost any reasonable level of demand growth.&rdquo; Expect gas prices to permanently exceed $50/barrel, and $100+/barrel will be reached with startling speed. With that in mind, a long term sales trend for hybrid cars in the US might be projected to look like this: </p> <div style="text-align: center;"> <picture> <source srcset="/blog/images/prophecy4.webp" type="image/webp"> <source srcset="/blog/images/prophecy4.png" type="image/png"> <img src="/blog/images/prophecy4.png" title="Forth prophecy" class="center halfsize liImg2 rounded shadow" alt="Forth prophecy" /> </picture> </div> <p> This projection assumes that historical sales figures are not a significant factor when gauging future sales of this hybrid car model, and that factors on the market that the product sells into have not yet been fully felt or internalized by consumers. </p> <h2 id="belief">Self-Fullfilling Prophecy</h2> <p> What if a hard-nosed automotive executive, looking at ROI on a quarter-by-quarter basis decided that the return was not immediate enough, and terminated the hybrid car based on sales history to date? That automotive manufacturer would soon miss the world&rsquo;s largest market, China, as the demand for fuel-efficient cars will soon be supported by a Chinese middle class newly able to afford more expensive hybrid cars. Unless the manufacturer was somehow able to introduce another technology, such as hydrogen powered vehicles, and successfully put together a national infrastructure network to support hydrogen fuel stations, they will have traded off short-term profitability for long-term viability. </p> <p> Returning to the original idea behind this blog posting, one&rsquo;s expectations and beliefs do color what one might think to be reasonable. If you believe that most people are kind and generous, your behavior will probably reinforce this belief &ndash; with the result that even when you meet Scrooge incarnate, he might crack a smile despite himself. If you think that sales of product XYZ are going down, down, down and that nothing can be done about it then that product is as good as dead. If instead, you come up with an alternate world view, communicate higher expectations and provide sufficient resources over the appropriate time period, that product&rsquo;s best years may be yet to come. </p> Marshall Brain is a very smart man 2005-04-03T00:00:00-05:00 https://mslinn.github.io/blog/2005/04/03/marshall-brain-is-very-smart-man <p> He founded <a href='https://howstuffworks.com/' target="_blank" rel="nofollow">How Stuff Works</a>, and wrote many interesting and informative articles for the site. </p> <p> We share an interest in personal robotics. Like many US citizens, Mr. Brain's focus in Robotic Nation is on the potentially negative aspects of a integrating robots into society, such as mass unemployment. His writing reminds me of the endless fear-mongering that CNN's Lou Dobbs serves up daily on the evils of outsourcing and illegal immigration. I have a more optimistic point of view on personal robotics. </p> <p> The Japanese attitude for robotics is quite positive, and their enthusiasm for personal robots resembles their enthusiasm for advanced cell phones. iMode mobile phone systems have features beyond what is known in North America, and Japanese society has evolved to incorporate behavioral modes that depend on the advanced features of the iMode phone network. Beyond mobile phones, the Japanese culture celebrates electronic gadgetry, and they already accept robots, including personal robots, into their daily experience. </p> <p> Change is coming, and we can either embrace the future or bemoan the loss of the past. I think personal robotics will become a plaything of the rich in 2015; Mr. Brain predicts the mass market will explode in 2030. It's not like we didn't have any warning. </p> <p>I think I need to start learning Japanese.</p> Top 12 Klingon Programmer Sayings 2005-04-02T00:00:00-05:00 https://mslinn.github.io/blog/2005/04/02/top-12-klingon-programmer-sayings <style> ol { list-style-type:none; counter-reset:item 13; } ol > li { counter-increment:item -1; } ol > li:before { content:counter(item) ". "; } </style> <ol> <li>Specifications are for the weak and timid!</li> <li>This machine is a piece of GAGH! I need dual Opteron processors if I am to do battle with this code!</li> <li>You cannot really appreciate Dilbert unless you've read it in the original Klingon.</li> <li>Indentation?! &mdash; I will show you how to indent when I indent your skull!</li> <li>What is this talk of &lsquo;release?&rsquo; Klingons do not make software &lsquo;releases.&rsquo; Our software &lsquo;escapes,&rsquo; leaving a bloody trail of designers and quality assurance people in its wake.</li> <li>Klingon function calls do not have &lsquo;parameters&rsquo; &mdash; they have &lsquo;arguments&rsquo; &mdash; and they ALWAYS WIN THEM.</li> <li>Debugging? Klingons do not debug. Our software does not coddle the weak.</li> <li>I have challenged the entire quality assurance team to a Bat-Leth contest. They will not concern us again.</li> <li>A TRUE Klingon Warrior does not comment his code!</li> <li>By filing this PTR you have challenged the honor of my family. Prepare to die!</li> <li>You question the worthiness of my code? I should kill you where you stand!</li> <li>Our users will know fear and cower before our software! Ship it! Ship it and let them flee like the dogs they are!</li> </ol> <p> I lifted this from Monsters Island, now defunct. </p> Zample in a Blog Post 2005-01-05T00:00:00-05:00 https://mslinn.github.io/blog/2005/01/05/zample-in-blog <p> Erik Dasque, Product Manager for <a href="https://mono-project.com/" target="_blank" rel="nofollow">Mono</a> at <a href="https://www.novell.com/linux/ximian.html" target="_blank" rel="nofollow">Ximian</a> (now part of Novell) was the first person to include a Zample in a blog post. Way to go, Erik! May there be many more bloggers using Zamples to display live code in their blogs. </p> <p> I have had some trouble doing this &ndash; when I used SnipSnap it simply wouldn't accept raw HTML. I wrote a SnipSnap plugin to allow raw HTML, but then my SnipSnap installation b0rked and I gave up on it. WordPress really mangles the HTML for a Zamples button; here is my best effort (can't get the 'Try It!' button to remain vertically centered.) </p> <table> <tbody> <tr style="vertical-align: middle;" valign="middle"> <td><form action="" class="formInput" method="post" target="TomcatMain"> <input name="cmd" type="hidden" value="// Define a runnable C# class here. You must include a // &quot;main&quot; method. Command line arguments go in the text area above. public class Test { public static void Main() { System.Console.WriteLine(&quot;I love Zamples!&quot;); } }"> <input name="imports" type="hidden" value="# Command line arguments go here"> <input name="applet" type="hidden" value=""> <input name="format" type="hidden" value="4608"> <input name="pre" type="hidden" value="true"> <input name="stylesheet" type="hidden" value="/assets/css/style.css"> <input class="formInput/" type="submit" value="Try It!"></form> </td> </tr> </tbody> </table> <pre data-lt-active='false'>public class Test { public static void Main() { System.Console.WriteLine("I love Zamples!"); } }</pre> <p>If you would like to include a live code example in your blog, first create it using the Zamplizer (easiest if you click on one of the selections under the Create Sample left-hand menu on the Zamples Solutions page &ndash; that will set up the Zamplizer with a short working program in the language you want, which you can then modify.)</p> <div style="text-align: center;"> <picture> <source srcset="/assets/images/zamples.webp" type="image/webp"> <source srcset="/assets/images/zamples.png" type="image/png"> <img src="/assets/images/zamples.png" title="Zamples logo" class="center liImg2 rounded shadow" alt="Zamples logo" /> </picture> </div> Another Software Expert Assignment 2004-12-24T00:00:00-05:00 https://mslinn.github.io/blog/2004/12/24/another-software-litigation-assignment <p> I&lsquo;ve just picked up another software litigation case, this time doing research for a software vendor prior to going to court. My client is concerned that two former employees used their code in a product sold by a new company that the former employees started. </p> <p> I've worked on this type of case before. They are often settled out of court. It is interesting to see how human interactions affect technology and business decisions. </p> <p>I know what I'll be doing over the Christmas/New Year's holidays!</p> Nabu cast a long shadow 2004-12-18T00:00:00-05:00 https://mslinn.github.io/blog/2004/12/18/nabu-cast-long-shadow <p> Twenty years ago an Ottawa-based startup was formed by the merger of nine Canadian companies. Nabu sold a cable set-top box with a computer that booted off the cable for under $1000. I was one of the early employees. </p> <p> Nabu was first in many ways, but it never became the commercial success that the founders had hoped. Many years have since passed, and I forgot about the experience. In the last couple of years I have been contacted by various people regarding Nabu&rsquo;s products and services, mostly former customers. </p> <p> &ldquo;Nabu was the first computer that I had when I was a kid. I remember playing games like Mummy&rsquo;s Tomb, Kiddy Park etc. Talk about being ahead of it's time! ... I think maybe people that were influenced by NABU when they were young are probably at the point in their life where they can stop and take a look around... Anyway, I thought I would drop you a message and say that this Ottawa boy (who now lives in Seattle, WA) still remembers the old Nabu days. I&rsquo;m now a professional video game developer, which I&rsquo;m sure was influenced by playing the Nabu so long ago. Of course today&rsquo;s machines have a little more power! What you guys did was a pretty amazing accomplishment IMHO.&rdquo; </p> <p> &ldquo;As someone who was fortunate enough to have access to the NABU network in my after school program at Regina Public School in Ottawa... I really enjoyed it as a kid.&rdquo; </p> <p> Recently I was contacted by a legal firm that has retained me to assist them in a patent litigation case. They are going to cite some of Nabu&rsquo;s products as prior art. I&rsquo;m having fun going back and reconnecting with people that I knew from that bygone era, who I lost track of. So far no-one that I knew has died of old age, which pleases me. </p> Live Code Examples for JDK 6 (Mustang) 2004-12-09T00:00:00-05:00 https://mslinn.github.io/blog/2004/12/09/live-code-examples-for-j2se-6-mustang <p> No sooner did Sun push J2SE 5 (also known as JDK 1.5) out the door, then talk of the next version began. Java releases are generally 18 months apart, but for the first time Sun is letting the world see development builds as they are created. </p> <p> We at Zamples took advantage of this new openness and proudly bring you Live Code Examples for J2SE 6. Let me know if you discover any cool new features! </p> <div style="text-align: center;"> <picture> <source srcset="/assets/images/zamples.webp" type="image/webp"> <source srcset="/assets/images/zamples.png" type="image/png"> <img src="/assets/images/zamples.png" title="Zamples logo" class="center liImg2 rounded shadow" alt="Zamples logo" /> </picture> </div> All Global 2000 Businesses Are Software Developers 2004-11-04T00:00:00-05:00 https://mslinn.github.io/blog/2004/11/04/web-services-means-all-global-2000 <p> With the advent of web services, any company that wishes to expose a programmable interface to their business processes also inherits the issues faced by software vendors. Connecting up a supply chain isn't quite as simple as one might want it to be. </p> <p> Software vendors have an urgent need for their channel partners and customers to adopt their technology quickly. The faster their products are understood, the more money they make. Shorter adoption times drive top line revenue. </p> <p> Similar dynamics affect companies that use web services to transact business. Businesses with custom interfaces need to somehow show system integrators how to program to their web services. Executives need to be able to wire their enterprises together quickly and easily in order to take advantage of business opportunities. Web services are supposed to make this easy... but nothing is ever quite as easy as it is supposed to be. </p> <p> The answer for both software vendors and businesses with custom web services interfaces is a comprehensive developer relations program. Not suprisingly, live online code examples are an important aspect of such a program. I have begun writing a follow-on to my article entitled <a href='https://productmanagement.buzz/index.php/2004/11/10/top_10_issues_f/' target="_blank" rel="nofollow">Top Ten Issues for Developer Relations Managers</a>. I would be happy to hear from you if you would like to review a draft before it is published. </p> Reading Java Properties Files from Bash 2004-11-03T00:00:00-05:00 https://mslinn.github.io/blog/2004/11/03/reading-java-properties-files-from-bash <p>Ever notice that Java properties files are fairly similar to bash scripts? Here is a short bash script that takes a valid Java properties file called <tt>language.properties</tt>, massages it into a format that is compatible with bash syntax, writes it to a temporary file and sources it.</p> <pre data-lt-active='false'>#!/bin/bash TEMPFILE=$(mktemp) cat language.properties | \ sed -re 's/"/"/'g | \ sed -re 's/=(.*)/="\1"/g' &gt; $TEMPFILE source $TEMPFILE rm $TEMPFILE</pre> <p>After running this script, all of the properties in the Java properties file become available as environment variables.</p>