Das Bityard - Networking/2023-12-10T00:49:37-05:00Filling in the Gaps: HTTPS/TLS Certificates2023-12-07T00:00:00-05:002023-12-10T00:49:37-05:00Charlestag:None,2023-12-07:articles/2023/December/filling-in-the-gaps-httpstls-certificates.html<figure>
<img src="images/tls/lock.png">
</figure>
<p><em>(Disclaimer: No AIs were used in the writing of this article. My intelligence has been frequently questioned, but has thus far never been accused of being less than natural.)</em></p>
<p>At this point, it's fair to say most tech-oriented cyberspace denizens have at least a passing familiarity with the basic concepts …</p><figure>
<img src="images/tls/lock.png">
</figure>
<p><em>(Disclaimer: No AIs were used in the writing of this article. My intelligence has been frequently questioned, but has thus far never been accused of being less than natural.)</em></p>
<p>At this point, it's fair to say most tech-oriented cyberspace denizens have at least a passing familiarity with the basic concepts of <a href="https://en.wikipedia.org/wiki/HTTPS">HTTPS</a>. Back in Ye Olden Times, that meant buying a cert from a company<sup id="fnref:company"><a class="footnote-ref" href="#fn:company">1</a></sup> once a year and installing it on your web server. These days, we have free public certificate authorities whose APIs are baked right into the server software or are easily bolted on. So there is not generally as much need to understand the vagaries of the certs themselves as there was before.</p>
<p>But what happens when the magical cert-fetching mechanisms fail and you don't know why? Or you need something that a public CA cannot provide for whatever reason? Well, when that happens, please do feel free to drop me an email to ask about my rates. :) Or, if you must, fall back to Plan B and read the remainder of this article to learn about the most important, <em>practical</em> bits of how HTTPS and SSL/TLS work.</p>
<p>This is loosely based on a talk I gave at work, where a few developers on a neighboring team had questions about how to configure and troubleshoot certificates in their server software. The goal here is to show how certs work at the most basic conceptual level, without straying too far into RFCs and X.509 and what have you.</p>
<h1>Background</h1>
<p>First off, you may be asking yourself, what's the difference between <a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">SSL</a> and <a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">TLS</a>? The answer is: None. None at all. They are just different versions of the same thing. What happened is that the protocol was <em>initially</em> named SSL (Secure Socket Layer) and as it went through revisions, someone decided to call it TLS (Transport Layer Security) instead. That was decades ago but due to inertia and the inability of anything on the Internet to ever truly die<sup id="fnref:die"><a class="footnote-ref" href="#fn:die">2</a></sup>, people still call the technology both "SSL" and "TLS" and use the two acronyms completely interchangeably. So that's fun.</p>
<p>The rest of this article, I'm going use "TLS" because that feels like the most correct thing to do.</p>
<p>HTTPS is ostensibly just HTTP (hypertext transfer protocol) with TLS layered on top of it. Layers are good because that means they can be interchangeable and individually upgradable. TLS can be layered on top of other protocols such as FTP, IMAP, SMTP, IRC, etc. But HTTPS is far and away the most popular use case.<sup id="fnref:case"><a class="footnote-ref" href="#fn:case">3</a></sup></p>
<p>TLS gives us two fundamental properties of a secure information exchange: encryption and trust. <strong>Encryption</strong> is the process of transforming information of any kind into an opaque blob of data that only the <em>sender</em> and <em>receiver</em> can decode back into its plaintext form. Someone else in the path of the encrypted data can see that it's there, but cannot read the information while it is encrypted because it just <a href="https://en.wikipedia.org/wiki/Ciphertext_indistinguishability">looks like random nonsense</a>. The <strong>trust</strong> component ensures that you are talking to someone who is who they say they are. The SSL/TLS designers decided that encryption was totally and utterly useless without trust, so both of these were combined into the one thing, thus ensuring Internet flamewars for generations to come.</p>
<p>Any software that hosts a TLS service (e.g. HTTPS) needs at least two things for a valid configuration:</p>
<ul>
<li>A private key</li>
<li>One or more certificates</li>
</ul>
<p>Private keys are considered <strong>secrets</strong>. (Like your passwords, but totally random and much longer.) "Keep it secret. Keep it safe."</p>
<p>TLS certificates and keys are <a href="https://en.wikipedia.org/wiki/X.509">X.509</a> structured binary data, but we most often see them encoded in <a href="https://en.wikipedia.org/wiki/Privacy-Enhanced_Mail">PEM</a> format for easy transport between systems, software, and people. (And clipboards.)</p>
<p>A private key looks like this:</p>
<div class="highlight"><pre><span></span><code>-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDdwRCbSS3MBX0Z
V063D0GRGqTMJp2Tla+3ICuVXffJR2QPH41r08vHOPz3bikFL3lu3d6uZMlkOKJU
... many lines of data ...
LwhR0trxBvI9mPXw9NbiLzdabL7UJ+bW36o5FBsevDoJ8i6tHmrMZ55H/W6G0h5d
zq3Af0aj6XNl7ky2fbgw78Q=
-----END PRIVATE KEY-----
</code></pre></div>
<p>A certificate looks like this:</p>
<div class="highlight"><pre><span></span><code>-----BEGIN CERTIFICATE-----
MIIG8zCCBdugAwIBAgIRALuuT/h77hmJFbbXENERxHIwDQYJKoZIhvcNAQELBQAw
gZUxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO
... many lines of data ...
b1nH99Re7ArYen3dKsgk/chlIHowdIKk7mTL3DTIck5PUaWmn9P8wg851EONxPnP
YtTVdTaXNg==
-----END CERTIFICATE-----
</code></pre></div>
<p>I <em>highly</em> encourage you to play around with tools that let you generate your own certificates. Try it with raw OpenSSL, or GnuTLS, or download a sketchy shell script from a GitHub gist, whatever. This article doesn't have any instructions for this because there are already eleventy-billion blogs on the web that do.</p>
<h1>Chain of Trust, Part 1: Vocabulary</h1>
<p>A <strong>Certificate Authority</strong> (CA) is an entity which signs certificates. An "entity" here is usually an organization of some kind. But it could be a small team, a department within your company, or a single individual. A CA is intended to be <a href="https://en.wikipedia.org/wiki/Trusted_third_party">someone you <em>trust</em></a> to secure network communications on your behalf.<sup id="fnref:ca"><a class="footnote-ref" href="#fn:ca">4</a></sup></p>
<p>When a CA <strong>signs</strong> another certificate, they are vouching for the trustworthiness of the person or organization in control of the private key for that certificate. It does this using some fancy mathematics that we call cryptography. If your software trusts a CA, it uses the same fancy mathematics to verify that the certificates it receives was signed by a CA that you trust.</p>
<figure>
<a href="https://upload.wikimedia.org/wikipedia/commons/8/87/Chain_of_trust_v2.svg">
<img src="images/tls/chain-o-trust.png">
</a>
<figcaption><a href="https://commons.wikimedia.org/wiki/File:Chain_of_trust_v2.svg">Used with Permission</a></figcaption>
</figure>
<p>There are three kinds of certificates which form a <a href="https://en.wikipedia.org/wiki/Chain_of_trust">chain of trust</a>:</p>
<ol>
<li>Root certificates</li>
<li>Intermediate certificates</li>
<li>Leaf certificates<sup id="fnref:leaf"><a class="footnote-ref" href="#fn:leaf">5</a></sup></li>
</ol>
<p>A <strong>root certificate</strong> (a.k.a. root CA cert) is generated by a CA and is <em>self-signed</em>, meaning the Issuer and Subject fields are identical. It is also marked as a CA cert in the certificate extensions. Although technically possible, root certificates are not typically used for directly signing leaf certificates because the time and effort cost of replacing them is usually high. They are generally created with a validity of many years, perhaps a decade or so. Their private key is (or should be) held in a secure location and is only brought out on special occasions to sign new intermediate certs.</p>
<p>An <strong>intermediate certificate</strong> (a.k.a. subordinate CA cert) is also generated by a CA, but has a shorter validity than a root, and is used to sign either other intermediate certs, or leaf certs. These can be signed either by a root CA cert or another intermediate cert. These are also marked as CA certs in the cert's extension fields. Intermediate certs are <em>never self-signed</em>.</p>
<p>A <strong>leaf certificate</strong> (a.k.a. end-entity cert, or server cert, or service cert) contains the server's public key and (usually) proof of ownership/trust by virtue of being signed by a trusted CA. A leaf certificate can also be self-signed, in which case the TLS connection still works, but users will get warned by their client that the certificate is not trusted. (Unless the cert is added it to their browser as a trusted cert.) Leaf certs contain special fields to show what the cert is authenticating, which in the case of HTTPS is the DNS name of the site.</p>
<h1>Anatomy of a Certificate</h1>
<p>There are oodles of tools that can inspect an certificate, but <a href="https://www.openssl.org">OpenSSL</a> is ubiquitous:</p>
<div class="highlight"><pre><span></span><code>openssl x509 -in <cert_file> -text
</code></pre></div>
<p>This is what you get:</p>
<!-- ![openssl cert info](https://img.bityard.net/blog/tls/cert3web-opt.svg) -->
<figure>
<img src="images/tls/cert.svg" width="700px">
</figure>
<p>A cert has many bits of information, but these are the most important ones:</p>
<ul>
<li><strong>Issuer</strong>: Who issued (signed) the certificate.</li>
<li><strong>Subject</strong>: What the certificate is for. If the Subject and Issuer are exactly the same, that is a self-signed cert and possibly a root cert.<ul>
<li>Side-note: The subject contains a CN (CommonName) attribute. Once upon a time, this was used to specify the domain that the certificate was created for. In modern times, the CN attribute is largely ignored by TLS clients, which only look at the Subject Alternative Name (SAN) if it exists. See below. (But note that some "legacy" software may still try to validate the CN.)</li>
</ul>
</li>
<li><strong>Validity</strong>: When the certificate is valid.</li>
<li><strong>X509 Extensions</strong>:<ul>
<li><strong>Subject Alternative Name</strong> (a.k.a. SAN): In a leaf cert, this contains one or more DNS names and is what links the site security to the certificate. This can be an FQDN or a wildcard (e.g. <code>*.example.com</code>). It's important to know that wildcards do not include subdomains, so <code>*.example.com</code> will not work for <code>foo.bar.example.com</code>.</li>
<li><strong>Key Usage</strong>: Root and intermediate certs will say "Certificate Sign", leaf certs will say, "Digital Signature"</li>
<li><strong>Basic constraints</strong>: Root and intermediate certs will say "CA: True"</li>
<li><strong>Extended Key Usage</strong>: Leaf certs will say "TLS Web Server Authentication"</li>
</ul>
</li>
</ul>
<p>Every certificate has a matching <strong>private key</strong> that should be kept secret. Ideally, it should be generated on the device/host it is for and never leave that host for any reason. But the reality is that sometimes we have to copy keys around just to get the job done. Avoid putting the key in email, or on some public or semi-public system. If you <em>must</em> copy it somewhere, at least take some time to think about the most secure way of getting it there.</p>
<h1>Chain of Trust, Part 2: Machinery</h1>
<p>Web browsers, operating systems, and other client software ship with a set of root certificates from organizations that they consider trustworthy. There is generally a vetting/audit process that browsers and OSes make CA's jump through in order to get their root CA cert shipped in their software. If the root CA cert is installed on your system, then it is trusted. This is essentially the entire basis of security on the web as we know it today. (For better or worse, depending on your experience and opinions.)</p>
<p>Users and companies can optionally decide to trust their own root CA certificates by placing them in the root certificate store of the OS and browsers of the computers they manage. Et voila, you now have an internal certificate authority. This is a very practical choice when using public CAs is onerous for whatever reason, or when you require a higher degree of trust/security than the public CA vendors can promise, at the non-trivial cost of extra administration tasks and workflows.</p>
<p>TLS clients decide whether a certificate is trusted by walking backward from the leaf cert to the root. Using a web browser to illustrate an <em>absurdly</em> oversimplified example:</p>
<ol>
<li>When the browser connects to an HTTPS website, the server sends one or more certificates:<ul>
<li>The leaf cert</li>
<li>Zero or more intermediate certs (one to two is most common)</li>
</ul>
</li>
<li>The browser looks at the leaf cert and compares the domain in the URL bar with the SANs in the leaf cert. If there is a match, the browser knows that the certificate is at least the correct one for the website, and proceeds to verify the chain of trust.</li>
<li>The browser looks at the leaf cert's subject and signature, and tries to find an already-trusted root cert matching those. If it does not, it looks at any intermediate certs that were sent by the server.</li>
<li>The browser looks at the intermediate cert's subject and signature, and tries to find an already-trusted root cert matching those. If it does not, it looks for any further intermediate certs that were sent by the server. This step repeats until a root cert is found.</li>
<li>If the browser finds a matching root cert in its certificate store, then all of the certs downstream are considered trusted, and the HTTPS connection is allowed to continue.</li>
</ol>
<p>If any of the above steps fail, the browser declares the chain of trust to be broken, and will throw an SSL/TLS error. Quite often accompanied by a cute cartoon that some UI designer thought might somehow lessen your frustration with their product.</p>
<h1>Common Configuration Patterns</h1>
<p>In the days of yore, server software required these to be split up into multiple files and specified as separate configuration items. (Many still do, especially those that can be configured only through a web interface.) For example, <a href="https://httpd.apache.org">Apache</a> used this:</p>
<div class="highlight"><pre><span></span><code># leaf cert, intermediate cert (or bundle), and key respectively
SSLCertificateFile /etc/ssl/certs/foo.example.crt
SSLCACertificateFile /etc/ssl/certs/EXAMPLE-CA.bundle
SSLCertificateKeyFile /etc/ssl/private/foo.example.net.key
</code></pre></div>
<p>Nowadays, you can get away with:</p>
<div class="highlight"><pre><span></span><code># everything in one file
SSLCertificateFile /etc/ssl/private/foo.example.net.crt
</code></pre></div>
<p>Some software only has fields for the certificate plus intermediate certs, and the private key.</p>
<p>However, most server software is moving toward the practice of bundling all certificate materials into one file. When this is done, you generally must specify the certs/key in a particular order in the file, that order being:</p>
<ol>
<li>The leaf certificate</li>
<li>Any intermediate certs (each cert below the one it signed)</li>
<li>The private key</li>
</ol>
<p>Note that the root CA is nowhere in here. Servers don't need to send root certs to clients, although it usually will not hurt anything if they do.</p>
<p>How to know if your server software supports concatenating the certs and key into one file?</p>
<ol>
<li>RTFM</li>
<li>Just try it</li>
</ol>
<h1>Verifying Certificate Configuration</h1>
<p>It's important to know that web browsers <a href="https://serverfault.com/questions/279984/how-do-i-clear-chromes-ssl-cache">can</a> and <a href="https://superuser.com/questions/351516/do-intermediate-certificates-get-cached-in-firefox">do</a> cache intermediate certificates that they have seen before. So if you have inadvertently left out an intermediate certificate on a server configuration, the web browser will say, "oh, I've seen that one before, I'll just grab it from my cache" and proceed to verify the chain of trust using the intermediate cert(s) that it dug out from between the couch cushions rather than noticing that the server didn't even send one. And it will show the site as trusted and everything will look normal, until another user with a different browser visits the site and gets an SSL error and kicks over every trash can on his way to your desk to berate you for "breaking the network."</p>
<p>This is why you should always verify that the certificates are correct with another tool. Naturally, OpenSSL can do this:</p>
<div class="highlight"><pre><span></span><code>openssl s_client -connect <hostname>:<port> -servername <hostname>
</code></pre></div>
<p>The output has essentially has these sections:</p>
<ol>
<li>The chain of trust. This shows the subject of each certificate in the trust chain, starting with the root and ending with the leaf, with any intermediates in between.</li>
<li>The "Certificate chain" as sent by the server, starting with the leaf cert, followed by intermediate certs. It shows the subject, issuer, and validity period of each cert. This is generally the one you want to pay the most attention to.</li>
<li>The leaf cert in PEM format.</li>
<li>TLS handshake details. The most important part here is near the end a line that says, "Verify return code" which will be <code>0</code> if the whole chain is trusted.</li>
</ol>
<h1>Non-Public Root CA Certs</h1>
<p>All of this depends on your OS having the correct root certs installed, and OpenSSL knowing where to find them. For most publicly-trusted CAs, this generally Just Works but if you have a custom root CA in your company or team, you probably have to manually install the root CA cert in your browser and/or on your operating system. Figuring out how to do that for your OS or browser is left as an exercise to the reader since it varies <em>greatly</em> by product.</p>
<p>But for those running a Debian or Ubuntu Linux distribution, I'll make it easy because they make it easy:</p>
<ol>
<li>Copy the root CA cert to <code>/usr/local/share/ca-certificates</code></li>
<li>Run <code>update-ca-certificates</code> as root.</li>
</ol>
<h1>Ways to Do It Wrong (From Experience)</h1>
<p>Ask me how I know!</p>
<h2>Forget the Intermediate Cert</h2>
<p>The HTTPS client needs to see all of the intermediate certs that make up the chain of trust, if you forget to include it (or them, if there are multiple), then the client has no way to match up the leaf cert with one of the root CA certs in it's trusted certificate store and the chain of trust is broken.</p>
<p>It doesn't help that a LOT of server software obfuscates or completely omits how to configure an intermediate cert in their documentation and UI, which makes it easy to forget. But intermediate certs are a standard and usually required thing, so there's always a way to do it, it just may not be obvious.</p>
<h2>Put Certs/Keys in the Wrong Order</h2>
<p>When multiple certs and the key are concatenated into a single file, they must generally be presented in a particular order, as detailed above. I don't know why this is, as I feel that any TLS libraries <em>should</em> be smart enough to examine each PEM section individually and work what each one is for. This is one of those things that contributes to the graying of hair for no good reason.</p>
<h2>Don't Test After Deployment</h2>
<p>Your cert works fine in development but users get a TLS error after deployed to production, eh? It's not entirely your fault. To make it work correctly, you have to get everything set up just right. To break it entirely, you only need to get one thing wrong.</p>
<p>So the first thing I do after deploying a new service or a new certificate is to validate that TLS is working correctly. Generally the OpenSSL command I mentioned under "Verifying Certificate Configuration" is the first thing I do. If that works, there's a 99% chance I did it right. If I'm in a very <em>thorough</em> mood, I will also bring the site up in a few different browsers, and in a couple different OSes if I can manage it.</p>
<h1>Suggested Best Practices</h1>
<p>A few things that could make your life easier.</p>
<h2>Comment Your Certs</h2>
<p>When saving a cert to a file, always save the text of the certificate along with the PEM encoding. Think of it like adding comments to obtuse code. This makes the cert both human-readable and machine-readable, and will save you a ton of time when debugging a certificate issue, or just figuring out which cert is which. OpenSSL can do this:</p>
<div class="highlight"><pre><span></span><code>openssl x509 -in cert_file.crt -text > annotated_cert_file.crt
</code></pre></div>
<h2>Consider Keys to be Immutable</h2>
<p>It is generally considered safest and securest if the TLS key is generated on the host it's meant for, and never ever leaves that host. Copying a key around leaves room for it to get accidentally copied (or left) somewhere it shouldn't be. And then you gotta re-key and re-cert the host if you want any semblance of actual security.</p>
<p>Related: When a cert is expiring soon and you are tasked with renewing it, you may as well re-generate a new key. It's only one extra command and provides a little bit of insurance against your previous key leaking out unbeknownst to you.</p>
<h2>Automate What You Can</h2>
<p><a href="https://letsencrypt.org">Let's Encrypt</a> proved that certificate automation was not only possible but even potentially easy. If you're managing purchased certs by hand like some neanderthal, you will find it very worth your time to look into how to automate your cert issuance and deployment, if for no other reason than to avoid a surprise cert expiration at 3 a.m. on a Sunday morning.</p>
<h1>Aside: Certificate Files and Formats</h1>
<p>You are probably most used to seeing certificate files and keys in PEM format with its <code>BEGIN CERTIFICATE</code> and <code>END CERTIFICATE</code> markers with the base64-encoded data in between. This data is encoded for easily handling but certificate data is actually binary data in an x509 structure. That said, there are other formats out there that you may have to deal with:</p>
<ul>
<li>PEM (Privacy-Enhanced Mail): Described above. The most common extension for these is <code>.crt</code> for certs, but some software and CAs will use <code>.cer</code> or <code>.pem</code>. Keys usually have the extension <code>.key</code>.</li>
<li>DER (Distinguished Encoding Roles): Simply the binary format for certs, the same data that gets base64-encoded for PEM certs. The extension is usually <code>.der</code> but sometimes <code>.crt</code>.</li>
<li>PKCS12: a format used to bundle certs and keys into one file. Primarily used by Java. The file extension is usually <code>.p12</code>.</li>
<li>PKCS7: Used by Windows and their CA. The extension is usually <code>.p7b</code>.</li>
</ul>
<p>There are lots of tools for converting between formats, here is how to do it with OpenSSL:</p>
<h2>PEM to DER</h2>
<div class="highlight"><pre><span></span><code>openssl x509 -in cert.crt -outform der -out cert.der
</code></pre></div>
<h2>DER to PEM</h2>
<div class="highlight"><pre><span></span><code>openssl x509 -in cert.der -inform der -outform pem -out cert.crt
</code></pre></div>
<h2>PEM to PKCS12</h2>
<p>Convert the key and cert into a PKCS12 file. Note that a passphrase is actually required here.</p>
<div class="highlight"><pre><span></span><code>openssl pkcs12 -export -in example.crt -inkey example.key -out example.p12 -name example
</code></pre></div>
<p>Verify that it worked (if you have Java's keytool installed):</p>
<div class="highlight"><pre><span></span><code>keytool -v -list -storetype pkcs12 -keystore example.p12
</code></pre></div>
<h2>PKCS12 to Java Keystore (JKS)</h2>
<div class="highlight"><pre><span></span><code>keytool -importkeystore -deststorepass changeme \
-destkeypass changeme -destkeystore example.jks \
--srckeystore example.p12 -srcstoretype PKCS12 \
--srcstorepass changeme -alias example
</code></pre></div>
<h2>PKCS7 to PEM</h2>
<div class="highlight"><pre><span></span><code>openssl pkcs7 -in example.p7b -outform pem -out example.crt -print_certs
</code></pre></div>
<h1>Don't Panic 👍</h1>
<p>If you're new to managing certificates this might seem like quite a lot of ground to cover, but try not to worry. The good news is that this stuff has been around for a long time and the vast majority of your questions are only a quick search away. Once you understand the fundamentals of the chain of trust and exactly what the browsers are looking for, you're about 95% of the way to being a TLS expert. In my experience, becoming known around your sphere of influence as, "That Guy/Gal Who <em>Knows Things</em> About TLS" is generally all upside. But even if that is not your goal, hopefully this write-up still made the whole thing a bit more approachable.</p>
<h1>Rejected Titles</h1>
<ul>
<li>Certs, HTF Do They Even Work?</li>
<li><em>Encryption Intensifies</em></li>
<li>If You Liked it Then You Should Have Put a Cert on It</li>
</ul>
<h1>Addendum</h1>
<p>Over on the <a href="https://lobste.rs/s/vbsnup/filling_knowledge_gap_https_tls">Lobste.rs thread</a>, mcpherrinm provided <a href="https://lobste.rs/s/vbsnup/filling_knowledge_gap_https_tls#c_x3prdc">some excellent additional tips and clarifications</a> on cert formats/extensions.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:company">
<p>Their only business model, mind you, was to take money in one end, and emit some mathematically-derived bits out the other end. As it were. <a class="footnote-backref" href="#fnref:company" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:die">
<p>Except for privacy and Rick Astley memes. <a class="footnote-backref" href="#fnref:die" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:case">
<p>As it happens, newer versions of the HTTP protocol make TLS a mandatory part of the protocol. This negates the architectural advantages of separating orthogonal concerns from each other but in theory promotes better security, as long as users of the new protocols get all of their other security practices right. You win some, you lose some. <a class="footnote-backref" href="#fnref:case" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:ca">
<p>Theoretically speaking, <em>you</em> could be a CA, if you wanted to. Because generating and signing certificates is actually really easy. Getting others to trust you is the hard part. <a class="footnote-backref" href="#fnref:ca" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:leaf">
<p>There is, surprisingly, no one industry-standard common term for the final certificate in a chain of trust. Most of the time, people just call it "the certificate," or sometimes "service certificate". I find these to be ambiguous enough to be worth avoiding. A long time ago, I read something that used the phrase "leaf certificate." That's what I'm going with because I like the tree analogy and because it fits in well with the hierarchical DAG arrangement of certificate trust. <a class="footnote-backref" href="#fnref:leaf" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
</ol>
</div>Various Ways of Sending Mail via SMTP2023-01-30T00:00:00-05:002023-12-10T00:49:37-05:00Charlestag:None,2023-01-30:articles/2023/January/various-ways-of-sending-mail-via-smtp.html<p>Internet Mail, or email, or whatever kids these days call it, was one of those things that terrified me very early on when I was a strapping young System Administrator. Everything else that I was doing at the time seemed comparitively easy: Linux/BSD installs, system setup, automation, and such …</p><p>Internet Mail, or email, or whatever kids these days call it, was one of those things that terrified me very early on when I was a strapping young System Administrator. Everything else that I was doing at the time seemed comparitively easy: Linux/BSD installs, system setup, automation, and such. Learning how various Unix shells and relational databases worked was a joy. But mail server administration... now <em>that</em> scared the hell out of me.</p>
<p>E-mail was and still is a complicated, fragile system. You can do everything right and <em>still</em> end up arse-deep in alligators due to someone else's mistake or bad hair day. There's just so much that can go wrong. To run a successful mail server means that you have to--at a bare minimum--concern yourself with such trivialities as:</p>
<ol>
<li>Getting <em>all</em> the DNS records exactly just so.</li>
<li>Make sure you start with a "clean" IP address... and keep it that way.</li>
<li>Set up user accounts and authentication.</li>
<li>Know how to configure the SMTP server.</li>
<li>Know how to configure the POP/IMAP server.</li>
<li>Oh yes, and most importantly: don't let the mail server become a spammer's playground.</li>
</ol>
<p>One of my first jobs was at a managed web hosting provider. Back then, if you wanted to become an expert on Apache, PHP, and email, then working the phones at a company like this was the quickest path "grizzled veteran" status. It's safe to say I learned me some email at that job.</p>
<figure>
<img src="images/smtp/exim.png">
</figure>
<p>I'm pretty comfortable with mail administration and troubleshooting nowadays. Heck, I even host my own personal mail server. Not out of necessity or anything, mostly just to annoy people on Reddit and HN who say it's impossible. My own setup is pretty stable and very rarely needs any attention. But either at work or at home, I sometimes find myself needing to troubleshoot occasional mail-related issues.</p>
<p>When The Mail Doth Not Flow, one of the most basic things you find yourself doing is sending test messages. Sometimes from systems that don't even have a proper mail server, client, or relay. For better or worse, it turns out that the "simple" in <a href="https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol">SMTP</a> is not as much of a lie as "lightweight" in <a href="https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol">LDAP</a>, and you don't often need a lot of ceremony just to fire off a simple message or two for testing or notifications from a barebones system. This article describes a few methods for doing so.</p>
<p><strong>Important:</strong> I'm going to use <code>example.com</code> as the domain here for illustrative purposes. This is not, in fact, a real mail server and it will never accept your messages or advances, even if you come armed with flowers and chocolate.</p>
<p>If you want to try some of these out but don't have your own mail server to fool around on, probably the best option is to fire up a Docker container or VM and install Postfix inside it. You can <em>try</em> to send mail to major mail providers using these methods but strive to contain your inevitable outrage if it doesn't work, especially if you are sending from a residential IP address.</p>
<h2>SMTP Basics</h2>
<p>It's important to know that when you (or perhaps even your mail client) send a message via SMTP, you're not just blasting a request at a server and hoping for a response, as with HTTP. Instead, SMTP more closely resembles a <em>conversation</em>. You say something, the server replies. You say another thing, the server replies again, and so on, until everything that needs to be said has been said and the discussion ends amicably. If you say something out of order, or that the mail server doesn't understand, it will act confused or just rudely hang up on you.</p>
<figure>
<img src="images/smtp/protip.png">
</figure>
<p>It's also worth pointing out early on that the SMTP standards require CR+LF line endings. (That's a carriage return character <code>0x0d</code>, followed by a linefeed character <code>0x0a</code>.) Most mail servers will happily accept stand-alone LF or (heaven forbid) CR line endings, but you shouldn't always count on that. When troubleshooting, you generally want to do things the way they are supposed to be done so as not to be lead down the garden path by your own incompetence, ask me how I know.</p>
<p>Finally, SMTP servers generally listen on TCP port 25, among others. (Other ports might mandate TLS, or refuse to continue without STARTTLS, or require authentication.)</p>
<h3>1. Greeting</h3>
<p>When you connect to an SMTP server, it will tell you its name and then wait for you to greet it with yours. The first thing you say to a mail server is almost literally, "Hello, I am (insert name here)." You greet the server and tell it the hostname of the machine you're sending mail from.<sup id="fnref:hostname"><a class="footnote-ref" href="#fn:hostname">1</a></sup> If you didn't offend it somehow, the mail server responds simply with <code>250 OK</code>. In the following example, I have connected to the mail server at <code>mail.example.com</code> and told it that my own hostname is <code>blog.bityard.net</code>:</p>
<figure>
<img src="images/smtp/helo.png">
</figure>
<p>Different mail servers reply with different text, the important bit is that the response starts with <code>250</code>. That's SMTPese for, "I don't hate you yet, let's keep talking."</p>
<p>You can also use <code>EHLO</code> instead of <code>HELO</code>. All this does is tell the server that you're a client that can handle SMTP features invented within the last 30 years or so. For the purposes of courageous troubleshooting or intrepid messing around, it doesn't really matter much which one you use but I'll be using <code>EHLO</code> from now on because it sounds more British.</p>
<h3>2. Envelope</h3>
<p>Next we say who the message is from and who the message is to.</p>
<figure>
<img src="images/smtp/envelope.png">
</figure>
<h3>3. Message</h3>
<p>If you've made it this far, there's a <em>fair</em> chance the server will accept the message <em>and</em> it might actually even deliver it. So we tell it that we're about to send the message:</p>
<figure>
<img src="images/smtp/data.png">
</figure>
<p>This means the server is ready to accept the message. Each mail message consists of two parts, the headers and the body. These must be separated by a blank line. (If you're reading this article, I'll presume you know what email headers are.) Note that the mail server is helpfully telling you how to signal the end of the message: A blank line, a dot, and another blank line. Here's an example of what to send:</p>
<figure>
<img src="images/smtp/message.png">
</figure>
<p>Note that different mail servers tend to respond to confusion in the headers in different ways. The <code>To</code> and <code>From</code> headers don't <em>always</em> have to match what you put in the envelope (this is to allow for things like mailing lists and forwarding to work), and technically a <code>Subject</code> header is optional. But a lot of things will go easier for you in life if you don't try to optimize for the smallest possible character count.</p>
<p>If the server was not terribly displeased by your inane ramblings, it accepts the message. Note that the mail server can still do whatever it wants with the message after acceptance. Up to and including:</p>
<ol>
<li>deliver it into a user's mailbox</li>
<li>forward it to another mail server</li>
<li>drop it on the floor (unceremoniously delete it)</li>
<li>broadcast it into outer space via radio signal to show alien civilizations that there is no intelligent life here</li>
</ol>
<p>If at any point you feel like you have made a sufficient fool of yourself, you can always bail out with the <code>QUIT</code> command or just close the TCP connection.</p>
<h2>The Telnet Way</h2>
<p><a href="https://en.wikipedia.org/wiki/Telnet">Telnet</a> is ostensibly its own protocol, not just some low-level TCP client. But in practice it often works as one anyway, and we can use it to manually simulate a number of other protocols such as SMTP. This is the closest some of us will ever get to being one of those super-cool computer hackers in action movies that save the day by cracking a military-grade encryption algorithm with seconds left to spare.</p>
<p>To send a message, run the <code>telnet</code> command with the server hostname and TCP port number as arguments:</p>
<div class="highlight"><pre><span></span><code>$ telnet mail.example.com 25
Trying 127.0.0.1...
Connected to mail.example.com
Escape character is '^]'.
220 mail.example.com ESMTP Postfix (Ubuntu)
</code></pre></div>
<p>Although as we noted above that regular newlines will <em>probably</em> work, the proper and correct thing to do is to switch <code>telnet</code>'s line endings to CR+LF. To do that, type <code>^]</code> followed by Enter and then:</p>
<div class="highlight"><pre><span></span><code>^]
telnet> toggle crlf
Will send carriage returns as telnet <CR><LF>.
</code></pre></div>
<p>From this point on, continue sending your message:</p>
<div class="highlight"><pre><span></span><code>HELO blog.bityard.net
250 OK
MAIL FROM:<alice@bityard.net>
250 OK - mail from alice@bityard.net
RCPT TO:<bob@example.com>
250 OK
data
354 End data with <CR><LF>.<CR><LF>
From: alice@bityard.net
To: bob@example.com
Subject: Why are fish so easy to weigh?
Because they have their own scales.
.
250 OK
quit
221 BYE
</code></pre></div>
<h2>The Netcat Way</h2>
<p>If your Unix machine is far too modern and hip to have an old fossil like <code>telnet</code> lying around, then perhaps it has <code>netcat</code>? If so, the process is largely similar, except you start the program with the <code>-C</code> flag to tell <code>netcat</code> to use CR+LF line endings:</p>
<div class="highlight"><pre><span></span><code>$ nc -C mail.example.com 25
</code></pre></div>
<p>From here, your port is open and you can just bash out the conversation on your keyboard.</p>
<p>Since <code>netcat</code> is, after all, designed to be stuffed into pipelines, you could conceivably put your half of the conversation into a file and just blast it at the server, right? Well, you could try, and you will sometimes even get away with it. Remember what I said above: SMTP is a conversation. If you start barking multiple commands at the server without waiting for a response in between, it will complain because that isn't a <em>conversation</em>.[^pipelining]</p>
<p>There is a cheap hacky work-around to this, though: you can tell <code>netcat</code> to wait a certain amount of time between sending lines, in order to give the server time to respond to commands. This will often work, but you'll want to pay attention and adjust the interval when working with particularly lethargic mail servers. (And I certainly do NOT recommend doing this for any kind of permanent solution. It is very brittle.)</p>
<p>Here is what a file called <code>test.smtp</code> might look like (be sure to use CR+LF line endings in the file!):</p>
<div class="highlight"><pre><span></span><code>HELO blog.bityard.net
MAIL FROM:<alice@bityard.net>
RCPT TO:<bob@example.com>
DATA
From: alice@bityard.net
To: bob@example.com
Subject: Originally, I didn't like having a beard.
But then it grew on me.
.
QUIT
</code></pre></div>
<p>And this is how you would send it.</p>
<div class="highlight"><pre><span></span><code>nc -C -i 1 mail.example.com 25 < test.smtp
</code></pre></div>
<h2>The Python Way</h2>
<p>One of the better ways to send a message from a host that has <a href="https://python.org">Python</a> installed is with a short script. This is made possible by virtue of Python's built-in <a href="https://docs.python.org/3/library/smtplib.html">smptlib</a> module. The nice thing about this is that it's highly flexible and doesn't require any other local mail server or tools.</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/usr/bin/env python3</span>
<span class="kn">import</span> <span class="nn">smtplib</span>
<span class="kn">from</span> <span class="nn">email.message</span> <span class="kn">import</span> <span class="n">EmailMessage</span>
<span class="n">msg</span> <span class="o">=</span> <span class="n">EmailMessage</span><span class="p">()</span>
<span class="n">msg</span><span class="p">[</span><span class="s1">'From'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'alice@bityard.net'</span>
<span class="n">msg</span><span class="p">[</span><span class="s1">'To'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'bob@example.com'</span>
<span class="n">msg</span><span class="p">[</span><span class="s1">'Subject'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'Every time you swallow some food coloring...'</span>
<span class="n">msg</span><span class="o">.</span><span class="n">set_content</span><span class="p">(</span><span class="s1">'...you dye a little inside.'</span><span class="p">)</span>
<span class="n">smtp</span> <span class="o">=</span> <span class="n">smtplib</span><span class="o">.</span><span class="n">SMTP</span><span class="p">(</span><span class="s1">'mail.example.com'</span><span class="p">,</span> <span class="mi">25</span><span class="p">)</span>
<span class="n">smtp</span><span class="o">.</span><span class="n">set_debuglevel</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="n">smtp</span><span class="o">.</span><span class="n">send_message</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
<span class="n">smtp</span><span class="o">.</span><span class="n">quit</span><span class="p">()</span>
</code></pre></div>
<h2>The Sendmail Way</h2>
<p>Unix greybeards will remember Sendmail, possibly as a motivation for taking up a burning interest in the hobby of drinking to excess. As a mail server, it has mostly been supplanted by more modern and sensible options. But several parts of its legacy live on and one of those is the <code>sendmail</code> client for sending mail from the command-line.</p>
<p>The <code>sendmail</code> command allows one to to write (or of course generate) a message in a standard format and then send it on its way to a mail server. If the host you're logged into has a mail server running on it (such as Sendmail, Postfix, Exim, etc), then the <code>sendmail</code> command is likely available. There are also stand-alone mail transfer agents that <em>only</em> accept messages and forward them along to some "real" mail server. (The one that I usually reach for is <a href="https://marlam.de/msmtp/">msmtp</a>.)</p>
<p>If a <code>sendmail</code> command exists on the host, you can use it to send messages which were written as text files. Let's assemble the following message as <code>my_message.eml</code> in the text editor of your choice:</p>
<div class="highlight"><pre><span></span><code>From: alice@bityard.net
To: bob@example.com
Subject: I recently developed an irrational fear of elevators
Since then, I have been taking steps to avoid them.
</code></pre></div>
<p>Notice that the top of the message has <em>headers</em> followed by a blank line, followed by the message. Theoretically, only the <code>To</code> header is required, but it depends on which <code>sendmail</code> variant you have installed. It's a good idea to include all three in any case, it will possibly make your life less interesting-but-in-a-bad-way.</p>
<p>You can send it with:</p>
<div class="highlight"><pre><span></span><code>sendmail -vt < my_message.eml
</code></pre></div>
<p>The <code>-v</code> flag tells the command to report what it's doing (useful when troubleshooting) and the <code>-t</code> flag tells it to read the recipient(s) from the headers in the message itself. Your <code>sendmail</code> implementation may have other options to investigate. Feel free to peruse them with <code>man sendmail</code>.</p>
<h2>The Swaks Way</h2>
<p>On systems with <a href="https://www.perl.org">Perl</a> (or a package manager that can install one), <a href="https://github.com/jetmore/swaks">Swaks</a> may be an option.</p>
<p>Swaks describes itself as a "Swiss Army Knife for SMTP". The nice thing about Swaks is that it lets you test and verify aspects of your SMTP configuration that would otherwise take a lot of setup or custom code. You can use it to test encryption (TLS, STARTTLS), authentication, SMTP protocol variants, sockets, proxies, and a whole bunch more.</p>
<p>See <a href="https://github.com/jetmore/swaks/blob/v20201014.0/doc/base.pod">the docs</a> for full details, but a simple test message can be sent with:</p>
<div class="highlight"><pre><span></span><code>swaks --to alice@example.com --server mail.example.com
</code></pre></div>
<h2>The Bash Way</h2>
<p>I present this way last because of the ways presented to far, this one is the most ill-advised. It's here mainly for completeness and and probably should not be used for anything serious except by those afflicted with chronic self-loathing. In any case, you do you.</p>
<p><a href="https://tiswww.case.edu/php/chet/bash/bashtop.html">Bash</a> has this <a href="https://tiswww.case.edu/php/chet/bash/bashref.html#Redirections">one weird trick</a> where you can open a TCP (or UDP) port to another host and read and write to it with a file descriptor. This means you can (in theory) write a Bash script to communicate with any Internet service. Now, Bash is good at a great many things, but writing a robust SMTP client would be quite a challenge. Nevertheless, if we sacrifice our sanity a little and don't mind some repetition, we can get away with the bare minimum needed to send a message.</p>
<p>The following script was lightly modified for clarity but was based on <a href="https://stackoverflow.com/a/10001357">this answer</a> from Stack Overflow.</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/usr/bin/env bash</span>
<span class="nb">readonly</span><span class="w"> </span><span class="nv">smtp_host</span><span class="o">=</span>mail.example.com
<span class="nb">readonly</span><span class="w"> </span><span class="nv">smtp_port</span><span class="o">=</span><span class="m">25</span>
<span class="nb">readonly</span><span class="w"> </span><span class="nv">msg_from</span><span class="o">=</span>alice@bityard.net
<span class="nb">readonly</span><span class="w"> </span><span class="nv">msg_to</span><span class="o">=</span>bob@example.com
<span class="nb">readonly</span><span class="w"> </span><span class="nv">msg_subject</span><span class="o">=</span><span class="s1">'Why did the scarecrow win an award?'</span>
<span class="nb">readonly</span><span class="w"> </span><span class="nv">msg_body</span><span class="o">=</span><span class="s1">'Because he was outstanding in his field.'</span>
<span class="c1"># send a line ending in a carriage return followed by an implicit line feed</span>
<span class="c1"># (`echo` prints a line feed at the end of each line automatically)</span>
send<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>-e<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">\r"</span><span class="w"> </span>><span class="p">&</span><span class="m">3</span>
<span class="o">}</span>
<span class="c1"># check the status code returned by the server</span>
<span class="c1"># if it's not what we expect, bail out</span>
check_status<span class="w"> </span><span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nv">expect</span><span class="o">=</span><span class="m">250</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="nv">$#</span><span class="w"> </span>-eq<span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nv">expect</span><span class="o">=</span><span class="s2">"</span><span class="nv">$3</span><span class="s2">"</span>
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="nv">$1</span><span class="w"> </span>-ne<span class="w"> </span><span class="nv">$expect</span><span class="w"> </span><span class="o">]</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Error: </span><span class="nv">$2</span><span class="s2">"</span><span class="w"> </span>><span class="p">&</span><span class="m">2</span>
<span class="w"> </span><span class="nb">exit</span>
<span class="w"> </span><span class="k">fi</span>
<span class="o">}</span>
<span class="c1"># open a TCP connection to the mail server</span>
<span class="nb">exec</span><span class="w"> </span><span class="m">3</span><>/dev/tcp/<span class="nv">$smtp_host</span>/<span class="nv">$smtp_port</span>
<span class="nb">read</span><span class="w"> </span>-u<span class="w"> </span><span class="m">3</span><span class="w"> </span>status<span class="w"> </span>text
check_status<span class="w"> </span><span class="s2">"</span><span class="nv">$status</span><span class="s2">"</span><span class="w"> </span><span class="s2">"</span><span class="nv">$text</span><span class="s2">"</span><span class="w"> </span><span class="m">220</span>
<span class="c1"># greet the server</span>
send<span class="w"> </span><span class="s2">"HELO </span><span class="k">$(</span>hostname<span class="w"> </span>-f<span class="k">)</span><span class="s2">"</span>
<span class="nb">read</span><span class="w"> </span>-u<span class="w"> </span><span class="m">3</span><span class="w"> </span>status<span class="w"> </span>text
check_status<span class="w"> </span><span class="s2">"</span><span class="nv">$status</span><span class="s2">"</span><span class="w"> </span><span class="s2">"</span><span class="nv">$text</span><span class="s2">"</span>
<span class="c1"># send the envelope</span>
send<span class="w"> </span><span class="s2">"MAIL FROM: </span><span class="nv">$msg_from</span><span class="s2">"</span>
<span class="nb">read</span><span class="w"> </span>-u<span class="w"> </span><span class="m">3</span><span class="w"> </span>status<span class="w"> </span>text
check_status<span class="w"> </span><span class="s2">"</span><span class="nv">$status</span><span class="s2">"</span><span class="w"> </span><span class="s2">"</span><span class="nv">$text</span><span class="s2">"</span>
send<span class="w"> </span><span class="s2">"RCPT TO: </span><span class="nv">$msg_to</span><span class="s2">"</span>
<span class="nb">read</span><span class="w"> </span>-u<span class="w"> </span><span class="m">3</span><span class="w"> </span>status<span class="w"> </span>text
check_status<span class="w"> </span><span class="s2">"</span><span class="nv">$status</span><span class="s2">"</span><span class="w"> </span><span class="s2">"</span><span class="nv">$text</span><span class="s2">"</span>
<span class="c1"># send the message</span>
send<span class="w"> </span><span class="s2">"DATA"</span>
<span class="nb">read</span><span class="w"> </span>-u<span class="w"> </span><span class="m">3</span><span class="w"> </span>status<span class="w"> </span>text
check_status<span class="w"> </span><span class="s2">"</span><span class="nv">$status</span><span class="s2">"</span><span class="w"> </span><span class="s2">"</span><span class="nv">$text</span><span class="s2">"</span><span class="w"> </span><span class="m">354</span>
send<span class="w"> </span><span class="s2">"From: </span><span class="nv">$msg_from</span><span class="s2">"</span>
send<span class="w"> </span><span class="s2">"To: </span><span class="nv">$msg_to</span><span class="s2">"</span>
send<span class="w"> </span><span class="s2">"Subject: </span><span class="nv">$msg_subject</span><span class="s2">"</span>
send
send<span class="w"> </span><span class="s2">"</span><span class="nv">$msg_body</span><span class="s2">"</span>
send
send<span class="w"> </span><span class="s2">"."</span>
<span class="nb">read</span><span class="w"> </span>-u<span class="w"> </span><span class="m">3</span><span class="w"> </span>status<span class="w"> </span>text
check_status<span class="w"> </span><span class="s2">"</span><span class="nv">$status</span><span class="s2">"</span><span class="w"> </span><span class="s2">"</span><span class="nv">$text</span><span class="s2">"</span>
</code></pre></div>
<h2>In Conclusion</h2>
<p>I am terrible at writing conclusions. This is the end of the article, I hope you enjoyed it.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:hostname">
<p>Real Mail Servers out there in Cyberspace <em>may</em> try to verify that you are who you say you are with a DNS lookup or two and might close the connection if they think you are lying. But on an internal mail relay or somesuch, you can often get away with some degree of subterfuge. <a class="footnote-backref" href="#fnref:hostname" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:piplining">
<p>The more experienced readers among us will note that all modern mail servers these days support <a href="https://datatracker.ietf.org/doc/html/rfc2920">pipelining</a>, but pipelining only helps you blast <em>some</em> commands at a server in rapid-fire fashion, not all. <a class="footnote-backref" href="#fnref:piplining" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>OpenSSH: The Poor Man's SOCKS Proxy2009-01-21T01:17:00-05:002023-12-10T00:49:37-05:00Charlestag:None,2009-01-21:articles/2009/January/openssh-the-poor-mans-socks-proxy.html<p>Just when I think I know everything I need to know about <a href="http://openssh.org/">OpenSSH</a>, I end up learning something new and tremendously useful. Today, that would be the -D argument.</p>
<p>Many times I have been stuck on an "untrusted" Internet connection and need to log in (insecurely) to a certain site …</p><p>Just when I think I know everything I need to know about <a href="http://openssh.org/">OpenSSH</a>, I end up learning something new and tremendously useful. Today, that would be the -D argument.</p>
<p>Many times I have been stuck on an "untrusted" Internet connection and need to log in (insecurely) to a certain site. My university, for example, uses a system that has no way of logging in via HTTPS, nor does it secure the traffic to and from the browser. I have moderate faith that the folks at my ISP aren't snooping my traffic (since I know the company pretty well and used to work with them), so I don't have a huge problem logging into their site at home. I also have a colocated server at the web hosting company I work for, so I know the layout of their network even better and trust them not to snoop or interfere with my traffic. But when I'm on the road connected to some dodgy insecure hotel wifi, I acquire no small amount of anxiety over the fact that anyone with a packet sniffer can get access to all of my personal and academic details.</p>
<p>For the past few years, I've had this plan to get <a href="http://openvpn.net/">OpenVPN</a> set up for my network and laptop so that I can always have a secure connection to my home and colocated server. And for the past few years, I've kept putting it off. While OpenVPN is easier to use than many other VPN solutions I could name, it's still at least a good hour of my time getting all the settings right and testing it out.</p>
<p>I was already aware of OpenSSH's -L option which simply forwards a local port through an SSH tunnel to a port on the remote machine. Very handy when you want to connect surely to a site hosted on that server and happen to have a shell account on it. But to do much more than that ranges from the complex to impossible. This is where -D comes in.</p>
<p>The -D arg tells OpenSSH to be a <a href="http://en.wikipedia.org/wiki/SOCKS">SOCKS</a> proxy. So you simply log in to the endpoint via SSH with the -D arg like:</p>
<div class="highlight"><pre><span></span><code>ssh -D 1234 user@host.example.com
</code></pre></div>
<p>And then tell your web browser to use a SOCKS v5 proxy on localhost at the specified port and bingo, you have a secure connection to your endpoint. In fact, any application with SOCKS support can have its traffic routed through the SSH tunnel via SOCKS. Firefox supports SOCKS just fine, Opera doesn't. Konqueror is supposed to, but judging from the Google responses I got, support might be a little flaky.</p>
<p>The final test was whether I'd be able to use this newfangled (to me) proxy method on my Nokia N800, a device that I browse and email with quite often whilst traveling. Obviously OpenSSH has to be installed as it doesn't come with the firmware. And the N800's web browser, MicroB, uses the Gecko engine. The UI has no widgets for entering a SOCKS proxy, but you can set the preferences manually with about:config:</p>
<div class="highlight"><pre><span></span><code>network.proxy.socks localhost
network.proxy.socks_port 1234
network.proxy.type 1
</code></pre></div>
<p>The result? Portable proxy surfing!</p>