MAIL CLUB NIGHT POST

Mail club · night post the one that follows a request home in the dark

Reference · /learn/

ISSUE 03

How this gets to be on the internet

Issue 02 built something worth serving. This one is about the distance between “it runs on my machine” and “other people can type a name and reach it” — which turns out to be about five separate pieces, none of which is the code.

Where this is going: Part A walks one request backwards, from the name you type to the process that answers, one hop per section. Part B re-reads the four commands you actually ran, then hands you a panel of buttons that go and ask this machine what is true right now.

the specimen is this site read-only probes systemd · DNS · TLS one open question
Stickers
  • the big picture
  • worth noting
  • the simple version
  • a quick aside
  • zooming right in

PART A — THE JOURNEY

01Part A

The whole journey, in one picture

Five things stand between a name someone types and the program that answers. Only the last one is yours.

When you visit alice.charliehub.net/learn/, it feels like one action. It is not. It is a relay, and each leg is handled by a different piece of software that knows nothing about the others. The reason that matters: when something breaks, it breaks at one hop, and knowing the hops is most of knowing where to look.

THE REQUEST, LEFT TO RIGHT browser you type a name — DNS name → address 10.254.0.1 TLS proves the name Let's Encrypt proxy picks a machine on hub2 port 8000 picks a program here, in this CT your app kept alive by systemd THE RESPONSE COMES BACK THE SAME WAY YOURS — INSIDE THIS CONTAINER SHARED — RUN BY THE HUB five handovers, and only the last two are things you can change from here
Each box is a different program, usually on a different machine. None of them knows what the next one does — they agree only on how to hand the request over.

Two things are worth noticing before the detail. First, most of the chain isn't yours. DNS, the certificate and the proxy are shared infrastructure that Charles runs; the port and the process are the parts that live in this container. Second, the chain is indirect on purpose. Nothing in it says “this name means this program”. Each step only knows how to get one step closer, which is exactly why any one of them can be changed without the others noticing.

The thing to hold on to

A name gets you to a machine. A port gets you to a program. A process manager keeps that program alive. Almost every confusing hosting problem is one of those three sentences failing, and telling them apart is most of the fix.

Section Review Card

no. 01 of 08
Concept:
Getting online is a relay of five handovers, not one action.
Mine:
Only the last two hops — the port and the process.
One thing I actually understood today:
Rating:

02Part A

The name

Computers have never once cared about your domain name. Something has to translate it first.

Machines on a network are found by address, not by name. alice.charliehub.net means nothing to the networking layer; it is a label for humans. Before anything else can happen, the browser has to turn that label into an address, and the system that does it is DNS — the Domain Name System, a globally distributed lookup table.

You can do exactly what the browser does:

the lookuprun it yourself
$ getent hosts alice.charliehub.net
10.254.0.1      alice.charliehub.net

That number is the answer, and it is more interesting than it looks.

Addresses beginning 10. belong to a range reserved for private networks. They are deliberately not routable across the public internet — every office and home network in the world is free to use the same 10.x.x.x numbers, precisely because no packet carrying one ever leaves the local network. There are three such ranges, and you will meet all of them eventually:

RangeWhere you'll see it
10.0.0.0 – 10.255.255.255Larger private networks. This one.
172.16.0.0 – 172.31.255.255Docker's default, among others.
192.168.0.0 – 192.168.255.255Almost every home router.

So your site has a real, ordinary domain name that resolves to an address only reachable from inside CharlieHub. That combination is what “internal-only” actually means, and it is a much more precise statement than the phrase. It isn't hidden, or secret, or password-protected. Anyone can look up the name. They simply cannot get a packet to the answer.

Why names, not numbers

The address is written down in exactly one place: the DNS record. If your service moved to a different machine tomorrow, you would change that record and every visitor would follow, without knowing anything happened. That is the whole reason for the layer of indirection — names are stable, addresses are not.

Section Review Card

no. 02 of 08
Concept:
DNS turns a name into an address; mine is a private one.
Command:
getent hosts alice.charliehub.net
One thing I actually understood today:
Rating:

03Part A

The padlock

A certificate is not a lock. It is a signed statement that you are talking to who you think you are.

HTTPS is ordinary HTTP wrapped in TLS, and TLS does two separate jobs that are easy to blur together:

Encryption on its own is nearly worthless. An impostor can encrypt too. The identity half is what the certificate is for: a certificate authority checks that whoever asked for it genuinely controls the name, and then signs a statement saying so. Your browser trusts a list of these authorities, which is why the padlock appears without you doing anything.

Here is the certificate this site presents, read exactly as a browser reads it:

the certificatethe observatory can fetch this live
subject   alice.charliehub.net
issuer    US, Let's Encrypt, YR2
valid     Aug  7 16:26:18 2026 GMT  ->  Nov  5 16:26:17 2026 GMT
protocol  TLSv1.3

Note the dates: ninety days. That is Let's Encrypt's standard lifetime and it is short deliberately. A certificate that lasted three years would be renewed by someone remembering, once, badly. Ninety days is short enough that renewal must be automated — and an automated renewal that runs every couple of months is one you know works, because it just worked.

Stargazing note

Something here doesn't add up, and it is the most interesting thing on this page.

A certificate authority normally proves you control a name by visiting it — it fetches a file from http://your-name/.well-known/… and checks the contents. But we established in the last section that this name points at 10.254.0.1, a private address. Let's Encrypt, sitting out on the public internet, cannot reach this site at all. So how did it verify anything?

The answer is almost certainly that the check was done a different way — by publishing a specific DNS record rather than serving a file, which proves control of the name without anyone needing to reach the machine. That method exists exactly for names like this one. But I have not confirmed that is what Charles does, so treat it as a good guess and a good question, not a fact.

Section Review Card

no. 03 of 08
Concept:
TLS proves identity as well as encrypting.
Ours:
Let's Encrypt, 90-day lifetime, renewed automatically.
Open:
How is a private-only name validated?
One thing I actually understood today:
Rating:

04Part A

The receptionist

One machine answers for dozens of sites. Something at the door has to read the label and decide where each request goes.

Your DNS lookup returned 10.254.0.1. That is not this container — it is hub2, the machine that fronts everything. Many different hostnames resolve to that same address. So when a request arrives there, hub2 faces an obvious question: which of these sites did you want?

The answer is in the request itself. Every HTTP request carries a Host header naming the site it is for:

what the browser actually sendsthe label on the envelope
GET /learn/delivery.html HTTP/1.1
Host: alice.charliehub.net
Accept: text/html

A reverse proxy reads that header and forwards the request onward to whichever machine and port is configured for that name. CharlieHub uses Traefik for this. “Reverse” because a normal proxy sits in front of you and hides which sites you visit; this one sits in front of the servers and hides which machine answered.

It also handles the TLS from the previous section. The certificate lives on the proxy, not here — which is why your app speaks plain HTTP on port 8000 and visitors still get a padlock. The proxy decrypts on the way in and re-encrypts on the way out.

You can watch the whole chain work by asking this site for its own health check the long way round — not via 127.0.0.1, but via the public name, so the request leaves this container, reaches hub2, gets matched to a route, and comes back:

out and backevery hop, in one command
$ curl -s https://alice.charliehub.net/api/health
{"ok":true,"entries":1}

The observatory below runs that and times it. Locally, a request to 127.0.0.1:8000 answers in about 7 ms. The same request via the name takes closer to 120 ms. That gap is the round trip out to hub2, through TLS, through the proxy, and back — a rare case where an abstraction has a number attached to it.

The honest gap in this page

I can show you that the proxy exists and that it works, but not the specific rule that maps your hostname to this container's port 8000. That configuration lives on hub2, and the documentation at docs.charliehub.net currently returns 410 Gone for every page — which means “this existed and was deliberately removed”, not “broken”. Rather than guess at Charles's setup and write something confident and wrong, this section stops here. Ask him: what defines the route for alice.charliehub.net, and how did the certificate get validated?

Section Review Card

no. 04 of 08
Concept:
A reverse proxy routes by the Host header.
Also does:
TLS — the certificate lives there, not here.
Evidence:
~7 ms local vs ~120 ms via the name.
One thing I actually understood today:
Rating:

05Part A

The port

The address gets a request to the right machine. The port decides which program on it gets to answer.

A machine runs many programs at once, and they cannot all answer the door. A port is just a number, from 1 to 65535, that a program claims so the operating system knows where to deliver incoming traffic. Address and port together identify one destination: this machine, that program.

You can see the claim being held right now:

who has port 8000ss = socket statistics
$ ss -ltnH sport = :8000
LISTEN 0      2048         0.0.0.0:8000      0.0.0.0:*
-l listening, -t tcp, -n numeric ports, -H no header row

0.0.0.0 means “on every network interface this machine has”, rather than only on the loopback address that nothing outside can reach. That is why your unit file says --host 0.0.0.0: bind to 127.0.0.1 instead and the app would work perfectly when tested from inside this container and be completely unreachable from hub2.

The rule that matters: only one program can hold a given address and port at a time. This is not a policy, it is how the kernel works — the second program to ask gets an error. It is also the reason the swap you ran had to stop the placeholder before starting the guestbook, in that order. They were both trying to be the answer to the same question.

Why 8000, and why reuse it

There is nothing special about 8000; it is convention for “a web thing in development”. What makes it special here is that the proxy route for your hostname points at it. The port number is half of an agreement made on another machine — which is exactly why building a new app means reusing port 8000 rather than picking a fresh number and then needing a second route to be configured.

Ports below 1024 are normally restricted to root, which is why web servers traditionally needed elevated privileges to bind 80 and 443. This container has that limit lowered to 80, so you could bind them directly — but you have no reason to, because the proxy handles the public-facing ports for you.

Section Review Card

no. 05 of 08
Concept:
A port selects the program; only one can hold it.
Why 0.0.0.0:
Binding loopback only would hide it from hub2.
Command:
ss -ltnH sport = :8000
One thing I actually understood today:
Rating:

06Part A

The keeper

A program you start by typing dies when you stop watching it. Something has to care whether it is alive.

Run python3 -m uvicorn app:app in a terminal and you have a working server — until you close the window, or the connection drops, or the machine reboots, or the process hits a bug and exits. A website that only exists while you are logged in is not a website.

systemd is the program that starts and supervises other programs. You describe what you want in a unit file; systemd is then responsible for making it true. Yours is short, and every line earns its place:

~/.config/systemd/user/guestbook.servicethe whole arrangement
[Service]
Type=simple
WorkingDirectory=/home/alice/projects/guestbook
ExecStart=/usr/bin/python3 -m uvicorn app:app --host 0.0.0.0 --port 8000
Restart=on-failure     ← bring it back if it crashes
RestartSec=2           ← but wait 2s, don't spin

[Install]
WantedBy=default.target  ← what "enable" hooks it onto

Two words get confused constantly, and the difference is worth learning once:

WordMeansSurvives a reboot?
startRun it now.No
enableRun it at boot from now on.Yes — but doesn't start it now
enable --nowBoth.Yes

Which explains the shape of the commands you ran: disable --now is “stop it, and don't bring it back at boot”, and enable --now is “start it, and do bring it back”. Two independent switches, one flag to set both.

These are user services — yours, not the system's, which is why every command has --user in it. Normally a user's services stop when they log out; lingering is enabled for your account, which tells systemd to keep them running anyway. Without it, your site would go down every time you closed your SSH session.

Try this one for real

This is the experiment worth doing, and it belongs in your terminal rather than in a button, because the whole point is that you do something destructive and watch it heal.

Open the observatory below and press Is the service running? Write down the MainPID and the NRestarts number. Then, in a terminal:

kill -9 $(systemctl --user show guestbook.service -p MainPID --value)

Now press the button again. The MainPID will be a different number and NRestarts will have gone up by one. Nothing you did brought it back — systemd noticed the process was gone and started a new one about two seconds later, which is Restart=on-failure and RestartSec=2 doing exactly what the file says.

The -9 is not decoration, and it is the real lesson here. A plain kill sends SIGTERM — a polite “please shut down” that uvicorn catches and obeys, closing tidily and exiting with status 0. systemd sees a program that was asked to stop and stopped, records Result=success, and does not restart it, because on-failure means on failure and quitting when asked is not a failure. Your site would simply stay down.

kill -9 sends SIGKILL, which cannot be caught, blocked or handled: the kernel stops the process where it stands. systemd records Result=signal, counts that as a failure, and brings it back. Two commands one character apart, opposite outcomes.

If you do try the plain kill to see it for yourself — worth doing — start it again with systemctl --user start guestbook.service.

Section Review Card

no. 06 of 08
Concept:
systemd keeps the process alive; the terminal doesn't.
start vs enable:
now, versus at every boot.
Proof:
kill -9 it — NRestarts goes up, a new PID appears.
One thing I actually understood today:
Rating:

PART B — SEEING IT

07Part B

The four commands you ran

You typed these before any of the words above had meanings. Here they are again, now that they do.

the swapwhat each line was for
$ cp ~/projects/guestbook/guestbook.service ~/.config/systemd/user/
put the description somewhere systemd looks

$ systemctl --user daemon-reload
re-read the unit files; it does not do this by itself

$ systemctl --user disable --now hello-web.service
stop the placeholder AND unhook it from boot - it has to let go of port 8000 first

$ systemctl --user enable --now guestbook.service
start the real app AND hook it onto boot - now it can claim the port

Reading them in order, nothing about this is really about web servers. Line one is filing a description. Line two is asking systemd to read its files again, which it does not do automatically because unit files are edited far more often than they are meant to take effect. Lines three and four are a handover of one number — port 8000 — from one process to another, in the only order that can work.

What did not happen is as instructive. No DNS record changed. No certificate was issued. Nothing on hub2 was touched. The route that sends alice.charliehub.net to this container's port 8000 was already there and never noticed the swap — from the proxy's side, the same port answered before and after. That is the indirection from section 01 paying off: you replaced the entire application without any other layer needing to know.

Rolling it back

The same two switches, in reverse. Nothing here is one-way:

systemctl --user disable --now guestbook.service
systemctl --user enable --now hello-web.service

One genuine difference to know about: the FastAPI app returns 404 for a bare directory URL where http.server used to show a file listing. Any folder with an index.html behaves identically; a folder you were browsing as a listing will not.

Section Review Card

no. 07 of 08
Concept:
The swap moved one port between two processes.
Untouched:
DNS, the certificate, and the proxy route.
Reversible:
Two commands, either direction.
One thing I actually understood today:
Rating:

08Part B

The observatory

Six buttons. Each one runs a real command on this machine and shows you what came back, just now.

Everything above is a claim. These are the claims being checked, live, against the running system — not a recording, not a mock-up. Press one and a request goes to this site's own API, which runs a command and returns its output. What was run is printed above each result — four of the six are commands you can paste straight into a terminal and compare; the other two say what they did, because reading a certificate is a few lines of Python rather than one command.

Read-only probes

checking…
Pick a probe above.

Note: every one of these only reads. Nothing here starts, stops, writes or deletes — and the next section explains exactly what stops it from doing so.

The pairing worth doing

Press What is holding port 8000? and then Is the service running?. The MainPID from the second is the process that owns the port in the first. Two different commands, asking two different parts of the system, agreeing about the same program — sections 05 and 06 meeting in one place.

If the buttons are dead

If the status pill reads API not running, port 8000 is being served by something without this API — most likely the static placeholder, if the swap has been rolled back. Everything above still reads correctly; only this panel needs the FastAPI service.


09Part B

Why these buttons are safe

A web page that runs commands on your server is the most dangerous thing in this whole issue. It is worth knowing precisely why this one isn't.

The obvious way to build the panel above is also a catastrophe. It looks like this, and versions of it exist in the wild:

never write thisnot once, not for testing
# the browser sends a command and the server runs it
subprocess.run(request.query_params["cmd"], shell=True)

That endpoint does not run your commands. It runs anyone's. Whoever can reach the page can read your files, take your keys out of ~/.env, and delete whatever they like — and shell=True means a single ; or backtick in the string turns one command into several.

What the real endpoint does instead: the browser never sends a command at all. It sends a key — a short word like service — which is looked up in a dictionary written by hand in the source file. If the key isn't already in that dictionary, the answer is 404 and no process is started.

~/projects/guestbook/probes.pythe menu, and the lookup
COMMANDS = {
    "service": {"label": "Is the service running?",
                 "argv": ["systemctl", "--user", "show", UNIT, …]},
    "dns":     {"label": "What does the name resolve to?",
                 "argv": ["getent", "hosts", HOSTNAME]},
    # ... four more, all written here, none built at runtime
}

if key not in COMMANDS:
    raise HTTPException(status_code=404)   ← before anything runs

subprocess.run(argv, shell=False, timeout=5, capture_output=True)

Look at what the caller can influence: which row of a list it selects. Nothing else. It cannot add a row, edit one, or pass an argument into one. And shell=False with a list of arguments means there is no shell present to interpret a semicolon, a quote or a backtick — those characters would simply be part of a filename that doesn't exist.

You have met this idea before

This is the same lesson as the guestbook's SQL, wearing different clothes. There, the message text went to Postgres as a parameter rather than being formatted into the query string — which is why Robert'); DROP TABLE was stored as a joke instead of executed as a command.

Here, the key goes to a lookup rather than to a shell. Both are the same principle: data must never be promoted into instructions. Nearly every serious security bug you will ever read about is some version of that line being crossed.

Three more limits sit behind the lookup, because “read-only” is a statement of intent and intent is not enforcement:

The honest boundary: this is safe because it is a menu. The moment you add a text box that lets a visitor type the command, every one of these protections is gone and you have built the catastrophe at the top of this section. If you ever want that, the answer is not to make the box careful — it is to not have the box.

Section Review Card

no. 08 of 08
Concept:
A fixed menu, never a string from the browser.
Same as:
SQL parameters — data is never promoted to instructions.
Enforced by:
404 first, shell=False, timeout, truncation.
One thing I actually understood today:
Rating: