Hosting a Tor Hidden Service Decently

Posted on Tue 03 January 2017 in Security

Background music ideas: (something... rap-rave?, deathcore, and... something else electropunkish)

These days people may need to publish stuff anonymously. Since many people fuck that up and I'm really, really bored, here's a guide on how not to be dread pirate roberts to host a hidden service decently. That should cover most things related to hosting your political blog in Trump-era and staying safe afterwards.

There are many guides on setting up the server itself but not much about the OPSEC related and required to run a hidden service, and that's the part you have to know before you start to do anything to stay safe. They also mostly cover very simple static sites, but dynamic apps require a whole other level of security, especially to stay hidden.
This is not a full guide for anyone to copy/paste blindly in a terminal, but more an example of good practices you can learn from and use.

This is targeted at the average power-user wanting to host a Tor hidden service properly, who want to stay as anonymous as possible including from an oppressive government, and who cannot invest too much time or money in the project.
Don't get me wrong, be sure you cannot make it fast and secure and keep your money. This is just a way I like because it's reasonably efficient on those three qualities.
The most advanced example here will be able to host with a dynamic app from an average 5$/month VPS without sacrificing security, along with information to do more.

This is obviously for educational purpose and I am responsible for NOTHING. And I am in no way telling you to do illegal shit.
If you get caught hosting illegal content, lost money, or fucked up something else, I can't and won't help you.
If you save the world, thank you, please send me an anonymous note and some bitcoins.

Note: I'd love to make this as a service, like a VPS provider with only Tor networking and no public IP access. Could anyone confirm me I wouldn't end up in jail? (France) (I don't mean a bulletproof "please-sue-us" host based on an island, a normal VPS provider respecting the law but selling basically Tor hosting)

TODO:

  • unprivileged containers: really painful at the moment because of systemd. Should be tried harder later.

First

Things you need before starting:

  • money (let's say 0.5BTC, you can do with less, but you don't want to move money after that, it will get dedicated to this project)
  • Tails for temporary things (moving bitcoins, random tor browsing...)
  • a clean and mostly unmonitored network access
  • a clean workstation to run your local VMs ("a newly installed PC with Qubes or VirtualBox and lots of RAM") ("the host")

A VPN is great if you can't trust your ISP or country, try the one linked on the left, it's reasonably safe and secure - I can guarantee that because I've made it - and it will give me work and money and keep me alive to write nice stuff.

As usual the secret to being good at this is to know very well your environment and tools, their features, safeties, limits, and potential flaws.
Most tools you can find work but are fragile and complicated; the way you use them is more important than the tool itself.

I'll give many on-topic information on this page but it's not even close to enough, merely tips or remainders.
Take your time and get familiar with these: (try them, read manuals, understand how they work)

  • Bitcoin:
    • wallets, addresses, transactions, clients, blockchain as a large public database of transactions
  • Tor:
  • Tails, Whonix, Qubes, or whatever OS you also use:
  • LXC, OpenVZ, KVM/Xen, ... depending in which you will choose for your server-side VMs:
    • how to run and manage VMs, how much isolation each provides
  • Anything else you may use.

The more you know and understand, the more you can defend against.

OPSEC crash course

First, love The Grugq. The Grugq teaches nice things that can save you. Learn until everything is obvious:

To sum it up as text and add my own part:

  • Don't say anything to anyone. As good as it feels, pride will fuck you up and your friendships.

  • You will have a new identity and a new associated VM. That's a wonderful tool to isolate everything. Consider it your portal to that identity and project.
    Everything linked to it must happen inside the VM, everything inside the VM must be related to this identity only.
    Perfect compartmentation is all you need right now, but since it's impossible we'll add layers to make any link useless and/or hard to follow.

  • Each communication between the two identities is a huge risk, aim for none.
    That's why you move money at the start and make it hard to link back. That's why you need a VM that exclusively use Tor.

  • Each communication from your secret identity is a variable risk, aim for none too.
    If someone doesn't really need to know something, it's better kept secret.
    Everything you say to anyone can and will be recorded and used against you. Think about every single word and piece of information you let go.
    If you don't want to say something, just say you don't know, and make that your default answer to any question.

  • Also remember: crime doesn't pay, mostly because money laundering is a frustrating pain in the ass. If that was your plan, it's a bad one and you really should give that up.
    If you get caught, doing crime for profit is way worse and will only add charges. It has very often been used to find criminals and nail people that were strongly suspected and couldn't have been convicted without looking at where their money came from.

  • Fixing OPSEC and erasing leaked information is incredibly difficult, expensive and time-consuming.
    Think hard before you send anything to anyone, from the start to the end. (If you think about killing someone you did it wrong.)

  • If you ever do a mistake and leak information, accept it and clean it quickly. Erase everything you can, keep quiet, don't ever admit it or even talk about it ever again; it would just repeat it.
    Just act like this thing never happened as far as you know. If anyone ask anything, you don't know: that's the best and only right reaction in nearly all possible cases. (works with a leaked information or a whole identity, with anyone, at anytime.)
    If it's necessary, you can and should drop everything and start over with a clean identity later. It would only get worse.

Be paranoid, and I mean close to the clinical definition. Always imagine everybody is possibly an enemy and they will all work together to get you, and remember that all the safeties you put in place can fail or be broken. Don't let anyone or any group have power over you, except the very few things that you need to trust and that defines the quality of your anonymity.

With this article you should be safe from all of them unless they work together:

  • People attacking and taking control of your site (and its host) or your remote ISP and remote government
  • The Tor network
  • VPN, if any, and its government
  • Your local ISP and local government

("government": regular law enforcement. Not all the NSA being specifically after you. If that's your threat I advise you to drop this article and become Putin's best friend.)

Local environment setup

For both the local workspace and remote server, we will use the only good enough architecture I found: Tor isolated as much as possible, the rest having no direct access to the Internet or any of your other stuff. It the way Whonix works, also what Tails tries to do in a more compact way.

Remotely, we will set it up ourselves to make it simple and lightweight and with as few useless parts as possible, and use less resources by using containers instead if needed.
Locally, you can do so too but since RAM and storage is cheaper on this side, Whonix is fine.

You should also know that it's possible to do the same with a router that transparently proxy any traffic through Tor and physically goes between a dedicated computer instead of the VM and the Internet.

VMs:
Install Whonix on your platform (VirtualBox, Qubes, ...) by following their well detailed guides on https://www.whonix.org/.
You can replace the Workstation with any OS you're comfortable with, but use the same networking settings you can find on their docs. You don't have to configure any proxy on the workstation, it's all transparently going through Tor on the Gateway.

After installation you should always make sure the Gateway has a NAT and internal network, the Workstation has only an internal network, and it's currently using Tor https://check.torproject.org/.

Some platforms (like VirtualBox) keeps logs. Those could be correlated to your online presence, so you should disable logging or delete them regularly or switch to a platform more adapted to your activity.

You should really fully encrypt your host's disk, and each VM's disk using full disk encryption inside the VMs (better).

If you're using a VPN on the host, make sure to use a NATed interface, or if using a bridge to connect it to the VPN interface. Using a bridge on the physical interface would bypass the VPN.

Password manager:
That's also a necessity for your normal identity, but it's especially important here. It would be a terrible idea to reuse your passwords, and even more across identities, so get a password manager to put all your new secrets in.
An offline one would be best, so you can keep that database safe in your work VM.

The password manager will not only help you keep your passwords unique and secure, but also keep a list of accounts you have. Use it to remember where your identity is present, what accounts to quickly make disappear if needed, and on which sites you have access, power or assets stored.

Identity & Preparations

At this point, everything should be done and stored inside that Tor-only work VM, unless stated otherwise.
Shit's getting real. (on that track already?)

Identity

This part can be really boring. I really don't expect anyone to actually look up this information, so let's make it quick.
http://www.fakenamegenerator.com/

This will be mostly useful for two things:

  • pay your servers without looking sketchy
  • get a consistent fake identity.

All those years being a closeted trans person or playing RPGs all day will finally be useful for something, yay!

If you're more determined you can make a better fake life with social media profiles and make friends and all that stuff. You can also make multiple of them to make everything even more complicated and hard to follow. But we won't use them for much in this guide, so a name and location can be enough.
Save all that in a file somewhere safe so you don't end up wondering what your name is or where you live in the middle of a conversation.

To make more random differences from the real you, think about changing and making up habits, like:

  • Favorite tools (if you like Vim, try emacs or nano, same with browsers)
  • Writing style (favorite words or phrases, typos, emojis)
    For example, Deadpool's frequent use of "anyhow" in the 2016 movie could help identify Wade Wilson if he gave a fuck.
  • Times you are online (show another timezone or only appear once in a while)
  • Favorite music, movies, ...

All of these can be used to fingerprint an identity, and with enough time and data to match it to your real one. That identity is not you and should not be anything close to you.

A very simple example would be to tweet on how you fixed something on your .onion on Twitter. Even without any direct link, it could be used to confirm a link.
Another would be to use last.fm and talk about the song you're listening to with criminal friends. With just one undercover LEO or leaked logs and some data from last.fm you could reduce the set of suspects from hundred of millions to a few thousands or less. Compare time zones and twitter timelines and you're again much closer.
Fingerprinting can be very impressive and dangerous so avoid sending useful data. If you can't, mix it with misleading data.

Money

Your hidden identity will need money, to buy servers or just make the identity look real by paying common stuff.
You can either find a job with your fake identity, or send it some bitcoins.

To minimize the exchanges between identities, you will want to get there enough money to pay for every need of your project, including in the future. Think about servers for a long period, backups servers or storage, maybe a secure mail account, any other donation or expense, and the tumbler or conversion fee.

First, you need to find a list of tumblers, there are a few on reddit.

Bitcoin is anonymous, but traceable. It's not really pseudonymous, you never know who owns which address and for how long, and you can't tie up addresses together either without guessing.
So you will have to make the trace hard to follow and look like it changed owners a few times. Split the bitcoins, mix the bitcoins, as randomly and slowly as possible.
That's why we need tumblers: many people put their bitcoin on the same address, it gets mixed and sent randomly and in the end everyone gets other peoples' bitcoins out.
You of course have to trust the tumbler to not log transactions, and anyone can still compare the amounts of bitcoins each address sent and received.
We will use at least two as you never know which agency is operating them, and only the ones who can split between multiple outputs and randomize amounts. Otherwise, they're useless against basic blockchain analysis.

To relay money, boot up a Tails VM and create as many Electrum wallets as you need. Use one wallet per operation, to make sure no addresses are common between the different steps. Use different Electrum servers too. To keep your money between Tails sessions, write down the seeds somewhere, that's enough to import back your wallet at any time later.
Once it's finished, import the outputs in your work VM.

On a long timespan (weeks, months), with different tumblers, try to do something like that:

                                            ---> output 1
        +----> tumbler 1 -------> tumbler 2 ---> output 2
input --|
        +----> tumbler 2 -------> tumbler 1 ---> output 3

On each -->, do as much random mixing as you want (on the same branch, you don't want to make the tumbling useless), and let it sit there for some time. That's a basic thing but you can add outputs, inputs, layers, pauses, ...
Also that's not a scientifically proved scheme, it just looks complicated enough.

  • Do not mix two branches directly.
  • Do not pay the same thing with two outputs.
  • Do not send two outputs to the same tumbler input either.
  • If you need at least two outputs for one payment, tumble it again with the third tumbler.

Failure to follow these rules will create obvious patterns that will stand out if anyone tries to analyze the bitcoin flow.
Think like you will be analyzing that and remember tumblers have many inputs and many outputs. It should spread. If you see any two branches going back together in both ways it will look like the owner didn't really change and they're just trying to hide it. And that's a very good reason to continue looking.

You need something that doesn't accept Bitcoins?
https://wirexapp.com/ gives nice legit virtual credit cards with any name, with relatively high limits without requiring an ID, and you can directly send bitcoins on it. I've used it as a personal virtual credit card for some time, it works great everywhere.

Remote environment: Hosting & Server

Providers: There are many. Thousands of small, cheap, very shitty, sometimes acceptable, VPS providers, everywhere in the world.
https://lowendbox.com/ is a good place to start.

Requirements are very different for two categories of site you may want to host.

Static, meaning that you will just drop files somewhere and serve them. No PHP or Python behind doing dynamic things.
Any VPS will do, from 256MB RAM.

Dynamic, meaning that your site will be some kind of program running and doing things depending on stuff, especially user input. (PHP, or anything else)
That's much harder to host as your app can be compromised and an enemy can attack from the inside; so you have to isolate that app from your main server, so that even if compromised it stays contained and hidden. (again, layers. Even if your server is paid and accessed anonymously, you don't want it to get any attention on it.)

That requires virtualization or containers. Try to find bare metal dedicated server, or a Xen or KVM VPS. OpenVZ shares (provider) host kernel, if you really can't find better ask your provider if they support LXC (sometimes advertised as docker support) and dmcrypt.
VPS/Cloud providers that would support both: DigitalOcean, OVH, LeaseWeb, many more.

As said before, I'm aiming at a low-cost but secure result, so I'll try to make this fit in 512MB RAM (available around 5$ per month on most VPS providers) and use containers.

If you can get more RAM (2GB) and a bare metal server, you can use full VMs instead of containers (especially privileged containers). They will be much more secure. (but would result in poor performances nested inside another VM)
Security: LXC < OpenVZ < KVM/Xen
Cheapness: LXC > OpenVZ > KVM/Xen

Simple, static site hidden service setup

That's the most common thing, already tried and described many times, there's not much to say about that.
A simple web server listening on localhost with Tor relaying to it.

https://www.thecthulhu.com/setting-up-a-hidden-service-with-nginx/

It's often ignored but you can make nice websites without big dynamic CMS and no server interaction, and most common dynamic features can be reproduced client-side on a fully static site (see how Sphinx and Rustdoc handle searching, for instance).
Pelican, Jekyll and Lektor are good static sites generators that make it easy to manage your content and render it into a static site.

It's also one of most secure kinds of site you can imagine: (close to) no interaction = (close to) no attack surface.
The next step this way would be a Freenet Freesite for example, where the website just floats in the network identified by a public key.

If you really want to improve that you could put it entirely in a container or VM inside your host in the same way as the Tor node VM on the next part, to add a layer of isolation between the outside server and Tor/nginx, and encrypt your data without encrypting the whole host (it's the safest way to decrypt the inner VM through SSH and not through your provider's serial console, and the only way with many providers)

Advanced, isolated, dynamic app hidden service setup

(in this part I'm using interchangeably VM and container, they're used for the same thing.)

Target architecture :

                +------------------------------------------------+
                |    +-----------------+    +-----------------+  |
Internet  ------|-|-<| tor node        |>-->| one secret site |  |
                | |  |                 |    | (php, db, ...)  |  |
                | |  +----<nat  vm>----+    +--<isolated vm>--+  |
                | |                                              |
                | +> host ssh server, maybe a decoy http server  |
                +------------------<host server>-----------------+

This will ensure two very important things:

  • You can't guess the content (and existence of the hidden site) by probing the host from the Internet
  • You can't guess anything about the host by probing or attacking the app from Tor

Common attacks solved by this approach:

  • Compare SSH public key from the hidden service :22 and from the host :22, it's easy with tools like Shodan.
    Very common mistake if you want SSH access to your .onion.

  • HTTP port open on the host, letting people query the same pages than on the .onion, or related pages.
    It's less common because of how obvious it is but has happened anyway.
    How Silk Road was probably located

  • Data leaking out an app because it's using an external API not configured to use Tor as a proxy.
    This is extremely common and make the external API know the real location of you host. By isolating the VM with a transparent Tor router, that can't happen and everything running in the VM will go through Tor without even being aware of it.

  • Data leaking out of a compromised app and betraying the host's location and address, like with a remote code execution. Harder to predict and may happen at any time.
    Same as for the previous point, with this architecture you cannot bypass Tor without being able to exploit Tor or the hypervisor or kernel.

  • Co-hosting leaks (seen on ~25% of onions) are basically impossible when each site is contained in its own VM or container and has its own dedicated instance of nginx and/or any other software used.

Having your site and Tor node in a VM or container also lets you encrypt their disks even if your VPS provider does not support that, and you can enter the keys from your usual SSH shell.
It won't stop advanced attacks (full snapshot of the VPS, or logging the keys) but it's likely the first bad thing that could happen to your VPS is getting powered off and staying encrypted forever.

You can keep the Tor node as a simple process on the host too without doing anything too stupid, but I recommend putting it in a container or VM too. It will mostly make it very easy to set it up as a full Tor gateway that redirects any traffic from one interface to Tor, avoid giving the host direct access to the internal network (and giving the secret vm reach to any service potentially running on the host), and make it dependent on routing to stay up, rather than dependent on iptables rules to stay secure: your whole security is not annihilated by a script not running.

You should put each hidden service in its own VM or container. Not doing so will risk leaks in any software common to any two sites, and consequences to other sites from attacks to one that could also be used to link the two sites.
With containers it's an extremely resource-efficient way to do virtual hosting with no risk of leak except from Tor itself, and you could even run a second node.

Network configuration:

Internal interface: brint on LXC host, ethint on containers
Internal network prefix: 10.6.6.0/24
Tor node VM IP: 10.6.6.1
App VM IP: 10.6.6.2

> with LXC:
NAT'd interface: brnat on LXC host, ethnat on containers
NAT'd network prefix: 10.5.5.0/24
Host IP: 10.5.5.1
Tor node VM IP: 10.5.5.2

The next step is in two part - one for containers and one for VMs.

Installing the host [with containers, LXC]

Start with a clean Debian stable install (SSH server + standard system utilities). You can use any OS you feel comfortable with, I don't see any real benefit of Debian except my own experience with it and the overall stability. CentOS or Ubuntu Server would probably work at least as good.

Don't forget to change the root password after installation, if it was set through your provider.

We'll start by installing basic required packages, and creating our two containers.

apt install vim
apt install lxc iptables bridge-utils

lxc-create -t debian -n tornode
# > note root password!

lxc-create -t debian -n app
# > note root password!

Basic Debian hardening:

  1. Change the default SSH port:
    It won't make anything more resilient but will get your host ignored from all automated attacks and probing. It will keep your logs clean so you can spot any targeted attack.

    Edit /etc/ssh/sshd_config and set the Port to a less common port, like 222 or 1022.

    If that's your idea, don't even tell me "security by obscurity is bad", it's complete bullshit and you know it. It's a relatively efficient layer you shouldn't ignore because of baseless ideology.

  2. Disable password authentication: Set up a SSH keypair: SSH Key Auth
    Edit /etc/ssh/sshd_config and set PasswordAuthentication to no.
    That will solve the SSH brute-force problem and make authentication faster, safer and easier.

  3. Remove logs:
    Logs can be a very useful security feature and a very efficient evidence collection.
    It's not good if the evidence is against you. You can either:

    • Lower the log retention:
      Change stuff in /etc/logrotate.d/ to not keep anything longer than a day.
      Keep in mind that it will still be written on disk, so you should add the shred option to make logrotate use shred to delete the file by overwriting enough times to make it basically unrecoverable.
      Sadly it will not be that effective on a SSD, but it will still make it unrecoverable from the VM. (you can't just get it with strings /dev/sda)

    • Disable syslog

      systemctl stop rsyslog
      systemctl disable rsyslog
      
      # and for good measure
      shred -uvn1 /var/log/*.log*
      

Host network configuration:

Append to /etc/network/interfaces :

auto brnat
iface brnat inet static
    bridge_ports none
    address 10.5.5.1
    netmask 255.255.255.0

auto brint
iface brint inet manual
    bridge_ports none

Edit /var/lib/lxc/tornode/config and replace any lxc.network line with these:

lxc.network.type = veth
lxc.network.link = brnat
lxc.network.veth.pair = vethtornat
lxc.network.flags = up
lxc.network.name = ethnat
lxc.network.ipv4 = 10.5.5.2/24
lxc.network.ipv4.gateway = 10.5.5.1


lxc.network.type = veth
lxc.network.link = brint
lxc.network.veth.pair = vethtorint
lxc.network.flags = up
lxc.network.name = ethint
lxc.network.ipv4 = 10.6.6.1/24

Same for the app VM in /var/lib/lxc/app/config :

lxc.network.type = veth
lxc.network.link = brint
lxc.network.veth.pair = vethappint
lxc.network.flags = up
lxc.network.name = ethint
lxc.network.ipv4 = 10.6.6.2/24
lxc.network.ipv4.gateway = 10.6.6.1

Our host will act as a router for the Tor VM, so we will enable IP forwarding. Create /etc/sysctl.d/forward.conf with this content:

net.ipv4.conf.all.forwarding=1

Then apply it with sysctl -p /etc/sysctl.d/forward.conf It should confirm the new value.

Next, the firewall and NAT.

Create /etc/iptables4.up.rules: (replace the SSH port if you changed it earlier!)

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
# NAT for Tor node
-A POSTROUTING -s 10.5.5.2/32 -o eth0 -j MASQUERADE
COMMIT

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]

# Reject packets from wrong interface
-A INPUT ! -i lo -d 127.0.0.0/8 -j REJECT
# Accept once a connection is open, drop invalids
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
# Allow localhost
-A INPUT -i lo -j ACCEPT
# Clearnet interface: Allow SSH
-A INPUT -i eth0 -p tcp --dport 22 -j ACCEPT

-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
# Allow forwarding from Tor node
-A FORWARD -i brnat -o eth0 -s 10.5.5.2/32 -j ACCEPT
# Allow forwarding between VMs
-A FORWARD -i brint -o brint -j ACCEPT

# It's easier not to filter any OUTPUT here.
# We trust the host, and potentially untrusted apps are in a VM
# that is already filtered by the (not)FORWARD(ed) part.
COMMIT

And /etc/iptables6.up.rules:

*nat
:PREROUTING DROP [0:0]
:INPUT DROP [0:0]
:OUTPUT DROP [0:0]
:POSTROUTING DROP [0:0]
COMMIT

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
COMMIT

Try the new settings and make sure you can still SSH in afterwards:

iptables-restore < /etc/iptables4.up.rules
ip6tables-restore < /etc/iptables6.up.rules

If it's all good, create /etc/network/if-pre-up.d/iptables to load it right before the interfaces come up:

#!/bin/sh
/sbin/iptables-restore < /etc/iptables4.up.rules
/sbin/ip6tables-restore < /etc/iptables6.up.rules

And chmod +x that file:

chmod +x /etc/network/if-pre-up.d/iptables

Container filesystem encryption:

For LXC containers, we will encrypt disk images and mount them from the host. This approach is also useful to limit their total size.
To make it easier, I made a few scripts to automate the transfer and mounting:

Create a encrypt_container.sh:

#!/bin/bash
set -e
c="$1"
size="$2"

size_pattern='^[0-9]+$'

function usage() {
    echo "usage: ./encrypt_container.sh <container name> <max size in MB>"
}
if [ -z "$c" ]; then usage; exit 1; fi
if [ -z "$size" ]; then usage; exit 1; fi
if ! [[ "$size" =~ $size_pattern ]]; then echo "invalid size"; usage; exit 1; fi

set -v 

# https://wiki.archlinux.org/index.php/Dm-crypt/Device_encryption#Encryption_options_for_LUKS_mode
LUKS_OPTIONS="--cipher aes-xts-plain64 --key-size 256 --hash sha256 --iter-time 2000"

PLAIN_PATH="/var/lib/lxc/$c/rootfs.plain"
IMG_PATH="/var/lib/lxc/$c/rootfs.cryptimg"
ROOT_PATH="/var/lib/lxc/$c/rootfs"
DM_NAME="lxcroot_$c"
DM_DEV="/dev/mapper/$DM_NAME"

if [ -d $ROOT_PATH ] && [ ! -d $PLAIN_PATH ]; then
    mv $ROOT_PATH $PLAIN_PATH
    mkdir $ROOT_PATH
fi

if [ -d $IMG_PATH ]; then
    echo "$IMG_PATH already exists."
    exit 1
fi

# Create a sparce file for the encrypted FS
dd of=$IMG_PATH bs=1M count=0 seek=$size

cryptsetup -v $LUKS_OPTIONS --verify-passphrase luksFormat $IMG_PATH
cryptsetup open $IMG_PATH $DM_NAME

mkfs.ext4 $DM_DEV
mount $DM_DEV $ROOT_PATH
rsync -va $PLAIN_PATH/ $ROOT_PATH/ 

rm -rf /var/lib/lxc/$c/rootfs.plain

Create a open_container.sh:

#!/bin/sh
set -e
c="$1"
if [ -z "$c" ]; then echo "usage: ./open_container.sh <name>"; exit 1; fi
set -v

DM_NAME="lxcroot_$c"
cryptsetup open /var/lib/lxc/$c/rootfs.cryptimg $DM_NAME
mount /dev/mapper/$DM_NAME /var/lib/lxc/$c/rootfs

Create a close_container.sh:

#!/bin/sh
set -e
c="$1"
if [ -z "$c" ]; then echo "usage: ./close_container.sh <name>"; exit 1; fi
set -v

DM_NAME="lxcroot_$c"
umount /dev/mapper/$DM_NAME
cryptsetup close $DM_NAME

And run that to create a encrypted image and copy your containers' filesystem on it:

# Install required dependencies
apt install cryptsetup e2fsprogs rsync

# Move the old root, create and format and mount a new encrypted image,
# then rsyncs and remove the old root.
chmod +x encrypt_container.sh open_container.sh close_container.sh
./encrypt_container.sh tornode 4000
./encrypt_container.sh app 4000

encrypt_container.sh just did it for you, but on next reboot remember to run this before you start the containers with lxc-start:

./open_container.sh tornode
./open_container.sh app

Now to make encrypted backups of the containers, you can copy /var/lib/lxc/<name>/rootfs.cryptimg to a safe location while the container is stopped and after running ./close_container.sh <name>.

And that's it for the host setup.

You can start your containers and open a console on them that way:

# Start:
lxc-start -n tornode -d
lxc-start -n app -d

# Console:
lxc-console -n tornode
# <Ctrl-a q> to quit the console when you're done.
# <Ctrl-d> or `exit` first or it will stay logged in

If you later need to wipe the server, encrypted containers make it easy and safe.
Stop them, close the filesystems with ./close_container.sh <name> and execute for each container: shred -vun2 /var/lib/lxc/<NAME>/rootfs.cryptimg
This step will securely wipe them on HDDs, and make them a bit harder to recover on SSDs. On a VPS, it's likely there are backups or images copies everywhere, but that's why we encrypted them with a secure encryption key that must never be saved on the server itself.

Installing the host [with VMs, any hypervisor]

Getting an hypervisor:

  • Ready to use server, like ESXi or Proxmox. Nothing much more to do.
    Keep it secure and it will look just like any other dedicated virtualization server, that's good.
  • Your own Xen or KVM: A full hypervisor installation guide doesn't really belong here, but see the Debian wiki on KVM, Xen.

Creating the VMs:

  • Memory: The Tor node VM can work fine with as little as 256MB, 512MB can't hurt.
  • Disk: 4GB minimum for the Tor node VM, I recommend at 8GB if you can. Adapt the app VM's disk size to your app.
  • Network: For the installation ONLY, connect both to a NAT on the host to speed up the download and make network config easier with DHCP.
    That's usually the easiest way to access the Internet from a VM with any hypervisor and doesn't require an additional public IP address.

Installing Debian for both VMs:

Download and boot on the Debian netinst iso, downloadable here:
https://www.debian.org/CD/netinst/

  • Locale: That's mostly for the timezone, English/US is fine and generic.
  • Hostname: tornode and app
  • Domain: none
  • Partitioning:
    1. Create a /boot partition of 100M in ext4. (Bootable flag: on)
    2. Create a / partition of the rest in ext4.
    3. Go to Configure encrypted volumes and encrypt the / partition (should be /dev/sda2).
    4. The defaults should be sane, AES-256 on xts-plain64. AES-128 is fine too.
      Note: choosing a 256b key will effectively use a 512b key size in XTS mode and AES-256.
    5. Once back on the Partition disks menu, make sure the new encrypted device (sda2_crypt) has one large / partition in ext4.
    6. You will be asked about the lack of swap, just answer you like living on the edge.
  • Software selection: uncheck everything except SSH server and standard system utilities.
  • GRUB: Install it on your only device, should be /dev/sda.

Important: You connected both VMs to the NAT network for the installation. This is as insecure as possible to host a Tor hidden service, so do not miss this step: The app VM should only have an internal network interface.

  • Connect the Tor node VM to:
    • a host NAT, or bridge with a dedicated public IP;
    • an internal network;
  • Connect the app VM to:
    • the same internal network only.

We now have to configure networking. Open a console directly on your VMs and login as root on both.

Interfaces: First, we will rename interfaces to make configuration easier and less error-prone.
Look at your VM's configuration and which has received an IP address via DHCP in ip addr to recognize them and find their MAC address.

On the Tor node VM, create or edit /etc/udev/rules.d/70-persistent-net.rules and add:

SUBSYSTEM=="net", DRIVERS=="?*", ATTR{address}=="11:11:11:11:11:11", NAME="ethnat"
SUBSYSTEM=="net", DRIVERS=="?*", ATTR{address}=="22:22:22:22:22:22", NAME="ethint"

Replace 11:11:11:11:11:11 and 22:22:22:22:22:22 by your NAT interface and internal interface, respectively, and reboot. Run again ip addr and you should see the new names ethnat and ethint.
The system is freshly installed and doesn't have nice editors, so use nano or echo >>.

Repeat this step for the app VM, with only one line about that's about ethint.

IP addresses & DNS:

On the Tor node VM, replace /etc/network/interfaces by:

auto lo
iface lo inet lookback

allow-hotplug ethnat
iface ethnat inet dhcp

allow-hotplug ethint
iface ethint inet static
  address 10.6.6.1/24

And on the app VM, by:

auto lo
iface lo inet lookback

allow-hotplug ethint
iface ethint inet static
  address 10.6.6.2/24

And reboot again or run /etc/init.d/networking restart.

You should be able to ping 8.8.8.8 from the Tor node VM, 10.6.6.1 from the app VM, but not 8.8.8.8 from the app VM.

Installing the Tor node VM

We will start by installing the basic packages and finishing network configuration:

# the resolver imported from the host may not work here, but check without first
# it will only be used for this VM's upgrades anyway, the rest will go through Tor.
echo nameserver 8.8.8.8 > /etc/resolv.conf

# If this doesn't work, check your resolver and network configuration
apt update && apt upgrade

# Required packages
apt install iptables tor

# Useful tools (missing in a minimal Debian)
apt install iputils-ping dnsutils curl wget vim

Append the local Tor configuration to /etc/tor/torrc:

# Listen as a transparent proxy for the internal network
VirtualAddrNetworkIPv4 10.128.0.0/9
AutomapHostsOnResolve 1
TransPort 127.0.0.1:9040
TransPort 10.6.6.1:9040
DNSPort 127.0.0.1:53
DNSPort 10.6.6.1:53

# Serve a hidden proxy from the app VM
HiddenServiceDir /var/lib/tor/hidden_service/
HiddenServicePort 80 10.6.6.2:80

Restart Tor and get your newly generated hostname:

service tor restart

cat /var/lib/tor/hidden_service/hostname
# > note your shiny .onion hostname

Tor node VM iptables configuration:

Notice there is no IP forwarding or masquerading happening here. This VM is not a "real" router but a Tor node that redirects all TCP and DNS inputs from one interface to Tor. Tor itself will then communicate on the other interface making this box a router that safely and transparently puts everything through Tor, without the other VM being able to do anything about it.

/etc/iptables4.up.rules:

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
# redirect all DNS traffic to Tor's resolver
-A PREROUTING -i ethint -p udp --dport 53 -j REDIRECT --to-ports 53
# redirect all TCP traffic to Tor's TransPort
-A PREROUTING -i ethint -p tcp --syn -j REDIRECT --to-ports 9040
COMMIT

*filter
:INPUT DROP [0:0]
# allow localhost and replies to established connections
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
# allow DNS and TCP on TransPort
-A INPUT -i ethint -p udp --dport 53 -j ACCEPT
-A INPUT -i ethint -p tcp --dport 9040 -j ACCEPT
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT

/etc/iptables6.up.rules:

# You could setup an IPv6 address for this and manage the IPv6 stack too but...
# let's just ignore it and keep it simple.
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
COMMIT

As previously, put in /etc/network/if-pre-up.d/iptables:

#!/bin/sh
/sbin/iptables-restore < /etc/iptables4.up.rules
/sbin/ip6tables-restore < /etc/iptables6.up.rules

Then chmod +x and apply it. You don't have to test it before since we're not using SSH on the VM anyway.

chmod +x /etc/network/if-pre-up.d/iptables
/etc/network/if-pre-up.d/iptables

Installing the hidden service VM

Same as for the other VM, net and packages first:

# Use the Tor node as DNS resolver
echo nameserver 10.6.6.1 > /etc/resolv.conf

# If this doesn't work, check your internal network configuration and the Tor node
# It will be slow, it's expected, it's going through Tor.
apt update && apt upgrade

# Required packages (add what you need, like php5-fpm)
apt install iptables nginx

# Useful tools (missing in a minimal Debian)
apt install dnsutils curl wget vim

/etc/iptables4.up.rules:

*filter
:INPUT DROP [0:0]
# Accept anything on localhost, open connection replies, and incoming
# connection on port 80 (your hidden service).
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -i ethint -p tcp --dport 80 -j ACCEPT
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT

/etc/iptables6.up.rules:

# Block everything because we have an internal IPv4-only network, it has
# nowhere to go anyway.
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
COMMIT

/etc/network/if-pre-up.d/iptables:

#!/bin/sh
/sbin/iptables-restore < /etc/iptables4.up.rules
/sbin/ip6tables-restore < /etc/iptables6.up.rules

Then one last time, chmod +x and apply it.

chmod +x /etc/network/if-pre-up.d/iptables
/etc/network/if-pre-up.d/iptables

Now the base is done, you can install nginx or any other web server:

Put in /etc/nginx/sites-available/hidden_service:

server {
    listen 10.6.6.2:80;
    server_name <YOUR .ONION NAME>.onion;

    root /var/www/html;
}

The remove the default vhost and enable this one:

rm /etc/nginx/sites-enabled/default
ln -s /etc/nginx/sites-available/hidden_service /etc/nginx/sites-enabled/hidden_service
service nginx reload

At this point you should be able to go on that .onion host and see the nginx welcome page.

You now have the framework to host any Tor hidden service, including running safely anything dynamic like a PHP blog with comments or forum.

Obviously you still have to take care of the app's own security. In real VMs it would be pretty hard to get out of this one even with root, but with containers there is a lot more in common. I don't know of any way to evade LXC containers that work at the time of writing this, but it's more likely than a way to evade a real VM.
Anyway, root is still important and you should keep it secure as on any server.

Keeping it secure

  • Updates and patching: regularly run apt update && apt upgrade on all your VMs. (not too regularly or at the same time but you get the point)
    Don't forget the local VMs, and to download the most recent Tails image when available, and not just before you intend to use it.

  • If you have dynamic apps, keep it up to date and learn to secure it. That's the most important part, not only for your safety but for your site's and visitors'. Apart from nginx and PHP itself, it's the only surface that's likely to get targeted attacks and deserves all your attention, and usually has the poorest security in a web server.

  • Search for potential issues: OnionScan

  • Don't ever do memory snapshots. It will save in-memory encryption keys.
    Disk snapshots are great and easy for encrypted backups of individual VMs, but make sure it doesn't save memory too.

  • Have a plan for anything that could happen.

    • Watch your logs and know how to spot unusual behavior;
    • Have (daily, encrypted) backups (on another server) and be ready to restore them anytime;
    • Be ready to reinstall your app VM is needed;
    • Be ready to clean up everything and move to another server or provider if needed;