John Newton's blog

☰

Published: November 10, 2024

Tags: linux, lxc

Dev Environments with LXC and Cloud-init

Recently, I have started using LXC containers, managed by LXD, as development environments. In this post I’ll explore the rationale for this type of setup and provide an example that can be copied and tweaked as needed. The following gist can also be referenced: https://gist.github.com/eternal-turtles/f3d3b2eccc466012a561ad67ddd95d96.

Rationale

In case you’re not familiar, LXC containers, being system containers, are unlike OCI app containers such as Docker or Podman, in that they are persistent (by default) and do not rely on a single process entrypoint. This means you can treat the containers similarly to virtual machines, except that they are much more lightweight because they share the host kernel rather than emulating hardware.

I have a sort of omnibus development container with a number of programming language runtimes installed, some installed through ASDF and others through the system package manager. It’s also of course possible to run a single service per container, and the only difference would be the cloud-init configuration and perhaps container networking configuration.

Reasons why this setup might make sense for you:

Reasons why you might pass on this sort of setup:

LXD installation

Install snap, enable the snapd service, then install and initialize LXD.

On Fedora, this looks like:

sudo dnf install -y snapd
sudo systemctl enable --now snapd.socket
sudo snap install lxd
sudo newgrp lxd
sudo usermod -aG jnewton
lxd init --minimal

Logout and then log back in to be associated with the new ‘lxd’ group.

Container up

You’ll need to modify the following script slightly. Replace jnewton with your actual username, and substitute enp87s0 with the name of your actual physical network interface.

Explanation of the following script:

lxc init images:fedora/40/cloud dev
lxc config set dev cloud-init.user-data - < cloud-init.yml
lxc config set dev security.nesting true
lxc config set dev security.privileged true
lxc config set dev limits.cpu=4
lxc config set dev limits.memory=16GB
lxc config set dev raw.idmap "both 1000 1000"
lxc network attach enp87s0 dev eth0
lxc config device add dev dev-code disk source=/home/jnewton/code/dev path=/home/user/code
lxc start dev
lxc exec dev -- /bin/bash

cloud-init.yml

#cloud-config
package_upgrade: true
packages:
  - bash-completion
  - tree
  - openssh
  - postgresql-server
  - postgresql-contrib
  - redis
  - curl
  - git
  - awscli2
  - openssl
  - openssl-devel
  - libyaml-devel
  - ncurses
  - ncurses-devel
  - ncurses-compat-libs
  - perl
  - "@development-tools"
  - gmp-devel
  - autoconf
  - sbcl
  - ecl
  - podman
timezone: America/Los_Angeles
write_files:
  - path: /run/scripts/setup-asdf.sh
    content: |
      #!/bin/bash -e

      git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.14.1
      echo '. "$HOME/.asdf/asdf.sh"' >> ~/.bashrc
      echo '. "$HOME/.asdf/completions/asdf.bash"' >> ~/.bashrc
      source ~/.bashrc
      asdf update
      asdf plugin add ruby
      asdf plugin add nodejs
      asdf plugin add erlang
      asdf plugin add elixir
      asdf install ruby latest
      asdf install nodejs latest
      asdf install erlang latest
      asdf install elixir latest
      asdf global ruby latest
      asdf global nodejs latest
      asdf global erlang latest
      asdf global elixir latest
      npm install -g npm
      npm install -g yarn
    owner: root:root
    permissions: '0744'
  - path: /run/scripts/setup-qlot.sh
    content: |
      #!/bin/bash -e

      curl --proto '=https' --tlsv1.2 -LsSf https://qlot.tech/installer > /tmp/install-qlot
      cat /tmp/install-qlot
      chmod +x /tmp/install-qlot
      /tmp/install-qlot

      echo -e '\nexport PATH="/home/user/.qlot/bin:$PATH"' >> /home/user/.bashrc
  - path: /run/scripts/setup-haskell.sh
    content: |
      #!/bin/bash -e

      curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org > /tmp/get-ghcup
      cat /tmp/get-ghcup
      chmod +x /tmp/get-ghcup
      /tmp/get-ghcup
    owner: root:root
    permissions: '0744'
  - path: /run/scripts/setup-postgresql.sh
    content: |
      #!/bin/bash -e

      postgresql-setup --initdb --unit postgresql
      systemctl enable postgresql
      systemctl start postgresql
      sudo -u postgres createuser user
      echo "local   all    user    trust" >> /etc/postgresql/15/main/pg_hba.conf
      systemctl reload postgresql
    owner: root:root
    permissions: '0744'
  - path: /run/scripts/setup-redis.sh
    content: |
      #!/bin/bash -e

      systemctl enable redis
      systemctl start redis
    owner: root:root
    permissions: '0744'
runcmd:
  - chown -R user:user /home/user
  - chmod +x /run/scripts/setup-asdf.sh
  - chmod +x /run/scripts/setup-qlot.sh
  - chmod +x /run/scripts/setup-haskell.sh
  - chown user:user /run/scripts/setup-asdf.sh
  - chown user:user /run/scripts/setup-qlot.sh
  - chown user:user /run/scripts/setup-haskell.sh
  - cd /home/user && sudo -H -u user /run/scripts/setup-asdf.sh
  - cd /home/user && sudo -H -u user /run/scripts/setup-qlot.sh
  - cd /home/user && sudo -H -u user /run/scripts/setup-haskell.sh
  - /run/scripts/setup-postgresql.sh
  - /run/scripts/setup-redis.sh
users:
  - name: user
    gecos: User
    primary_group: user
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    groups: sudo
    shell: /bin/bash

Debugging

Bridged networking

Remove the eth0 interface and re-add, attaching to lxdbr0.

lxc stop dev
lxc config device remove dev eth0
lxc config device add dev eth0 nic nictype=bridged parent=lxdbr0 name=eth0

Container down

Stop the container: lxc stop dev.

Delete the container: lxc delete dev.

Other considerations