User Tools

Site Tools


projects:base-infrastructure

Base Infrastructure

Overview

The base needs infrastructure to be operational.
One key basic needs of every human beeing besides a room, power, water and Mate is of cause Internet

When we had the go to move into the room there was already a Internet connection and we are allowed to use it.
The three parties in the building should be segmented from each other.

The resulting infrastructure description should be generic and the concept usable by other Hackerspaces or even smaller companies.

Concept

The WAN port of the Fritzbox is unusable, because it is only usable for DSL. We have FTTB
Every party will get one port on the Fritzbox.

The fritzbox must stay at the base. It will be a shared environment, but it cannot be avoided for now.
The Qnap needs to be moved to the top floor.

New highlevel layout

                           +-------------+
                           |   Fritzbox  |
                           | 192.168.1.1 |
                           +-+----+----+-+
                             |    |    |
          +------------------+    |    +-----------------------+
          |                       |                            |
3rd floor |              2nd floor|                   1st+floor|
          |                       |                            |USB-NIC (vmbr1, external)
          |                       |                            |
+---------+---------+   +---------+---------+   +--------------+--------------------------------------+
|    WLAN Router    |   |    WLAN Router    |   |                                                     |
|   192.168.1.30    |   |   192.168.1.20    |   |               +------------------+------------------+
|        NAT        |   |        NAT        |   |               |192.168.1.10 vmbr1|192.168.1.11 vmbr1|
|  192.168.30.0/24  |   |  192.168.20.0/24  |   |    Proxmox    |     +------+     |     +------+     |
+-------------------+   +-------------------+   | 192.168.11.10 |    pfSense VM    |     rproxy01     |
                                                |     vmbr0     |     +------+     |     +------+     |
                                                |               |192.168.10.1 vmbr0|                  |
                                                |               |192.168.11.1 vmbr2|                  |
                                                |               +------------------+------------------+
                                                |                                                     |
                                                +-----------+---+-------------------------------------+
                                                            |   |
                            Virtual Servers (vmbr2, servers)|   |Onboard-NIC (vmbr0, Internal)
                                                            |   |
                                                 +----------+   +----------------+
                                                 |                               |
                                  192.168.10.0/24|                               |192.168.11.0/24
                                                 |                               |
                                        +--------+-------+               +-------+------+
                                        | Virtual Switch |               |    Switch    |
                                        +------+--+------+               +-----+---+----+
                                               |  |                            |   |
                                          +----+  +----+                 +-----+   +-----+
                                          |            |                 |               |
                               +----------+--+   +-----+-------+   +-----+------+  +-----+------+
                               |   auth VM   |   |  chat VM    |   |     AP     |  |  Freifunk  |
                               |192.168.10.11|   |192.168.10.12|   |192.168.11.3|  |            |
                               +-------------+   +-------------+   +------------+  +------------+

House segments

Fritzbox
192.168.1.0/24
This is the shared network and it belongs to the building itself.
The Fritzbox itself is managed by trinitor

Every floor can have 10 static IPs in the shared network
1st floor = 192.168.1.10 - 19
2nd floor = 192.168.1.20 - 29
3rd floor = 192.168.1.30 - 39
There is also a DHCP range, but it is more for emergency use.

/dev/base Structure

Our /dev/base port will be connected to a Proxmox server = our virtual environment.
This port will be dedicated to a VM where we have a firewall running. In this case a pfSense.
The firewall VM will have three virtual NICs.
1. WAN = Fritzbox = Internet
2. virtual switch = Server = vSwitch with no physical NICs attached. All VMs will be connected to the vSwitch. The traffic will stay inside the host.
3. physical switch = the /dev/base internal network

The firewall will protect the /dev/base infrastructure from the Internet and the other parties in the building.
The VMs are in a segmented VLAN to have them also controlled in a better way.

IP addresses

IP Hostname Comment
External
192.168.1.1 router fritzbox
Firewall
192.168.1.10 fw01 WAN = fritzbox connection
192.168.10.1 fw01 virtual switch
192.168.11.1 fw01 physical switch
Hardware
192.168.11.2 switch01
192.168.11.3 accesspoint01 radius AP
192.168.11.4 accesspoint01 freifunk AP
192.168.11.10 virt01 proxmox server
192.168.11.11 virt02 proxmox server
192.168.11.20 music01 RuneAudio RPi
Virtual Machines
192.168.1.11 rproxy01 nginx reverse proxy
192.168.10.11 auth01 UCS
192.168.10.12 chat01 rocket chat

Virtualisation (main)

There are some options (ESXi, XenServer, oVirt, …), but the simplest one seems to be Proxmox.
An old Dell Notebook with an additional USB NIC will be used for now.

Network

Interface Comment
Network Cards
eno1 internal NIC - connected to the /dev/base switch
enx9410… wireless (unused)
enx9ceb… USB NIC - connected to the Fritzbox
Bridges (virtual switches)
vmbr0 Internal Bridge, eno1, 192.168.11.10
vmbr1 External Bridge, enx9ceb, no IP assigned
vmbr2 Server Bridge, no NIC connected, no IP assigned

Other settings
vi /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet consoleblank=10" 

grub-mkconfig -o /boot/grub/grub.cfg

Management:
https://192.168.11.10:8006

Virtualisation (secondary)

There is also a second Proxmox server with only one NIC.
It is running on a MacMini and will only be turned on if needed to safe power.

Proxmox runs on Apple Hardware, but there are some driver issues.
Disable modules:

cat >> /etc/modprobe.d/pve-blacklist.conf <<EOF                               
blacklist pcspkr
blacklist b43
EOF

don't enable readondriver (which breaks to console)
vi /etc/defaults/grub

GRUB_CMDLINE_LINUX_DEFAULT="nomodeset debug"

update-grub

Management:
https://192.168.11.11:8006

Firewall

pfsense installed inside a VM
Default settings

Network Setup

Proxmox Bridge Proxmox device name Internal device nameInterface Name IP
vmbr1 net0 em0 WAN 192.168.1.10/24
vmbr0 net1 em1 LAN 192.168.11.1/24
vmbr2 net2 em2 SERVER 192.168.10.1/24

DHCP
enabled for LAN and SERVER
192.168.10.100-199
192.168.11.100-199

Firewall Rules

NAT
disabled on pfsense. The House Network is routed, so we could share resources (printers?)
Static routes are configured on the fritzbox for this reason

Management:
https://192.168.10.1
https://192.168.11.1

Auth

  • Univention Corporate Server (UCS) 4.2
  • Create new UCS domain
  • Oganization Name: devbase.org
  • FQDN: auth01.devbase.org
  • LDAP base: dc=devbase,dc=org
  • Components
    • Active Directory compatible Domain Controller (Samba)
    • Radius
  • Reboot
  • Mode: Server only

Structure:

  • Members and Guests can have accounts
  • admins have two accounts, one for normal usage and and dedicated admin account (prefix “admin-”)

Groups:

  • roles
    • devbase_role_admins
      • member of: Administrators, Domain Admins, Enterprise Admins, Schema Admins, DC Backup Hosts, Group Policy Creator Owners
      • members: admin-trinitor
      • Policy:cn=default-umc-all
    • devbase_role_guests
    • devbase_role_members
      • members: trinitor
    • devbase_role_orgas
      • members: trinitor, …
  • delegation groups
    • devbase_delegate_wireless_access
      • members: devbase_role_members
      • Radius: allow access enabled
    • devbase_delegate_rocketchat_access
      • members: devbase_role_members, devbase_role_guest
    • devbase_delegate_wiki_members
      • members: devbase_role_members
    • devbase_delegate_wiki_orgas
      • devbase_role_orgas

LDAP:

  • create service account (per system)
    • Lastname: s-systemname
    • Username: s-systemname
    • Options
      • Kerberos principal
      • POSIX account
      • Samba account
    • test
      • Install Apache Directory Studio
        • Hostname: auth01.devbase.org
        • Port: 389
        • Encryption: StartTLS
        • Bind DN: cn=s-systemname,cn=users,dc=devbase,dc=org
        • Bind Password: xxx

Management:
https://192.168.10.11

Wireless

The Access Point will broadcast three SSIDs.

/dev/base PSK WPA2/AES Personal fallback
/dev/base WPA2/AES Enterprise member network
/dev/guest WPA2/AES Personal guest only

For not both share the same subnet and are not segmented.
Advantage: might be easier to change the password, implement radius

Longterm there should be a Freifunk for guests

Segmenting the guest network means we have to use VLANs in Proxmox, pfsense, switch, and access point.
Without VLANs we can just take the cable from the AP and plug it into the Fritzbox.
With VLANs we have no easy fallback in case theres an issue with the Proxmox server

Access Point Config

SSID /dev/base
Version WPA2
Encryption AES
Radius Server IP 192.168.10.11
Radius Port 1812
Radius Password xxx
Group Key Update Period 0

UCS Config

vi /etc/freeradius/clients.conf

client 192.168.11.3 {
        secret=xxx
        shortname=ap01
}

univention-radius-check-access –username=trinitor

Chat (Rocket Chat)

  • Install Ubuntu 16.04 LTS
  • apt-get update ; apt-get dist-upgrade ; reboot
  • snap install rocketchat-server
  • systemctl enable snap.rocketchat-server.rocketchat-server.service
  • systemctl status snap.rocketchat-server.rocketchat-server.service

LDAP:

  • Administration → LDAP
    • Enable: yes
    • Login Fallback: True
    • Host: auth01.devbase.org
    • Port: 389
    • Encryption: StartTLS
    • CA Cert: —–BEGIN CERTIFICATE—– …
    • Reject: Unauthorized
    • Domain Base: dc=devbase,dc=org
    • Use Custom Domain Search
{"filter": "(&(objectCategory=person)(objectclass=user)(memberOf:1.2.840.113556.1.4.1941:=CN=devbase_delegate_access_rocketchat,CN=Groups,DC=devbase,DC=org)(sAMAccountName=#{username}))", "scope": "sub", "userDN": "cn=s-rocketchat,cn=users,dc=devbase,dc=org", "password": "xxx"}
  • enable LDAP user group filter: false
  • Username Field: sAMAccountName
  • Unique Identifier Field: objectGUID,ibm-entryUUID,GUID,dominoUNID,nsuniqueId,uidNumber,cn
  • Sync Data: True
  • Sync User Avatar: True
  • User Data Field Map: {“cn”:“name”, “userPrincipalName”:“email”}
  • Merge existing users: False
  • Import LDAP users: True

Login:
http://192.168.10.12:3000/

Reverse Proxy

There will be multiple services on the internal network with need certificates
Lets encrypt should be used when possible

port 80 will point to one server

Fritzbox configuration:

  • forward port 80 and 443 to 192.168.1.11

fw01.devbase.org configuration

  • Firewall → NAT
  • new
    • Interface: WAN
    • Destination: WAN address
    • Destination port rang: 3000 - 3000
    • Redirect target IP: 192.168.10.12
    • Redirect target port: 3000
  • Firewall → Rules → WAN
  • move newly created NAT rule to the right place in the ruleset
  • don't forget to save and activate

Server Configuration:

  • Install Ubuntu 16.04 LTS
  • vi /etc/network/interfaces
iface ens18 inet static
  address 192.168.1.11/24
  gateway 192.168.1.1
  dns-nameservers 192.168.1.1
  • apt-get install software-properties-common
  • add-apt-repository ppa:certbot/certbot
  • apt-get update
  • apt-get install certbot nginx
  • mkdir /var/www/chat_devbase_org
  • vi /etc/nginx/sites-available/chat_devbase_org
server {
    listen 80;
    server_name chat.devbase.org;
    index index.html index.htm;
    location / {
        alias /var/www/chat_devbase.org/;
    }
}

server {
    #client_max_body_size 80M;
    listen 443 ssl default_server;
    server_name chat.devbase.org;

    ssl          on;
    ssl_certificate /etc/letsencrypt/live/chat.devbase.org/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/chat.devbase.org/privkey.pem;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;
        proxy_pass http://192.168.1.10:3000;
        proxy_redirect off;
    }
}
  • ln -s /etc/nginx/sites-available/chat_devbase_org /etc/nginx/sites-enabled
  • systemctl restart nginx.service
  • certbot certonly –webroot -w /var/www/chat_devbase.org -d chat.devbase.org
  • systemctl restart nginx.service
  • echo “* 3 * * * root certbot renew” » /etc/crontab

VPN

Certificate

  • pfsense
  • System → Cert Manager
  • New CA
    • Create an internal CA
    • key length 4096
    • Digest Algorithm 256bit
    • common name fw01.devbase.org
  • ssh auth01.devbase.org
  • vi /etc/freeradius/clients.conf
client 192.168.11.1 {
	secret=xxx
	shortname=fw01
}
  • systemctl restart freeradius.service
  • System → User Manager
  • Authentiction Servers
    • Name: auth01.devbase.org radius
    • IP: 192.168.10.11
    • shared secret: xxx
  • VPN → OpenVPN
  • Wizard
  • Type: Radius
  • Radius Server: auth01.devbase.org radius
  • new server certificate
    • name: vpn.devbase.org
    • key length: 4096
  • Server Setup
    • Interface: WAN
    • Protocol: UDP
    • local port: 1194
    • Tunnel Network: 192.168.12.0/24
  • Firewall rules
    • Firewall Rule: checked
    • OpenVPN rule: checked
  • Install pfsense package openvpn-client-export
  • VPN → OpenVPN
  • Client Export
    • Host Name Resolution: Other
      • Hostname: vpn.devbase.org
    • Verify Server CN: Do not verify
    • Use Random Local Port: checked
  • export config
  • Rulebase for OpenVPN
    • allow * to LAN NET
    • allow * to Server NET
    • allow * to * 53 UDP
    • allow * to trusted sites
    • allow * to ICMP
    • block * to *

nextcloud

  • install Ubuntu 16.04 LTS
  • configure network
  • apt update && sudo apt upgrade
  • apt-get install apache2 mariadb-server libapache2-mod-php7.0
  • apt-get install php7.0-gd php7.0-json php7.0-mysql php7.0-curl php7.0-mbstring php7.0-intl php7.0-mcrypt php-imagick php7.0-xml php7.0-zip php7.0-ldap
  • mkdir /opt/install && cd /opt/install
  • tar xf nextcloud*.tar.bz2
  • mv nextcloud /var/www/
  • chown -R www-data.www-data /var/www/nextcloud/
  • cat >/etc/apache2/sites-available/nextcloud.conf «EOF
Alias /nextcloud "/var/www/nextcloud/"
 
<Directory /var/www/nextcloud/>
   Options +FollowSymlinks
   AllowOverride All
 
   <IfModule mod_dav.c>
     Dav off
   </IfModule>
 
   SetEnv HOME /var/www/nextcloud
   SetEnv HTTP_HOME /var/www/nextcloud
</Directory>
EOF
  * ln -s /etc/apache2/sites-available/nextcloud.conf /etc/apache2/sites-enabled/
  * mysql_secure_installation
    * mysql -u root -p
      * CREATE USER 'nextcloud'@'localhost' IDENTIFIED BY 's3cret';
      * CREATE DATABASE nextcloud;
      * GRANT ALL PRIVILEGES ON nextcloud.* TO 'nextcloud'@'localhost';
  * cat >> /etc/php/7.0/apache2/php.ini <<EOF
<code>
opcache.enable=1
opcache.enable_cli=1
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=10000
opcache.memory_consumption=128
opcache.save_comments=1
opcache.revalidate_freq=1
EOF
  • systemctl restart apache2.service
    • enter new user credentials for admin user
    • configure database
  • enable apps:
    • calendar
    • contacts
    • deck
    • tasks
    • LDAP user and group backend
      • Admin → LDAP
        • Advanced
          • Turn off SSL certificate validation = checked
        • Server
          • Server: ldaps:/ /auth01.devbase.org:636
          • User: cn=s-nextcloud,cn=users,dc=devbase,dc=org
          • Base DN: dc=devbase,dc=org
        • Users
          • persons
        • Login Attributes
          • LDAP user
        • Group
          • devbase_delegate_access_nextcloud

gitlab

gitlab_rails['ldap_enabled'] = false
gitlab_rails['ldap_servers'] = YAML.load <<-'EOS'
  main: # 'main' is the GitLab 'provider ID' of this LDAP server
    label: 'LDAP'
    host: 'auth01.devbase.org'
    port: 389
    uid: 'sAMAccountName'
    bind_dn: 'CN=s-gitlab,CN=Users,DC=devbase,DC=org'
    password: 'mylittlepassword'
    encryption: 'start_tls' # "start_tls" or "simple_tls" or "plain"
    verify_certificates: false
    active_directory: true
    allow_username_or_email_login: false
    lowercase_usernames: false
    block_auto_created_users: false
    base: 'CN=Users,DC=devbase,DC=org'
#     user_filter: ''
EOS
  • gitlab-ctl reconfigure
  • gitlab-rake gitlab:ldap:check

Music

Background music should not be interrupted by rebooting clients or phones leaving the building.
A dedicated client should play the music and controllable by all kind of devices.

  • download runeaudio for RPi
  • flash to SD card and boot
  • browse http://ip
    • settings
      • hostname: music01
      • airplay: on
      • airplay name: music01
      • UPnP: on
      • UPnP name: music01
  • ssh root@ip
    • passwd
    • cat » /etc/mpd.conf «EOF
audio_output {
   type            "httpd"
   name            "My HTTP Stream"
   encoder         "flac"       # optional, vorbis or lame
   port            "8000"
   bind_to_address "0.0.0.0"    # optional, IPv4 or IPv6
   quality         "5.0"        # do not define if bitrate is defined
   # bitrate         "128"      # do not define if quality is defined
   format          "44100:16:1"
   max_clients     "0"          # optional 0=no limit
}
EOF

Management:
http://music01.devbase.org

Ubiquiti Controller

To manage the Unfi WiFi access points a Controller is needed

  • Install Ubuntu 16.04 LTS
  • echo 'deb http://www.ubnt.com/downloads/unifi/debian stable ubiquiti' | sudo tee /etc/apt/sources.list.d/100-ubnt-unifi.list
  • apt-key adv –keyserver keyserver.ubuntu.com –recv 06E85760C0A52C50
  • apt-get install unifi
  • https:<ip>:8443 ==== Wiki ==== ==== Matrix - testing only ==== * Install Ubuntu 16.04 LTS * apt update && sudo apt upgrade * add-apt-repository https://matrix.org/packages/debian/ * wget https://matrix.org/packages/debian/repo-key.asc -O key * apt-key add - < key * apt update * apt install matrix-synapse python-matrix-synapse-ldap3 * Server Name: matrix.devbase.org * cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 34 | head -1 * vi /etc/matrix-synapse/homeserver.yaml * registration_shared_secret: your_random_string * systemctl enable matrix-synapse.service * systemctl start matrix-synapse.service * register_new_matrix_user -c /etc/matrix-synapse/homeserver.yaml https://localhost:8448 * name root * set password * make admin = yes Public DNS Recort for federation SRV matrix_tcp.devbase.org
projects/base-infrastructure.txt · Last modified: 2018-05-13 08:58 by trinitor