Server Infrastructure Zbigniew Kulesza zjk.pl

A warm welcome to my private, home-based FreeBSD/UNIX servers, Sieradz. Poland

Today is: 2026-07-10   Last update: [an error occurred while processing this directive]

[Polski]



PLEASE NOTE - all parameters provided on this website are NOT fiction, theoretical simulations, or ideal scenarios. This is an absolutely REAL, live environment that you are using while reading these words.
However, certain elements have been anonymized for security purposes.
If you doubt the authenticity of the presented data, rest assured that I usually understate the actual figures.
Furthermore - due to ongoing development work - in some components there may differ in their stage of deployment (I typically wait several months to ensure a specific technology performs stably).
Certain configuration elements are occasionally phased out - if in doubt, please inquire via email regarding the current status, as this website serves primarily an illustrative purpose rather than strict documentation. Concurrently, specific hardware parameters and historical events described in the history do hold documented value.
The diagrams were created independently or with the assistance of ChatGPT and Microsoft Copilot. The texts were originally written by myself and "refined" using ChatGPT. Translations were performed with the help of ChatGPT.
If you find any linguistic or technical errors - please report them and forgive the oversight; keeping this server room and other responsibilities running takes up a significant amount of my time ;P

Table of Contents: A. Project Philosophy: B. Infrastructure Parameters: C. Infrastructure Architecture D. The infrastructure consists of the following layers: D.1 Edge Layer – communication with the outside world D.1.1 Edge Description: D.2 Network Layer – pure transport infrastructure D.2.1 Network Description: D.3 Service & Application Layer – offered functionalities / services D.3.1 Mail System Description: D.3.2 Web Cluster Description: D.3.3 DNS Description: D.4 Storage & Data Layer – data storage and backend D.4.1 ZFS Description: D.4.2 MooseFS Cluster Description: D.4.3 SeaweedFS Cluster Description: D.4.4 Database Description: D.4.5 Backup Description: D.5 Client & User Layer – who utilizes the infrastructure D.6 Operations & Auxiliary Systems Layer – typically hidden in other homelabs, but showcased at zjk.pl D.7 Power & Reliability Layer – deserves its own dedicated layer at zjk.pl D.7.1 Power Layer Description: D.8 Summary: outstanding technologies and security

A. Project Philosophy:

Is it possible to build a professional Datacenter without a corporate cloud?

Nowadays, the web is dominated by large corporations, advertisements, and tracking scripts. However, the Internet can look different. For 24 years, a fully independent, private server infrastructure, zjk.pl, has been developed, which serves as proof that a free and secure digital world still exists. This is not just a regular computer sitting in a corner, but an advanced ecosystem based on the FreeBSD system.

For people looking for inspiration, computer science students, or technology enthusiasts, this page serves as a practical case study. It presents how Enterprise-class solutions function in a real environment:

Foundations and security: Implementation of the highest website security standards (HTTP/3, strict SSL/TLS headers), an independent and spam-resistant mail system, and applications isolated in secure containers (Jails).
High Availability (HA): A cluster architecture based on the CARP protocol and the HAProxy load balancer, eliminates downtime in service operations in the event of hardware failures.
Storage Management: A scalable and secure data space realized using the ZFS file system, as well as the distributed systems MooseFS and object-based SeaweedFS.

The following description is a unique record of the path – from a single server to a distributed architecture. This is technical documentation of how modern UNIX art functions in practice.

B. Infrastructure Parameters:

• service availability 24/7/365, full uninterruptible power supply (UPS)
  
• redundancy of selected services and infrastructure components,
• OPNsense HA (master/slave + CARP),
• two independent ISP connections,
• LACP/LAGG for critical network connections,
• physical and virtual servers,
• storage, backup, and monitoring,
• systematic hardware and software updates (continuous mode, reboots after essential updates regardless of uptime),
• ability to host own domains and services.

The above diagram and the description below are for illustrative purposes only (for security reasons).

C. Infrastructure Architecture

The following diagrams and descriptions are for illustrative purposes only for security reasons.

Stable Network and Distributed Space: Fault-Tolerant Architecture

True engineering begins where a single drive and a single server end. Building a reliable environment requires effective management of the growing volume of data, tying two independent ISPs, and ensuring the continuity of service operations, even in the event of a physical machine failure.

This chapter is a detailed description of the topology and structure of zjk.pl. It presents an architecture devoid of single points of failure (SPOF), where the network layer relies on OPNsense HA (CARP) firewalls and link aggregation via LACP/LAGG. A key element of this configuration is a highly scalable storage, realized based on MooseFS and object-oriented SeaweedFS. The following material documents how these advanced data storage and replication solutions function in the daily production practice of the cluster.


Description of the zjk.pl server room ecosystem: "ecosystem", how zjk.pl operates as a whole with an emphasis on the most important technologies.

D. The Infrastructure Consists of the Following Layers:

D.1 Edge Layer – communication with the world

• OPNsense HA (master/slave),
• CARP failover,
• two independent ISP connections,
• routing and automatic failover,
• firewall / NAT / filtering,
• HAProxy / reverse proxy,
• TLS termination,
• HTTP/3 / QUIC.

D.1.1 Edge Description:

a. the layer consists of two modems from different Internet Service Providers,
b. two computers with OPNsense - running CARP in a Master-Slave configuration. The diagram illustrates this system: 2 modems (Vectra and Netia), 2 x OPNsense,
c. but it extends to the next layer – the network: two switches (one managed and one unmanaged) and a WiFi access point.

D.2 Network Layer – pure transport infrastructure

• managed switch,
• backup switch / fallback,
• LACP / LAGG,
• VLAN / network segmentation,
• trunks and aggregation,
• WiFi access point,
• LAN network.

D.2.1 Network Description:

a. LAGG/LACP connects the servers with the managed switch at 2x1Gb/s,
b. however, in selected areas LAGG is also implemented in failover and loadbalance modes.

D.3 Service & Application Layer (Service Layer) - offered functionalities / services

• distributed Web servers,
• Sendmail mail, Dovecot,
• Nextcloud, Piwigo,
• WordPress,
• Jellyfin,
• Git / SVN / Forgejo,
• monitoring - including Monitorix, Mrtg, Ganglia, Symon,
• logging / syslog,
• DNS,
• application reverse proxy,
• service HA.

D.3.1 Mail System Description:

a. two mail servers mail.zjk.pl (mail1.zjk.pl) and mail2.zjk.pl, both running Sendmail + various security tools: rspamd, Mailscanner, spamassassin, and fetchmail along with mechanisms such as: DKIM, SPF, openDMARC, and LetsEncrypt keys.
b. at the client access layer, it uses dovecot with highly modern mdbox (and an interesting detail: the mail data operates on a MooseFS-mounted /usr/home with a separate directory for fast local indices).

D.3.2 Web Cluster Description:

Providing a description of the data flow:
a. data arrives at OPNsense,
b. from there, HAproxy routes it using the Round-Robin method with sticky connections to
c. 4 web servers (checking which one is currently available),
d. the 4 servers utilize directories shared on MooseFS (1 server, one directory),
e. or (test/backup mode) 4 SeaweedFS (similarly there: 1 directory, 1 filler),
f. cache - local to each server: opcache and the apache memory/diskcache module,
g. and redis - redis is complex, consisting of 6 machines with one selected as master by 6 sentinels - operational synchronization takes place via HAproxy on OPNsense, and the web servers utilize the selected master,
h. OPNsense adds the HTTP/3 protocol (in testing phase) at its output over the UDP protocol,
i. explanation - the directories on both MooseFS and SeaweedFS are actually a single MooseFS directory mounted on each web server, and similarly for SeaweedFS - there is one directory, but there are 4 separate caches storing copies of this directory on each computer individually.

D.3.3 DNS Description:

a. External DNS: 3 providers (home.pl, seohost.pl, porkbun.com) - entries include not only standard records, but also SPF, DKIM, DMARC, etc. - for several domains handled by the zjk.pl servers,
b. Internal DNS BIND9: 4 machines: one master + 3 slaves, describing the full diagram of the internal zjk.pl server room structure.

D.4 Storage & Data Layer - storage of data and backend

• ZFS + ZFS RAID,
• MooseFS,
• SeaweedFS,
• PostgreSQL cluster,
• MySQL cluster,
• backup,
• data replicas,
• object storage,
• snapshots and backup,

D.4.1 ZFS Description:

File system present mainly on data storage drives (meaning MooseFS utilizes ZFS drives, as do all archive and backup drives), in one pool type raidz2-0 – the equivalent of RAID 6 (tolerates the failure of 2 drives).
System drives with minor exceptions (on ZFS) are UFS2.

D.4.2 MooseFS Cluster Description:

a. 1 master – this is the Community Edition version,
b. however, 2 servers can assume the master role via a script,
c. 12 chunkservers,
d. approx. 30 mount points, among which it is worth noting – mounting the /usr/home directory for servers (which is quite a challenging matter to maintain), and the mount points are FreeBSD and Windows,
e. it is worth highlighting that there are HDD and SSD drive classes (triple replication) and a special class for 4 drives, which serve as local machines delivering files for the 4 web servers (quite unique).

D.4.3 SeaweedFS Cluster Description:

a. 3 masters,
b. 7 volume servers,
c. synchronized on 4 different ports separately in HAproxy on OPNsense,
d. 4 volume servers operate for web directories, the remaining ones are mount points with 2x and 3x replication, but separately for fast SSDs and standard HDDs - note - the 3 main volume servers have two HDD and SSD drives each, web volume servers have SSD drives, making a total of 10 drives,
e. SeaweedFS stores metadata in PostgreSQL - 3 separate servers operating in Hot Redundancy mode, with one selected as master - the selection and data transmission of the metadata database takes place through HAproxy running on OPNsense.

D.4.4 Databases Description:

a. PostgreSQL in Hot Redundancy mode: 3 servers with one selected as master, its selection takes place via HAproxy on OPNsense - with access to the database through a shared port on OPNsense - this database stores SeaweedFS metadata and among others, Bareos backup data,
b. MySQL - 5 servers in Cluster Replication mode, communicating with OPNsense to select a single master, while a shared port for data access is simultaneously exposed on OPNsense. MySQL mainly provides databases for the web servers,
c. note - databases are backed up using native tools as dumps with a retention period of at least 15 days.

D.4.5 Backup Description:

a. each of the 8 servers has two drives: a main SSD and an HDD with local dumps (created by my own script - monthly),
b. data is sent to a shared network drive (NFS, not MooseFS),
c. and from there to an external archive drive,
d. Bareos backups (classic: daily, weekly, and monthly) on separate NFS drives,
e. Urbackup - separate NFS drive
e. MooseFS and SeaweedFS drives are sent to the aforementioned external archive drive,
f. /etc and /usr/local/etc configuration backups - etckeeper (testing phase).

D.5 User and Client Layer (Client Layer) - who utilizes the infrastructure

• home computers,
• laptops,
• mobile devices,
• IoT devices,
• test systems,
• users' network devices,
• workstations.

D.6 Operations & Auxiliary Systems Layer (Operations Layer) – typically hidden in other homelabs, but showcased at zjk.pl

• monitoring,
• alerting,
• log aggregation,
• backup orchestration,
• automation,
• HA scripts,
• configuration synchronization,
• infrastructure documentation.

D.7 Power & Reliability Layer – deserves its own dedicated layer at zjk.pl

• UPS #1,
• UPS #2,
• 12 V DC distribution,
• battery buffer,
• PSU redundancy,
• power monitoring,
• OR-ing / Schottky,
• power failover,

D.7.1 Power Layer Description:

a. two Fideltronik KI PRO 2000 UPS units,
b. their output features a dual switch that detects power loss - meaning the power line selection is cascaded; if one UPS shuts down, the second one takes over the load,
c. power is delivered to two lines - one is the primary line to a 660 W 80Platinum power supply - providing the main 12 V power rail to all servers,
d. servers (all of them) have Picopsu power supplies on board (find the description on the Internet),
e. a second mini-ITX 80Platinum power supply generates a secondary 12 V power line,
f. both 12 V lines combine via Schottky diodes onto the power rail for the servers and other critical assets like OPNsense, switch, WiFi, etc. - so that power to them is always available,
g. a third high-power power supply generates 12 V, but only for the utility/desktop computers - so that they are isolated.

D.8 Summary: Outstanding Technologies and Security

• FreeBSD,
• OPNsense,
• HAProxy,
• MooseFS,
• SeaweedFS,
• PostgreSQL HA,
• MySQL Cluster,
  
• Redis,
• HTTP/2 and HTTP/3,
• SPF / DKIM / DMARC,
• security monitoring,
• systematic updates.

Historical name:

The former name "cookie server" (not to be confused with modern mini-ITX cases). The English name for subminiature computers is "biscuit computer". In Polish, "biscuit" translates to "herbatnik" or "sucharek" (also English "rusk"). Concurrently, it can be noted that "ciasteczko" is rather the English "cookie". However, in Polish, "herbatnikowy serwer" or "sucharkowy serwer" sounds rather awkward... Therefore, I ultimately left it as "ciasteczkowy serwer" (cookie server). More about these remarkably small, low-power-consuming little computers can be found in the "hardware" section. Welcome... PLEASE NOTE - the former "cookies" have been replaced with more modern constructions, but also with a very low power consumption - description also in the hardware section.


Internet connections

The infrastructure utilizes two independent operator connections to increase service availability.

1. Current Internet Service Provider - Vectra branch Sieradz - Vectra business connection
(since June 18, 2021): 600/60 Mb/s (which means 75 MB/s and 7.5 MB/s)
IP: 88.156.77.167 formerly 232
2. Second connection: Netia on Orange BSA, fiber optic
(since May 11, 2021): 1000/300 Mb/s (which means 125 MB/s and 37.5 MB/s) - Netia promotion
IP: 83.238.166.222

The server has its name registration at NASK, currently home.pl - officially: zjk.pl

Powrót na stronę główną - Informacje o stronie, prawa autorskie, legalność itd. tutaj Informacje o przetwarzaniu i ochronie danych osobowych, kontakt i zapytania itd. tutaj Prywatne serwery Zbigniewa Kuleszy zjk.pl. Aktualny dostawca Internetu - Vectra.pl, Wszelkie prawa zastrzeżone. Zespół redakcyjny zjk.pl: zjk7@wp.pl W sprawie treści i działania strony oraz w sprawie funkcjonowania i udostępniania treści na serwerach zjk.pl - kontakt z administratorem: webmaster@zjk.pl lub zjk7@wp.pl