Skip to content

Commit 952b7f6

Browse files
titanismtitanism
committed
feat: redis -> valkey fixes, organized docs, fixed and tested backup scripts, added pm2 error emails and log rotation, added calendar events endpoints and tests (closes #455)
1 parent e67accf commit 952b7f6

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

85 files changed

+3294
-129122
lines changed

‎.env.defaults‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,15 @@
1+
# Postfix relay for server alerts
2+
POSTFIX_USERNAME=
3+
POSTFIX_PASSWORD=
4+
POSTFIX_RCPTS=
5+
16
SSL_CERT_PATH=
27
SSL_KEY_PATH=
38
SSL_CA_PATH=
49
BACKUP_SECRET=
510
MONGO_BACKUP_BUCKET=forwardemail-backups
611
REDIS_BACKUP_BUCKET=forwardemail-backups
7-
REDIS_DATA_DIR=/var/lib/redis
12+
REDIS_DATA_DIR=/var/lib/valkey
813

914
###################################
1015
## ignored self test domains ##
@@ -270,7 +275,6 @@ EMAILS_MONGO_URI="mongodb://{{EMAILS_MONGO_HOST}}:{{EMAILS_MONGO_PORT}}/{{EMAILS
270275
REDIS_TLS=false
271276
REDIS_USERNAME=
272277
REDIS_PORT=6379
273-
REDIS_TLS_PORT=6380
274278
REDIS_HOST=localhost
275279
# Generate a 64-character random password (Redis can handle long passwords)
276280
# openssl rand -base64 48

‎.env.schema‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,8 @@
1+
# Postfix relay for server alerts
2+
POSTFIX_USERNAME=
3+
POSTFIX_PASSWORD=
4+
POSTFIX_RCPTS=
5+
16
# MongoDB Configuration
27
MONGO_HOST=
38
MONGO_BACKUP_BUCKET=forwardemail-backups
@@ -282,7 +287,6 @@ EMAILS_MONGO_URI=
282287
REDIS_TLS=
283288
REDIS_USERNAME=
284289
REDIS_PORT=
285-
REDIS_TLS_PORT=
286290
REDIS_HOST=
287291
REDIS_PASSWORD=
288292
WEB_REDIS_TLS=

‎ansible/docs/DISASTER_RECOVERY.md‎

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
1-
# Disaster Recovery and Failover Guide
1+
# Disaster Recovery Guide
22

3-
This guide provides comprehensive procedures for recovering MongoDB and Redis databases from catastrophic failures and migrating to new servers with minimal downtime.
3+
> [!CAUTION]
4+
> This guide provides comprehensive procedures for recovering MongoDB and Redis databases from catastrophic failures and migrating to new servers with minimal downtime.
45
6+
> [!IMPORTANT]
7+
> Review and test these procedures regularly. Don't wait for an actual disaster!
58
## Table of Contents
69

710
1. [DNS TTL Configuration](#dns-ttl-configuration)
@@ -53,14 +56,18 @@ redis.example.com A 1.2.3.5 TTL: 300
5356
redis.example.com AAAA 2001:db8::2 TTL: 300
5457
```
5558
56-
**Important:** Set these TTL values **before** an outage occurs. Changing TTL during an outage will not take effect until the old TTL expires.
59+
> [!WARNING]
60+
> **Important:** Set these TTL values **before** an outage occurs. Changing TTL during an outage will not take effect until the old TTL expires.
5761
5862
---
5963
6064
## MongoDB Disaster Recovery
6165
6266
### Scenario 1: Complete Server Failure
6367
68+
> [!CAUTION]
69+
> **Critical scenario requiring immediate action**
70+
6471
**Symptoms:**
6572
- MongoDB server is completely unreachable
6673
- Hardware failure, data center outage, or catastrophic software failure
@@ -203,6 +210,9 @@ db.adminCommand({ fsyncUnlock: 1 })
203210

204211
### Scenario 1: Complete Server Failure
205212

213+
> [!CAUTION]
214+
> **Critical scenario requiring immediate action**
215+
206216
**Symptoms:**
207217
- Redis server is completely unreachable
208218
- Hardware failure or data center outage
@@ -230,11 +240,11 @@ sudo systemctl stop redis-server
230240

231241
# Restore encrypted backup
232242
aws s3 cp "s3://forwardemail-backups/${BACKUP_PATH}" - --endpoint-url="$AWS_ENDPOINT_URL" | \
233-
gpg --decrypt --batch --yes --passphrase "$BACKUP_SECRET" > /var/lib/redis/dump.rdb
243+
gpg --decrypt --batch --yes --passphrase "$BACKUP_SECRET" > /var/lib/valkey/dump.rdb
234244

235245
# Set correct permissions
236-
sudo chown redis:redis /var/lib/redis/dump.rdb
237-
sudo chmod 640 /var/lib/redis/dump.rdb
246+
sudo chown redis:redis /var/lib/valkey/dump.rdb
247+
sudo chmod 640 /var/lib/valkey/dump.rdb
238248

239249
# Start Redis
240250
sudo systemctl start redis-server
@@ -290,9 +300,9 @@ sudo systemctl stop redis-server
290300

291301
aws s3 cp "s3://forwardemail-backups/redis/LATEST_BACKUP.rdb.gpg" - \
292302
--endpoint-url="$AWS_ENDPOINT_URL" | \
293-
gpg --decrypt --batch --yes --passphrase "$BACKUP_SECRET" > /var/lib/redis/dump.rdb
303+
gpg --decrypt --batch --yes --passphrase "$BACKUP_SECRET" > /var/lib/valkey/dump.rdb
294304

295-
sudo chown redis:redis /var/lib/redis/dump.rdb
305+
sudo chown redis:redis /var/lib/valkey/dump.rdb
296306
sudo systemctl start redis-server
297307
```
298308

‎ansible/docs/MONGODB_OPERATIONS_GUIDE.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# MongoDB Operations Guide
22

3-
This guide covers advanced MongoDB operations including version upgrades, migrations, performance analysis, troubleshooting, and index optimization.
3+
> [!NOTE]
4+
> This guide covers advanced MongoDB operations including version upgrades, migrations, performance analysis, troubleshooting, and index optimization.
45
56
## Table of Contents
67

‎ansible/docs/README.md‎

Lines changed: 257 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,257 @@
1+
# Ansible Documentation
2+
3+
> [!NOTE]
4+
> This directory contains comprehensive documentation for deploying and managing the Forward Email infrastructure using Ansible.
5+
6+
## 📚 Table of Contents
7+
8+
- [Getting Started](#getting-started)
9+
- [Deployment Guides](#deployment-guides)
10+
- [Operations & Maintenance](#operations--maintenance)
11+
- [Performance Tuning](#performance-tuning)
12+
- [Disaster Recovery](#disaster-recovery)
13+
- [Security & Auditing](#security--auditing)
14+
15+
---
16+
17+
## 🚀 Getting Started
18+
19+
> [!IMPORTANT]
20+
> Before deploying any services, ensure you have:
21+
> - Ansible 2.9+ installed
22+
> - SSH access to target servers
23+
> - Required environment variables configured
24+
> - SSL/TLS certificates ready
25+
26+
### Quick Start
27+
28+
```bash
29+
# 1. Install Ansible dependencies
30+
ansible-galaxy install -r ansible/requirements.yml
31+
32+
# 2. Configure environment variables
33+
export [email protected]
34+
export POSTFIX_PASSWORD=<secure_password>
35+
export [email protected]
36+
37+
# 3. Deploy security baseline
38+
ansible-playbook ansible/playbooks/security.yml -i hosts.yml
39+
40+
# 4. Deploy your services
41+
ansible-playbook ansible/playbooks/node.yml -i hosts.yml
42+
```
43+
44+
---
45+
46+
## 📖 Deployment Guides
47+
48+
### Database Deployment
49+
50+
**[MongoDB & Redis/Valkey Deployment Guide](README_MONGO_REDIS.md)**
51+
52+
Complete guide for deploying MongoDB v6 and Valkey (Redis fork) with:
53+
- ✅ SSL/TLS encryption
54+
- ✅ UFW firewall configuration
55+
- ✅ Automated backups to Cloudflare R2
56+
- ✅ Email alerting system
57+
- ✅ Security hardening
58+
59+
> [!TIP]
60+
> Start here if you're deploying database services for the first time.
61+
62+
### Mail Server Deployment
63+
64+
**[Mail Server Deployment Guide](MAIL_DEPLOYMENT.md)**
65+
66+
Step-by-step guide for deploying SMTP, IMAP, POP3, and other mail services:
67+
- 📧 SMTP server configuration (ports 25, 587, 465, 2525, 2587, 2465, 2455, 2555)
68+
- 📬 IMAP server setup (ports 993, 2993)
69+
- 📮 POP3 server setup (ports 995, 2995)
70+
- 🔐 TLS/SSL certificate management
71+
- 🛡️ Security best practices
72+
73+
> [!WARNING]
74+
> Mail servers require proper DNS configuration (MX, SPF, DKIM, DMARC) before deployment.
75+
76+
---
77+
78+
## 🔧 Operations & Maintenance
79+
80+
### MongoDB Operations
81+
82+
**[MongoDB Operations Guide](MONGODB_OPERATIONS_GUIDE.md)**
83+
84+
Comprehensive operational procedures including:
85+
- 🔄 Backup and restore procedures
86+
- 📊 Monitoring and health checks
87+
- 🔍 Query optimization
88+
- 🗄️ Index management
89+
- 📈 Capacity planning
90+
- 🚨 Troubleshooting common issues
91+
92+
> [!NOTE]
93+
> This guide covers day-to-day MongoDB administration tasks.
94+
95+
### Service User Management
96+
97+
**[Service User Audit](SERVICE_USER_AUDIT.md)**
98+
99+
Documentation of service users and their permissions:
100+
- 👤 User roles and responsibilities
101+
- 🔑 Permission matrices
102+
- 📁 File ownership guidelines
103+
- 🔒 Security considerations
104+
105+
---
106+
107+
## ⚡ Performance Tuning
108+
109+
### MongoDB Performance
110+
111+
**[MongoDB Performance Tuning Guide](MONGODB_PERFORMANCE_TUNING.md)**
112+
113+
Optimize MongoDB for production workloads:
114+
- 🎯 WiredTiger cache configuration
115+
- 💾 Memory allocation strategies
116+
- 🔄 Connection pool tuning
117+
- 📊 Query performance optimization
118+
- 🗂️ Index strategies
119+
- 💿 Storage engine tuning
120+
121+
> [!TIP]
122+
> Apply these optimizations after initial deployment and load testing.
123+
124+
### Redis/Valkey Performance
125+
126+
**[Redis Performance Tuning Guide](REDIS_PERFORMANCE_TUNING.md)**
127+
128+
Maximize Redis/Valkey performance:
129+
- 🚀 Memory optimization
130+
- ⚡ I/O threading configuration
131+
- 🔄 Persistence strategies
132+
- 📈 Monitoring and metrics
133+
- 🎯 Eviction policies
134+
- 🔧 Kernel parameter tuning
135+
136+
---
137+
138+
## 🆘 Disaster Recovery
139+
140+
**[Disaster Recovery Guide](DISASTER_RECOVERY.md)**
141+
142+
Complete disaster recovery procedures:
143+
- 💾 Backup strategies and schedules
144+
- 🔄 Restore procedures
145+
- 🚨 Incident response workflows
146+
- 📋 Recovery checklists
147+
- 🧪 Testing procedures
148+
- 📞 Escalation paths
149+
150+
> [!CAUTION]
151+
> Review and test disaster recovery procedures regularly. Don't wait for an actual disaster!
152+
153+
### Backup Schedule
154+
155+
| Service | Frequency | Retention | Storage |
156+
|---------|-----------|-----------|---------|
157+
| MongoDB | Every 6 hours | 30 days | Cloudflare R2 |
158+
| Redis/Valkey | Every 6 hours | 30 days | Cloudflare R2 |
159+
| System configs | Daily | 90 days | Cloudflare R2 |
160+
161+
> [!NOTE]
162+
> Backups older than 7 days are consolidated to one per day to save storage space.
163+
164+
---
165+
166+
## 🔒 Security & Auditing
167+
168+
### Email Alerting System
169+
170+
All critical system events are monitored and reported via email:
171+
172+
- 🚫 **fail2ban** - IP ban notifications
173+
- 📦 **unattended-upgrades** - System update alerts
174+
- 💾 **MongoDB backups** - Backup failure alerts
175+
- 💾 **Redis backups** - Backup failure alerts
176+
- 🔴 **PM2 errors** - Application crash notifications
177+
- ⚠️ **systemd failures** - Service failure alerts
178+
179+
> [!IMPORTANT]
180+
> Configure `POSTFIX_RCPTS` environment variable to receive alerts.
181+
182+
### Rate Limiting
183+
184+
Email alerts are rate-limited to prevent flooding:
185+
- **Limit**: 10 emails per hour per service
186+
- **Tracking**: JSON-based in `/var/lib/email-rate-limits/`
187+
- **Logging**: All rate limit events logged to syslog
188+
189+
---
190+
191+
## 🔗 Related Resources
192+
193+
### External Documentation
194+
195+
- [MongoDB Official Documentation](https://docs.mongodb.com/)
196+
- [Redis Documentation](https://redis.io/documentation)
197+
- [Valkey Documentation](https://valkey.io/docs/)
198+
- [Ansible Documentation](https://docs.ansible.com/)
199+
- [PM2 Documentation](https://pm2.keymetrics.io/docs/)
200+
201+
### Ansible Roles Used
202+
203+
- `trfore/ansible-role-mongodb-install` v3.0.5 - MongoDB installation
204+
- `hifis.toolkit` collection v6.2.2 - System hardening and unattended upgrades
205+
206+
> [!NOTE]
207+
> Valkey (Redis fork) is installed directly via APT packages without using Galaxy roles for maximum control and compatibility.
208+
209+
---
210+
211+
## 📝 Document Conventions
212+
213+
Throughout this documentation, you'll see these GitHub-style alerts:
214+
215+
> [!NOTE]
216+
> General information and helpful context
217+
218+
> [!TIP]
219+
> Suggestions and best practices
220+
221+
> [!IMPORTANT]
222+
> Critical information that must be followed
223+
224+
> [!WARNING]
225+
> Potential issues or risks to be aware of
226+
227+
> [!CAUTION]
228+
> Dangerous operations that could cause data loss or downtime
229+
230+
---
231+
232+
## 🤝 Contributing
233+
234+
When adding new documentation:
235+
236+
1. Use GitHub-style markdown alerts for important information
237+
2. Include practical examples and commands
238+
3. Add cross-references to related documents
239+
4. Keep the table of contents updated
240+
5. Test all commands before documenting them
241+
242+
---
243+
244+
## 📧 Support
245+
246+
For questions or issues:
247+
248+
1. Check the relevant guide in this directory
249+
2. Review the troubleshooting sections
250+
3. Check application logs: `/var/log/pm2/`, `/var/log/mongodb/`, `/var/log/redis/`
251+
4. Contact the infrastructure team
252+
253+
---
254+
255+
**Last Updated**: November 25, 2025
256+
**Version**: 2.0.0
257+
**Maintained By**: Forward Email Infrastructure Team

0 commit comments

Comments
 (0)