Complete guide to setting up your own Linux Kernel mirror, configuring rsync synchronization, and establishing proper repository linkbacks to official kernel.org sources
Follow this comprehensive guide to establish a production-ready Linux Kernel mirror that serves your community while respecting kernel.org's guidelines.
Minimum 500GB SSD/HDD for kernel sources, plus 200GB growth buffer
1Gbps+ connection recommended for serving multiple simultaneous downloads
4+ CPU cores, 8GB+ RAM, Linux OS (Ubuntu 22.04+ or Debian 11+ recommended)
99.5%+ uptime commitment to be listed as official mirror
# Update system packages sudo apt update && sudo apt upgrade -y # Install rsync, nginx, and monitoring tools sudo apt install rsync nginx wget curl htop iotop -y # Install fail2ban for security sudo apt install fail2ban -y # Install certbot for HTTPS (Let's Encrypt) sudo apt install certbot python3-certbot-nginx -y
# Create base mirror directory sudo mkdir -p /var/www/kernel-mirror/pub/linux/kernel # Set proper permissions sudo chown -R www-data:www-data /var/www/kernel-mirror sudo chmod -R 755 /var/www/kernel-mirror # Create log directory sudo mkdir -p /var/log/kernel-mirror sudo chown www-data:www-data /var/log/kernel-mirror
Create a sync script to pull kernel sources from official mirrors:
# Create sync script
sudo nano /usr/local/bin/sync-kernel-mirror.sh
# Add the following content:
#!/bin/bash
# Linux Kernel Mirror Sync Script
# Syncs from kernel.org official mirrors
MIRROR_DIR="/var/www/kernel-mirror/pub/linux/kernel"
LOG_FILE="/var/log/kernel-mirror/sync-$(date +%Y%m%d-%H%M%S).log"
LOCK_FILE="/var/run/kernel-mirror-sync.lock"
# Check if sync is already running
if [ -f "$LOCK_FILE" ]; then
echo "Sync already in progress. Exiting." | tee -a "$LOG_FILE"
exit 1
fi
# Create lock file
touch "$LOCK_FILE"
# Start sync
echo "========================================" | tee -a "$LOG_FILE"
echo "Kernel Mirror Sync Started: $(date)" | tee -a "$LOG_FILE"
echo "========================================" | tee -a "$LOG_FILE"
# Sync from kernel.org (use closest official mirror)
rsync -avH --delete --delete-delay \
--timeout=300 \
--exclude="*.sign" \
--exclude="*.asc" \
--stats \
rsync://mirrors.edge.kernel.org/pub/linux/kernel/ \
"$MIRROR_DIR/" 2>&1 | tee -a "$LOG_FILE"
RSYNC_EXIT=$?
# Log completion
echo "========================================" | tee -a "$LOG_FILE"
echo "Sync Completed: $(date)" | tee -a "$LOG_FILE"
echo "Exit Code: $RSYNC_EXIT" | tee -a "$LOG_FILE"
echo "========================================" | tee -a "$LOG_FILE"
# Remove lock file
rm -f "$LOCK_FILE"
# Keep only last 30 days of logs
find /var/log/kernel-mirror -name "sync-*.log" -mtime +30 -delete
exit $RSYNC_EXIT
# Make script executable sudo chmod +x /usr/local/bin/sync-kernel-mirror.sh # Test the sync script sudo /usr/local/bin/sync-kernel-mirror.sh
Official kernel.org mirror policy requires:
# Create nginx configuration
sudo nano /etc/nginx/sites-available/kernel-mirror
# Add the following configuration:
server {
listen 80;
listen [::]:80;
server_name mirror.example.com; # Replace with your domain
root /var/www/kernel-mirror;
# Enable directory listing
autoindex on;
autoindex_exact_size off;
autoindex_localtime on;
# Logging
access_log /var/log/nginx/kernel-mirror-access.log;
error_log /var/log/nginx/kernel-mirror-error.log;
# Security headers
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
# Mirror information page
location = / {
return 301 /pub/linux/kernel/;
}
# Serve kernel files
location /pub/linux/kernel/ {
autoindex on;
# Cache headers for static files
expires 1h;
add_header Cache-Control "public, immutable";
}
# Block access to hidden files
location ~ /\. {
deny all;
access_log off;
log_not_found off;
}
# Bandwidth limiting (optional)
location ~* \.(tar\.xz|tar\.gz|tar\.bz2)$ {
limit_rate 10m; # Limit to 10MB/s per connection
}
}
# Enable the site
sudo ln -s /etc/nginx/sites-available/kernel-mirror /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl restart nginx
# Obtain SSL certificate sudo certbot --nginx -d mirror.example.com # Auto-renewal is configured automatically # Test renewal sudo certbot renew --dry-run
Your kernel mirror is now configured. Next steps:
Complete list of official kernel.org mirrors and repositories for syncing and linkbacks.
Kernel files are organized by major version number:
# Download kernel and signature wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.6.tar.xz wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.6.tar.sign # Import kernel developers' PGP keys gpg --locate-keys torvalds@kernel.org gregkh@kernel.org # Verify signature xz -cd linux-6.6.tar.xz | gpg --verify linux-6.6.tar.sign - # Expected output: "Good signature from..."
For security, always verify PGP signatures before deploying kernels in production. This ensures authenticity and prevents tampered downloads.
Maintaining proper attribution and linkbacks to kernel.org is essential for mirror operation and community standards.
All kernel mirrors must clearly indicate:
<!-- Add to your mirror's index.html or footer -->
<div class="mirror-attribution">
<h3>Mirror Information</h3>
<p>
This is an official mirror of the
<a href="https://www.kernel.org" target="_blank">Linux Kernel Archives</a>
hosted at kernel.org.
</p>
<p>
<strong>Official Source:</strong>
<a href="https://cdn.kernel.org/pub/linux/kernel/">
https://cdn.kernel.org/pub/linux/kernel/
</a>
</p>
<p>
<strong>Last Sync:</strong> <span id="last-sync">2024-10-24 14:30 UTC</span>
</p>
<p>
<strong>Mirror Maintainer:</strong>
<a href="mailto:mirror@example.com">mirror@example.com</a>
</p>
<p>
<strong>Geographic Location:</strong> United States (US-East)
</p>
</div>
Prevent search engines from indexing mirror content separately:
# /var/www/kernel-mirror/robots.txt User-agent: * Disallow: /pub/linux/kernel/ # Allow access to mirror information page Allow: / # Sitemap (if you create one) Sitemap: https://mirror.example.com/sitemap.xml # Crawl delay to reduce server load Crawl-delay: 10
If serving HTML pages, add canonical links pointing to kernel.org:
<!-- In HTML <head> section --> <link rel="canonical" href="https://www.kernel.org/" /> <!-- For specific kernel version pages --> <link rel="canonical" href="https://cdn.kernel.org/pub/linux/kernel/v6.x/" />
To: ftpadmin@kernel.org Subject: New Mirror Registration - mirror.example.com Mirror Details: --------------- Domain: mirror.example.com Geographic Location: New York, United States Bandwidth: 1 Gbps Protocols: HTTP, HTTPS, Rsync Sync Frequency: Every 6 hours Upstream Source: rsync://mirrors.edge.kernel.org/pub/linux/kernel/ Maintainer: Your Name Contact Email: mirror@example.com Organization: Example Organization Mirror URLs: HTTP: http://mirror.example.com/pub/linux/kernel/ HTTPS: https://mirror.example.com/pub/linux/kernel/ Rsync: rsync://mirror.example.com/kernel/ Server Specs: - 8 CPU cores - 16GB RAM - 1TB SSD storage - Expected uptime: 99.9% Notes: We commit to maintaining this mirror with regular syncs and prompt security updates. We will notify ftpadmin@kernel.org of any extended downtime or discontinuation of service. Thank you, Your Name
Setup automated syncing, monitoring, and alerting for production-ready mirror operations.
# Edit crontab sudo crontab -e # Add sync job - runs every 6 hours at :15 past the hour # (stagger timing to avoid thundering herd) 15 */6 * * * /usr/local/bin/sync-kernel-mirror.sh >> /var/log/kernel-mirror/cron.log 2>&1 # Daily cleanup of old logs (runs at 3 AM) 0 3 * * * find /var/log/kernel-mirror -name "sync-*.log" -mtime +30 -delete # Weekly disk space check (runs Sunday at 2 AM) 0 2 * * 0 /usr/local/bin/check-mirror-space.sh
# Create monitoring script
sudo nano /usr/local/bin/check-mirror-space.sh
#!/bin/bash
# Monitor disk space and send alerts
THRESHOLD=85
PARTITION="/var/www/kernel-mirror"
EMAIL="admin@example.com"
USAGE=$(df -h "$PARTITION" | awk 'NR==2 {print $5}' | sed 's/%//')
if [ "$USAGE" -gt "$THRESHOLD" ]; then
echo "WARNING: Disk usage on $PARTITION is at ${USAGE}%" | \
mail -s "Mirror Disk Space Alert" "$EMAIL"
fi
# Log disk usage
echo "$(date): Disk usage at ${USAGE}%" >> /var/log/kernel-mirror/disk-usage.log
# Make executable
sudo chmod +x /usr/local/bin/check-mirror-space.sh
# Create sync monitor
sudo nano /usr/local/bin/monitor-sync.sh
#!/bin/bash
# Monitor last sync time and alert if stale
MIRROR_DIR="/var/www/kernel-mirror/pub/linux/kernel"
LOG_FILE="/var/log/kernel-mirror/monitor.log"
EMAIL="admin@example.com"
MAX_AGE_HOURS=8 # Alert if no sync in 8 hours
# Find newest file in mirror
NEWEST_FILE=$(find "$MIRROR_DIR" -type f -printf '%T@ %p\n' | sort -n | tail -1 | cut -d' ' -f2-)
if [ -z "$NEWEST_FILE" ]; then
echo "ERROR: No files found in mirror directory" | tee -a "$LOG_FILE"
exit 1
fi
# Calculate age in hours
NOW=$(date +%s)
FILE_TIME=$(stat -c %Y "$NEWEST_FILE")
AGE_HOURS=$(( (NOW - FILE_TIME) / 3600 ))
echo "$(date): Last sync was $AGE_HOURS hours ago" >> "$LOG_FILE"
if [ "$AGE_HOURS" -gt "$MAX_AGE_HOURS" ]; then
echo "WARNING: Mirror sync is $AGE_HOURS hours old (max: $MAX_AGE_HOURS)" | \
mail -s "Mirror Sync Alert - Stale Data" "$EMAIL"
fi
# Make executable
sudo chmod +x /usr/local/bin/monitor-sync.sh
# Add to crontab (check every hour)
# 0 * * * * /usr/local/bin/monitor-sync.sh
# Install log analysis tool
sudo apt install goaccess -y
# Generate real-time HTML report
goaccess /var/log/nginx/kernel-mirror-access.log \
-o /var/www/kernel-mirror/stats.html \
--log-format=COMBINED \
--real-time-html
# Or generate static report daily
# Add to crontab:
# 0 4 * * * goaccess /var/log/nginx/kernel-mirror-access.log -o /var/www/kernel-mirror/stats.html --log-format=COMBINED
# Comprehensive health check
sudo nano /usr/local/bin/mirror-health-check.sh
#!/bin/bash
# Complete mirror health check
EMAIL="admin@example.com"
ERRORS=0
# Check nginx is running
if ! systemctl is-active --quiet nginx; then
echo "ERROR: Nginx is not running" | mail -s "Mirror Alert: Nginx Down" "$EMAIL"
((ERRORS++))
fi
# Check disk space
DISK_USAGE=$(df -h /var/www/kernel-mirror | awk 'NR==2 {print $5}' | sed 's/%//')
if [ "$DISK_USAGE" -gt 90 ]; then
echo "CRITICAL: Disk usage at ${DISK_USAGE}%" | mail -s "Mirror Alert: Disk Critical" "$EMAIL"
((ERRORS++))
fi
# Check if mirror is accessible
HTTP_CODE=$(curl -o /dev/null -s -w "%{http_code}" https://mirror.example.com/pub/linux/kernel/)
if [ "$HTTP_CODE" != "200" ]; then
echo "ERROR: Mirror not accessible (HTTP $HTTP_CODE)" | mail -s "Mirror Alert: Not Accessible" "$EMAIL"
((ERRORS++))
fi
# Check SSL certificate expiry
CERT_DAYS=$(echo | openssl s_client -servername mirror.example.com -connect mirror.example.com:443 2>/dev/null | \
openssl x509 -noout -dates | grep "notAfter" | cut -d= -f2 | xargs -I {} date -d {} +%s)
NOW=$(date +%s)
DAYS_LEFT=$(( (CERT_DAYS - NOW) / 86400 ))
if [ "$DAYS_LEFT" -lt 30 ]; then
echo "WARNING: SSL certificate expires in $DAYS_LEFT days" | \
mail -s "Mirror Alert: SSL Expiring" "$EMAIL"
fi
# Log results
echo "$(date): Health check completed - $ERRORS errors found" >> /var/log/kernel-mirror/health.log
exit $ERRORS
# Make executable
sudo chmod +x /usr/local/bin/mirror-health-check.sh
# Run every hour
# 30 * * * * /usr/local/bin/mirror-health-check.sh
Add these optimizations to your Nginx configuration:
# Add to /etc/nginx/nginx.conf in http block
# Worker processes (set to number of CPU cores)
worker_processes auto;
worker_rlimit_nofile 65535;
events {
worker_connections 4096;
use epoll;
multi_accept on;
}
http {
# Enable sendfile
sendfile on;
tcp_nopush on;
tcp_nodelay on;
# Keepalive
keepalive_timeout 65;
keepalive_requests 100;
# Gzip compression
gzip on;
gzip_vary on;
gzip_min_length 1024;
gzip_types text/plain text/css application/json application/javascript text/xml;
# File caching
open_file_cache max=10000 inactive=30s;
open_file_cache_valid 60s;
open_file_cache_min_uses 2;
open_file_cache_errors on;
# Connection limiting
limit_conn_zone $binary_remote_addr zone=addr:10m;
limit_conn addr 10; # 10 concurrent connections per IP
}