- Fix UFW firewall to allow outbound traffic (was blocking all outbound) - Add HOST parameter support to shell Makefile target - Fix all ansible-lint errors (trailing spaces, missing newlines, document starts) - Add changed_when: false to check commands - Fix variable naming (vault_devGPU -> vault_devgpu) - Update .ansible-lint config to exclude .gitea/ and allow strategy: free - Fix NodeSource repository GPG key handling in shell playbook - Add missing document starts to host_vars files - Clean up empty lines in datascience role files Reviewed-on: #2
4.9 KiB
Timeshift Snapshot and Rollback Guide
Overview
Timeshift is a system restore utility that creates snapshots of your system before Ansible playbook execution. This allows you to easily rollback if something goes wrong during configuration changes.
How It Works
When you run a playbook, the Timeshift role automatically:
- Checks if Timeshift is installed (installs if missing)
- Creates a snapshot before making any changes
- Tags the snapshot with "ansible" and "pre-playbook" for easy identification
Usage
Automatic Snapshots
Snapshots are created automatically when running playbooks:
# Run playbook - snapshot created automatically
make dev HOST=dev02
# Or run only snapshot creation
make timeshift-snapshot HOST=dev02
List Snapshots
# List all snapshots on a host
make timeshift-list HOST=dev02
# Or manually on the host
ssh ladmin@192.168.20.28 "sudo timeshift --list"
Restore from Snapshot
# Restore from a specific snapshot
make timeshift-restore HOST=dev02 SNAPSHOT=2025-12-17_21-30-00
# The command will:
# 1. Show available snapshots if SNAPSHOT is not provided
# 2. Ask for confirmation before restoring
# 3. Restore the system to that snapshot
Manual Snapshot
# Create snapshot manually on host
ssh ladmin@192.168.20.28
sudo timeshift --create --comments "Manual snapshot before manual changes"
Manual Restore
# SSH to host
ssh ladmin@192.168.20.28
# List snapshots
sudo timeshift --list
# Restore (interactive)
sudo timeshift --restore
# Or restore specific snapshot (non-interactive)
sudo timeshift --restore --snapshot '2025-12-17_21-30-00' --scripted
Configuration
Disable Auto-Snapshots
If you don't want automatic snapshots, disable them in host_vars or group_vars:
# inventories/production/host_vars/dev02.yml
timeshift_auto_snapshot: false
Customize Snapshot Settings
# inventories/production/group_vars/dev/main.yml
timeshift_snapshot_description: "Pre-deployment snapshot"
timeshift_snapshot_tags: ["ansible", "deployment"]
timeshift_keep_daily: 7
timeshift_keep_weekly: 4
timeshift_keep_monthly: 6
Important Notes
Disk Space
- Snapshots require significant disk space (typically 10-50% of system size)
- RSYNC snapshots are larger but work on any filesystem
- BTRFS snapshots are smaller but require BTRFS filesystem
- Monitor disk usage:
df -h /timeshift
What Gets Backed Up
By default, Timeshift backs up:
- ✅ System files (
/etc,/usr,/boot, etc.) - ✅ System configuration
- ❌ User home directories (
/home) - excluded by default - ❌ User data
Recovery Process
-
Boot from recovery (if system won't boot):
- Boot from live USB
- Install Timeshift:
sudo apt install timeshift - Run:
sudo timeshift --restore
-
Restore from running system:
- SSH to host
- Run:
sudo timeshift --restore - Select snapshot and confirm
Best Practices
-
Always create snapshots before major changes
make timeshift-snapshot HOST=dev02 make dev HOST=dev02 -
Test rollback process before you need it
# Create test snapshot make timeshift-snapshot HOST=dev02 # Make a test change # ... # Practice restoring make timeshift-list HOST=dev02 make timeshift-restore HOST=dev02 SNAPSHOT=<test-snapshot> -
Monitor snapshot disk usage
ssh ladmin@192.168.20.28 "df -h /timeshift" -
Clean up old snapshots if needed
ssh ladmin@192.168.20.28 "sudo timeshift --delete --snapshot 'OLD-SNAPSHOT'"
Troubleshooting
Snapshot Creation Fails
# Check Timeshift status
ssh ladmin@192.168.20.28 "sudo timeshift --list"
# Check disk space
ssh ladmin@192.168.20.28 "df -h"
# Check Timeshift logs
ssh ladmin@192.168.20.28 "sudo journalctl -u timeshift"
Restore Fails
- Ensure you have enough disk space
- Check that snapshot still exists:
sudo timeshift --list - Try booting from recovery media if system won't boot
Disk Full
# List snapshots
sudo timeshift --list
# Delete old snapshots
sudo timeshift --delete --snapshot 'OLD-SNAPSHOT'
# Or configure retention in group_vars
timeshift_keep_daily: 3 # Reduce from 7
timeshift_keep_weekly: 2 # Reduce from 4
Integration with Ansible
The Timeshift role is automatically included in the development playbook and runs first to create snapshots before any changes are made. This ensures you always have a restore point.
# playbooks/development.yml
roles:
- {role: timeshift, tags: ['timeshift', 'snapshot']} # Runs first
- {role: base}
- {role: development}
# ... other roles
See Also
- Timeshift Documentation
- Ansible Vault Guide - For securing passwords
- Maintenance Guide - For system maintenance