Some checks failed
CI / lint-and-test (push) Successful in 1m16s
CI / ansible-validation (push) Successful in 5m49s
CI / secret-scanning (push) Successful in 1m33s
CI / dependency-scan (push) Successful in 2m48s
CI / sast-scan (push) Successful in 5m46s
CI / license-check (push) Successful in 1m11s
CI / vault-check (push) Failing after 5m25s
CI / playbook-test (push) Successful in 5m32s
CI / container-scan (push) Successful in 4m32s
CI / sonar-analysis (push) Successful in 6m53s
CI / workflow-summary (push) Successful in 1m6s
- Fix UFW firewall to allow outbound traffic (was blocking all outbound) - Add HOST parameter support to shell Makefile target - Fix all ansible-lint errors (trailing spaces, missing newlines, document starts) - Add changed_when: false to check commands - Fix variable naming (vault_devGPU -> vault_devgpu) - Update .ansible-lint config to exclude .gitea/ and allow strategy: free - Fix NodeSource repository GPG key handling in shell playbook - Add missing document starts to host_vars files - Clean up empty lines in datascience role files Reviewed-on: #2
212 lines
4.9 KiB
Markdown
212 lines
4.9 KiB
Markdown
# Timeshift Snapshot and Rollback Guide
|
|
|
|
## Overview
|
|
|
|
Timeshift is a system restore utility that creates snapshots of your system before Ansible playbook execution.
|
|
This allows you to easily rollback if something goes wrong during configuration changes.
|
|
|
|
## How It Works
|
|
|
|
When you run a playbook, the Timeshift role automatically:
|
|
1. Checks if Timeshift is installed (installs if missing)
|
|
2. Creates a snapshot before making any changes
|
|
3. Tags the snapshot with "ansible" and "pre-playbook" for easy identification
|
|
|
|
## Usage
|
|
|
|
### Automatic Snapshots
|
|
|
|
Snapshots are created automatically when running playbooks:
|
|
|
|
```bash
|
|
# Run playbook - snapshot created automatically
|
|
make dev HOST=dev02
|
|
|
|
# Or run only snapshot creation
|
|
make timeshift-snapshot HOST=dev02
|
|
```
|
|
|
|
### List Snapshots
|
|
|
|
```bash
|
|
# List all snapshots on a host
|
|
make timeshift-list HOST=dev02
|
|
|
|
# Or manually on the host
|
|
ssh ladmin@192.168.20.28 "sudo timeshift --list"
|
|
```
|
|
|
|
### Restore from Snapshot
|
|
|
|
```bash
|
|
# Restore from a specific snapshot
|
|
make timeshift-restore HOST=dev02 SNAPSHOT=2025-12-17_21-30-00
|
|
|
|
# The command will:
|
|
# 1. Show available snapshots if SNAPSHOT is not provided
|
|
# 2. Ask for confirmation before restoring
|
|
# 3. Restore the system to that snapshot
|
|
```
|
|
|
|
### Manual Snapshot
|
|
|
|
```bash
|
|
# Create snapshot manually on host
|
|
ssh ladmin@192.168.20.28
|
|
sudo timeshift --create --comments "Manual snapshot before manual changes"
|
|
```
|
|
|
|
### Manual Restore
|
|
|
|
```bash
|
|
# SSH to host
|
|
ssh ladmin@192.168.20.28
|
|
|
|
# List snapshots
|
|
sudo timeshift --list
|
|
|
|
# Restore (interactive)
|
|
sudo timeshift --restore
|
|
|
|
# Or restore specific snapshot (non-interactive)
|
|
sudo timeshift --restore --snapshot '2025-12-17_21-30-00' --scripted
|
|
```
|
|
|
|
## Configuration
|
|
|
|
### Disable Auto-Snapshots
|
|
|
|
If you don't want automatic snapshots, disable them in `host_vars` or `group_vars`:
|
|
|
|
```yaml
|
|
# inventories/production/host_vars/dev02.yml
|
|
timeshift_auto_snapshot: false
|
|
```
|
|
|
|
### Customize Snapshot Settings
|
|
|
|
```yaml
|
|
# inventories/production/group_vars/dev/main.yml
|
|
timeshift_snapshot_description: "Pre-deployment snapshot"
|
|
timeshift_snapshot_tags: ["ansible", "deployment"]
|
|
timeshift_keep_daily: 7
|
|
timeshift_keep_weekly: 4
|
|
timeshift_keep_monthly: 6
|
|
```
|
|
|
|
## Important Notes
|
|
|
|
### Disk Space
|
|
|
|
- Snapshots require significant disk space (typically 10-50% of system size)
|
|
- RSYNC snapshots are larger but work on any filesystem
|
|
- BTRFS snapshots are smaller but require BTRFS filesystem
|
|
- Monitor disk usage: `df -h /timeshift`
|
|
|
|
### What Gets Backed Up
|
|
|
|
By default, Timeshift backs up:
|
|
- ✅ System files (`/etc`, `/usr`, `/boot`, etc.)
|
|
- ✅ System configuration
|
|
- ❌ User home directories (`/home`) - excluded by default
|
|
- ❌ User data
|
|
|
|
### Recovery Process
|
|
|
|
1. **Boot from recovery** (if system won't boot):
|
|
- Boot from live USB
|
|
- Install Timeshift: `sudo apt install timeshift`
|
|
- Run: `sudo timeshift --restore`
|
|
|
|
2. **Restore from running system**:
|
|
- SSH to host
|
|
- Run: `sudo timeshift --restore`
|
|
- Select snapshot and confirm
|
|
|
|
### Best Practices
|
|
|
|
1. **Always create snapshots before major changes**
|
|
```bash
|
|
make timeshift-snapshot HOST=dev02
|
|
make dev HOST=dev02
|
|
```
|
|
|
|
2. **Test rollback process** before you need it
|
|
```bash
|
|
# Create test snapshot
|
|
make timeshift-snapshot HOST=dev02
|
|
|
|
# Make a test change
|
|
# ...
|
|
|
|
# Practice restoring
|
|
make timeshift-list HOST=dev02
|
|
make timeshift-restore HOST=dev02 SNAPSHOT=<test-snapshot>
|
|
```
|
|
|
|
3. **Monitor snapshot disk usage**
|
|
```bash
|
|
ssh ladmin@192.168.20.28 "df -h /timeshift"
|
|
```
|
|
|
|
4. **Clean up old snapshots** if needed
|
|
```bash
|
|
ssh ladmin@192.168.20.28 "sudo timeshift --delete --snapshot 'OLD-SNAPSHOT'"
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Snapshot Creation Fails
|
|
|
|
```bash
|
|
# Check Timeshift status
|
|
ssh ladmin@192.168.20.28 "sudo timeshift --list"
|
|
|
|
# Check disk space
|
|
ssh ladmin@192.168.20.28 "df -h"
|
|
|
|
# Check Timeshift logs
|
|
ssh ladmin@192.168.20.28 "sudo journalctl -u timeshift"
|
|
```
|
|
|
|
### Restore Fails
|
|
|
|
- Ensure you have enough disk space
|
|
- Check that snapshot still exists: `sudo timeshift --list`
|
|
- Try booting from recovery media if system won't boot
|
|
|
|
### Disk Full
|
|
|
|
```bash
|
|
# List snapshots
|
|
sudo timeshift --list
|
|
|
|
# Delete old snapshots
|
|
sudo timeshift --delete --snapshot 'OLD-SNAPSHOT'
|
|
|
|
# Or configure retention in group_vars
|
|
timeshift_keep_daily: 3 # Reduce from 7
|
|
timeshift_keep_weekly: 2 # Reduce from 4
|
|
```
|
|
|
|
## Integration with Ansible
|
|
|
|
The Timeshift role is automatically included in the development playbook and runs first to create snapshots before any changes are made.
|
|
This ensures you always have a restore point.
|
|
|
|
```yaml
|
|
# playbooks/development.yml
|
|
roles:
|
|
- {role: timeshift, tags: ['timeshift', 'snapshot']} # Runs first
|
|
- {role: base}
|
|
- {role: development}
|
|
# ... other roles
|
|
```
|
|
|
|
## See Also
|
|
|
|
- [Timeshift Documentation](https://github.com/teejee2008/timeshift)
|
|
- [Ansible Vault Guide](./vault.md) - For securing passwords
|
|
- [Maintenance Guide](../reference/makefile.md) - For system maintenance
|
|
|