ilia/slack-sieve
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document Levkin Mattermost import and channel renames in prepare-export.
CI / skip-ci-check (pull_request) Successful in 8s
Details
CI / secret-scan (pull_request) Failing after 6s
Details
CI / python-ci (pull_request) Successful in 21s
Details

Browse Source 
Record the completed migration (channel map, mmetl/mmctl steps, caveats) and teach prepare-export to apply mattermost_channel renames plus user_emails for matching existing MM accounts.
This commit is contained in:
ilia
2026-07-10 13:36:09 -04:00
parent 4b034d19f6
commit 934b674995
4 changed files with 123 additions and 122 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/HANDOFF.md
+76 -111
View File
@@ -3,15 +3,60 @@
**Repo:** https://git.levkin.ca/ilia/slack-sieve (public)
**Clone:** `git clone git@git.levkin.ca:ilia/slack-sieve.git`
**Vikunja:** [todo.levkin.ca → Homelab → slack-sieve](https://todo.levkin.ca) (`SLK`)
**Last updated:** 2026-06-03
**Last updated:** 2026-07-10
---
## Status: Levkin Slack → Mattermost **done** (2026-07-10)
Filtered archive is live on **https://slack.levkin.ca** (team `levkin`, VM `10.0.10.237`).
- **18 channels**, ~**299 posts** + attachments (from ~765 cherry-picked Slack messages)
- Users: Slack `jedi_jew` → MM `ilia`, `ira_feeds` → `ira` (both added to imported channels)
- Boy/girl name lists verified in **Zoey** (`kids-zoey`)
### Channel map (Slack → Mattermost)
| Slack | MM name | Display name |
|-------|---------|--------------|
| `122harmony` | `prop-122harmony` | 122 Harmony |
| `153niagara` | `prop-153niagara` | 153 Niagara |
| `284richmond` | `prop-284richmond` | 284 Richmond |
| `45ritson` | `prop-45ritson` | 45 Ritson |
| `6keefer` | `prop-6keefer` | 6 Keefer |
| `york` | `prop-york` | York |
| `airbnb` | `prop-airbnb` | Airbnb |
| `realestate` | `prop-realestate` | Real estate |
| `zoey` | `kids-zoey` | Zoey |
| `zane` | `kids-zane` | Zane |
| `izik` | `kids-izik` | Izik |
| `moishe` | `kids-moishe` | Moishe |
| `parenting` | `kids-parenting` | Parenting |
| `after-school-activities` | `kids-after-school` | After school |
| `banking` | `family-banking` | Banking |
| `purchase` | `family-purchase` | Purchase |
| `medical` | `family-medical` | Medical |
| `levkin` | `family-levkin` | Levkin |
Skipped for import: noise channels (`random`, `jam`, `meetings`, …), `29wilson`, `520steeles`, empty picks.
### Import caveats
- Thread replies whose **parent was not cherry-picked** were dropped by mmetl (`couldn't find rootPost`).
- First `mmctl` job failed updating `ilia` roles (“Cannot demote last System Admin”); fixed by omitting existing users from the JSONL and re-running. Channels from the first attempt were reused.
- Local artifacts (gitignored): `prepared-export/`, `prepared-export.zip`, `.tools/mmetl`, `/tmp/mm-import/` on the Mac; `/tmp/mattermost-bulk-import.zip` on the VM.
### Share with Irina
> Slack archive is on Mattermost: https://slack.levkin.ca
> Team **levkin** — channels like **Zoey**, **Zane**, **45 Ritson**, **Banking**.
> Login **ira**. Filtered archive (files/facts, not all chat). Name lists in **Zoey**.
---
## What this is
Local triage UI for migrating **Levkin** Slack (`levkine.slack.com`) → homelab Mattermost (`https://slack.levkin.ca`, VM `10.0.10.237`).
Pipeline:
Local triage UI for migrating Slack workspace exports → Mattermost.
```text
Official Slack export (JSON)
@@ -20,7 +65,7 @@ Official Slack export (JSON)
→ make review # manifest.json (gitignored)
→ make prepare-export # prepared-export/ (filtered Slack shape)
→ mmetl transform slack
→ mmctl import
→ mmctl import --bypass-upload (on MM host)
```
---
@@ -30,10 +75,10 @@ Official Slack export (JSON)
| Path | Role |
|------|------|
| `~/Downloads/Levkin Slack export Sep 29 2016 - May 27 2026` | Raw official export (~27MB JSON only) |
| `~/Downloads/Levkin Slack export enriched` | **Use for review** — JSON + `__uploads/` (~1109 files) |
| `~/Downloads/Levkin Slack export enriched` | Review + prepare source — JSON + `__uploads/` |
| `~/Documents/code/slack-sieve/manifest.json` | Triage decisions (local only, not in git) |
Hydrate copied **1109 / 1122** attachments; **13** old files returned HTTP 401 (expired export token).
Hydrate: **1109 / 1122** attachments; **13** old files HTTP 401 (expired export token).
---
@@ -45,99 +90,35 @@ cd ~/Documents/code/slack-sieve
export SLACK_EXPORT_PATH="$HOME/Downloads/Levkin Slack export enriched"
make review # http://127.0.0.1:8765
make hydrate # re-download attachments if needed
make preset-skips # apply presets/levkin-skips.json (safe to re-run; keeps your channel decisions)
make cherry-pick # bulk data/files picks on priority channels (optional)
make prepare-export # after review — filtered export for mmetl
make test # pytest + Playwright
make hydrate
make preset-skips
make cherry-pick # bulk data/files picks (optional)
make prepare-export # filtered export for mmetl
make test
```
---
### Re-run Mattermost import (if needed)
## Review progress (manifest snapshot)
As of preset + partial manual review:
| Status | Count | Channels |
|--------|-------|----------|
| **skip** (preset Tier A/B) | 22 | `#jewbottest`, `#meetings`, `#help`, `#baby`, `#garden`, `#recipes`, `#grants`, `#integromat`, `#clickup`, `#general`, `#after-school-activities`, `#cleaner`, `#medical`, `#purchase`, `#temp`, `#viewings`, `#moishe`, `#520steeles`, `#29wilson`, `#york`, `#jam`, Reading List |
| **import** | 1 | `#153niagara` |
| **partial** (cherry-pick) | 3 | `#122harmony`, `#6keefer`, `#zoey` |
| **undecided** | 14 | See review order below |
Refresh counts after more review:
```bash
python3 -c "
import json
from pathlib import Path
m = json.loads(Path('manifest.json').read_text())
from collections import Counter
c = Counter(v.get('status','?') for v in m.get('channels',{}).values())
print(dict(c))
"
```
1. Set `mattermost_channel` / `user_emails` in `manifest.json`, then `make prepare-export`.
2. Zip `prepared-export/` → run [mmetl](https://github.com/mattermost/mmetl) v0.4+
`mmetl transform slack --team levkin --file prepared-export.zip -o mattermost_import.jsonl --bot-owner ilia --default-email-domain levkin.local`
3. Map Slack usernames → existing MM users in the JSONL; **do not** re-import users that would change System Admin roles.
4. Package `data/` (attachments) + `mattermost_import.jsonl` → zip; copy to VM;
`mmctl --local import process --bypass-upload /tmp/mattermost-bulk-import.zip`
---
## Suggested review order
1. `#284richmond` → `#45ritson` → finish `#122harmony`, `#6keefer`, `#zoey` (partial)
2. `#banking` → `#realestate` → `#airbnb`
3. `#izik` → `#levkin`
4. **Tier D (later):** `#random`, `#professional`, `#parenting`, `#adhd`, `#baking-biz`, `#standup`, `#zane`
### Inside a channel
- **Shift+S** — skip whole channel
- **Shift+I** — import whole channel
- **Cherry-pick mode** — only checked messages export (use **Import whole thread** on parent)
- **Skip:** join/leave noise, bot posts, bare “ok” / link-only
- **Keep:** attachments, leases, repairs, banking, decisions with dates/names
---
## Presets
`presets/levkin-skips.json` — Tier A/B channel names for `make preset-skips`.
Copy and edit for another workspace.
---
## UI / server notes
- **Port:** 8765 (`SLACK_IMPORT_PORT`)
- **Attachments:** served at `/uploads/…` and `/api/file-download?file_id=…`
- **DOCX preview:** server-side + browser mammoth; spaces in filenames need URL decode (fixed in server)
- Restart after code changes: `make review` (kill old process on 8765 if needed)
---
## What not to use
| Tool | Why |
|------|-----|
| `slack-advanced-exporter` + rotating `xoxe.xoxp` token | Token format rejected; enrich zip stayed ~4MB |
| `slackdump hydrate` | Rejects `?token=xoxe-…` URLs; use **this repo’s** `make hydrate` instead |
| `slackdump` browser login | Failed in headless/automated runs; not required if official export exists |
---
## Open tasks (source of truth)
Tracked in **Vikunja → Homelab → slack-sieve**. Update here when scope changes.
## Open tasks
| P | Task | Owner | Status |
|---|------|-------|--------|
| **P1** | **Channel review in UI** — `make review`, finish 14 undecided + 3 partial ([order](#suggested-review-order)); save manifest | @you | ⏳ **next** |
| **P1** | Pilot import: one channel via `make prepare-export` → `mmetl` → `mmctl` on MM VM `10.0.10.237` | @agent | ⏳ blocked on review |
| **P2** | User mapping: Slack ID → Mattermost username for `mmetl` | @agent | backlog |
| **P3** | Re-hydrate 13 files that returned HTTP 401 (expired export token) | @you | optional |
| — | `scripts/prepare-export.py` + `make prepare-export` | @agent | ✅ done |
| — | Levkin triage + Mattermost import | @you / @agent | ✅ done 2026-07-10 |
| **P3** | Re-hydrate 13 files that returned HTTP 401 | @you | optional |
| **P3** | Optionally re-import missing thread parents | @you | optional |
### Mattermost / homelab
- SSO: [authentik-apps.md#mattermost](https://git.levkin.ca/ilia/ansible/src/branch/master/docs/guides/authentik-apps.md#mattermost) (ansible)
- SSO: [authentik-apps.md#mattermost](https://git.levkin.ca/ilia/ansible/src/branch/master/docs/guides/authentik-apps.md#mattermost)
- Import docs: https://docs.mattermost.com/administration-guide/onboard/migrate-from-slack.html
---
@@ -146,33 +127,17 @@ Tracked in **Vikunja → Homelab → slack-sieve**. Update here when scope chang
```text
slack-sieve/
server.py # HTTP API + static UI
slack_export.py # Parse export, file index, search
static/ # Review UI
scripts/
hydrate-export.sh
preset-skips.py
auto-cherry-pick.py
prepare-export.py
build-test-fixture.py
tests/ # pytest + Playwright (mini-export fixture)
presets/
levkin-skips.json
manifest.json # gitignored — your work in progress
docs/
HANDOFF.md # this file
server.py / slack_export.py / static/
scripts/ hydrate, preset-skips, auto-cherry-pick, prepare-export, build-test-fixture
tests/ pytest + Playwright
presets/ levkin-skips.json
docs/HANDOFF.md
manifest.json # gitignored
```
---
## Moved from ansible
Slack import tooling was removed from `git@git.levkin.ca:ilia/ansible.git` (2026-05-29). Ansible guide pointer: `docs/guides/slack-mattermost-import.md`.
---
## Security
- Do not commit `manifest.json` or Slack export folders.
- Do not commit `manifest.json`, Slack export folders, or import zips.
- Rotate any Slack tokens/cookies pasted in chat during setup.
- `~/.config/hermes/gitea.env` used for Gitea API only — unrelated to Slack.