ilia/slack-sieve
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document Levkin Mattermost import and channel renames in prepare-export.
Some checks failed
CI / skip-ci-check (pull_request) Successful in 8s
Details
CI / secret-scan (pull_request) Failing after 6s
Details
CI / python-ci (pull_request) Successful in 21s
Details

Browse Source 
Record the completed migration (channel map, mmetl/mmctl steps, caveats) and teach prepare-export to apply mattermost_channel renames plus user_emails for matching existing MM accounts.
This commit is contained in:
ilia 2026-07-10 13:36:09 -04:00
parent 4b034d19f6
commit 934b674995
4 changed files with 123 additions and 122 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
.gitignore vendored
View File

@ -1,5 +1,8 @@
manifest.json
prepared-export/
prepared-export.zip
*.bulk-import.zip
.tools/
node_modules/
test-results/
playwright-report/

13
README.md
View File

@ -3,7 +3,7 @@
Sieve your [Slack workspace export](https://slack.com/help/articles/201658943-Export-your-workspace-data) before [Mattermost](https://mattermost.com) import — triage channels, cherry-pick messages, preview attachments.
```text
Slack export → hydrate (attachments) → review UI → manifest.json → (prepare-export) → mmetl → mmctl
Slack export → hydrate → review UI → manifest.json → prepare-export → mmetl → mmctl
```
## Quick start
@ -59,9 +59,12 @@ Edit `presets/levkin-skips.json` (or copy it) for workspace-specific channels to
## Mattermost import (after review)
1. `make prepare-export` — writes `prepared-export/` from `manifest.json` (pytest-covered; not yet run through mmetl/mmctl).
2. [mmetl](https://github.com/mattermost/mmetl) `transform slack` on `prepared-export/`
3. [mmctl](https://docs.mattermost.com/administration-guide/manage/mmctl-command-line-tool.html) `import`
1. Set optional `mattermost_channel` renames and `user_emails` in `manifest.json`.
2. `make prepare-export` — writes `prepared-export/` (pytest-covered).
3. Zip that folder; [mmetl](https://github.com/mattermost/mmetl) `transform slack --team <team> …`
4. [mmctl](https://docs.mattermost.com/administration-guide/manage/mmctl-command-line-tool.html) `import process --bypass-upload` on the Mattermost host
**Levkin:** import completed 2026-07-10 — see [docs/HANDOFF.md](docs/HANDOFF.md) for channel map and caveats.
Docs: [Migrate from Slack](https://docs.mattermost.com/administration-guide/onboard/migrate-from-slack.html)
@ -72,7 +75,7 @@ Docs: [Migrate from Slack](https://docs.mattermost.com/administration-guide/onbo
## Handoff
Ongoing Levkin migration status, review order, and TODOs: **[docs/HANDOFF.md](docs/HANDOFF.md)**
Levkin migration status and channel map: **[docs/HANDOFF.md](docs/HANDOFF.md)**
## Remote

187
docs/HANDOFF.md
View File

@ -3,15 +3,60 @@
**Repo:** https://git.levkin.ca/ilia/slack-sieve (public)
**Clone:** `git clone git@git.levkin.ca:ilia/slack-sieve.git`
**Vikunja:** [todo.levkin.ca → Homelab → slack-sieve](https://todo.levkin.ca) (`SLK`)
**Last updated:** 2026-06-03
**Last updated:** 2026-07-10
---
## Status: Levkin Slack → Mattermost **done** (2026-07-10)
Filtered archive is live on **https://slack.levkin.ca** (team `levkin`, VM `10.0.10.237`).
- **18 channels**, ~**299 posts** + attachments (from ~765 cherry-picked Slack messages)
- Users: Slack `jedi_jew` → MM `ilia`, `ira_feeds` → `ira` (both added to imported channels)
- Boy/girl name lists verified in **Zoey** (`kids-zoey`)
### Channel map (Slack → Mattermost)
| Slack | MM name | Display name |
|-------|---------|--------------|
| `122harmony` | `prop-122harmony` | 122 Harmony |
| `153niagara` | `prop-153niagara` | 153 Niagara |
| `284richmond` | `prop-284richmond` | 284 Richmond |
| `45ritson` | `prop-45ritson` | 45 Ritson |
| `6keefer` | `prop-6keefer` | 6 Keefer |
| `york` | `prop-york` | York |
| `airbnb` | `prop-airbnb` | Airbnb |
| `realestate` | `prop-realestate` | Real estate |
| `zoey` | `kids-zoey` | Zoey |
| `zane` | `kids-zane` | Zane |
| `izik` | `kids-izik` | Izik |
| `moishe` | `kids-moishe` | Moishe |
| `parenting` | `kids-parenting` | Parenting |
| `after-school-activities` | `kids-after-school` | After school |
| `banking` | `family-banking` | Banking |
| `purchase` | `family-purchase` | Purchase |
| `medical` | `family-medical` | Medical |
| `levkin` | `family-levkin` | Levkin |
Skipped for import: noise channels (`random`, `jam`, `meetings`, …), `29wilson`, `520steeles`, empty picks.
### Import caveats
- Thread replies whose **parent was not cherry-picked** were dropped by mmetl (`couldn't find rootPost`).
- First `mmctl` job failed updating `ilia` roles (“Cannot demote last System Admin”); fixed by omitting existing users from the JSONL and re-running. Channels from the first attempt were reused.
- Local artifacts (gitignored): `prepared-export/`, `prepared-export.zip`, `.tools/mmetl`, `/tmp/mm-import/` on the Mac; `/tmp/mattermost-bulk-import.zip` on the VM.
### Share with Irina
> Slack archive is on Mattermost: https://slack.levkin.ca
> Team **levkin** — channels like **Zoey**, **Zane**, **45 Ritson**, **Banking**.
> Login **ira**. Filtered archive (files/facts, not all chat). Name lists in **Zoey**.
---
## What this is
Local triage UI for migrating **Levkin** Slack (`levkine.slack.com`) → homelab Mattermost (`https://slack.levkin.ca`, VM `10.0.10.237`).
Pipeline:
Local triage UI for migrating Slack workspace exports → Mattermost.
```text
Official Slack export (JSON)
@ -20,7 +65,7 @@ Official Slack export (JSON)
→ make review # manifest.json (gitignored)
→ make prepare-export # prepared-export/ (filtered Slack shape)
→ mmetl transform slack
→ mmctl import
→ mmctl import --bypass-upload (on MM host)
```
---
@ -30,10 +75,10 @@ Official Slack export (JSON)
| Path | Role |
|------|------|
| `~/Downloads/Levkin Slack export Sep 29 2016 - May 27 2026` | Raw official export (~27MB JSON only) |
| `~/Downloads/Levkin Slack export enriched` | **Use for review** — JSON + `__uploads/` (~1109 files) |
| `~/Downloads/Levkin Slack export enriched` | Review + prepare source — JSON + `__uploads/` |
| `~/Documents/code/slack-sieve/manifest.json` | Triage decisions (local only, not in git) |
Hydrate copied **1109 / 1122** attachments; **13** old files returned HTTP 401 (expired export token).
Hydrate: **1109 / 1122** attachments; **13** old files HTTP 401 (expired export token).
---
@ -45,99 +90,35 @@ cd ~/Documents/code/slack-sieve
export SLACK_EXPORT_PATH="$HOME/Downloads/Levkin Slack export enriched"
make review # http://127.0.0.1:8765
make hydrate # re-download attachments if needed
make preset-skips # apply presets/levkin-skips.json (safe to re-run; keeps your channel decisions)
make cherry-pick # bulk data/files picks on priority channels (optional)
make prepare-export # after review — filtered export for mmetl
make test # pytest + Playwright
make hydrate
make preset-skips
make cherry-pick # bulk data/files picks (optional)
make prepare-export # filtered export for mmetl
make test
```
---
### Re-run Mattermost import (if needed)
## Review progress (manifest snapshot)
As of preset + partial manual review:
| Status | Count | Channels |
|--------|-------|----------|
| **skip** (preset Tier A/B) | 22 | `#jewbottest`, `#meetings`, `#help`, `#baby`, `#garden`, `#recipes`, `#grants`, `#integromat`, `#clickup`, `#general`, `#after-school-activities`, `#cleaner`, `#medical`, `#purchase`, `#temp`, `#viewings`, `#moishe`, `#520steeles`, `#29wilson`, `#york`, `#jam`, Reading List |
| **import** | 1 | `#153niagara` |
| **partial** (cherry-pick) | 3 | `#122harmony`, `#6keefer`, `#zoey` |
| **undecided** | 14 | See review order below |
Refresh counts after more review:
```bash
python3 -c "
import json
from pathlib import Path
m = json.loads(Path('manifest.json').read_text())
from collections import Counter
c = Counter(v.get('status','?') for v in m.get('channels',{}).values())
print(dict(c))
"
```
1. Set `mattermost_channel` / `user_emails` in `manifest.json`, then `make prepare-export`.
2. Zip `prepared-export/` → run [mmetl](https://github.com/mattermost/mmetl) v0.4+
`mmetl transform slack --team levkin --file prepared-export.zip -o mattermost_import.jsonl --bot-owner ilia --default-email-domain levkin.local`
3. Map Slack usernames → existing MM users in the JSONL; **do not** re-import users that would change System Admin roles.
4. Package `data/` (attachments) + `mattermost_import.jsonl` → zip; copy to VM;
`mmctl --local import process --bypass-upload /tmp/mattermost-bulk-import.zip`
---
## Suggested review order
1. `#284richmond` → `#45ritson` → finish `#122harmony`, `#6keefer`, `#zoey` (partial)
2. `#banking` → `#realestate` → `#airbnb`
3. `#izik` → `#levkin`
4. **Tier D (later):** `#random`, `#professional`, `#parenting`, `#adhd`, `#baking-biz`, `#standup`, `#zane`
### Inside a channel
- **Shift+S** — skip whole channel
- **Shift+I** — import whole channel
- **Cherry-pick mode** — only checked messages export (use **Import whole thread** on parent)
- **Skip:** join/leave noise, bot posts, bare “ok” / link-only
- **Keep:** attachments, leases, repairs, banking, decisions with dates/names
---
## Presets
`presets/levkin-skips.json` — Tier A/B channel names for `make preset-skips`.
Copy and edit for another workspace.
---
## UI / server notes
- **Port:** 8765 (`SLACK_IMPORT_PORT`)
- **Attachments:** served at `/uploads/…` and `/api/file-download?file_id=…`
- **DOCX preview:** server-side + browser mammoth; spaces in filenames need URL decode (fixed in server)
- Restart after code changes: `make review` (kill old process on 8765 if needed)
---
## What not to use
| Tool | Why |
|------|-----|
| `slack-advanced-exporter` + rotating `xoxe.xoxp` token | Token format rejected; enrich zip stayed ~4MB |
| `slackdump hydrate` | Rejects `?token=xoxe-…` URLs; use **this repo’s** `make hydrate` instead |
| `slackdump` browser login | Failed in headless/automated runs; not required if official export exists |
---
## Open tasks (source of truth)
Tracked in **Vikunja → Homelab → slack-sieve**. Update here when scope changes.
## Open tasks
| P | Task | Owner | Status |
|---|------|-------|--------|
| **P1** | **Channel review in UI** — `make review`, finish 14 undecided + 3 partial ([order](#suggested-review-order)); save manifest | @you | ⏳ **next** |
| **P1** | Pilot import: one channel via `make prepare-export` → `mmetl` → `mmctl` on MM VM `10.0.10.237` | @agent | ⏳ blocked on review |
| **P2** | User mapping: Slack ID → Mattermost username for `mmetl` | @agent | backlog |
| **P3** | Re-hydrate 13 files that returned HTTP 401 (expired export token) | @you | optional |
| — | `scripts/prepare-export.py` + `make prepare-export` | @agent | ✅ done |
| — | Levkin triage + Mattermost import | @you / @agent | ✅ done 2026-07-10 |
| **P3** | Re-hydrate 13 files that returned HTTP 401 | @you | optional |
| **P3** | Optionally re-import missing thread parents | @you | optional |
### Mattermost / homelab
- SSO: [authentik-apps.md#mattermost](https://git.levkin.ca/ilia/ansible/src/branch/master/docs/guides/authentik-apps.md#mattermost) (ansible)
- SSO: [authentik-apps.md#mattermost](https://git.levkin.ca/ilia/ansible/src/branch/master/docs/guides/authentik-apps.md#mattermost)
- Import docs: https://docs.mattermost.com/administration-guide/onboard/migrate-from-slack.html
---
@ -146,33 +127,17 @@ Tracked in **Vikunja → Homelab → slack-sieve**. Update here when scope chang
```text
slack-sieve/
server.py # HTTP API + static UI
slack_export.py # Parse export, file index, search
static/ # Review UI
scripts/
hydrate-export.sh
preset-skips.py
auto-cherry-pick.py
prepare-export.py
build-test-fixture.py
tests/ # pytest + Playwright (mini-export fixture)
presets/
levkin-skips.json
manifest.json # gitignored — your work in progress
docs/
HANDOFF.md # this file
server.py / slack_export.py / static/
scripts/ hydrate, preset-skips, auto-cherry-pick, prepare-export, build-test-fixture
tests/ pytest + Playwright
presets/ levkin-skips.json
docs/HANDOFF.md
manifest.json # gitignored
```
---
## Moved from ansible
Slack import tooling was removed from `git@git.levkin.ca:ilia/ansible.git` (2026-05-29). Ansible guide pointer: `docs/guides/slack-mattermost-import.md`.
---
## Security
- Do not commit `manifest.json` or Slack export folders.
- Do not commit `manifest.json`, Slack export folders, or import zips.
- Rotate any Slack tokens/cookies pasted in chat during setup.
- `~/.config/hermes/gitea.env` used for Gitea API only — unrelated to Slack.

42
scripts/prepare-export.py
View File

@ -99,8 +99,10 @@ def prepare_export(
channels_meta = []
manifest_channels = manifest.get("channels") or {}
email_map = manifest.get("user_emails") or {}
included_names: set[str] = set()
stats: dict[str, int] = {}
rename_map: dict[str, str] = {}
for ch in channels_meta:
name = ch.get("name")
@ -110,25 +112,52 @@ def prepare_export(
if not channel_included(entry):
continue
included_names.add(name)
mm_name = (entry.get("mattermost_channel") or "").strip()
if mm_name and mm_name != name:
rename_map[name] = mm_name
# channels.json subset
filtered_meta = [c for c in channels_meta if c.get("name") in included_names]
# channels.json subset (optionally renamed for Mattermost)
filtered_meta = []
for c in channels_meta:
name = c.get("name")
if name not in included_names:
continue
out_ch = dict(c)
if name in rename_map:
out_ch["name"] = rename_map[name]
filtered_meta.append(out_ch)
(out_root / "channels.json").write_text(
json.dumps(filtered_meta, indent=2, ensure_ascii=False) + "\n",
encoding="utf-8",
)
# users.json — copy whole (mmetl needs user mapping)
# users.json — copy + inject emails so mmetl can match existing MM users
users_src = export.paths.users_json
if users_src.is_file():
shutil.copy2(users_src, out_root / "users.json")
users = load_json(users_src)
if isinstance(users, list) and email_map:
for u in users:
if not isinstance(u, dict):
continue
uid = u.get("id")
uname = u.get("name")
email = email_map.get(uid) or email_map.get(uname)
if email:
profile = dict(u.get("profile") or {})
profile["email"] = email
u["profile"] = profile
(out_root / "users.json").write_text(
json.dumps(users, indent=2, ensure_ascii=False) + "\n",
encoding="utf-8",
)
uploads_out = out_root / "__uploads"
uploads_src = export.paths.root / "__uploads"
for name in sorted(included_names):
entry = manifest_channels[name]
ch_out = out_root / name
out_name = rename_map.get(name, name)
ch_out = out_root / out_name
ch_out.mkdir(parents=True)
count = 0
for date in export.list_channel_date_files(name):
@ -164,12 +193,13 @@ def prepare_export(
json.dumps(kept, indent=2, ensure_ascii=False) + "\n",
encoding="utf-8",
)
stats[name] = count
stats[out_name] = count
summary = {
"channels": len(stats),
"messages": sum(stats.values()),
"per_channel": stats,
"renames": rename_map,
}
(out_root / "prepare-export-summary.json").write_text(
json.dumps(summary, indent=2) + "\n",