21 — Web portal & data export

Everything in notes 11–13 is about the phone. This note is about the other half: a browser-based portal for programme staff, at /portal on the same Frappe backend the app talks to. Surveyors keep using the app; nobody is asked to change how they collect data.

It is live on staging and not yet on production — see Where this stands.

Who sees what

Two audiences, one screen set, separated by role.

The scoping is the same rule the app uses, so a surveyor sees the same records in the browser as on their phone. It is enforced on the server, not just hidden in the interface — a surveyor who guesses at an export address gets a permission error, not data.

Swasti Manager is an administrator-level role, not a read-only reporting role. It also grants edit, create and bulk-import rights on member and programme records. Hand it to named individuals who are meant to have that, one account each — not a shared login.

The list screens

Each of Members, Schemes, Documents, Health and Livelihood is a filterable table: filter by geography, partner, donor, surveyor, status and date; sort; choose which columns to show; and save that layout so it is there next time. Follow-ups have their own screen, which the phone does not offer at all.

Two ways to get data out, and they answer different questions

This distinction is the thing most worth understanding, because the two used to be conflated.

1. Download, on a list screen — “give me what I am looking at.” Exports that one form, honouring whatever filters or ticked rows are on screen. Built in the browser, arrives immediately. Best for a working subset.

2. Data export, under Administration — “assemble a dataset.” Pick any combination of forms, filter them, and Swasti prepares one ZIP in the background. No row limit. This is the only place a multi-form export starts.

Past about 5,000 records a list screen stops offering to build the file in the browser — at production volumes that would mean a several-hundred-megabyte file assembled inside a browser tab — and sends you to Data export instead, with the form you were viewing already selected. A list screen will never hand you all eleven forms again.

What a data export contains

• One CSV per form you selected, plus export_manifest.csv listing every file, its form and its row count. If a form you selected has no rows, its CSV still appears with its headers and a note in the manifest — nothing is ever silently missing.
• Readable values, not internal codes. Village, block, donor, partner, scheme and surveyor all read as names.
• Multi-select answers joined into one cell, e.g. Aadhaar Card; PAN Card; Voter ID.
• No staff email addresses anywhere. Worth stating because Frappe’s own built-in export does leak them — which is why that option was removed rather than kept alongside.
• Dates that open correctly in Excel. Created and Modified used to carry microseconds, which Excel could not read as a date and rendered as an elapsed time like 08:49.0.

Created vs Modified — read Created

Created is when the surveyor made the record. Modified is the last time anything touched the row on the server, including our own data migrations — which is why records collected in March can show a Modified date of 10 July. The date filter on Data export therefore filters on Created. If you are asking “what was collected last month”, Created is the column you want.

Scheme status: 9 options instead of 21

The Scheme Application status filter used to list 21 options with heavy overlap — Approved, Approved & Awaiting Benefit, Approved & Benefits Received, Application Approved & Benefits Received and more, several of which match no records at all. They are now grouped into nine:

Approved — benefits received · Approved — awaiting benefits · Approved — benefits not received · In progress · Delayed · Rejected · Status not checked · Not interested · Closed

Nothing stored changes — this is a grouping over the existing values, so the phone is unaffected. Exports carry both a raw Status column and a Status (grouped) column, so a sheet can be reconciled against Desk and still pivot cleanly.

Closed is deliberately its own group: Delayed therefore means “delayed and still open”, which is usually what someone asking for delayed cases wants.

A note for whoever specified this (SWF-2026-69174): the canonical list in the ticket could not be used as written. Those nine values are workflow pipeline stages, but what the system actually stores is the derived follow-up outcome. Eight of the nine proposed values match zero records, and the value covering roughly 90% of production — Application Approved & Benefits Received — was not in the list at all. The nine groups above are built from the values that genuinely occur. Worth confirming they answer the reporting need.

Why the download link appears on the page, not in your inbox

A prepared export is emailed to you and shown on the Data export page as a private download link. Right now the on-page link is the one that works: neither staging nor production has an outgoing email account configured yet — nothing has ever been sent from either. DevOps is provisioning it. When that lands, the email starts arriving with no further change and no redeploy.

Either way the link is private to the person who requested it and expires 48 hours after the export is prepared, after which the file is deleted automatically. One export at a time per person; a second request while one is still running is refused rather than queued.

Where this stands

• Staging: live and verified. Exercised end to end as both a manager and a surveyor.
• Production: not yet. Held pending PM sign-off on the export changes, then a DevOps deploy. Production has no portal yet at all, so this is a first-time rollout rather than an update: the Swasti Manager role and per-user layout storage are created by the deploy, and manager accounts need creating afterwards.
• Not yet measured at production scale. Staging holds about 8,150 members; production has 557,873 members and 595,285 scheme applications. A full staging export is ~1.4 MB and finishes in seconds; the production figure is an extrapolation until a real run is measured, and the 5,000-record browser threshold is an estimate rather than a measurement.

Related: note 14 for repos and environments, note 16 for creating users, note 19 for the production checklist.