Metadata-Version: 2.4
Name: acmad-upload-helper
Version: 0.1.0
Summary: Helper for uploading data to the ACMAD platform.
Author-email: Jan Hettenhausen <j.hettenhausen@griffith.edu.au>, Isaac Jennings <i.jennings@griffith.edu.au>
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: acmadauth>=1.3.3
Requires-Dist: httpx>=0.28.1
Requires-Dist: openpyxl>=3.1.5
Requires-Dist: pyyaml>=6.0
Requires-Dist: typer>=0.26.8
Dynamic: license-file

# ACMAD Upload Helper

This package extracts clinic data, transforms it into a validated canonical
representation, and loads it into the ACMAD platform.

## Architecture

The package follows a staged, ports-and-adapters design:

```text
source package
    │
    ▼
sources/        filesystem and Excel extraction
    │
    ▼
domain/         canonical values, provenance and structured issues
    │
    ▼
application/    ETL orchestration and stage results
    │
    ▼
ports/          destination-independent interfaces
    │
    ▼
adapters/       REST API and other destination implementations
```

Source adapters do not import API code. Domain models do not import source or
HTTP libraries. The application layer depends on protocols, allowing each
stage to be tested independently and allowing clinics to supply alternative
extractors or loaders.

The pre-refactor implementation is retained under `acmad_uploader.legacy` as
reference material. New code must not depend on it.

Every record type ("slice") follows the same shape, illustrated here with
patient records:

```text
PatientWorkbookExtractor -> PatientTransformer -> PatientLoader
                                                   |
                                                   v
                                         PatientRepository port
                                                   |
                                                   v
                                         ACMAD REST API adapter
```

The transformer owns template-version validation and mappings into the API's
controlled vocabulary. The repository owns destination lookup and persistence.
Neither workbook extraction nor a canonical domain record depends on HTTP.
Each slice adds its own `sources/`, `domain/`, `transformers/`, `ports/`,
`application/` and `adapters/api/` module, plus tests, without widening
another slice's responsibilities. See [CLAUDE.md](CLAUDE.md) for the shared
infrastructure (workbook discovery, form-field validation helpers, the
`JsonApiClient`, natural-key lookup helpers) that later slices reuse instead
of re-deriving, and [DATA_QUALITY_FINDINGS.md](DATA_QUALITY_FINDINGS.md) for
known real-world data-quality issues each slice's validation surfaces.

## Installation

The package is published on a private package index. Install it with pip or uv.

``` bash
pip install --extra-index-url=https://pypi.acmad.cloud.edu.au/ acmad-upload-helper  
```

## Source inspection

Inventory a package without parsing or uploading clinical records:

```bash
acmad-upload inspect /path/to/package
acmad-upload inspect /path/to/package --json
```

## Record types

Every record type below has its own `validate`/`upload` subcommand pair,
each accepting a package directory (or a single relevant workbook) and
printing one line per canonical record found:

```bash
acmad-upload <slice> validate /path/to/package
acmad-upload <slice> upload /path/to/package
```

| Slice | Reads | Notes |
| --- | --- | --- |
| `patient` | `patient_record.xlsx` | Idempotent by `site_patient_id`. |
| `diagnosis` | `patient_record.xlsx` | Primary/secondary diagnoses per patient. |
| `gait-assessment` | `physical_exam.xlsx` / `function_proms.xlsx` / `mocap.xlsx` | Joins sibling workbooks under one `gait_assessment_<age>` by natural key; must run before anything hanging off it. |
| `function-assessment` | `function_proms.xlsx` | GMFCS, FMS, FAC. |
| `mocap-data` | `mocap.xlsx` | One mocap session per test condition. |
| `biomechanics-artifact` | files under each `mocap_<condition>/` folder | Uploads raw/derived motion-capture files (C3D, TRC, MOT, STO, the three "derived" CSVs, model/mesh files) as metadata-tagged artifacts; the API parses the derived CSVs server-side. |
| `physical-examination` | `physical_exam.xlsx` | Anthropometrics plus the side-keyed range-of-motion/strength/tone/foot-structure detail rows. |
| `proms` | `function_proms.xlsx` | FAQ, OFAQ, PODCI, MGaitEfficiency, Walk12, GoalAssessment. |
| `surgery` | `surgery.xlsx` | One episode with its numbered procedure slots. |

`gait-assessment`, `function-assessment`, `mocap-data`,
`biomechanics-artifact`, `physical-examination` and `proms` all resolve an
already-created `GaitAssessment` (and, for `biomechanics-artifact`, an
already-created `MocapData` session too) by natural key and fail if it
doesn't exist yet at the destination — upload `patient` and
`gait-assessment` first. `surgery` only depends on `patient` (it creates
its own surgery-type `PatientEvent`), so its position relative to the
gait-assessment family doesn't matter. See
[CLAUDE.md](CLAUDE.md) for each slice's exact natural key and destination
quirks.

## Validating and uploading a whole package

`validate`/`upload` (no slice name) run every slice above, in the
dependency order described above, against the same package directory:

```bash
acmad-upload validate /path/to/package
acmad-upload upload /path/to/package
```

Each slice's issues are printed under its own `== <slice> ==` header; the
command exits non-zero if any slice failed. Point these at a patient's
whole directory tree, not a hand-picked subfolder — folder names are a
human-browsing convenience, not the source of truth, and every workbook
already carries its own identifying fields (see
[CLAUDE.md](CLAUDE.md)).

## Authentication

Every `upload` command (per-slice or package-level) authenticates through
the OIDC device flow provided by `acmadauth`. It opens the identity
provider login page by default and stores tokens in the operating system
keyring; it never receives or stores the user's password. Use
`--no-open-browser` to print the login URL without opening it, or
`--memory-token` to avoid persisting tokens.

The built-in defaults target a local development environment:

- API: `http://localhost:8000/api/v1/`
- OIDC realm: `http://localhost:8080/realms/localdev`
- OIDC client: `localdevoidc`

Override these per-invocation with `--api-url`, `--realm-url`, and
`--client-id`, or persist them for a package in `acmad.yaml` (see
[Configuration file](#configuration-file)). TLS verification remains
enabled unless explicitly disabled. The selected Keycloak client must be
public and have the OAuth 2.0 Device Authorisation Grant enabled, as
required by `acmadauth`.

## Configuration file

Settings you'd otherwise repeat on every invocation can be persisted in an
`acmad.yaml` file at the package root:

```yaml
# ACMAD upload helper settings for this package
site: QCH
api_url: https://acmad.example/api/v1/
realm_url: https://id.acmad.example/realms/prod
client_id: prod-cli
```

Every key is optional. Resolution is always **CLI flag > `acmad.yaml` >
built-in default**, so a flag never gets silently shadowed by the config
file. An unrecognised key (e.g. `api-url` instead of `api_url`) or a
non-string value is reported as an error rather than quietly ignored.

| Key | Overrides | Built-in default |
| --- | --- | --- |
| `site` | `--site` | none (falls back to each workbook's own `site_id`) |
| `api_url` | `--api-url` | `http://localhost:8000/api/v1/` |
| `realm_url` | `--realm-url` | `http://localhost:8080/realms/localdev` |
| `client_id` | `--client-id` | `localdevoidc` |

The endpoint keys apply to every `upload` command. `site` applies to
`patient`, `diagnosis`, `gait-assessment` and `surgery` — the only slices
that read a workbook's own `site_id` field.

### Site acronym

The site acronym fills in `site_id` when a workbook doesn't declare one at
all (or leaves it blank) — most usefully for `gait-assessment`, whose
`function_proms.xlsx`/`mocap.xlsx` inputs never carry `site_id`. When a
workbook *does* declare a value, the supplied acronym must agree with it;
a mismatch fails validation rather than silently overriding real workbook
data:

```bash
acmad-upload patient validate /path/to/package --site QCH
acmad-upload upload /path/to/package --site QCH
```

Neither the flag nor the config file infers an acronym from folder names,
even though sample data happens to be organised into per-site folders —
see [CLAUDE.md](CLAUDE.md) for the full design rationale.

## Idempotency

Every slice upserts by its own natural key rather than always creating a
new record, so re-running `upload` against an unchanged package is safe
and mostly a no-op (e.g. `patient` resolves an existing patient through the
API's `identifiers` filter; `biomechanics-artifact` skips a file entirely
if its content hash hasn't changed). Exact natural keys and destination-side
quirks differ per slice and are documented in
[CLAUDE.md](CLAUDE.md).
