diachron/TODO.md

high importance

• Add unit tests all over the place.

  • ⚠️ Huge task - needs breakdown before starting

• migrations, seeding, fixtures

CREATE SCHEMA fw;
CREATE TABLE fw.users (...);
CREATE TABLE fw.groups (...);

CREATE TABLE app.user_profiles (...);
CREATE TABLE app.customer_metadata (...);

• flesh out mgmt and develop (does not exist yet)

4.1 What belongs in develop

• Create migrations
• Squash migrations
• Reset DB
• Roll back migrations
• Seed large test datasets
• Run tests
• Snapshot / restore local DB state (!!!)

develop fails if APP_ENV (or whatever) is production. Or maybe even testing.

• Add default user table(s) to database.

• Add authentication

  • password
  • third party?

• Add middleware concept

• Add authorization

  • for specific routes / resources / etc

• Add basic text views Partially done; see the /time route. But we need to figure out where to store templates, static files, etc.

• fix process management: if you control-c master process sometimes it leaves around master-bin, logger-bin, and diachron:nnnn processes. Huge problem.

medium importance

• Add a log viewer

  • with queries
  • convert to logfmt and is there a viewer UI we could pull in and use instead?

• add nested routes. Note that this might be easy to do without actually changing the logic in express/routes.ts. A function that takes an array of routes and maps over them rewriting them. Maybe.

  • related: add something to do with default templates and stuff... I think we can make handlers a lot shorter to write, sometimes not even necessary at all, with some sane defaults and an easy to use override mechanism

• time library

• fill in the rest of express/http-codes.ts

• fill out express/content-types.ts

• identify redundant "old skool" and ajax routes, factor out their commonalities, etc.

• figure out and add logging to disk

• I don't really feel close to satisfied with template location / rendering / etc. Rethink and rework.

• Add email verification (this is partially done already)

• Reading .env files and dealing with the environment should be immune to the extent possible from idiotic errors

• Update check script:

  • shellcheck on shell scripts
  • go vet on go files
  • golangci-lint on go files
  • Run go fmt on all .go files
  • Eventually, run unit tests

• write docs

  • upgrade docs
  • starting docs
  • taking over docs
  • reference
  • internals

• make migration creation default to something like yyyy-mm-dd_ssss (are 9999 migrations in a day enough?)

• clean up cmd and mgmt: do the right thing with their commonalities and make very plain which is which for what. Consider additional commands. Maybe develop for specific development tasks, operate for operational tasks, and we keep cmd for project-specific commands. Something like that.

low importance

• add a prometheus-style /metrics endpoint to master

• create a metrics server analogous to the logging server

  • accept various stats from the workers (TBD)

• move master-bin into a subdir like master/cmd or whatever is idiomatic for golang programs; adapt master wrapper shell script accordingly

• flesh out the sync.sh script

  • update framework-managed node
  • update framework-managed pnpm
  • update pnpm-managed deps
  • rebuild golang programs

• If the number of workers is large, then there is a long lapse between when you change a file and when the server responds

  • One solution: start and stop workers serially: stop one, restart it with new code; repeat
  • Slow start them: only start a few at first

• in express/user.ts: FIXME: set createdAt and updatedAt to start of epoch

finished

• Reimplement fixup.sh

  • run shfmt on all shell scripts (and the files they source)
  • Run go fmt on all .go files
  • Run prettier biome on all .ts files and maybe others

• Adapt master program so that it reads configuration from command line args instead of from environment variables

  • Should have sane defaults
  • Adding new arguments should be easy and obvious

• Add wrapper script to run master program (so that various assumptions related to relative paths are safer)

• Add logging service

  • New golang program, in the same directory as master
    • Intended to be started by master
    • Listens to a port specified command line arg
    • Accepts POSTed (or possibly PUT) json messages, currently in a to-be-defined format. We will work on this format later.
    • Keeps the most recent N messages in memory. N can be a fairly large number; let's start by assuming 1 million.

• Log to logging service from the express backend

  • Fill out types and functions in express/logging.ts

• Add first cut at database access. Remember that ORMs are not all that!

• Create initial docker-compose.yml file for local development

  • include most recent stable postgres
  • include beanstalkd
  • include memcached
  • include redis
  • include mailpit