feat: add automated migration scripts for NDF by dcshzj · Pull Request #1725 · opengovsg/isomer

dcshzj · 2025-11-13T08:21:05Z

Problem

NDF is some special site that requires a significant automation work.

Solution

Breaking Changes

Yes - this PR contains breaking changes
No - this PR is backwards compatible

Features:

Add a new scripts package to handle all recurring automation work.
Add script for migration NDF collection.

socket-security · 2025-11-13T08:27:26Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Quality	Maintenance
	@vitest/coverage-istanbul@2.1.2 ⏵ 2.1.9		^-1
	msw-storybook-addon@2.0.4 ⏵ 2.0.6	⁺¹	⁺¹	⁺⁴
	@tiptap/extension-text@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@tiptap/extension-document@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@chakra-ui/theme-tools@2.2.7 ⏵ 2.2.9	⁺¹	⁺³	^-2
	@tanstack/react-query-devtools@5.85.3 ⏵ 5.90.2	⁺¹	⁺³	^-4
	tailwindcss-react-aria-components@1.1.4 ⏵ 1.2.0	⁺¹	⁺³	⁺¹
	@tiptap/extension-table-row@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@tiptap/extension-dropcursor@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@chakra-ui/utils@2.2.3 ⏵ 2.2.5	⁺¹	⁺³	^-2
	@tiptap/extension-gapcursor@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@tiptap/extension-list-item@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@tiptap/extension-paragraph@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@tiptap/extension-table-header@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@tiptap/extension-table-cell@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@tiptap/extension-history@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@tiptap/extension-underline@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@tanstack/react-table@8.21.2 ⏵ 8.21.3	⁺¹	⁺¹
	@tiptap/extension-subscript@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@tiptap/extension-superscript@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@tiptap/extension-blockquote@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@next/eslint-plugin-next@14.2.13 ⏵ 14.2.33
	@babel/preset-typescript@7.27.1 ⏵ 7.28.5			⁺¹
	@tiptap/extension-hard-break@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@tiptap/html@2.25.0 ⏵ 2.27.1			⁺¹
	@types/pg@8.15.5 ⏵ 8.15.6	⁺¹	⁺¹	⁺²
	@tiptap/extension-bullet-list@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@tiptap/extension-heading@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	@tiptap/extension-strike@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	isomorphic-dompurify@2.25.0 ⏵ 2.31.0		^-2	⁺⁴
	@tiptap/extension-horizontal-rule@2.22.3 ⏵ 2.27.1	⁺¹	⁺³	⁺¹
	typescript-eslint@8.33.1 ⏵ 8.46.4		⁺¹	⁺¹
See 87 more rows in the dashboard

View full report

adriangohjw · 2025-11-17T03:47:35Z

@dcshzj this seems like a one-off script for NDF and no need to be merged so i won't review - do correct me if im wrong!

dcshzj · 2025-11-17T03:54:43Z

@dcshzj this seems like a one-off script for NDF and no need to be merged so i won't review - do correct me if im wrong!

Hmm conflicted, I created this PR cos we will run this for at least 6 times - July to December 2026, but also a bit pointless to review honestly. Can review if got time! Super not urgent.

mergify · 2025-12-17T03:41:54Z

This pull request has been stale for more than 30 days! Could someone please take a look at it @opengovsg/isomer-engineers

Copilot

Pull request overview

This PR adds automated migration scripts for the National Drug Formulary (NDF), introducing a new scripts package to handle recurring automation work for migrating NDF collections.

Key Changes:

Creates a new @isomer/scripts package with TypeScript configuration and dependencies for HTML/CSV processing
Implements migration scripts for NDF active ingredient and product information collections
Integrates TipTap extensions and jsdom for HTML content transformation

Reviewed changes

Copilot reviewed 8 out of 10 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
tooling/scripts/tsconfig.json	TypeScript configuration for the scripts package
tooling/scripts/package.json	Package definition with dependencies for HTML/CSV processing
tooling/scripts/ndf-collection/config.ts	Configuration file containing CSV file paths
tooling/scripts/ndf-collection/utils.ts	Utility functions for CSV parsing, HTML processing, and data transformation
tooling/scripts/ndf-collection/template.ts	Template generators for monograph and product information pages
tooling/scripts/ndf-collection/index.ts	Entry point with interactive CLI for selecting migration type
tooling/scripts/ndf-collection/getHtmlAsJson.ts	HTML to JSON converter using TipTap and jsdom
tooling/scripts/ndf-collection/createProductInformationCollection.ts	Script to migrate product information from CSV to JSON
tooling/scripts/ndf-collection/createActiveIngredientCollection.ts	Script to migrate active ingredient data from CSV to JSON
package-lock.json	Dependency lock file with version updates and new package entries

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-05T09:39:17Z

tooling/scripts/ndf-collection/createActiveIngredientCollection.ts

+    // eslint-disable-next-line @typescript-eslint/no-non-null-assertion
+    const permalink = entry["Monograph ID"]!;


The non-null assertion operator is used here without validation. If "Monograph ID" is missing from the CSV data, this will cause a runtime error. Consider adding validation to ensure the field exists before using it, or handle the undefined case explicitly with a meaningful error message.

Suggested change

// eslint-disable-next-line @typescript-eslint/no-non-null-assertion

const permalink = entry["Monograph ID"]!;

const permalink = entry["Monograph ID"];

if (!permalink) {

// Skip entries without a valid Monograph ID to avoid runtime errors

continue;

}

Copilot · 2026-01-05T09:39:17Z

tooling/scripts/ndf-collection/template.ts

+                            {
+                              text: "Subsidised brands of vaccines",
+                              type: "text",
+                              marks: [


The non-null assertion operator is used without validation. While the array access with optional chaining ensures the array element exists, the assertion assumes the "Guidance Recommendations_" field will always be present at that index. Consider using optional chaining (?.) instead or providing a default empty string to handle cases where the field might be missing.

Copilot · 2026-01-05T09:39:17Z

tooling/scripts/package.json

+  "description": "",
+  "main": "index.ts",
+  "scripts": {
+    "ndf": "dotenv tsx ndf-collection/index.ts"


The script name "ndf" is not descriptive and could be confused with other commands. Consider using a more specific name like "ndf-migrate" or "migrate-ndf" to clearly indicate this is a migration script for NDF collections.

Suggested change

"ndf": "dotenv tsx ndf-collection/index.ts"

"ndf-migrate": "dotenv tsx ndf-collection/index.ts"

Copilot · 2026-01-05T09:39:18Z

tooling/scripts/ndf-collection/createActiveIngredientCollection.ts

+    );
+    const generalAvailability = getSimilarKeysAsArray(
+      entry,
+      "General Availability in  Public Healthcare Institutions_"


There's a typo in the key name: "General Availability in Public Healthcare Institutions_" contains two spaces between "in" and "Public". This should be "General Availability in Public Healthcare Institutions_" with a single space. This typo could cause the function to not find matching keys in the CSV data if the actual column name has only one space.

Suggested change

"General Availability in Public Healthcare Institutions_"

"General Availability in Public Healthcare Institutions_"

Copilot · 2026-01-05T09:39:18Z

tooling/scripts/ndf-collection/createProductInformationCollection.ts

+    // eslint-disable-next-line @typescript-eslint/no-non-null-assertion
+    const permalink = entry["Licence number (SIN number)"]!;


The non-null assertion operator is used here without proper validation. If "Licence number (SIN number)" is not present in the CSV data, this will cause a runtime error. Consider adding validation to ensure the field exists and is not empty before using it, or handle the undefined case explicitly.

Suggested change

// eslint-disable-next-line @typescript-eslint/no-non-null-assertion

const permalink = entry["Licence number (SIN number)"]!;

const licenceNumberField = entry["Licence number (SIN number)"];

if (!licenceNumberField) {

console.warn(

'Skipping entry without "Licence number (SIN number)" field:',

entry

);

continue;

}

const permalink = licenceNumberField;

Copilot · 2026-01-05T09:39:18Z

tooling/scripts/ndf-collection/getHtmlAsJson.ts

+const dom = new JSDOM(
+  `<html>
+      <div class="element"></div>
+    </html>`
+);
+const window = dom.window;
+const document = window.document;
+global.document = document;
+global.window = window as any;


Setting global.document and global.window in this way can have unintended side effects if this module is imported in other contexts. This approach mutates the global environment which can cause issues in testing or when the code runs in different environments. Consider:

Using a more isolated approach with jsdom instances

Documenting this side effect clearly

Ensuring this module is only imported when needed for HTML parsing

Copilot · 2026-01-05T09:39:19Z

tooling/scripts/ndf-collection/config.ts

+  "/Users/zhongjun/Downloads/general-monograph.csv";
+
+// This is the CSV file for the collection of all product information
+export const NDF_PRODUCT_INFORMATION_CSV_FILEPATH =
+  "/Users/zhongjun/Downloads/product-information.csv";


The file paths are hardcoded to a specific user's local machine ("/Users/zhongjun/Downloads/"). These should be removed or replaced with environment variables or relative paths to avoid exposing personal information and to make the code portable across different development environments.

Suggested change

"/Users/zhongjun/Downloads/general-monograph.csv";

// This is the CSV file for the collection of all product information

export const NDF_PRODUCT_INFORMATION_CSV_FILEPATH =

"/Users/zhongjun/Downloads/product-information.csv";

process.env.NDF_GENERAL_MONOGRAPH_CSV_FILEPATH ?? "general-monograph.csv";

// This is the CSV file for the collection of all product information

export const NDF_PRODUCT_INFORMATION_CSV_FILEPATH =

process.env.NDF_PRODUCT_INFORMATION_CSV_FILEPATH ?? "product-information.csv";

Copilot · 2026-01-05T09:39:19Z

tooling/scripts/ndf-collection/utils.ts

+export const decodeHtmlEntities = (str: string) => {
+  const txt = document.createElement("textarea");
+  txt.innerHTML = str;
+  return txt.value;
+};


The decodeHtmlEntities function uses the global document object to decode HTML entities. This approach is problematic because:

It relies on a browser-specific API (document.createElement) that may not work consistently in all environments

The global document is being set up using jsdom in getHtmlAsJson.ts, but this function is also used in other files that may not have that setup

Using innerHTML to decode entities can be a security risk if the input is not trusted

Consider using a dedicated HTML entity decoding library or ensuring that jsdom is properly initialized before this function is called.

Copilot · 2026-01-05T09:39:19Z

tooling/scripts/ndf-collection/createProductInformationCollection.ts

+    // if (permalink === "SIN16609P") {
+    //   break;
+    // }


The commented-out code creates ambiguity about whether early termination is intended for testing or production use. If this was used for debugging, it should be removed. If it's intended as a feature for testing single entries, consider replacing it with a proper command-line flag or environment variable (e.g., TEST_MODE=true or --limit=1).

Suggested change

// if (permalink === "SIN16609P") {

// break;

// }

mergify · 2026-02-28T08:28:18Z

This pull request has been stale for more than 30 days! Could someone please take a look at it @opengovsg/isomer-engineers

adriangohjw

TBH didn't really review very deep, skimmed through it

will be helpful if can include a README, or sample CSVs for the inputs for reference, in case someone needs to run this when you are not around

but non-blocker and good-to-have, so approving to clean up the PRs

thanks!

adriangohjw · 2026-03-10T05:48:14Z

tooling/scripts/ndf-collection/config.ts

+export const NDF_GENERAL_MONOGRAPH_CSV_FILEPATH =
+  "/Users/zhongjun/Downloads/general-monograph.csv";
+
+// This is the CSV file for the collection of all product information
+export const NDF_PRODUCT_INFORMATION_CSV_FILEPATH =
+  "/Users/zhongjun/Downloads/product-information.csv";
+
+// This is the CSV file for the list of Pharmacological Classifications
+export const NDF_PHARMACOLOGICAL_CLASSIFICATIONS_CSV_FILEPATH =
+  "/Users/zhongjun/Downloads/pharmacological-classifications.csv";


can update to relative path instead? in case someone else needs to run this script. thanks!

tooling/scripts/ndf-collection/createProductInformationCollection.ts

@@ -0,0 +1,128 @@
+import fs from "fs";
+import path from "path";
+import Papa from "papaparse";


dcshzj added 3 commits November 13, 2025 16:16

feat: add automated migration scripts for NDF

7d69815

fix: update based on feedback

3f44752

chore: remove empty file

a86d8f8

dcshzj requested a review from a team as a code owner November 13, 2025 08:21

chore: update links

abd8743

chore: update scripts based on feedback

4566ef0

Copilot AI review requested due to automatic review settings January 5, 2026 09:35

Copilot started reviewing on behalf of dcshzj January 5, 2026 09:36 View session

Copilot AI reviewed Jan 5, 2026

View reviewed changes

dcshzj added 5 commits January 9, 2026 15:31

fix: change to title case and split pipe

4c49004

feat: generate pharmacological classification list

bdd8b7a

fix: add synonyms to description

0d2c407

fix: sort product names within routes of administration

de92c3b

chore: add missing exemptions and use active ingredients

4bd6172

adriangohjw approved these changes Mar 10, 2026

View reviewed changes

fix: update based on NDF feedback

56bc7ca

github-code-quality bot found potential problems Mar 13, 2026

View reviewed changes

tooling/scripts/ndf-collection/createProductInformationCollection.ts

@@ -0,0 +1,128 @@

import fs from "fs";

import path from "path";

import Papa from "papaparse";

dcshzj added 3 commits March 13, 2026 15:54

chore: update text

9076e91

chore: update name

a3610e3

chore: add README and update paths

8ef374d

dcshzj mentioned this pull request Mar 20, 2026

feat: add isomer-admin tooling scripts #1904

Merged

2 tasks

chore: update legend

a7b0c0d

		// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
		const permalink = entry["Monograph ID"]!;

-    // eslint-disable-next-line @typescript-eslint/no-non-null-assertion
-    const permalink = entry["Monograph ID"]!;
+    const permalink = entry["Monograph ID"];
+    if (!permalink) {
+      // Skip entries without a valid Monograph ID to avoid runtime errors
+      continue;
+    }

	"ndf": "dotenv tsx ndf-collection/index.ts"
	"ndf-migrate": "dotenv tsx ndf-collection/index.ts"

	"General Availability in Public Healthcare Institutions_"
	"General Availability in Public Healthcare Institutions_"

		// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
		const permalink = entry["Licence number (SIN number)"]!;

-    // eslint-disable-next-line @typescript-eslint/no-non-null-assertion
-    const permalink = entry["Licence number (SIN number)"]!;
+    const licenceNumberField = entry["Licence number (SIN number)"];
+    if (!licenceNumberField) {
+      console.warn(
+        'Skipping entry without "Licence number (SIN number)" field:',
+        entry
+      );
+      continue;
+    }
+    const permalink = licenceNumberField;

Conversation

dcshzj commented Nov 13, 2025

Problem

Solution

Uh oh!

socket-security bot commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adriangohjw commented Nov 17, 2025

Uh oh!

dcshzj commented Nov 17, 2025

Uh oh!

mergify bot commented Dec 17, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Feb 28, 2026

Uh oh!

adriangohjw left a comment

Choose a reason for hiding this comment

Uh oh!

adriangohjw Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

socket-security bot commented Nov 13, 2025 •

edited

Loading