Community Batch Imports endpoint

[scottbarnes]

Closes #7705

Feature.

This adds a (currently admin-only) endpoint at /import/batch/new for community batch imports.

Technical

This:

• creates an endpoint at /import/batch/new that takes JSONL data via a multipart POST;
• allows # as a comment in JSONL, just to make life a bit easier.
• validates incoming data and requires the same things required to create a rec for import via load();
• uses source_records[0] for the ia_id;
• uses the Batch class for imports;
• relies on there being a 'submitter' column in the import_item table;
• only works for people with in the /usergroup/admin group; and
• generates a hash based on the incoming JSON byte string to use as the name of the import batch (so that an attempted import of the same items will get the same name). Note: this isn't super sophisticated as the hash would change if the order changed.

Some outstanding questions:

• We've mentioned using both /import/batch/new and /api/import/bulk; which do we want to use, if either?
• If there should be a limit on the number of items in the batch, what should that limit be?
• If there should be a limit on the batch size that's given to Batch.add_items(), what should that limit be?
• Is the hash strategy for generating a batch name acceptable? I was trying to find something that would:
  • be unique for any given set of items (in a given order...); and
  • allow for different batch names to be generated within a few seconds of one another.
• If the hash strategy is acceptable, is there any need to truncate the hash? I was just doing that to try to keep the table readable.

This was also tested against some of the data from @Billa05's work in #8551, found here: #8551 (comment)

Testing

Visit http://localhost:8080/import/batch/new

Try to upload a JSONL file with some errors:

{"title": "Blob Book 1", "source_records": "blob_source", "authors": [{"name": "Blob Author 1"}], "publishers": "Fail Publishers", "publish_date": "January 1, 2000", "isbn_10": "1111111111"}
{'blah': True}
{"title": "Blob Book 2", "source_records": "blob_source", "authors": [{"name": "Blob Author 2"}], "publishers": "Fail Publishers", "publish_date": "January 2, 2000", "isbn_10": "2222222222"}
{"source_records": ["blob_source"], "authors": [{"name": "Blob Author 2"}], "publishers": ["Not Fail Publishers"], "publish_date": "January 2, 2000", "isbn_10": "2222222222"}
{"title": "Blob Book 3", "source_records": ["blob_source"], "authors": [{"name": "Blob Author 2"}], "publishers": ["Not Fail Publishers"], "publish_date": "January 2, 2040", "isbn_10": "2222222222"}
{"identifiers": {"open_textbook_library": ["1581"]}, "source_records": ["open_textbook_library:1581"], "title": "Legal Fundamentals of Healthcare Law", "languages": ["eng"], "description": "Healthcare, a field dedicated to the well-being of individuals and communities, operates within an intricate web of legal principles. Understanding these laws is not simply a professional necessity for doctors, nurses, administrators, and researchers; it\u2019s also an ethical imperative for anyone who interacts with the healthcare system. This book is your compass, guiding you through the labyrinth of legal fundamentals that shape the landscape of healthcare.", "subjects": ["Medicine", "Law"], "publishers": ["University of West Florida Pressbooks"], "publish_date": "2024", "authors": [{"name": "Tiffany Jackman"}], "lc_classifications": ["RA440", "KF385.A4"]}
{"identifiers": {"open_textbook_library": ["1580"]}, "source_records": ["open_textbook_library:1580"], "title": "Introduction to Literature: Fairy Tales, Folk Tales, and How They Shape Us", "languages": ["eng"], "description": "Introduction to Literature: Fairy Tales, Folk Tales, and How They Shape Us introduces college freshmen to the study of literature through a focus on texts that, generally, they already know, or think they know, and how those texts aim to shape audiences to be compliant cultural objects. The book is organized around several prominent story groups, including various genres and forms, meant to promote discussion and discovery leading to students\u2019 understanding that these texts function as cultural sculptors of readers\u2019 principles and behaviors. Students develop the skill of analyzing texts and creating sound arguments about them through class discussions and a series of writing assignments. Ideally, they leave the course understanding how to create a sound argument and, more pointedly, that there is no such thing as \u201cjust a story.\u201d", "subjects": ["Humanities", "Literature, Rhetoric, and Poetry"], "publishers": ["University of West Florida Pressbooks"], "publish_date": "2023", "authors": [{"name": "Judy Young"}], "lc_classifications": ["PE1408"]}

See that the validation errors are displayed to the patrons:

Fix the errors by... commenting out the problematic lines:

# {"title": "Blob Book 1", "source_records": "blob_source", "authors": [{"name": "Blob Author 1"}], "publishers": "Fail Publishers", "publish_date": "January 1, 2000", "isbn_10": "1111111111"}
# {'blah': True}
# {"title": "Blob Book 2", "source_records": "blob_source", "authors": [{"name": "Blob Author 2"}], "publishers": "Fail Publishers", "publish_date": "January 2, 2000", "isbn_10": "2222222222"}
# {"source_records": ["blob_source"], "authors": [{"name": "Blob Author 2"}], "publishers": ["Not Fail Publishers"], "publish_date": "January 2, 2000", "isbn_10": "2222222222"}
# {"title": "Blob Book 3", "source_records": ["blob_source"], "authors": [{"name": "Blob Author 2"}], "publishers": ["Not Fail Publishers"], "publish_date": "January 2, 2040", "isbn_10": "2222222222"}
{"identifiers": {"open_textbook_library": ["1581"]}, "source_records": ["open_textbook_library:1581"], "title": "Legal Fundamentals of Healthcare Law", "languages": ["eng"], "description": "Healthcare, a field dedicated to the well-being of individuals and communities, operates within an intricate web of legal principles. Understanding these laws is not simply a professional necessity for doctors, nurses, administrators, and researchers; it\u2019s also an ethical imperative for anyone who interacts with the healthcare system. This book is your compass, guiding you through the labyrinth of legal fundamentals that shape the landscape of healthcare.", "subjects": ["Medicine", "Law"], "publishers": ["University of West Florida Pressbooks"], "publish_date": "2024", "authors": [{"name": "Tiffany Jackman"}], "lc_classifications": ["RA440", "KF385.A4"]}
{"identifiers": {"open_textbook_library": ["1580"]}, "source_records": ["open_textbook_library:1580"], "title": "Introduction to Literature: Fairy Tales, Folk Tales, and How They Shape Us", "languages": ["eng"], "description": "Introduction to Literature: Fairy Tales, Folk Tales, and How They Shape Us introduces college freshmen to the study of literature through a focus on texts that, generally, they already know, or think they know, and how those texts aim to shape audiences to be compliant cultural objects. The book is organized around several prominent story groups, including various genres and forms, meant to promote discussion and discovery leading to students\u2019 understanding that these texts function as cultural sculptors of readers\u2019 principles and behaviors. Students develop the skill of analyzing texts and creating sound arguments about them through class discussions and a series of writing assignments. Ideally, they leave the course understanding how to create a sound argument and, more pointedly, that there is no such thing as \u201cjust a story.\u201d", "subjects": ["Humanities", "Literature, Rhetoric, and Poetry"], "publishers": ["University of West Florida Pressbooks"], "publish_date": "2023", "authors": [{"name": "Judy Young"}], "lc_classifications": ["PE1408"]}

Verify the records are not in the import_item table:

openlibrary=# SELECT * FROM import_item;
 id | batch_id | added_time | import_time | status | error | ia_id | data | ol_key | comments | submitter 
----+----------+------------+-------------+--------+-------+-------+------+--------+----------+-----------
(0 rows)

Then try to upload again:

See that the items ended up in import_item:

openlibrary=# SELECT * FROM import_item;
-[ RECORD 1 ]-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id          | 12
batch_id    | 3
added_time  | 2024-06-13 04:22:33.119869
import_time | 
status      | pending
error       | 
ia_id       | open_textbook_library:1581
data        | {"authors": [{"name": "Tiffany Jackman"}], "description": "Healthcare, a field dedicated to the well-being of individuals and communities, operates within an intricate web of legal principles. Understanding these laws is not simply a professional necessity for doctors, nurses, administrators, and researchers; it\u2019s also an ethical imperative for anyone who interacts with the healthcare system. This book is your compass, guiding you through the labyrinth of legal fundamentals that shape the landscape of healthcare.", "identifiers": {"open_textbook_library": ["1581"]}, "languages": ["eng"], "lc_classifications": ["RA440", "KF385.A4"], "publish_date": "2024", "publishers": ["University of West Florida Pressbooks"], "source_records": ["open_textbook_library:1581"], "subjects": ["Medicine", "Law"], "title": "Legal Fundamentals of Healthcare Law"}
ol_key      | 
comments    | 
submitter   | openlibrary
-[ RECORD 2 ]-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id          | 13
batch_id    | 3
added_time  | 2024-06-13 04:22:33.119869
import_time | 
status      | pending
error       | 
ia_id       | open_textbook_library:1580
data        | {"authors": [{"name": "Judy Young"}], "description": "Introduction to Literature: Fairy Tales, Folk Tales, and How They Shape Us introduces college freshmen to the study of literature through a focus on texts that, generally, they already know, or think they know, and how those texts aim to shape audiences to be compliant cultural objects. The book is organized around several prominent story groups, including various genres and forms, meant to promote discussion and discovery leading to students\u2019 understanding that these texts function as cultural sculptors of readers\u2019 principles and behaviors. Students develop the skill of analyzing texts and creating sound arguments about them through class discussions and a series of writing assignments. Ideally, they leave the course understanding how to create a sound argument and, more pointedly, that there is no such thing as \u201cjust a story.\u201d", "identifiers": {"open_textbook_library": ["1580"]}, "languages": ["eng"], "lc_classifications": ["PE1408"], "publish_date": "2023", "publishers": ["University of West Florida Pressbooks"], "source_records": ["open_textbook_library:1580"], "subjects": ["Humanities", "Literature, Rhetoric, and Poetry"], "title": "Introduction to Literature: Fairy Tales, Folk Tales, and How They Shape Us"}
ol_key      | 
comments    | 
submitter   | openlibrary

Try to upload the same file again and see that the duplicates are caught and reported, and that the total tried, added, and not queued are correct, along with the duplicate items:

[cdrini]

Converting this to a draft for now since it's kind of in limbo!

[cdrini]

Nice! This looks great! A few code fixes, and one annoying rename :P

[cdrini]

I think in the future we should move away from treating batch_name like a unqiue string; it should just be effectively a comment. The batch id is what should be... the id :P But future PR problem, I think there were some good reasons why we did this.

[cdrini]

Todo in a future PR: make this not extend web.storage

[cdrini]

Nice lgtm! Tested it erroring correctly; and tested it actually queueing items 🥳 Note I didn't see it actually get imported; caused around this time of a month we have a big import that goes through, so it'll take a whiiile for it to actually go through (note not ideal UX :/ but problem for another time and should only affect ~15-20th of every month).