Architecture and roadmaps for MoPepGen web application

[aholmes]

I will add diagrams and other information for architecting a MoPepGen web application.

Here is the PlantUML for this diagram. systems.puml.txt

[aholmes]

Rough relative roadmap of tasks. This diagram is just a sketch of when relative tasks might happen, none of these numbers are real.

PlantUML for roadmap. roadmap.puml.txt

[zhuchcn]

Really appreciate the work putting all those together @aholmes! I think this is more complicated than I thought! I have some questions about the diagram.

1. I see an arrow from user to the web UI on AWS but also arrows from user to UCLA cluster and Okta authentication. What's the difference between those? Will only users with access to UCLA mednet being able to get access to this? We definitely want people outside of UCLA could also use this tool.

2. I see this is using the web UI to trigger moPepGen running on our CDS cluster. This is really cool and will defnitely be very useful. A complete run of moPepGen will require a lot of resources (F16 or more) with the full human genome So my original thought was just allowing users to run only one gene at a time with the web server. This is saying, if users want to just see what peptides can be yield from mutations on a specific gene, great. But if they want to generate a database from the entire human genome, they should run the nextflow pipelines on a cluster. So if we are moving forward with this, do we still need to run moPepGen on our CDS clusters or can it run directly on AWS EC2/Elastic Beanstalk? I'm not sure which one will be easier to set up.

3. I also have a lot of questions of database. Is it going to be a database for only user information or are we actually going to store user data (input/output data of moPepGen) in it? I guess the former will be smaller and the later will be more complicated. Maybe we don't have to discuss it right now.

[aholmes]

I'm going to create a couple branches and make a PR between them so we can examine multiple diagrams and see the differences when things change.

[aholmes]

Hey Trevor, please see PR 737 for more thorough diagrams. These might clarify some of your questions.

1. I see an arrow from user to the web UI on AWS but also arrows from user to UCLA cluster and Okta authentication. What's the difference between those? Will only users with access to UCLA mednet being able to get access to this? We definitely want people outside of UCLA could also use this tool.

Got it! We can account for the external users. I indicated this with the new diagram. Currently the system has built-in support for SAML2; UCLA Health IT uses SAML2 through Okta, which is an external IDP. We can add other authentication options like OAuth, if needed, or we can extend to other IDPs. For example, if another university uses Okta we should be able to integrate with both UCLA Health's Okta instance and the other one.

Is anonymous / unauthenticated use needed as well?

As for the difference between the cluster and UI arrows, I was indicating that a user can use either the cluster or the web interface, but the direct cluster usage is irrelevant for this so I've removed it. The arrow to Okta is to indicate that the user currently must authenticate. I've added a separate item for other auth systems as well.

2. I see this is using the web UI to trigger moPepGen running on our CDS cluster. This is really cool and will defnitely be very useful. A complete run of moPepGen will require a lot of resources (F16 or more) with the full human genome So my original thought was just allowing users to run only one gene at a time with the web server. This is saying, if users want to just see what peptides can be yield from mutations on a specific gene, great. But if they want to generate a database from the entire human genome, they should run the nextflow pipelines on a cluster. So if we are moving forward with this, do we still need to run moPepGen on our CDS clusters or can it run directly on AWS EC2/Elastic Beanstalk? I'm not sure which one will be easier to set up.

We will need to do some benchmarking and see just how "light" we can make a run of moPepGen. For "specific gene" runs we can identify ways of enforcing this. Running on AWS infrastructure is significantly simpler at the moment because we are not able to directly connect AWS and the cluster; though we can find work arounds for this, it may take time to ensure OHIA and IT Security are on board. We also need to consider cost (EC2, EB). High-compute usage on Elastic Beanstalk (EB) can be very expensive - in fact this is likely the wrong infrastructure type for this kind of workload. The cost for EC2 is less than EB, but then we sacrifice capacity for running more than one "job" and risk choking the web application.

3. I also have a lot of questions of database. Is it going to be a database for only user information or are we actually going to store user data (input/output data of moPepGen) in it? I guess the former will be smaller and the later will be more complicated. Maybe we don't have to discuss it right now.

The database would only store application data and user authentication and authorization information. For example, the application needs a way to track that a job is running and how to find related information like who started the job, where files are located, etc. The user auth information is information like their email address and what kind of access to the system they have. Any moPepGen data storage depends on where we are able to run it. If on the cluster, we will use the same storage systems we currently use. If we run in AWS, the data will likely be stored in S3 or an alternate high-performance system; cost (S3, FSx) needs to be analyzed for this as well.

[zhuchcn]

The figures in the PR looks really nice!

I think we should priortize on minimalization. I confirmed it with Paul that we don't want to connect it to our cluster (at least for now). We might also priortize on anonymous usage. Because we are now not running it on our cluster, we then don't necessarily need authentication except for safty and administration.

Resources requirement for moPepGen to run a gene depends on how many input mutations it receives. We can put limitations to the parameters, including number of input mutations, to make sure it won't use too many resources to break the application.

[aholmes]

I confirmed it with Paul that we don't want to connect it to our cluster (at least for now).
...
Resources requirement for moPepGen to run a gene depends on how many input mutations it receives.

Cool - let's get some benchmarks and see what kind of resource usage is required, and the time it takes. That will inform our architecture decisions for running moPepGen itself.

Because we are now not running it on our cluster, we then don't necessarily need authentication except for safty and administration.

We will use auth for ourselves at a minimum. We should strongly consider requiring some kind of registration for external users, unless the intent is to allow anonymous usage.