Node engine, package manager and node modules not available in the run image

[musaffa]

Expected Behavior

Don't delete node engine, package manager and node_modules from launch layers.

Current Behavior

node-run-script removes node engine, package manager, node_modules from the launch layers. Starting from v0.5, node-run-script has a default script detection. If there's a script named build in package.json, this buildpack will be automatically detected. Thus node engine, package manager and node modules will be removed from the launch layers.

Possible Solution

The presence of a build script doesn't mean that node engine, package manager and node modules will not be needed in the run image anymore. node-run-script should not change the build plan. New env vars can be introduced in order to change the build plan. These env vars can be added to this buildpack or to another buildpack down the layer. Thus developers can choose whether they want to keep node engine, package manager and node modules in the launch layers or not.

Steps to Reproduce

1. A package.json file with a script named build.

Motivations

This issue may affect NodeJs servers and SSR enabled Front-end applications. In my case, it affects FastBoot enabled Ember applications.

[ryanmoran]

@musaffa, can you provide some more detail about how your application is being built? What other buildpacks are you using? Can you share build logs with as much context as possible?

[pbusko]

We tried to build a test application which contains the build script in the package.json file and both node engine and node_modules layers were added to the image, so we couldn't reproduce the issue.

List of buildpacks:

===> DETECTING
7 of 12 buildpacks participating
paketo-buildpacks/ca-certificates 3.4.0
paketo-buildpacks/node-engine     0.19.0
paketo-buildpacks/npm-install     0.10.8
paketo-buildpacks/node-module-bom 0.4.3
paketo-buildpacks/node-run-script 0.5.1
paketo-buildpacks/node-start      0.8.6
paketo-buildpacks/npm-start       0.10.3

Exported layers:

===> EXPORTING
Adding layer 'paketo-buildpacks/ca-certificates:helper'
Adding layer 'paketo-buildpacks/node-engine:node'
Adding layer 'paketo-buildpacks/npm-install:launch-modules'
Adding layer 'launch.sbom'
<truncated>

[musaffa]

It's because you are running npm-start after node-run-script. npm-start re-adds node engine, npm, node modules.

Some buildpacks may not have npm-start or yarn-start. For example, the meta buildpack web-servers, the one I'm trying to use for SSR enabled Ember project, doesn't have npm-start or yarn-start in its buildpack definition. So node engine, package manger and node modules are not available in the run image of this meta buildpack due to node-run-script.

[ryanmoran]

Ah. That is the expected design of the Web Servers buildpack. Its designed to enable serving frontend web applications with a file server like NGINX. The buildpack intentionally omits node and node_modules from the run image as they are not needed to serve the built JavaScript bundles. Can you explain some more of your usecase for why you'd need node and node_modules in the run image?

[musaffa]

@ryanmoran We are using Fastboot for SSR (Server Side Rendering) of Ember Apps. Instead of just serving files from the dist directory, Fastboot also boots up an Express.js server and sends the rendered HTML contents to the client (browser) for the incoming requests. This dramatically speeds up the initial loading time of modern web apps. SSR is also a must-to-have feature for Search Engine Optimization. For a node server like Express to run, process the incoming requests and send rendered contents, it needs the node engine and node_modules files.

SSR is a very common deployment practice for front-end web applications. All the major front-end frameworks, like React, Ember, Angular, Vue and Svelte, have this feature.

[ryanmoran]

I'm not familiar with Fastboot. I'm gonna play around with it, but just to get a quicker understanding of what is expected to happen within the built container, you will have compiled JS files, an NGINX server that serves them, and node, node_modules, and this Fastboot server running on a different port than NGINX.

That's 2 running processes in the container. Is that right?

[musaffa]

Yes.

Fastboot is an Ember addon. It serves both the dist files as well as the rendered HTMLs. NGINX acts as a reverse proxy for the Fastboot server.

I'll share a demo app tomorrow.

[ryanmoran]

That would be super helpful. Thanks!

[musaffa]

@ryanmoran Please check this demo application: https://github.com/musaffa/paketo-ember

The README contains the instructions how to build the project with Paketo Buildpacks. You will need Node 18.x and Yarn 1.x. For buildpack specific instructions, check this section.

The buildpack build will fail because there's no yarn-start in web-server meta buildpack. If you create a custom web-server buildpack with yarn-start, this app will build and run.

I checked it with a custom web server buildpack I built locally. There's a caveat though. Once started, the docker process cannot be stopped with Ctrl + c. It has be stopped manually with docker stop CONTAINER_ID.

5 files to check out in the repo.

1. The build script in package.json
2. nginx.conf: Check how it sets up an upstream server running in a unix socket. It is the Ember Fastboot server which runs in this socket. It follows the technique of heroku-buildpack-nginx.
3. bin/start-nginx: Starts FastBoot server in a unix socket and then starts NGINX server. Check this for reference.
4. Procfile: simply refers to bin/start-nginx
5. project.toml

[ryanmoran]

@musaffa thank you for the sample repo. I was able to get this working with a few modifications, but I think we need to figure out what the right behavior is here w.r.t. detection.

We have a node process that is running in the container which is what the Node.js buildpack is designed to facilitate, but we also have this nginx process running in the container which is what the Web Servers buildpack is designed to facilitate. This use-case sits somewhere in the middle. I'm inclined to add support into the Web Servers buildpack since the buildpack already has the requisite node-engine, npm-install, yarn, and yarn-install buildpacks and would simply need to make sure it requires those dependencies for launch.

Where I need your help is defining how the Web Servers buildpack would detect that it needs to include the node and node_modules dependencies. Where are the parts of the codebase that the buildpack could see that would indicate to it that this is an SSR application?

You mention that this is common across all the major front-end frameworks. Do they share common configuration or tooling that we could detect?

[musaffa]

I don't think they share any common configuration and tooling. There's no standard for implementing SSR.

[mhdawson]

Since it's been almost 2 year since the last discussion on this issue I want to see if its still an issue. @musaffa is this still something you would be interested in moving forward or has it been too long?

[musaffa]

It's because you are running npm-start after node-run-script. npm-start re-adds node engine, npm, node modules.

Some buildpacks may not have npm-start or yarn-start. For example, the meta buildpack web-servers, the one I'm trying to use for SSR enabled Ember project, doesn't have npm-start or yarn-start in its buildpack definition. So node engine, package manger and node modules are not available in the run image of this meta buildpack due to node-run-script.

We are using a hack as mentioned above. npm-start / yarn-start buildpack re-adds node engine, package manager and node_module which have been removed by node-run-script buildpack.

This issue hasn't been resolved in this buildpack yet.

[mhdawson]

Not fully up to speed yet, but wondering about:

Where I need your help is defining how the Web Servers buildpack would detect that it needs to include the node and node_modules dependencies. Where are the parts of the codebase that the buildpack could see that would indicate to it that this is an SSR application?

Would defining an additional environment variable during the build, and if that environment variable was defined not remove node engine, package manager, node_modules from the launch layers be a workable approach?

[musaffa]

That would be the right approach I think.