Firebase Functions with Yarn workspaces

Question

We're starting to adopt a monorepo setup using yarn workspaces and we'd like to have our firebase functions inside it. The repo structure is something like:

repo
    node_modules <- all dependencies
    packages
        core
        commom
        functions <- firebase functions

So, I have 2 problems with this setup:

1. The dependencies of the functions don't live on the same folder as the entry file from functions
2. The functions depends on other packages such as core and commom that are in the repo so yarn symlinks from node_modules to the packages in the repo.

Is there anyway I can handle this?

Answer 1

With Yarn 2 node_modules aren't fetched and placed into in the respective functions directory (as it would be the case with calling npm i in the functions directory). So when calling firebase deploy --project default --only function the node_modules folder is missing and firebase will complain about this and abort the deployment process with the following error (or similar):

Error parsing triggers: Cannot find module [...]
Try running "npm install" in your functions directory before deploying.

There are two github issues that are tracking this issue at the moment:

• Support mono-repos in deployment
• Functions deployment fails when firebase-functions has been hoisted by a monorepo manager like yarn/workspaces or lerna

In the two issues above, several clever workarounds are presented by firebase users, e.g. using webpack to create a build that contains all the local packages in the release or using rsync or other tools that rewire the packages before release.

Another solution is not hoisting your project packages, if that is possible. You can do this, be adding the following two directives to your .yarnrc.yml file.

# yarnrc.yml

# disables yarn's plugnplay style and uses node_modules instead
nodeLinker: node-modules
# makes sure the node_modules are not hoisted to the (monorepo) project root
nmHoistingLimits: "dependencies"

The two directives above are explained in the yarnrc configuration docs as follows:

nmHoistingLimits Defines the highest point where packages can be hoisted. One of workspaces (don't hoist packages past the workspace that depends on them), dependencies (packages aren't hoisted past the direct dependencies for each workspace), or none (the default, packages are hoisted as much as possible). This setting can be overriden per-workspace through the installConfig.hoistingLimits field.

nodeLinker Defines what linker should be used for installing Node packages (useful to enable the node-modules plugin), one of: pnp, node-modules.

Answer 2

The solution I found for this is Yarn's nohoist option in your root package.json file.

By default Yarn hoists dependencies to the root directory so they can be shared between your packages. Unfortunately this will not work with Firebase. This means you need to tell Yarn not to hoist the dependencies used by your Firebase functions.

The documentation for nohoist is less than ideal, but here is an official blog post about it here: https://yarnpkg.com/blog/2018/02/15/nohoist/

You probably want something like this:

{
  "workspaces": {
    "packages": [
      "packages/*"
    ],
    "nohoist": [
      "functions/core",
      "functions/common",
      "functions/**"
    ]
  }
}

Keep in mind that this uses the name field used in the package.json files of each workspace package. So in this example, it is assume that the functions directory has a package.json with "functions" as it's name.

functions/** tells yarn not to hoist any of the dependencies specified in packages/functions/package.json. This doesn't work for your shared yarn packages though, so functions/core and functions/common need to be specified separately.

You also need to include your workspaces as dependencies in your functions project, so add them to your package.json:

{
  "name": "functions",
  "dependencies": {
    "core": "*",
    "common": "*",
  }
}

Once you have added all that, you should delete your packages/functions/node_modules directory and run yarn install. After doing this, you should see all your dependencies included in packages/functions/node_modules (not symlinks).

Answer 3

I have developed a solution and wrote an article about this. Below is an excerpt, but here is the full article

The problem with Firebase

When deploying to Firebase it wants to upload a folder just like a traditional single package repository, containing the source files together with a manifest file declaring its external dependencies. After receiving the files in its cloud deployment pipeline, it then detects the package manager and runs an install and build.

In a monorepo, and especially a private one, your Firebase code typically depend on one or more shared packages from the same repository, for which you have no desire to publish them anywhere.

Once Firebase tries to look up those dependencies in the cloud they can not be found and deployment fails.

Hacking your way out

Using a bundler

In order to solve this you could try to use a bundler like Webpack to combine your Firebase code with the shared packages code and then remove those packages from the package.json manifest that is being sent to Firebase, so it doesn’t know these packages even existed.

Unfortunately, this strategy quickly becomes problematic…

If the shared packages themselves do not bundle all of their dependencies in their output, Firebase doesn’t know what the shared code depends on, because you are not including or installing those manifests.

You could try to bundle everything then, but if your shared package depends on things your Firebase package also depends on, you now have one part of your code running an internally bundled copy of a dependency and the other part using that same dependency from a different location installed by the package manager.

Also, some libraries really don’t like to be bundled, and in my experience that includes the Firebase and Google client libraries. You will quickly find yourself trying to externalize things via the bundler settings in order get thing to work.

And even if you managed to make all this work, you are probably creating large bundles which could then lead to problems with the cold-start times of your cloud functions.

Not exactly a reliable or scalable solution.

Packing and linking local dependencies

An arguably more elegant approach involves packing the local dependencies into a tarball (similar to how a package would be published to NPM), and copying the results to the build output before linking them in an altered manifest file.

This could work quite nicely, as it basically resembles how your Firebase code would have worked if these packages were installed from an external domain.

Whether you’re doing this manually, or write a shell script to handle things, it still feels very cumbersome and fragile to me, but I think it is a viable workaround if your local dependencies are simple.

However, this approach quickly becomes hairy once you have shared packages depending on other shared packages, because then you’ll have have multiple levels of things to pack and adapt.

My solution

I have created isolate-package. The name is generic because it doesn’t contain anything specific to Firebase, although I currently don’t know of any other use-cases for isolated output.

It takes a similar approach to what is described earlier in packing and linking dependencies, but does so in a more sophisticated way. It is designed to handle different setups and package managers and it completely hides the complexity from the user.

The isolate binary it exposes can simply be added to the Firebase predeploy hook, and that’s pretty much it!

This also allows you to deploy to Firebase from multiple different packages and keep the configuration co-located instead of littering the monorepo root directory.

It should be zero-config for the vast majority of use-cases, and is designed to be compatible with all package managers.

Answer 4

I am not sure I understand the question exactly, but I could give you my two cents on yarn workspaces based on whatever I understood from your question and from my experience using it.

Yarn workspaces consolidate all your dependencies into the node_modules present in project root as well as in a single package-lock.json to reduce conflicts and enables yarn to optimize the installation process giving you a faster yarn install. And also another advantage of it is, with a single pass yarn install can install dependencies of all packages under the workspace.

Edit: I think for some reason yarn link is not being called and instead only yarn install is being run, which will search the npm registries and throws the error mentioned in comment since it can't find the mentioned package on npm registry. So for a solution try creating an entry in the firebase's package.json like

"dependencies": {
  "a": "file:../dependency-package-name/",
}