Understanding Grails Dependency Fetching

April 13, 2012

If you have ever built or looked at a Grails project, you will notice that within the BuildConfig.groovy file there is a section called repositories that specifies all of the various repositories to pull in resources from. Grails is not built on any one type of repository system, instead it supports almost all of them. It does this through Ivy so that it can support Maven repositories, Grails repositories, standard HTTP repositories, etc. The problem with that, though, is that Grails via Ivy has to be pretty simplistic about how it achieves its lookups of dependencies. At the simplest of levels this means that it purely queries each repository for a given resource. To worsen the problem, the resources are usually specified in the GAV (group:artifact:version) pattern. Those resources could be ZIP files, JAR files, POM files, etc. Now, Ivy has to request each possible combination of allowed extensions to each repository until it finds a valid match. If you have a project that depends on several libraries, this could result in minutes to hours of download time the first time a project is built. Subsequent times will be fast as the resources are cached locally. I still personally hate having to wait forever for an initial build to run. So, how do you attempt to fix this in your project?

The answer is to make sure you order your repositories in an order that makes sense to ensure that the repositories most of your resources will come from are attempted first. Typically, the repositories are just nonchalantly thrown in any order and the result is long build times. Even worse, transitive dependencies could bring in their own repository configuration multiplying the number of requests to make per dependency. This is controlled by the inherits true statement, which is the default. So, what should the order be? It’s really up to you to decide. The first rule is to always include any local repository (maven local, grails home, etc) first so that those dependencies resolve quickly and non-matches fail quickly. The latter part is the key since you spend multiple requests to a repository for each dependency, which is costly for remote networks on the Internet. If you work at a company that has a central Maven repository, like Nexus, then I would suggest putting that next as it will resolve your own Maven libraries as well proxy Maven central and any other Maven repository. Plus, if you end up using many internal libraries, you will definitely want to utilize that repository first, especially since the request time and latency will be considerably less being in house rather than on the Internet.

Here are my recommended order of repositories:

// use Grails home first as it is local and quick and resolves all the standard Grails libraries quickly
// use maven home next if, and only if, you use maven so that your local dependencies are found quickly
// use your companies' Nexus repository that also proxies maven central and other repos
// if you do not use maven, then do not use this or include it much later in the process
// this only has value this high in the order if several of your dependencies use maven
// or if the repository is in house on the same network
mavenRepo "http://maven.company.org/..."
// use grails plugins next as it will resolve all the other plugins you use
// if you use more plugins than you do libraries, then reference grails plugins first
// before any maven repository
// use maven central as a last step...again, only include this if you use maven libraries
// also, do NOT use this if your in house Maven/Nexus repo proxies maven central
// include any other maven repository not included in mavenCentral or your Nexus configuration
// these are always one-offs typically so they should be last
mavenRepo "http://company.org/maven/..."

The most important part is to only include the repositories you know you need. For example, if you have a company Nexus repository that proxies maven central, then only include one or the other, not both. These rules are not meant in any way to be a firm stake, but something to give you context so you put thought into the ordering and how your environment, code, dependencies, and plugins work together. After all, development time is precious.

4 Responses to “Understanding Grails Dependency Fetching”

  1. This is excellent Nick. I had a hectic time resolving the dependencies for our current project, and when I finally fixed it, others in the team were complaining about the time it takes to download. Thanks to this. I will keep these suggestions in mind.

  2. [...] Understanding Grails Dependency Fetching [...]

  3. Excellent post Nick!

    I’ve been through the pains of dependency resolutions where our remote repository was behind a firewall and it took forever to identify that because Ivy was looking at all the possible combinations as you have mentioned.

    I couldn’t stop wondering why it was querying every other repository with the same information.

    This is the answer!

  4. [...] locate dependencies was long and cumbersome as it had to search every repository (see my previous blog post). Using pure Maven allows Maven to directly locate artifacts and better handle dependencies, [...]