Using gradle shadow plugin to resolve java version conflict

If you work with java you would have probably known what jar hell is. Recently when we started to build an indexing platform on top of hbase, we encountered the jar hell problem too: the current hbase client (1.3.0) depends upon proto2 (2.5.0), while our project uses proto3 (3.0.0). The indexing job we built is based on proto3. However, this would not run properly on spark/hbase as the default classloader will load proto2 classes instead, and we got mysterious failure serializing proto message using toByteArray():

java.lang.IllegalAccessError: tried to access field 
com.google.protobuf.AbstractMessage.memoizedSize from class Blahblah

Fortunately we are using the shadow plugin to generate a shadow jar (or fat jar) file that contains all the dependencies. And it has a nice feature to relocate packages:

shadowJar {
    relocate (‘com.google.protobuf’, ‘hbasewrapper.com.google.protobuf’)
}

According to the documentation, "Shadow uses the ASM library to modify class byte code to replace the package name and any import statements for a class. ". So this literally means that any class starts with package name "com.google.protobuf" will have new package name "hbasewrapper.com.google.protobuf" in the final jar file. And any code referencing the original package will reference the new package instead. With that we can simply create a wrapper project util/hbasewrapper/build.gradle:

group 'com.coupang.utils.hbasewrapper'
version '1.0-SNAPSHOT'

dependencies {
    compile group: 'org.apache.hbase', name: 'hbase-client', version: '1.3.0'
    compile group: 'org.apache.hbase', name: 'hbase-server', version: '1.2.2'
}

apply plugin: 'java'
apply plugin: 'com.github.johnrengelman.shadow'

shadowJar {
    baseName = 'hbasewrapper'
    version = null
    zip64 true
    relocate ('com.google.protobuf', 'hbasewrapper.com.google.protobuf')
    mergeServiceFiles()
}

Looking at the content of the generated jar file after "gradle build", you will find:

jar -tf build/libs/hbasewrapper-all.jar | grep com.google.protobuf
hbasewrapper/com/google/protobuf/
hbasewrapper/com/google/protobuf/HBaseZeroCopyByteString.class
hbasewrapper/com/google/protobuf/AbstractMessage$1.class
hbasewrapper/com/google/protobuf/AbstractMessage$Builder.class
...

All the original classes under com/google/protobuf/ are relocated to hbasewrapper/com/google/protobuf/. Remember that because current hbase version (1.3) depends on protobuf 2.5, you actually have all protobuf 2.5 related classes under hbasewrapper/com/google/protobuf/ and your hbase client/server will always reference them from there.
Now from your actual project, instead of depending upon hbase-client/hbase-server directly, make it depending upon the new hbasewrapper instead:

dependencies {
    ...
    compile group: 'com.google.protobuf', name: 'protobuf-java', version: '3.0.0'
    compile project(path: ':utils:hbasewrapper', configuration: 'shadow')
}

apply plugin: 'com.github.johnrengelman.shadow'

shadowJar {
    baseName = 'importer'
    version = null
    zip64 true
    relocate ('com.google.protobuf', 'importer.com.google.protobuf')
    mergeServiceFiles()
}

Let's see what are built into the shadow jar file:

jar -tf build/libs/importer-all.jar | grep com.google.protobuf
hbasewrapper/com/google/protobuf/
hbasewrapper/com/google/protobuf/HBaseZeroCopyByteString.class
hbasewrapper/com/google/protobuf/AbstractMessage$1.class
...
importer/com/google/protobuf/
importer/com/google/protobuf/util/
importer/com/google/protobuf/util/Durations.class
importer/com/google/protobuf/util/FieldMaskTree$1.class

We have proto 3 classes under importer/com/google/protobuf/, proto 2.5 under hbasewrapper/com/google/protobuf/. And both hbase and your own project will use the right version for them. Problem solved!