How to read and transfer large files with Apache Camel in any encoding?


“Encoding”, if you ask API developers these days, what encoding they’re accepting when they make and transfer/consume any data. You may get a common answer that “UTF-8” is what they accept which has become the standard in the movement of text-based data over REST calls.

But if you’re still stuck in the 20th Century, dealing with file systems API and you need to transfer a huge amount of data to another system by means of a file that is not encoded in the de facto “UTF-8”, you’ve your task cut out for you as you’ll encounter the most dreaded out of memory exceptions when you try out this task with Apache Camel.

In this post, we’ll help clear out this Jira ticket in your name and build a spring boot service that uses the Camel Integration framework to get transfer big files in GBs to a destination in any encoding you like.

We’ll be using the maven tool for building our project. Add the below plugins to your pom file. This will add the custom type converter which will convert the File payload to a Stream-based payload and register it in the Registry.

Now we need a class to implement the type convertor. This is where the magic of conversion of the payload of exchange happens. You can use anything for encoding instead of SHIFT-JS, I was experimenting with Japanese files so I had used that. There’s even a StandardCharsets class with some predefined encodings you can use.

That’s it, don’t use charset property now, as it’ll load the entire content of the file into memory to convert into the desired charset and transfer. A sample processor that is needed is shown here.

You can see behind the scenes action by enabling trace or debug logging.

You’re looking for a log where conversion to a stream-based exchange is guaranteed.

Cheers and hopefully if you found it useful enough, do consider leaving feedback. Thanks and welcome to this new year out of this pandemic!



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Varun Tripathi

Writer, Runner, Cyclist and software developer interested in intersection of various fields and tech based solutions to various problems.