Using Spring JMS with Amazon SQS

In this blog post, I’ll try to explain what you need to be careful about when using Spring JMS for consuming messages from Amazon SQS.

Even though there are many blog posts about this topic, I noticed that most of them don’t even mention how to handle failures properly.

I’m not sure if it’s because blog posts are poor-quality, or the consumers of them are just very lazy that they’re trying to build production systems based on random blog posts with zero questionings. Anyway…

Alright. So, you have an SQS queue that you want to consume and process messages. This message processing might involve some external systems so you should be prepared for failures during this message processing. SQS provides some nice features for you to handle these failures.

Visibility Timeout: SQS has out-of-the-box support for retries with fixed delays. If you don’t delete the message you consumed within this period, that message will be visible again to the consumers. Hence, retry mechanism with a fixed delay. You can configure the maximum number of retries with Redrive Policy.
What happens if you still can’t process message successfully and can’t delete the message from the queue after all these retries?

Dead-letter Queues: The messages that you couldn’t process within (maxReceiveCount * visibilityTimeout) seconds, will be sent to its dead-letter queue automatically by SQS, so that you can later on examine these messages you could not process and try to find out the root cause for the failures.

“Great!” you think, SQS already does a lot of stuff for handling failures, you just need to integrate Spring JMS with Amazon SQS, and then you can start writing message processing logic.

So you found this blog post for this integration and realize that SQS even has its own JMS library which you can integrate with Spring JMS, nice!

When you apply the very same config in the blog post above, if you have an error and your JMS Listener throws an exception during message processing, you’ll see that visibility timeout is not respected at all, and retry kicks in immediately. In this GitHub issue, you can find why that’s happening, and here you can find my hacky solution for it.

Basically, when you use CLIENT_ACKNOWLEDGE mode, if your listener method throws an exception, SQS JMS library changes that message’s visibility timeout to 0, so that message becomes visible immediately. This is not what we want, we all know that immediate retries often trouble rather than a cure for any problem.

It’s clear that we need to choose another acknowledge mode.

JMS Acknowledge Modes and SQS Implementations

These acknowledge modes come from the original JMS spec and client libraries (SQS JMS library, in this case) are responsible to implement them. Let’s have a look at our options and implementations of it.

There are 3 acknowledge modes in the JMS spec: AUTO_ACKNOWLEDGE, CLIENT_ACKNOWLEDGE, DUPS_OK_ACKNOWLEDGE

And there’s 1 acknowledge mode SQS JMS library provides: UNORDERED_ACKNOWLEDGE

To find out their SQS implementations, we need to look at SQSConnection class:

Let’s see these implementations one by one:

1. AutoAcknowledger: The implementation of AUTO_ACKNOWLEDGE.

As explained both in AutoAcknowledger and AUTO_ACKNOWLEDGE JavaDoc, this acknowledges the message (deletes it from the queue) when it first received it. So, even though your processing ends up with an exception, you’ll never get that message again from SQS. If you failed the process of this message, goodbye, it’s lost. Thus, if you can’t tolerate losing messages and want to leverage SQS' visibility timeout, do not use this acknowledge mode.

2. RangedAcknowledger: The implementation of CLIENT_ACKNOWLEDGE and DUPS_OK_ACKNOWLEDGE.

In this implementation, acknowledge deletes all messages within the unAckMessages queue it holds. Invocation of this acknowledge is done by Spring JMS from its commitIfNecessary method, if message listener doesn’t throw any exception. You need to be careful when using this acknowledge mode. If you use the solution above to not change visibility timeout to 0, you’ll end up removing rollbackOnExceptionIfNecessary implementation. Doing that may cause failed messages to be acknowledged. So, if you want to use fixed visibility timeout on the SQS queue you set and you can not tolerate losing messages, you should not use CLIENT_ACKNOWLEDGE mode either.

3. UnorderedAcknowledger: The implementation of UNORDERED_ACKNOWLEDGE.

As you see, this also deletes a single message just like AutoAcknowledger, except acknowledge method is not called by Spring JMS or SQS JMS libraries. If you’re using this mode, you’re the one who is responsible for calling the acknowledge.

All of these acknowledge modes do things very differently and you should choose the one that works for your use case. There is a number of things to consider, some of them:

• Can you tolerate losing messages?
• Is your message processing idempotent? Does the order of messages important for you?
• What’s the volume of messages you consume? Since SQS charges you per call, you may end up with 10x more costs if you don’t use CLIENT_ACKNOWLEDGE. (hint: take a look at AmazonSQSBufferedAsyncClient)
• Is your message processing asynchronous? If it is, you should be in charge of the calling acknowledge method, so you better not use anything other than UNORDERED_ACKNOWLEDGE.

My suggestion is, if you want to benefit from visibility timeout and dead-letter queues, go with UNORDERED_ACKNOWLEDGE by making sure that you acknowledge the message manually.

Nevertheless, as with anything in software development, there’s no silver bullet in this case too. You should know your tools/libraries well and be aware of the trade-offs you face.

Tracing Requests & Responses with Spring Boot Actuator

Spring Boot Actuator is a sub-project that provides endpoints allow you to monitor and interact with your application. You can take a look at complete list of endpoints but we will focus on trace and its customization in this post.

By default, trace endpoint is enabled for all HTTP requests and shows last 100 of them with 5 default properties:

• Request Headers
• Response Headers
• Cookies
• Errors
• Time Taken

So, when you hit /trace, the typical response will be like:

The question is: how can we customize these properties? Is there any way to exclude some of them or include even more properties? The answer is YES.

Here, you can see all available properties. Sometimes, the best documentation is the source code :).

In your configuration file you can specify the properties you want to trace:

management.trace.include = remote_address, parameters

Note that this will override default properties. If you want to keep them, you have to specify them along with additional properties you want.

In the introduction part we said that it’s enabled for all HTTP requests. It really is. If you played with it, probably you saw that requests made to /trace also traced. Let’s say you don’t want some endpoints to be traced such as /admin/**, /trace and endpoints provide static files like /css/** and /js/**.

To be able to prevent those endpoints to be traced, we need to extend WebRequestTraceFilter and override its shouldNotFilter method which inherited from OncePerRequestFilter.

shouldNotFilter’s default implementation was always returning false. Now with this implementation, it returns true for the endpoints we don’t want to trace. By annotating this class with @Component, we tell Spring to register it as a bean instead inherited WebRequestTraceFilter. No more configuration needed for this to work.

It’s not over yet, we need more customization!

In the introduction part we also said that, trace endpoint shows last 100 requests by default. Unfortunately, there is no way to change it from configuration file directly but still it’s a piece of cake.

By default, InMemoryTraceRepository is used as an implementation of TraceRepository. By extending it, we can expand the capacity, log requests explicitly or even persist them. It’s up to you.

Here is the source code of sample application.

Intercepting and modifying logs in Logback

Actually, Logback does not offer a direct way to modify logs but there is a workaround we can achieve our goal by using filter even though it looks like a bit hacky.

Let’s say that you logged some id no of some user hundreds of times, it scattered through all over the application and now you have a new requirement that says you have to encrypt this id number. Of course you’re smart enough to write an interceptor for this task instead of go find and change necessary logs manually. And also with this way, we can be sure that we’ll never log that id number accidentally.

For this case, we’re gonna extend TurboFilter and override its decide method.

If the log matches our criteria and we want to change it, we deny it by returning FilterReply.DENY and log again with the changed object.

Here, I used info as log level for the sake of simplicity but if you don’t want to change log level, you can easily check which level coming log has and use that level.

It’s that easy once you deal with recursion and not forget to declare your custom filter in configuration file properly.

CORS and Authentication

Let’s say that you have an API which stands on xdomain.com and you have a Javascript application that consuming this API from ydomain.com. For ydomain.com to consume the API, xdomain.com has to send CORS header Access-Control-Allow-Origin in its response.

The simplest way to do this in Spring is annotate whole controller or just handler method with the annotation CrossOrigin.

It’s all what you need when your API does not require authentication.

On the other hand, when your API requires authentication, things get a bit complicated. For example, let’s say that you authenticate your API with HTTP Basic Authentication. So, consumers of this API must send an Authorization header which contains user credentials in their request. Since this is a non-simple request, OPTIONS request will be send firstly. The thing is that, even if you send Authorization header with correct credentials in your request, you will face something like below:

The reason of this is, according to CORS spec, it excludes your Authorization header. Hence, you should permit OPTIONS requests in your security configuration class explicitly.

Last of all, I suggest you to read these two posts:

http://stackoverflow.com/a/15734032/3099704 https://code.google.com/archive/p/twitter-api/issues/2273

REST Authentication using Spring Security & Spring Session

In this post, I will try to demonstrate how easily we can implement an authentication mechanism for REST services using Spring Security and Spring Session with the help of Redis.

The authentication scheme that we will use:

1. Client requests an authenticated URL with its credentials. (first request to the server)
2. Server gives some unique string besides the normal response. (unique string is session id in our case)
3. Client has to send this string with every request using an HTTP header.
4. Server recognizes this unique string and logs client in.

Let’s start implementing the scheme above.

For this application to work, you must install Redis 2.8+ on localhost and run it with the default port (6379).

We will create very simple Spring Boot application which you can get source code and dependencies here.

Our security configuration looks like:

configureGlobal method is pretty straightforward, we just create a user with username sedooe and password password.

configure method is also straightforward but I want to emphasize two things here.

1. We are using HTTP Basic Authentication since it is simplest way to deal with authentication. As you probably know, Basic Authentication is just a standard HTTP header with the username:password encoded in base64: Authorization: Basic c2Vkb29lOnBhc3N3b3Jk Keep in mind that encoding is not encrypting. Therefore, you must use HTTPS instead of HTTP to make sure that user credentials are sent securely.

2. NullRequestCache prevents creation of session when user is not authenticated so that we have only authenticated users' session ids stored in Redis.

In httpSessionStrategy method, Spring Session comes into play. Thanks to HeaderHttpSessionStrategy, when a session is created, the HTTP response will have a response header of the specified name and the value of the session id. Default header name is x-auth-token and we will use it.

Here is our simple controller:

There is nothing to mention about this controller specifically so let’s start trying things out.

$ curl -v http://localhost:8080/api/resource

Yeah, authentication is required to access this resource. So try: $ curl -v http://localhost:8080/api/resource -u sedooe:password

Now you are authenticated and you have access to the resource. Take a look at response headers and you will see that: x-auth-token: 00661f53-8453-4daf-89f1-e748a3326040

For further requests, we can use x-auth-token instead of username and password. $ curl -v http://localhost:8080/api/resource -H "x-auth-token: 00661f53-8453-4daf-89f1-e748a3326040"

If you want to see all stored tokens: $ redis-cli keys '*'

To invalidate the session and delete it from Redis: $ curl -v http://localhost:8080/api/logout -H "x-auth-token: 00661f53-8453-4daf-89f1-e748a3326040"

Note:

You may wonder how our application connected to Redis without writing any code for that.

Since we use Spring Boot 1.3.3 in this project and Spring Session and Spring Data Redis are both on the classpath, Spring Boot will auto-configure Spring Session to connect to Redis.

If you use earlier Spring Boot versions than 1.3.1 or don’t use it at all, you should add @EnableRedisHttpSession to your configuration class.