Hot questions for Spring Cloud AWS

Hot questions for Spring Cloud AWS

Top 10 Java Open Source / Spring / Spring Cloud AWS

Question:

I have been fighting with this for several hours today. I started with the documentation at http://cloud.spring.io/spring-cloud-aws/spring-cloud-aws.html#_sending_mails which doesn't really say a lot about the specific steps. It just says that the developer can include a Bean XML and then autowire MailSender. I have tried that as well as many variants and have not been able to get it to work using spring-cloud-aws. I finally resorted to directly including aws-java-sdk-ses and manually configuring the class.

Here is a simple project demonstrating what I've tried: https://github.com/deinspanjer/aws-ses-test

This project compiles, but when I run it I get:

Parameter 0 of constructor in com.example.awssestest.AwsSesTestApplication required a bean of type 'org.springframework.mail.MailSender' that could not be found.
- Bean method 'mailSender' not loaded because @ConditionalOnClass did not find required class 'javax.mail.internet.MimeMessage'
- Bean method 'simpleMailSender' not loaded because @ConditionalOnClass did not find required class 'com.amazonaws.services.simpleemail.AmazonSimpleEmailService'
- Bean method 'javaMailSender' not loaded because @ConditionalOnClass did not find required class 'com.amazonaws.services.simpleemail.AmazonSimpleEmailService'

If I try adding javax-mail ( https://github.com/deinspanjer/aws-ses-test/tree/try-with-javax-mail-api ) then the error changes to:

Parameter 0 of constructor in com.example.awssestest.AwsSesTestApplication required a bean of type 'org.springframework.mail.MailSender' that could not be found.
- Bean method 'mailSender' not loaded because AnyNestedCondition 0 matched 2 did not; NestedCondition on MailSenderAutoConfiguration.MailSenderCondition.JndiNameProperty @ConditionalOnProperty (spring.mail.jndi-name) did not find property 'jndi-name'; NestedCondition on MailSenderAutoConfiguration.MailSenderCondition.HostProperty @ConditionalOnProperty (spring.mail.host) did not find property 'host'
- Bean method 'simpleMailSender' not loaded because @ConditionalOnClass did not find required class 'com.amazonaws.services.simpleemail.AmazonSimpleEmailService'
- Bean method 'javaMailSender' not loaded because @ConditionalOnClass did not find required class 'com.amazonaws.services.simpleemail.AmazonSimpleEmailService'

If instead, I try explicitly adding a dependency on aws-java-sdk-ses ( https://github.com/deinspanjer/aws-ses-test/tree/try-with-aws-java-sdk-ses ), I get this error instead:

Parameter 0 of constructor in com.example.awssestest.AwsSesTestApplication required a bean of type 'org.springframework.mail.MailSender' that could not be found.
- Bean method 'mailSender' not loaded because @ConditionalOnClass did not find required class 'javax.mail.internet.MimeMessage'
- Bean method 'javaMailSender' in 'MailSenderAutoConfiguration' not loaded because @ConditionalOnClass did not find required class 'javax.mail.Session'
- Bean method 'simpleMailSender' in 'MailSenderAutoConfiguration' not loaded because @ConditionalOnMissingClass found unwanted class 'org.springframework.cloud.aws.mail.simplemail.SimpleEmailServiceJavaMailSender'

For this error, I tried adding a @Qualifier("simpleMailSender") annotation to the @Autowired, but it did not help.

I hope someone might be able to steer me in the right direction.


Answer:

You may try below steps to fix your issue. I tried these changes in the forked repo from you and it works for me.

  1. Add dependency "com.amazonaws:aws-java-sdk-ses" in pom.xml file.
  2. Create an auto configuration class to configure the mail sender bean. Below is example. The AWSCredentialsProvider is configured and provided by spring-cloud-starter-aws out-of-the-box.

.

@Configuration
public class SimpleMailAutoConfig {

    @Bean
    public AmazonSimpleEmailService amazonSimpleEmailService(AWSCredentialsProvider credentialsProvider) {
        return AmazonSimpleEmailServiceClientBuilder.standard()
                .withCredentials(credentialsProvider)
                // Replace US_WEST_2 with the AWS Region you're using for
                // Amazon SES.
                .withRegion(Regions.US_WEST_2).build();
    }

    @Bean
    public MailSender mailSender(AmazonSimpleEmailService ses) {
        return new SimpleEmailServiceMailSender(ses);
    }
}

3. Use spring API to send mail using the configured mail sender.

Hope it helps.

Edit:

If you need to use JavaMailSender instead of MailSender (for instance when you want to send attachments), simply configure SimpleEmailServiceJavaMailSender instead of SimpleEmailServiceMailSender.

Like this:

    @Bean
    public JavaMailSender mailSender(AmazonSimpleEmailService ses) {
        return new SimpleEmailServiceJavaMailSender(ses);
    }

Question:

How can I disable the Cloudformation in a spring boot app that using spring cloud AWS?

I keep getting this error when running my app on amazon:

...
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.aws.core.env.stack.config.StackResourceRegistryFactoryBean]: Factory method 'stackResourceRegistryFactoryBean' threw exception; nested exception is com.amazonaws.AmazonServiceException: Stack for i-b5ce9e32 does not exist (Service: AmazonCloudFormation; Status Code: 400; Error Code: ValidationError; Request ID: 75b3076a-176d-11e6-90cc-b55a643dc6d6)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
        ... 82 more
Caused by: com.amazonaws.AmazonServiceException: Stack for i-b5ce9e32 does not exist (Service: AmazonCloudFormation; Status Code: 400; Error Code: ValidationError; Request ID: 75b3076a-176d-11e6-90cc-b55a643dc6d6)
        at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1389)
        at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:902)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.ja
...

I doesn't want to use cloudformation. (At least for now.)


Answer:

I have the following line in my application.properties (spring boot) file

cloud.aws.stack.auto=false

Official documentation: 4.4. CloudFormation configuration in Spring Boot

Question:

I'm using spring-cloud-aws's SqsListener to receive AWS SNS HTTP Notifications in JSON Format from AWS's Simple Queue Service (SQS).

This is the code for the listener:

@SqsListener(value = "my-queue", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
public void handle(final MyObject obj) throws Exception {
// ...
}

The documentation linked above is only about sending and reading plain serialized objects to the Queue and I thought that receiving SNS messages is expected to work out of the box. But I end up receiving conversion errors:

10:45:51.480 [simpleMessageListenerContainer-2] ERROR o.s.c.a.m.l.SimpleMessageListenerContainer - Exception encountered while processing message. org.springframework.messaging.MessagingException: An exception occurred while invoking the handler method; nested exception is org.springframework.messaging.converter.MessageConversionException: No converter found to convert to class com.myproject.model.MyObject, message=GenericMessage

I also tried creating a wrapper object that looks like the expected SNS Json Format linked above, but I keep getting the same exception. The only type that works is a String in the signature. Shouldn't the SNS be converted automatically?


Answer:

Yes it should. And it does actually.

In order to have the correct HandlerMethodArgumentResolver invoked (in this case NotificationMessageArgumentResolver) on deserialization, which in turn invokes the correct converter NotificationRequestConverter you simply need to add the annotation org.springframework.cloud.aws.messaging.config.annotation.NotificationMessage to your method signature. E.g.

@SqsListener(value = "my-queue", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
public void handle(final @NotificationMessage MyObject obj) throws Exception {
// ...
}

This way the Message part of your SNS gets extracted and converted to MyObject.

Question:

I have been trying to find documentation of @enablesns @enablesqs annotations but can't find them.

They seem to be required for the sqs and sns integration to work. But I'd just like to have a better understanding, and be sure I'm not using them wrong.

Any description would be of great help.


Answer:

Looking at the source code for those two annotations:

@Import({SnsConfiguration.class, SnsWebConfiguration.class})
public @interface EnableSns

Where @Import does the following:

@interface Import: Indicates one or more @Configuration classes to import.

Long story short: Those annotations only combine multiple @Configuration classes into one single annotation.

E.g. @EnableSns does the same as adding @SnsConfiguration and @SnsWebConfiguration which provide you with AmazonSNS, RegionProvider and AWSCredentialsProvider beans.

Question:

I'm trying to implement logic with manual deleting of AWS SQS message using spring-cloud-aws-messaging. This feature was implemented in scope of this ticket from the example in tests

@SqsListener(value = "queueName", deletionPolicy = SqsMessageDeletionPolicy.NEVER)
public void listen(SqsEventDTO message, Acknowledgment acknowledgment) {

    LOGGER.info("Received message {}", message.getFoo());
    try {
        acknowledgment.acknowledge().get();
    } catch (InterruptedException e) {
        LOGGER.error("Opps", e);
    } catch (ExecutionException e) {
        LOGGER.error("Opps", e);
    }
}

But faced with the unexpected exception

com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance oforg.springframework.cloud.aws.messaging.listener.Acknowledgment(no Creators, like default construct, exist): abstract types either need to be mapped to concrete types, have custom deserializer, or contain additional type information

Solution with SqsMessageDeletionPolicy.ON_SUCCESS works but I want to avoid throwing an exception.

What have I missed in the configuration?


Answer:

It took some fiddling around and trying different things from other SO answers.

Here is my code and I'll try to explain as best I can. I'm including everything that I'm using for my SQS consumer.

My config class is below. Only not-so-obvious thing to note below is the converter and resolver objects instantiated in the queueMessageHandlerFactory method. The MappingJackson2MessageConverter (in case it isn't obvious from the oh-so-obvious class name) class handles the deserialization of the payload from SQS.

It's also important that the strict content type match be set to false.

Also, the MappingJackson2MessageConverter allows you to set your own Jackson ObjectMapper, however if you do that you will need to configure it as follows:

objectMapper.configure(MapperFeature.DEFAULT_VIEW_INCLUSION, false);
objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);

You may not want to do that, so you can leave it null and it will create its own ObjectMapper.

I think the rest of the code is pretty self-explanatory...? Let me know if not.

One difference between our use-cases, it looks like you're mapping your own custom object (SqsEventDTO) and I assume that's working? In that case, I don't think you will need the MappingJackson2MessageConverter, but I could be wrong.

@Configuration
public class AppConfig {

@Bean
@Primary
public QueueMessageHandler queueMessageHandler(@Autowired QueueMessageHandlerFactory queueMessageHandlerFactory) {
    return queueMessageHandlerFactory.createQueueMessageHandler();
}

@Bean
@Primary
public QueueMessageHandlerFactory queueMessageHandlerFactory(@Autowired AmazonSQSAsync sqsClient) {

    QueueMessageHandlerFactory factory = new QueueMessageHandlerFactory();
    factory.setAmazonSqs(sqsClient);

    MappingJackson2MessageConverter messageConverter = new MappingJackson2MessageConverter();
    messageConverter.setSerializedPayloadClass(String.class);

    //set strict content type match to false
    messageConverter.setStrictContentTypeMatch(false);

    // Uses the MappingJackson2MessageConverter object to resolve/map 
    // the payload against the Message/S3EventNotification argument.
    PayloadArgumentResolver payloadResolver = new PayloadArgumentResolver(messageConverter);

    // Extract the acknowledgment data from the payload's headers, 
    // which then gets deserialized into the Acknowledgment object.  
    AcknowledgmentHandlerMethodArgumentResolver acknowledgmentResolver = new AcknowledgmentHandlerMethodArgumentResolver("Acknowledgment");

    // I don't remember the specifics of WHY, however there is 
    // something important about the order of the argument resolvers 
    // in the list
    factory.setArgumentResolvers(Arrays.asList(acknowledgmentResolver, payloadResolver));

    return factory;
}

@Bean("ConsumerBean")
@Primary
public SimpleMessageListenerContainer simpleMessageListenerContainer(@Autowired AmazonSQSAsync amazonSQSAsync, @Autowired QueueMessageHandler queueMessageHandler,
    @Autowired ThreadPoolTaskExecutor threadPoolExecutor) {

    SimpleMessageListenerContainer smlc = new SimpleMessageListenerContainer();
    smlc.setWaitTimeOut(20);
    smlc.setAmazonSqs(amazonSQSAsync);
    smlc.setMessageHandler(queueMessageHandler);
    smlc.setBeanName("ConsumerBean");
    smlc.setMaxNumberOfMessages(sqsMaxMessages);
    smlc.setTaskExecutor(threadPoolExecutor);

    return smlc;
}

@Bean
@Primary
public ThreadPoolTaskExecutor threadPoolTaskExecutor() {
    ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();

    executor.setCorePoolSize(corePoolSize);
    executor.setAllowCoreThreadTimeOut(coreThreadsTimeout);
    executor.setWaitForTasksToCompleteOnShutdown(true);
    executor.setMaxPoolSize(maxPoolSize);
    executor.setKeepAliveSeconds(threadTimeoutSeconds);
    executor.setThreadNamePrefix(threadName);
    executor.initialize();

    return executor;
}
}

My SQS consumer Service class is below.

@Service
public class RawConsumer {

@SqsListener(deletionPolicy = SqsMessageDeletionPolicy.NEVER, value = "${input.sqs.queuename}")
public void sqsListener(S3EventNotification event, Acknowledgment ack) throws Exception {
    // Handle event here
}

I hope that helps, let me know if you have any issues.

Question:

I'm using spring-cloud-aws-messaging in a Spring Boot project. I have SQS queue created manually in AWS.

It is being used like:

@SqsListener("${sqs.name.incoming}")
public void listen(String message) {
    ...
}

It works fine. But when I stop my application in IDE, or when the Spring Boot tests finish, it tries to stop the queue. It can't stop it and eventually times out. It throws this exception:

2019-10-29 15:40:07.949  WARN 10378 --- [       Thread-2] s.c.a.m.l.SimpleMessageListenerContainer : An exception occurred while stopping queue 'my-awesome-queue-name'

java.util.concurrent.TimeoutException: null
    at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204) ~[na:na]
    at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer.waitForRunningQueuesToStop(SimpleMessageListenerContainer.java:161) ~[spring-cloud-aws-messaging-2.1.3.RELEASE.jar:2.1.3.RELEASE]
    at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer.doStop(SimpleMessageListenerContainer.java:140) ~[spring-cloud-aws-messaging-2.1.3.RELEASE.jar:2.1.3.RELEASE]
    at org.springframework.cloud.aws.messaging.listener.AbstractMessageListenerContainer.stop(AbstractMessageListenerContainer.java:351) ~[spring-cloud-aws-messaging-2.1.3.RELEASE.jar:2.1.3.RELEASE]
    at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer.stop(SimpleMessageListenerContainer.java:45) ~[spring-cloud-aws-messaging-2.1.3.RELEASE.jar:2.1.3.RELEASE]
    at org.springframework.cloud.aws.messaging.listener.AbstractMessageListenerContainer.stop(AbstractMessageListenerContainer.java:239) ~[spring-cloud-aws-messaging-2.1.3.RELEASE.jar:2.1.3.RELEASE]
    at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer.stop(SimpleMessageListenerContainer.java:45) ~[spring-cloud-aws-messaging-2.1.3.RELEASE.jar:2.1.3.RELEASE]
    at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:238) ~[spring-context-5.1.10.RELEASE.jar:5.1.10.RELEASE]
    at org.springframework.context.support.DefaultLifecycleProcessor.access$300(DefaultLifecycleProcessor.java:53) ~[spring-context-5.1.10.RELEASE.jar:5.1.10.RELEASE]
    at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.stop(DefaultLifecycleProcessor.java:377) ~[spring-context-5.1.10.RELEASE.jar:5.1.10.RELEASE]
    at org.springframework.context.support.DefaultLifecycleProcessor.stopBeans(DefaultLifecycleProcessor.java:210) ~[spring-context-5.1.10.RELEASE.jar:5.1.10.RELEASE]
    at org.springframework.context.support.DefaultLifecycleProcessor.onClose(DefaultLifecycleProcessor.java:128) ~[spring-context-5.1.10.RELEASE.jar:5.1.10.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1018) ~[spring-context-5.1.10.RELEASE.jar:5.1.10.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:945) ~[spring-context-5.1.10.RELEASE.jar:5.1.10.RELEASE]

This waiting slows down application or tests shutdown.

How do I tell spring-cloud-aws-messaging that it is a manually created queue and it should not try to shut it down?


Answer:

It is not actually shutting down the the SQS queue rather it is shutting down the running task that is polling the queue. The default timeout to shut this down is 10 secs as defined in SimpleMessageListenerContainer.queueStopTimeout.

When the task is polling the queue, it waits for messages from 1 to 20 secs as set in SimpleMessageListenerContainerFactory.waitTimeOut. If this poll happens to be greater than 10 secs, then the above shutdown will timeout causing the TimeoutException.

So to fix this, either you increase the queueStopTimeout to be greater than 20

@PostConstruct
public void setUpListenerContainer() {
    listenerContainer.setQueueStopTimeout(25000);
}

or decrease the poll waitTimeOut to be less than 10.

@Bean
public SimpleMessageListenerContainerFactory simpleMessageListenerContainerFactory(AmazonSQSAsync amazonSqs) {
    SimpleMessageListenerContainerFactory factory = new SimpleMessageListenerContainerFactory();
    factory.setWaitTimeOut(5);
    return factory;
}

Question:

I'm using AmazonSQS & Spring Boot (spring-cloud-aws-messaging). I've configured a message listener to receive messages from the queue with the annotation @SqsListener.

@SqsListener(value = "indexerQueue", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
public void queueListener(String rawMessage) {
   ...
}

This is a very simple approach but I didn't find the way to make the queue name load from a configuration file because I have different environments. Any ideas on this regard?


Answer:

What version of spring-cloud-aws-messaging are you using? Version 1.1 should allow you to use a placeholder as a queue name, e.g.

@SqsListener(value = "${sqs.queue.indexer}", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
public void queueListener(String rawMessage) {
   ...
}

Then, in your application-env.properties files you can put different values. For instance in application-dev.properties:

sqs.queue.indexer=devIndexerQueue

and in application-production.properties

sqs.queue.indexer=indexerQueue

Question:

I could not find any API or documentation in Spring AWS Cloud to delete an object from S3 bucket. Can someone please let me know how to do it?

The documentation just talks about reading the content using ResourceLoader.

Only option right now I see is to explicitly inject AmazonS3 and call deleteObject.


Answer:

Spring's Resource API does not support the full lifecycle of operations. The two main interfaces are Resource and WritableResource. There is no API for deletion.

As an alternative you could use Spring Content for S3 instead. Spring Content Stores are all generic ResourceLoaders that return Resources that DO support deletion. That, by the way, works exactly as you are suggesting.

Take a look at the reference guide. The base Store interface is the one that should be a straight drop-in for the one from Spring Cloud AWS.

Simply cast the return from getResource to DeletableResource and call it's delete to delete the object in S3.

Let me know if this is not clear and I can update with more details.